Test and control design for retail
A store test compares stores that received something against stores that did not. It is the most practical experimental design available in retail and it fails more often through design than through execution.
Almost all the failures are decided before the test starts.
What decisions come first?
The unit of assignment, the assignment method, the sample size, the outcome measure, the window, and the analysis plan. Six decisions, all made before any store is selected, all documented.
The most consequential is sample size, because a test with too few stores cannot detect the effect it is looking for and will produce an inconclusive result that gets interpreted as a negative one.
How should stores be assigned?
Randomly within strata, so that assignment is not correlated with anything. Selecting test stores because they are cooperative, or because a field manager suggested them, builds the answer into the design.
Stratifying first — by size, region, format — and randomising within strata gives both balance and randomness, and it is straightforward to do.
How many stores are needed?
Enough to detect the smallest effect that would change a decision, given the variability of the measure. That calculation requires a variability estimate from historical data and a stated minimum effect.
Both are available and both are usually skipped. A test sized by what was convenient rather than by what is detectable is a test that cannot fail to be inconclusive.
What outcome should be measured?
The one the decision depends on, chosen in advance. Category sales, item velocity, basket size, margin — each answers a different question, and picking afterwards allows the measure to be selected for the result.
Specify a single primary outcome and treat everything else as secondary. A test with six co-equal outcomes will find something significant by chance.
How long should a test run?
Long enough to cover the behaviour being changed and to pass any novelty effect, which usually means several weeks at minimum. Too short and the result is trial; too long and other things change.
Set the end date in advance. Stopping when the result looks good is one of the most effective ways to generate a false positive.
What should be checked before analysis?
Pre-period comparability between groups, execution compliance, and whether anything unusual happened in either group during the test. A test where execution failed in a third of test stores is measuring execution, not the intervention.
Compliance data is the element most often missing. Without it, a null result cannot be distinguished from a failure to implement.
How should results be reported?
Effect size with a confidence interval, the pre-period comparability chart, the compliance rate, and the analysis plan as written before the test. Four elements, and together they let somebody else judge the result.
What about the decision threshold?
Agreed in advance, in the units of the outcome measure. A test without a pre-agreed threshold becomes a negotiation about whether the observed effect is large enough, conducted by people with an interest in the answer.
Frequently asked questions
Is randomisation really necessary?
It is what allows a causal claim, because it makes the groups comparable on everything including things nobody measured. Matched selection is the fallback when randomisation is impossible, and it carries a weaker claim that should be stated as such.
Can a test run in a single chain?
Yes, and the result then applies to that chain. Generalising a single-chain result to a fragmented independent channel is an extrapolation that should be labelled, because store characteristics and execution differ substantially.
What about contamination between groups?
Stores near each other can share shoppers, which dilutes a measured effect. Assigning by geographic cluster rather than by individual store reduces it, at the cost of needing more stores for the same power.
Should I stop a test early if it looks positive?
No, unless the plan included a formal interim analysis with adjusted thresholds. Stopping on a favourable look substantially inflates the false positive rate and is one of the most common ways commercial tests mislead.
How do I handle stores that drop out?
Report them, and analyse as assigned rather than only the compliant stores where possible. Analysing only the stores that complied compares volunteers to everybody, which reintroduces selection.
What if the effect varies by store type?
That is a legitimate secondary analysis and a weak primary one, because subgroup findings from a test sized for an overall effect are underpowered. Treat them as hypotheses for a next test.
Does a null result mean no effect?
It means the test did not detect one, which depends on how large the test was. A null from an underpowered test is uninformative, and reporting the detectable effect size alongside the result is what makes that clear.
How much does a proper test cost?
More than an uncontrolled rollout and less than a wrong decision at scale. The comparison that matters is against the cost of scaling something that does not work.
What is the single biggest design improvement?
A power calculation before the test. It is the step that converts a test from an exercise into an answer, and it is the one most often skipped. General guidance on statistical significance and its limits is worth reading alongside it.
### How should results be shared internally?
With the plan attached, so the analysis can be seen to match what was specified. A result presented without its plan invites the reasonable suspicion that the analysis was shaped by the outcome.
Share the null results as well. A programme that only reports its successes teaches nothing and loses credibility the first time somebody checks.
Working with retail data? See NRS Insights data solutions or contact NRS Insights.