Measuring incrementality without a holdout
The cleanest way to know whether an activity worked is to withhold it from a comparable group and compare. In retail that is frequently impossible: the activity is national, the retailer will not split, or withholding costs more than the answer is worth.
What remains is a set of weaker designs, each with a different claim strength, and knowing which one you used is most of the honesty in the exercise.
What is the hierarchy of designs?
A randomised holdout is strongest. A matched control group is next. A synthetic control built from a weighted combination of non-exposed units follows. A before-and-after comparison with modelled base is weaker again, and a simple before-and-after with no base model is barely evidence at all.
Naming which one was used tells a reader how much weight the conclusion carries, and that naming is usually absent from commercial results decks.
Why is before-and-after so weak?
Because everything else also changed. Seasonality, competitor activity, distribution, weather and price all move between periods, and a before-and-after difference contains all of them.
It can be strengthened with a modelled base, which is what promotional lift measurement does. It cannot be made equivalent to a controlled comparison, because the counterfactual is still estimated rather than observed.
What makes a matched control credible?
That the matching variables predict the outcome, that matching quality is reported, and that the groups tracked together before the intervention. The pre-period parallel is the evidence that the match works, and a chart showing it is the most persuasive single exhibit in this kind of analysis.
Groups that diverged before the activity started were never comparable, and no amount of statistical adjustment afterwards repairs that.
What is a synthetic control?
A weighted combination of non-exposed units chosen so that the combination tracks the exposed unit closely before the intervention. It is a principled way of building a comparison when no single unit matches. The method is established and it is more transparent than it sounds.
Its requirement is a good pre-period fit over a reasonable length of history. Without that, it produces a comparison with no evidential basis and a confident-looking chart.
What about geographic tests?
Matched-market designs are the practical middle ground for activity that cannot be split within a market. They work where markets are numerous enough to match and where the activity can be confined geographically.
In independent retail, store-level assignment is sometimes possible where market-level is not, which is an advantage of the channel worth using.
How should uncertainty be reported?
As a range, always. A point estimate from a non-randomised design carries assumption risk on top of sampling uncertainty, and quoting a single number misrepresents what is known.
That is uncomfortable in a commercial setting and it is the difference between a result that survives a challenge and one that does not.
What is the minimum acceptable design?
One with a comparison group of some kind and a pre-period that demonstrates comparability. Below that threshold, the honest description is "consistent with an effect" rather than "delivered an effect."
Frequently asked questions
Can modelling substitute for a control group?
It can approximate one and it cannot replace one, because the model supplies the counterfactual from assumptions rather than from observation. Modelled results are useful, and they should be labelled as modelled rather than presented as measurement.
How large does a control group need to be?
Large enough that the comparison is not driven by a handful of units, and the specific number depends on how variable the outcome is. Running a power calculation before the test is the right way to answer it, and doing so afterwards is usually an exercise in explaining why the result was inconclusive.
Is a competitor's region a valid control?
Only if it was comparable beforehand, which is an empirical question rather than an assumption. Plot both series for the pre-period first; if they do not track, the control is not a control.
What if the activity is already running everywhere?
Then a clean incremental measurement is no longer available, and the practical options are a staged withdrawal, a future test, or accepting a modelled estimate with its assumptions stated. Retrofitting a control after full rollout is not possible.
Does a bigger effect need less rigour?
A large effect is easier to detect and is not easier to attribute. The design question is about attribution, and a large unattributed movement is just a large unexplained movement.
How do I handle spillover between test and control?
Recognise it and account for it, because customers and stores are not sealed from each other. Spillover biases the estimate toward zero, so a result measured in its presence is usually cautious rather than inflated.
Can I test more than one thing at once?
With a design that separates them, yes. Running two changes simultaneously without a factorial design produces a combined effect that cannot be attributed to either, which is a common and avoidable waste.
Should the analysis plan be written first?
Always, including the measure, the window, the comparison and the threshold. Agreeing them afterwards allows the result to select the method, and any figure chosen after the numbers arrive is a rationalisation however reasonable it sounds.
What is the most useful single improvement to commercial testing?
A pre-period comparability chart on every result. It takes minutes and it tells a reader immediately whether the comparison means anything.
### What should be documented before the activity starts?
The design, the comparison group, the outcome measure, the window and the threshold. Writing them down before is what distinguishes a measurement from a post-hoc explanation.
Circulate the document. An analysis plan known only to the analyst can be quietly revised, and the revision will be invisible in the final report. General guidance on the limits of statistical significance is worth reading alongside it.
Working with retail data? See NRS Insights data solutions or contact NRS Insights.