Choosing test stores for a pilot

Good test stores look like the stores you plan to roll out to, and they come with matched control stores that look like them too. Test market selection is the process of choosing where a pilot runs and where it doesn't, so that the result can be trusted and generalized. It's the most important decision in a pilot, and often the least discussed.

A pilot run in the wrong stores can produce a clear result that doesn't hold anywhere else.

What makes a store a good test candidate?

Representativeness, first. If the plan is to roll out to small-format independent stores in several metros, the test stores should be small-format independent stores in those metros. Testing only in the busiest or most cooperative stores gives a result that describes those stores.

Stable data, second. A store with a consistent reporting history and no recent disruptions, such as a remodel or ownership change, gives a cleaner read. Stores with gaps or unusual swings add noise.

Relevant category presence, third. If the pilot involves a category, the test stores should carry it at a typical level. A store that barely sells the category won't show much, whatever the pilot does.

It's tempting to choose stores where the owner is enthusiastic. That enthusiasm can itself change results, since those owners may execute more carefully. Note it if it applies.

How do you match control stores?

On what predicts the outcome. Store size, format, location type and recent sales in the relevant category are the usual matching factors. The goal is a control group that would behave like the test group if nothing changed.

Check the match before the pilot. Plot test and control sales for several months before the start. If the lines track closely, the match is good. If they diverge, adjust the groups before starting.

The logic here is the same as A/B testing: hold everything constant except the change. In physical stores, you can't randomize perfectly, but careful matching gets close.

How many stores are enough?

It depends on how much sales vary. In a category with steady sales, fewer stores can detect a real effect. In a volatile category, more are needed.

A useful approach is to look at how much the measure varies from store to store and week to week before the pilot, and estimate how large an effect you'd need to detect. That tells you roughly how many stores and weeks the pilot needs. It's worth doing before committing, since a pilot that's too small often ends with an inconclusive result.

Length matters too. A pilot should run long enough to get past novelty effects and cover at least a few normal weeks. For promotions, include the weeks after, where a dip may appear.

What are the common traps?

Choosing stores after the fact. Deciding which stores count as the test group once results are in invites bias. Fix the list before the pilot starts.

Contamination. If test and control stores share a distributor, a neighborhood or promotional support, the change may spill over. Separating them geographically or operationally reduces this risk.

Overgeneralizing. A pilot in one region's independent stores supports conclusions about similar stores. It doesn't automatically apply elsewhere. For broader context on how categories move across independent stores, the NRS Insights monthly reports and its news page are useful references.