Statistical significance for brand managers

Statistical significance tells you whether a difference is larger than you'd expect from random noise alone. That's all it tells you. It doesn't say the difference is large, important or worth acting on, and it doesn't say the thing you tested caused it. Brand managers who treat "significant" as a synonym for "real and big" end up misreading test results in both directions.

The concept is useful. The myths around it cause most of the trouble.

What does a p-value actually say?

A p-value is the probability of seeing a difference at least as large as the one you observed, if there were truly no effect. A small p-value means the result would be surprising if nothing were going on. A common threshold is 0.05, or one in twenty. Below that, the result is conventionally called significant.

A hypothetical: a brand runs a display test in 30 stores, with 30 matched control stores. Test stores sell an average of 4 more units per store per week. The p-value is 0.02. That means if the display did nothing, a difference this large would show up only about 2 times in 100 by chance. It's reasonable evidence the display did something.

What the p-value doesn't say is the probability the display works. That's a different question, and confusing the two is one of the most common errors in business analysis. The statistical significance entry covers the distinction in more depth.

Why isn't significant the same as important?

Because significance depends on sample size as much as on the size of the effect. With enough stores and enough weeks, even a tiny difference becomes significant. A display that adds a fraction of a unit per store per week can pass the test in a large enough study.

Whether that matters is a business question. Does the extra volume cover the cost of the display? Does it justify the shelf space? Significance can't answer those. You need the size of the effect, in units or dollars, alongside the p-value.

The reverse is also true. A small test might find a large effect that isn't significant, simply because there weren't enough stores to rule out noise. That's not proof the effect doesn't exist. It's a sign the test was too small to tell.

How do store counts affect a test?

A lot. Stores vary. Some sell far more than others, and each store's sales bounce around from week to week. The more variation there is, the more stores you need to see a real effect through the noise.

That's why test design matters before the test runs. Estimating how many stores you need, based on how much sales usually vary, prevents the frustrating result of a test that ran for months and couldn't tell you anything. The logic mirrors A/B testing in digital settings, adapted to physical stores.

Matching helps too. Pairing each test store with a similar control store, by size and recent sales, reduces the noise and lets a smaller test detect the same effect.

What are the other common myths?

That significance proves causation. It doesn't. A significant difference between test and control stores points to the tested change only if the stores were comparable and nothing else differed. Poorly matched groups can produce significant results for reasons unrelated to the test.

That a non-significant result means no effect. It means the test couldn't distinguish the effect from noise. That's different from showing the effect is zero.

That running many tests and reporting the significant ones is fine. If you test twenty things, one will likely pass the one-in-twenty threshold by chance. Decide what you're testing before you look at results. The NRS Insights site and its news page offer category context for designing fair comparisons.