Significance, confidence, and what a p-value does not say
Significance testing is widely reported and widely misread in commercial analysis. It answers one narrow question — how surprising the observed data would be if there were no effect — and it is routinely interpreted as answering several others.
Getting the interpretation right matters because decisions are made on it.
What does a significance test actually say?
That the observed result would be unlikely under a specific assumption of no effect. That is the whole claim. It is not the probability that the effect is real, it is not a measure of how large the effect is, and it is not a guarantee that the result will replicate.
Significance is a statement about data under a hypothesis, not about the hypothesis given the data, and that inversion is the most common error in the field.
Why is effect size more useful commercially?
Because decisions depend on magnitude. A statistically significant effect that is commercially trivial should not change anything, and a large effect that missed significance in a small test may well be worth pursuing further.
Reporting effect size first and significance second reflects how the decision is actually made.
What is a confidence interval?
A range constructed so that, across repeated studies, a stated proportion of such intervals would contain the true value. Practically, it communicates precision: a wide interval says the estimate is imprecise regardless of whether it excludes zero.
Confidence intervals are more informative than a p-value in nearly every commercial context, because they carry both the estimate and its uncertainty in one object.
What happens when many things are tested?
Some will appear significant by chance. Testing twenty subgroups at a conventional threshold produces roughly one apparently significant result even when nothing is going on, which is the mechanism behind a great many findings that fail to replicate.
The defence is to declare the primary analysis in advance and to treat everything else as exploratory. Adjusting thresholds for multiplicity is the formal version and declaring the primary is the practical one.
How should uncertainty be presented to a commercial audience?
As a range with a plain-language sentence. "Our best estimate is a lift of this size, and the data is consistent with anything between these two figures" is understood by everybody and commits the analyst to the honest state of knowledge.
The instinct to give a single confident number is understandable and it transfers the uncertainty from the analysis to the decision without telling anybody.
What about practical significance thresholds?
Agreeing in advance what size of effect would change the decision converts a statistical question into a commercial one. If a five percent lift would not change the plan, a test powered to detect five percent is the wrong test.
That conversation belongs before the analysis, and holding it tends to improve the test design as well as the interpretation.
What should never be said?
That a non-significant result proves no effect, that a significant result proves the hypothesis, or that a p-value measures importance. Each of those is a specific error with a specific correction, and each appears regularly in commercial decks.
Frequently asked questions
Is a conventional threshold a rule?
It is a convention, not a law of nature, and it was never intended as a decision rule for commercial questions. What matters is that the threshold is set in advance and is appropriate to the consequences of being wrong in each direction.
What does a non-significant result mean?
That the data did not provide strong evidence against no effect, which is not the same as evidence for no effect. Reporting the confidence interval alongside makes the difference visible, because a wide interval containing zero says something very different from a narrow one centred on zero.
Should I report p-values at all?
They are fine as one element and poor as the headline. Effect size with an interval is the more useful presentation, with significance noted where an audience expects it.
How does sample size interact with this?
Large samples make small effects significant and small samples make large effects non-significant. That is why significance alone is a poor guide to importance and why effect size has to be read alongside it.
What about Bayesian approaches?
They answer the question people usually think they are asking — how probable the effect is given the data — at the cost of requiring a prior. They are a legitimate alternative and they do not remove the need to think about what would change the decision.
Does a confidence interval tell me the probability the effect is in the range?
Not literally under the standard interpretation, though it is commonly read that way and the reading is usually harmless in practice. What matters commercially is that a wide interval means imprecision, which is understood correctly either way.
How do I avoid multiplicity problems?
Declare the primary outcome and the primary subgroup before looking, and label everything else as exploratory. That single discipline removes most of the risk without any statistical machinery.
Should uncertainty go in the executive summary?
Yes, in one sentence. Burying it in an appendix means the decision is made on a point estimate while the analysis knew better, which is the worst available combination.
What is the single most useful habit?
Report the range. It changes how results are discussed, it protects the analysis, and it prompts the right question about whether the test was large enough.
### What should a commercial audience be taught about this?
Three things: that a range is more informative than a point, that a non-significant result is not proof of no effect, and that significance says nothing about size. Those three cover most of the misreadings that occur in practice.
Teaching them once, in the context of a real decision, works better than a general explanation. The next time a result arrives, the vocabulary is already shared.
Working with retail data? See NRS Insights data solutions or contact NRS Insights.