Reproducibility in commercial analytics
An analysis is reproducible when somebody else, given the same inputs, can follow the same steps and reach the same result. In commercial settings this is rarely achieved and rarely attempted, and the cost appears months later when a number is challenged and nobody can reconstruct it.
The requirements are modest and mostly organisational.
What has to be recorded?
The data source with its version and extraction date, the transformations applied, the filters and exclusions, and the code or steps used. Four elements, and the exclusions are the one most often lost.
An exclusion applied in a spreadsheet by deleting rows leaves no trace, which is why filtering rather than deleting is worth the habit.
Why does the extraction date matter?
Because data is restated. History moves as late files arrive, errors are corrected and methodologies change, and an analysis re-run against current data will not reproduce a result computed six months ago.
Stamping the extract is one field and it converts an irreproducible result into a reproducible one.
What breaks reproducibility most often?
Manual steps. A spreadsheet where somebody sorted, pasted, adjusted a cell and moved on cannot be reproduced, even by the person who did it, a month later.
The fix is not necessarily code. It is recording the steps in an order somebody could follow, which a well-organised spreadsheet with a documented process can achieve.
How much rigour is appropriate?
Proportionate to the consequence. A quick exploratory look needs no ceremony; a figure that will appear in a trade negotiation, a board pack or a public claim needs to be reconstructible.
Grading work by consequence is more workable than applying one standard to everything, which in practice means applying it to nothing.
What about the results themselves?
Archive the output alongside the inputs. A saved result with its extract date and method note is what lets somebody check a claim without re-running anything, and it is what catches a silent restatement.
How does this interact with automation?
Automation improves reproducibility of the steps and can worsen reproducibility of the results, because an automated report re-runs against current data by default and quietly updates history.
The defence is snapshotting: automated outputs that will be referred to again should be stored as of their run date rather than regenerated on demand.
What should a team standardise?
A location for analyses, a naming convention, a required header with source and date, and a rule about what gets archived. None of that is sophisticated and together it is most of the benefit.
Frequently asked questions
Does this require code?
No, though code makes it easier. A documented sequence of steps in a spreadsheet with filters rather than deletions is reproducible; an undocumented sequence in any tool is not.
Who should be able to reproduce it?
A competent colleague with access to the same data, without needing to ask the original analyst. That is the practical test and most commercial analyses fail it.
What about proprietary provider data?
Reproducibility within your own organisation is still achievable if you archive the extract you used. Where licence terms constrain retention, that constraint should be known before a claim depends on the data.
How long should analyses be retained?
As long as any decision based on them remains live, plus a margin. For trade terms and public claims that is usually years rather than months.
Is version control worth it for a small team?
For anything code-based, yes, and it is inexpensive. For spreadsheet-based work, a disciplined naming and archiving convention achieves much of the benefit.
What about a one-off analysis?
One-off analyses have a habit of being referred to again, which is precisely why the header and the archive are worth the two minutes.
Does reproducibility mean the analysis is right?
No — it means it can be checked. A reproducibly wrong analysis is still wrong, and it is much easier to find the error.
How do I start with an existing mess?
From now on rather than retrospectively. Apply the standard to new work and retrofit only the analyses that are still load-bearing.
What is the minimum viable practice?
Source, extract date, exclusions, and an archived output. Four things at the top of every analysis, and most of the problem disappears.
### What does this look like for a small team?
A shared folder, a naming convention, and a header block in every file. That is the whole implementation and it takes an afternoon to agree.
The discipline is harder than the setup. A convention that one person follows is a convention that fails the moment somebody else has to find something, which is exactly when reproducibility matters. Public statistical practice, such as the Census Bureau's documented methodology, is a useful model for what a durable record looks like.
What about analyses done in a provider's own tool?
Export and archive the result with its parameters, because tool contents change and access ends when a contract does. An analysis that exists only inside a supplier's platform is not a record you control.
### What should a brand actually do with this?
Apply the standard by consequence rather than universally. Exploratory work needs none of this; anything that will inform a trade negotiation, a public claim or an incentive calculation needs all of it.
Grading it this way is what makes the practice survive. A single standard applied to everything is abandoned within a quarter, and the high-consequence work then has no protection at all.
Working with retail data? See NRS Insights data solutions or contact NRS Insights.