Automating a report without automating an error
Automation removes manual effort and removes the person who would have noticed something was wrong. A manual report has an implicit check every time it is produced; an automated one runs whether or not the data behind it makes sense.
The controls that replace that implicit check are simple and they are usually added after the first incident rather than before.
What breaks in an automated report?
Upstream data that arrives incomplete, late, or with a changed structure. A source that adds a column, renames a field or restates history will not necessarily fail the job; it will produce a report that runs cleanly and says something different.
Silent success is the dangerous failure mode. A job that errors is noticed; one that completes with wrong numbers is not.
What controls are worth building?
Row counts against expectation, completeness against the expected universe, totals against the previous period within a tolerance, and a check that key dimensions have not gained or lost values. Four checks, all cheap, all catching the common failures.
A report that fails its own checks should say so on its face rather than in a log nobody reads.
How should exceptions be surfaced?
On the report itself, in a place a reader cannot skip. A banner stating that completeness was below threshold is worth more than an alert to an inbox, because the person reading the number is the person who needs to know.
Alerts to a monitoring channel are useful as well and they are not a substitute.
What about restatement?
An automated report re-run against current data quietly updates history. That is frequently desirable and occasionally disastrous, particularly for anything already circulated.
The defence is snapshotting outputs that will be referenced again, with the run date visible. A report whose historical figures change without notice cannot be used as a record.
How do you avoid automating a wrong definition?
By reviewing the logic at build time with somebody who understands the business meaning, not only the code. An automated report encodes a definition permanently, and a definitional error becomes a long-running silent misstatement.
Review the first several outputs manually against a hand calculation. That is the one-off cost that prevents a recurring error.
What should be documented?
The source, the logic, the checks, the owner, and the review date. Reports outlive the people who build them, and an unowned automated report will continue producing numbers long after anybody understands them.
A review date is what eventually retires reports nobody reads, which is a real and underrated benefit.
When should a report not be automated?
When the underlying question changes frequently, when judgment is needed each time, or when the output is used rarely enough that the build cost is not repaid. Automation suits stable, repeated questions.
Frequently asked questions
How do I know whether anybody reads a report?
Instrument it if the platform allows, or ask directly and act on the answer. Automated reports accumulate and a meaningful share of them are produced for a person who left.
What tolerance should a period-over-period check use?
Wide enough that normal variation does not trigger it and narrow enough to catch a structural break. Setting it from the series' own historical variation is more defensible than picking a round number.
Should a failing check stop the report?
For high-consequence reports, yes. For routine monitoring, publishing with a prominent warning is usually more useful than publishing nothing.
Who should own an automated report?
One named person, with the name on the report. Ownership recorded only in a system is ownership nobody feels.
How often should the logic be reviewed?
Annually, and whenever an upstream source changes. Source changes are the main cause of silent breakage and they are rarely announced in a way that reaches report owners.
What about reports built by people who have left?
They are the highest-risk category and they should be either adopted with a review or retired. An unowned report that nobody understands is a liability that keeps producing numbers.
Does automation improve accuracy?
It improves consistency, which is not the same thing. A consistently applied wrong definition is worse than an inconsistently applied right one, because it is harder to notice.
Should the checks be visible to readers?
A summary line is enough — completeness, run date, any exceptions. It puts the caveat where the number is read, which is the whole point.
What is the minimum viable control set?
Completeness, row count, and a period-over-period tolerance, all surfaced on the report. Three checks, and they catch most of what goes wrong.
What should happen when a check fires repeatedly?
Investigate the tolerance before investigating the data. A check that fires every period is either badly calibrated or describing genuine instability, and both need a decision rather than a habit of dismissal. Public series published by the Census Bureau pair estimates with quality indicators for the same reason.
### What should a brand actually do with this?
Add the checks when the report is built rather than after the first incident. Retrofitting controls onto a running report is more work and it happens only after something has already gone out wrong.
Then put an owner and a review date on it. Automated reports outlive their authors, and an unowned report producing numbers nobody understands is a liability rather than an asset.
Working with retail data? See NRS Insights data solutions or contact NRS Insights.