Forecast error: measuring it honestly
A forecast's value comes from being consistently and measurably better than the alternative, not from being right. Measuring error honestly is what turns forecasting from an opinion into a process that improves.
Most commercial forecasting is evaluated loosely or not at all, which is why it rarely improves.
What should be measured?
Both the size and the direction of the error. Size tells you how much uncertainty to plan around; direction tells you whether the process is biased, which is the more fixable problem.
A forecast that is consistently high by a similar amount is more valuable than one that is unbiased and wildly variable, because the bias can be corrected.
Which measures are used?
Absolute error and its percentage forms for size, and mean error for bias. Each has known weaknesses: percentage errors behave badly near zero and are asymmetric between over- and under-forecasting, which matters in intermittent-demand categories.
The practical answer is to report more than one and to state which is being used. A single measure quoted without its name invites the reader to assume the most favourable interpretation.
Why do percentage errors mislead?
Because the denominator changes. A large percentage error on a small item and a small percentage error on a large item can represent the same absolute miss, and an aggregate percentage error weighted by nothing in particular is dominated by the small items.
For stock and supply decisions, absolute error weighted by value is usually closer to the consequence. For a portfolio view, the weighting choice should be stated because it changes the answer.
At what level should error be measured?
At the level the forecast is used. A forecast used for store replenishment should be evaluated at store-item level; one used for production planning at the aggregate it feeds.
Error falls as things are aggregated, so a national-level accuracy figure describes a forecast that nobody uses for a decision at that level. Quoting the aggregate figure to describe a store-level process is flattering and misleading.
What is a naive benchmark?
The simplest possible forecast — last period repeated, or a seasonal naive — used as a baseline. A sophisticated forecast that does not beat the naive benchmark has not earned its complexity.
Running the benchmark is cheap and it is the single most informative comparison available. It is also the one most often omitted from forecast performance reporting.
How should bias be handled?
Measured, reported and corrected in the process rather than in the numbers. Persistent bias usually comes from an incentive or an assumption rather than from a model, and adjusting the output without addressing the cause reintroduces it next cycle.
That makes bias a management finding as much as an analytical one.
What should a forecast review contain?
Error by level, bias, comparison against naive, and the largest individual misses with explanations. The last element is where the learning is; the aggregate figures only tell you whether things are improving.
Frequently asked questions
What accuracy should I expect?
There is no universal figure and be sceptical of one offered without knowing the category, the horizon and the level. What matters is the trend in your own error and the comparison against a naive benchmark.
Should error be measured weekly or monthly?
At the frequency of the decision the forecast supports. Measuring monthly a forecast used weekly hides the misses that actually caused problems.
Does a longer horizon always mean more error?
Generally yes, which is why error should be reported by horizon rather than pooled. A single accuracy figure across all horizons conceals where the process breaks down.
How do I handle new items with no history?
They need a different process and a different expectation, and including them in an overall accuracy figure degrades it for reasons unrelated to the established process. Report them separately.
What about promotional periods?
They are usually the largest source of error and they deserve separate measurement. A base forecast evaluated including promotional weeks is being judged on something it was not designed to predict.
Is a low error always good?
Not if it was achieved by forecasting a flat line in a stable category, which a naive benchmark would have matched. Improvement over the benchmark is the meaningful claim.
Should forecasters see their own error?
Yes, promptly and at the level they work. Feedback delayed by a quarter and aggregated to a level they do not influence does not change anything.
Does more data always improve a forecast?
Not automatically. More relevant data helps; more of the same data usually does not, and added complexity without added signal tends to increase error out of sample.
What is the first improvement for a team with no error measurement?
Start measuring against a naive benchmark, at the level of use, with bias reported separately. That alone usually reveals where the effort should go. Standard treatments of forecasting practice set out the measures and their trade-offs.
### Who should see the error report?
Whoever influences the forecast, which usually includes commercial and supply as well as the forecasting team. Error driven by a commercial assumption cannot be fixed by the people who convert assumptions into numbers.
Reporting bias by source is uncomfortable and it is where most of the improvement is. A process that consistently over-forecasts in one region or one category has a cause with a name.
Working with retail data? See NRS Insights data solutions or contact NRS Insights.