Judgment Can Add Accuracy

Humans responsible for forecasting often feel the need to adjust the output of sophisticated forecasting systems.  In many contexts this makes forecasts worse, but it is worth noting this is not always the case.  In the Fall 2007 Foresight, Robert Fildes and Paul Goodwin looked at four British-based supply chain companies:

A nationwide retailer. A leading international food company.  A subsidiary of a U.S. pharmaceutical company.  A manufacturer of own-label domestic cleaning products.  We collected data on over 60,000 forecasts, interviewed the companies’ forecasters, and observed forecast review meetings where managers discussed and approved any adjustments that they thought were necessary. …

Those working for the food manufacturer adjusted 91% of the forecasts that had been generated by their expensive and sophisticated forecasting software. The four forecasters employed by the retailer adjusted only about 8% of their forecasts, but then they had over 26,000 forecasts to make each week, so there probably wasn’t enough time to put their mark on each forecast. The pharmaceutical company held 17 forecast review meetings every month, tying up about 80 person hours of valuable management time. On average 75% of the statistical forecasts in our companies were adjusted. …

Many of the adjustments were small, and in some cases very small. It was as if forecasters sometimes simply wanted to put their calling card on forecasts by tweaking them slightly to show that they were still doing their job. Indeed, we received anecdotal evidence from a consultant that people at review meetings tend to adjust more of the forecasts that are presented earlier in the meetings, rather than later on. As the meeting progresses they tire and feel that they have already done enough to justify the meeting, so later forecasts are simply waved through. …

Despite these concerns, judgmental adjustments to statistical forecasts can still play a useful role in improving accuracy. Our study found that on average they lowered the average percentage error (MAPE) by 3.6 percentage points for all companies except the retailer.  …  Larger adjustments tend to improve accuracy [more] and 2) negative adjustments tend to be much more beneficial than positive …

When we analyzed the accuracy of the retailer’s adjustments they looked awful. The positive adjustments its forecasters made more than doubled the MAPE from 32% to 65%. Moreover 83% of these adjustments were either too large or in the wrong direction. …  [However,] Most people would probably consider a forecast to be an estimate of the most likely level of future demand. It turned out that the retail forecasters were estimating a different quantity. Often they were trying to determine the levels of demand that only had a small chance of being exceeded  that is, the level that would limit stock-outs.