Demand forecasts, backtested
136 series over 1092 days (total, 3 regions, 12 stores, 120 store x product). Eight four-week blocks, each forecast from data before it only; intervals calibrated on the six blocks before those.
36%less error than seasonal naive on store x product (WAPE)
81.8%of actuals inside the 80% interval after conformal calibration (raw: 72.0%)
51%lower stock cost ordering at the 80th percentile instead of seasonal naive
9.7%of volume where unreconciled forecasts don't add up (0 after reconciliation)
Accuracy by level (WAPE; best in green)
| method | total | region | store | store x product | bias, store x product |
|---|---|---|---|---|---|
| seasonal naive | 0.220 | 0.255 | 0.340 | 0.676 | -6.6% |
| Holt-Winters | 0.217 | 0.244 | 0.311 | 0.600 | -7.6% |
| GBM (base) | 0.155 | 0.190 | 0.251 | 0.431 | -7.5% |
| GBM bottom-up | 0.110 | 0.140 | 0.207 | 0.431 | -7.5% |
| GBM + WLS | 0.123 | 0.156 | 0.220 | 0.451 | -2.9% |
| GBM + MinT | 0.119 | 0.152 | 0.217 | 0.446 | -4.4% |
80% interval coverage
| total | region | store | store x product | |
|---|---|---|---|---|
| raw quantiles | 82.6% | 76.2% | 76.1% | 72.0% |
| conformal | 84.8% | 85.7% | 83.7% | 81.8% |
Hard cases (WAPE)
| case | seasonal naive | Holt-Winters | GBM + MinT |
|---|---|---|---|
| after the structural break | 0.959 | 0.862 | 0.535 |
| store opened a year late | 0.786 | 0.661 | 0.567 |
Stock decisions (a unit short costs 4, a unit left over 1)
| policy | fill rate | units short | units left over | cost vs seasonal naive |
|---|---|---|---|---|
| seasonal naive (point) | 62.9% | 82,665 | 68,017 | +0.0% |
| GBM median | 74.7% | 56,306 | 39,481 | -33.6% |
| GBM 80th percentile | 90.9% | 20,290 | 113,252 | -51.2% |
| GBM 80th percentile, conformal | 91.9% | 18,114 | 123,773 | -50.8% |
Median forecasts of skewed demand sit below the mean, hence the negative bias; that is what minimises absolute error, and why stock is set from a higher quantile instead.