Evaluation reportGenerated by the project's CI evaluation on held-out data.About the projectCode on GitHub

Demand forecasts, backtested

136 series over 1092 days (total, 3 regions, 12 stores, 120 store x product). Eight four-week blocks, each forecast from data before it only; intervals calibrated on the six blocks before those.

36%less error than seasonal naive on store x product (WAPE)
81.8%of actuals inside the 80% interval after conformal calibration (raw: 72.0%)
51%lower stock cost ordering at the 80th percentile instead of seasonal naive
9.7%of volume where unreconciled forecasts don't add up (0 after reconciliation)

Accuracy by level (WAPE; best in green)

methodtotalregionstorestore x productbias, store x product
seasonal naive0.2200.2550.3400.676-6.6%
Holt-Winters0.2170.2440.3110.600-7.6%
GBM (base)0.1550.1900.2510.431-7.5%
GBM bottom-up0.1100.1400.2070.431-7.5%
GBM + WLS0.1230.1560.2200.451-2.9%
GBM + MinT0.1190.1520.2170.446-4.4%

80% interval coverage

totalregionstorestore x product
raw quantiles82.6%76.2%76.1%72.0%
conformal84.8%85.7%83.7%81.8%

Hard cases (WAPE)

caseseasonal naiveHolt-WintersGBM + MinT
after the structural break0.9590.8620.535
store opened a year late0.7860.6610.567

Stock decisions (a unit short costs 4, a unit left over 1)

policyfill rateunits shortunits left overcost vs seasonal naive
seasonal naive (point)62.9%82,66568,017+0.0%
GBM median74.7%56,30639,481-33.6%
GBM 80th percentile90.9%20,290113,252-51.2%
GBM 80th percentile, conformal91.9%18,114123,773-50.8%

Median forecasts of skewed demand sit below the mean, hence the negative bias; that is what minimises absolute error, and why stock is set from a higher quantile instead.