Demand Forecasting with Cost-Aware A/B Testing
The full Walmart M5 panel — all 30,490 store-SKU series — carried past the leaderboard to the question a real team has to answer: does the better model actually save money, and should it ship? A pre-registered A/B test on a newsvendor cost simulation says yes — SHIP at −14.1%. The whole pipeline runs on a 16 GB laptop in ~20 minutes.
01 Overview
M5 is usually a Kaggle accuracy contest. This project treats accuracy as the start, not the finish: a lower error metric doesn't matter unless it changes an ordering decision and saves money. So every forecast is turned into an order quantity, every ordering mistake is priced (understock costs 5× overstock), and the two systems are compared as a pre-registered A/B test.
What makes it credible is the discipline: the decision rule is written down before results, model selection and confirmation happen on separate time windows, and the honest failures — including two earlier HOLD verdicts — are preserved in the git history rather than edited out.
02 Results
Accuracy — 30,490 series, rolling-origin backtest (3 folds × 4-week horizon), volume-weighted WMAPE:
| Model | WMAPE | Bias | MASE (med) |
|---|---|---|---|
| ensemble (0.5·lgbm + 0.5·ets) | 0.3649 | −2.0% | 0.624 |
| lightgbm (one per store) | 0.3706 | −1.2% | 0.635 |
| ets (AutoETS, season=52) | 0.3757 | −2.8% | 0.636 |
| ma4 (best simple baseline) | 0.3875 | −1.3% | 0.650 |
| naive_last | 0.4283 | −5.3% | 0.733 |
The A/B test — control = the ordering system a retailer actually runs (ma4 + Gaussian safety stock); treatment = LightGBM trained with pinball loss to predict the cost-optimal order quantile directly:
| Pre-registered gate | Result | Pass |
|---|---|---|
| Mean cost reduction ≥ 5% | −14.1% (p ≈ 10⁻¹¹⁰) | ✓ |
| Stockout increase ≤ +2pp | +1.57pp | ✓ |
| Verdict | SHIP | ✓ |
03 Architecture
From 30,490 raw daily series to a shippable ordering policy — the analysis pipeline at a glance, then each step in detail below. Chosen to keep the whole thing on a laptop and the conclusion honest.
Chunked ETL to weekly grain
Melting all series at once needs 20–30 GB. Instead, per-store chunks (10 × ~3,049 series) build weekly features without ever materialising the 59M-row daily frame — 68s, 3.7 GB peak, sums reconcile exactly. Daily is 62% zero-sales noise; weekly cuts that to ~12% volume-weighted.
Decision-driving EDA
Ten views that each justify a modelling choice: the Syntetos-Boylan intermittency map (smooth / lumpy / erratic), SNAP as a dose-response (FOODS +22% in full-SNAP weeks), and two honest non-findings on price cuts and holidays.
LightGBM + classical challenger
One LightGBM per store (Tweedie objective for zero-inflated demand, recursive multi-step) vs AutoETS (season 52) as the classical challenger it must beat — and does. A 50/50 blend of the two is the accuracy champion; each family fixes the other's errors.
Newsvendor cost simulation
Every forecast becomes an order; the newsvendor-optimal order is the τ* ≈ 0.833 demand quantile (5:1 understock:overstock). The treatment model learns that quantile per SKU with pinball loss instead of "mean + z·σ" — same target, better estimator.
Pre-registered A/B test
The decision rule is fixed before results. τ is selected on folds 1–2 and judged once on fold 3 (chosen τ = 0.866). A paired design over ~30K series, with the ≥5% practical-significance gate — not the p-value — carrying the decision.
Verdict & sensitivity
SHIP at −14.1%, holding at 3:1 economics; against the same safety-stock policy on LightGBM-mean, quantile ordering still saves −9.4% — proving the ordering policy, not just the model, drives the gain.
04 What makes it rigorous
Money, not error
Every forecast becomes a priced order (understock 5:1 overstock), so the metric that decides shipping is dollars saved — not a WMAPE nobody can act on.
Pre-registered & separated
The decision rule was stated before results; τ selection (folds 1–2) and confirmation (fold 3) are kept apart, so the effect size — not a hackable p-value — carries the call.
No leakage, tested
Time-only splits; every lag/rolling feature is .shift()-ed. A unit test perturbs the current week and fails if any feature moves — 18 tests run in CI.
Full M5 on 16 GB
Per-store chunked ETL, aggressive dtypes, and per-store checkpointing keep peak RSS at 4.5 GB — any crash resumes instead of restarting.
Learn the quantile, not the mean
Rather than post-hoc safety stock, the treatment predicts the cost-optimal demand quantile as the order quantity — a better estimator of the exact thing that's being decided.
Honest development trail
The first version of this test was a HOLD — the accurate model lost money by under-ordering. That arc, including both earlier HOLDs, is preserved in git rather than hidden.
05 Stack
The full write-up is in the repo
Including the EDA gallery, the two earlier HOLD verdicts, and the complete A/B methodology.