Training that sees what serving sees
68,655 card transactions, 44,000 labelled moments ("will this customer buy in the next 14 days?"), 5.8% of events arriving a day or more late. The online store is replayed in arrival order and compared with every offline training row.
0%of point-in-time training rows differ from what serving returned
1.00 → 0.71AUC of a label-date join, offline then in production
0.812production AUC of the point-in-time model, exactly as measured offline
4 / 8features flagged when serving sends pence instead of pounds
What a leaky training set costs (AUC on later moments)
offline, what the team sees production, what customers get
| training set built from | offline | production | gap |
|---|---|---|---|
| point in time (this store) | 0.812 | 0.812 | +0.000 |
| event time, ignoring arrival | 0.811 | 0.811 | +0.001 |
| keyed on the label date | 1.000 | 0.707 | +0.293 |
| today's snapshot | 0.863 | 0.773 | +0.090 |
Rows that differ from serving
| training set built from | rows that differ from what serving returned |
|---|---|
| point in time (this store) | 0.00% |
| event time, ignoring arrival | 1.53% |
| keyed on the label date | 93.72% |
| today's snapshot | 95.23% |
Training/serving skew (PSI, alert at 0.2)
| feature | healthy serving | serving with amounts in pence |
|---|---|---|
| txn_count_7d | 0.023 | 0.023 |
| txn_count_30d | 0.056 | 0.056 |
| amount_sum_7d | 0.014 | 2.518 |
| amount_sum_30d | 0.044 | 5.982 |
| amount_mean_30d | 0.040 | 6.189 |
| amount_max_30d | 0.044 | 6.188 |
| categories_30d | 0.055 | 0.055 |
| days_since_last | 0.059 | 0.059 |
Online lookup p50 0.304 ms, p95 0.537 ms. Offline point-in-time join 3,233 rows per second.