Evaluation reportGenerated by the project's CI evaluation on held-out data.About the projectCode on GitHub

Training that sees what serving sees

68,655 card transactions, 44,000 labelled moments ("will this customer buy in the next 14 days?"), 5.8% of events arriving a day or more late. The online store is replayed in arrival order and compared with every offline training row.

0%of point-in-time training rows differ from what serving returned
1.00 → 0.71AUC of a label-date join, offline then in production
0.812production AUC of the point-in-time model, exactly as measured offline
4 / 8features flagged when serving sends pence instead of pounds

What a leaky training set costs (AUC on later moments)

offline, what the team sees production, what customers get

training set built fromofflineproductiongap
point in time (this store)0.8120.812+0.000
event time, ignoring arrival0.8110.811+0.001
keyed on the label date1.0000.707+0.293
today's snapshot0.8630.773+0.090

Rows that differ from serving

training set built fromrows that differ from what serving returned
point in time (this store)0.00%
event time, ignoring arrival1.53%
keyed on the label date93.72%
today's snapshot95.23%

Training/serving skew (PSI, alert at 0.2)

featurehealthy servingserving with amounts in pence
txn_count_7d0.0230.023
txn_count_30d0.0560.056
amount_sum_7d0.0142.518
amount_sum_30d0.0445.982
amount_mean_30d0.0406.189
amount_max_30d0.0446.188
categories_30d0.0550.055
days_since_last0.0590.059

Online lookup p50 0.304 ms, p95 0.537 ms. Offline point-in-time join 3,233 rows per second.