Cloud & platform · features.mlCI passing

Feature Store

Training sets that see exactly what serving sees, with the leakage it prevents measured.

View the code Evaluation report Demo tour Run it ›_ last push 5 days ago

0 of 44,000training rows that differ from what the online store served; offline AUC equals production AUC

1.00 to 0.71AUC of a label-date join offline and in production (honest model: 0.81 in both)

4 of 8features flagged when serving sends pence instead of pounds; the others stay quiet

Evaluation report: offline and production AUC for four ways of building training sets, and per-feature skew detection. Screenshot from the project's demo (synthetic data).

demo Demo tour

Screenshots from the running application, with its synthetic demo data. Select one to enlarge.

Offline and production AUC for four ways of building a training set; only point-in-time agrees.

Rows that differ from serving, and per-feature skew with a pence-for-pounds bug.

01 The problem

Models are trained on features computed one way and served features computed another. Joining today's feature table to old labels, or keying labels by the date the outcome was recorded, quietly leaks the future: the model looks brilliant offline and disappoints in production. Unit bugs in serving go unnoticed because models never crash.

02 How it works

Events arrive (some late)
Feature views defined once
Point-in-time training set
Online store, arrival order
Replay proves they match
PSI flags serving skew

03 What I built

Declarative, versioned window features with one aggregation function shared by the offline and online stores.
Point-in-time training sets built as known at each moment (event time inside the window and arrival time before the moment), and an online store replayed in arrival order to prove they match.
Training/serving skew detection with the population stability index per feature.

04 What I learned

Keying labels by the date the outcome was recorded gave a perfect offline AUC and a production AUC below the honest model; the leak looks exactly like a win.

Writing the online store independently and replaying the stream is the cheapest proof that training and serving agree.

05 Known limitations

Every repository documents what it doesn't do yet. The main ones:

In-memory stores; production needs a warehouse as-of join and a key-value store.
Window aggregations over one entity only.

06 Stack

PythonNumPyscikit-learnFeature engineeringMLOps

The repository has the design document, operations runbook, full evaluation report and known limitations. All data is synthetic, generated by code in the repository.

run Run it yourself

Every project runs on a laptop: no cloud account and no paid API needed.

git clone https://github.com/codeROCK-IZZY/feature-store.git
cd feature-store
pip install -e ".[dev]" && pytest
featurestore evaluate --gate          # writes eval/reports/report.html
featurestore lookup 7 --at 80