Cloud & platform · watch.mlopsCI passing

AWS MLOps Platform

Keeps a production model healthy after it ships.

View the code Demo tour Run it ›_ last push 1 week ago

0 monthsfrom each simulated change to its first alert

0 of 12false alarms on independent stable months

0.64 vs 0.58PR-AUC during drift, managed vs never retrained

Monitoring dashboard: model PR-AUC per month and drift per feature. Screenshot from the project's demo (synthetic data).

demo Demo tour

Screenshots from the running application, with its synthetic demo data. Select one to enlarge.

The champion model, pending approval and PR-AUC by month

Drift per feature and recall per equipment type

Approval: candidate vs. champion on the same month, every gate

A retrain blocked by the gates, with the reason

Versioned monthly datasets that passed the data contract

01 The problem

Most ML projects end at "the model works on the test set". In production the data keeps changing, models get quietly worse, scheduled retraining ships models nobody checked, and drift dashboards raise alarms nobody can act on.

02 How it works

Monthly data batch
Data contract + versioned S3
Tracked training (MLflow)
Gates vs. champion on same data
Human approval
SageMaker canary + auto-rollback
Drift + performance monitoring → retrain

03 What I built

Data contracts and versioned datasets; tracked, reproducible training with MLflow and a model registry.
Promotion gates that score the candidate and the champion on the same fresh data: quality floor, no regression, calibration, recall for every equipment type, latency.
Monitoring of input drift, prediction drift and measured performance; alerts start a retraining run by themselves.
A SageMaker-compatible model server and canary deployments that roll back automatically when error or latency alarms fire.

04 What I learned

A 0.02 PR-AUC tolerance raised false alarms on 6 of 12 stable months, because about 200 failures a month make PR-AUC wobble by ±0.03. Monitoring now uses 0.08; promotion keeps 0.02 because both models see the same month there.

A new failure mode was invisible to input-drift checks: only the measured PR-AUC collapse (0.77 to 0.39) revealed it.

05 Known limitations

Every repository documents what it doesn't do yet. The main ones:

Labels arrive at least 7 days late, so performance monitoring lags.
The AWS path is tested with mocked APIs and terraform validate, not a live account.

06 Stack

PythonMLflowXGBoostSageMakerTerraformFastAPIReact

The repository has the design document, operations runbook, full evaluation report and known limitations. All data is synthetic, generated by code in the repository.

run Run it yourself

Every project runs on a laptop: no cloud account and no paid API needed.

git clone https://github.com/codeROCK-IZZY/aws-mlops-platform.git
cd aws-mlops-platform
cp .env.example .env    # set the CHANGE_ME values
docker compose up -d --build
docker compose run --rm api python -m app.cli demo    # a year of data (~6 minutes)

Then open http://localhost:8088. The README has the demo logins and everything else.

07 Notes from this project

Why my drift monitor cried wolf on half the stable months · A 0.02 tolerance on monthly PR-AUC raised 6 alarms in 12 months where nothing had changed. The noise was in the measurement, not the model.