November 1, 2026 · from Fraud Detection Engine
A random split said 0.929. The honest number was 0.709.
How you split the data changes a fraud model's score more than how you build the model.
Fraud models are usually evaluated by shuffling transactions and holding out a random 20%. For the fraud engine, that would have reported a PR-AUC of 0.929.
In production, a model is trained on the past and scores the future. Fraud patterns move, cards and devices recur, and a random split lets the model peek at the same card's other transactions. Evaluated the way it will be used, trained on earlier weeks and tested on later ones, the same model scores 0.709.
What I report instead
- Time-based splits only.
- Results at the operating point the analysts actually use: about 31 alerts a day, catching 93% of fraud dollars, with 96% precision in the critical band.
- The random-split number, shown next to the honest one, so nobody mistakes it for performance.
A lower number that holds up beats a higher number that collapses on the first real month. It is also the number an employer can trust.