Autonomous software delivery: evaluation
Deploys of good and bad versions into a simulated service with background incidents. Canary settings tuned on separate worlds; held-out worlds and held-back regression kinds below.
97.9%of bad versions rolled back (big-bang + alert: 66.1%)
1 of 208healthy versions rolled back (threshold canary: 52)
4.7%of the requests big-bang deploys harmed
37.5%of tests run per change, catching 99.8% of bugs
Rollout policies (held-out + held-back)
| Policy | Bad caught | Caught at ≤5% | Requests harmed | Healthy rolled back | Minutes to 100% |
|---|---|---|---|---|---|
| Big-bang deploy + service alert | 66.1% | 0 | 344,625 | 22 of 208 | 0 |
| Canary vs pre-deploy thresholds | 80.2% | 141 | 74,371 | 52 of 208 | 60 |
| Progressive delivery (this platform) | 97.9% | 105 | 16,189 | 1 of 208 | 90 |
Caught by kind
| Regression | Big-bang | Threshold | Progressive |
|---|---|---|---|
| One endpoint slower | 4/13 | 7/13 | 13/13 |
| Errors everywhere | 21/25 | 25/25 | 25/25 |
| Errors growing (leak) | 13/13 | 13/13 | 13/13 |
| Errors in one region | 14/23 | 17/23 | 23/23 |
| Stalls on one endpoint | 20/30 | 27/30 | 28/30 |
| Errors in bursts | 40/44 | 44/44 | 44/44 |
| Errors in one cell | 15/44 | 21/44 | 42/44 |
Test selection
| Strategy | Tests run | Bugs caught | Red, no bug |
|---|---|---|---|
| Every test | 100.0% | 100.0% | 19.9% |
| Tests in the changed directories | 8.0% | 71.0% | 1.8% |
| Impact analysis (this platform) | 37.5% | 99.8% | 7.3% |
3 generated repositories, 6,000 changes; known-flaky tests retried once for every strategy.
Gate (regression limits)
| Check | Value | Limit | Result |
|---|---|---|---|
| heldout_caught_share_min | 0.9808 | 0.95 | pass |
| fresh_caught_share_min | 0.9773 | 0.93 | pass |
| false_rollback_share_max | 0.0048 | 0.02 | pass |
| harm_vs_big_bang_max | 0.049 | 0.1 | pass |
| impact_bugs_caught_share_min | 0.9982 | 0.99 | pass |