Evaluation reportGenerated by the project's CI evaluation on held-out data.About the projectCode on GitHub

Property management: evaluation

Maintenance requests in tenants' own words, triaged and dispatched in simulated months; and a year of rent through two ways of allocating payments.

96.8%of emergencies attended within 4 hours with triage, vs 59.2% with keyword rules and 51.4% in inbox order (fresh phrasings)
23 h vs 62 hlongest wait for an emergency: triage vs inbox order
90.4%of emergency messages caught, vs 14.8% for keyword rules, at 19.3% false alarms
0 vs 1,756late fees charged while all rent was paid: rent-first allocation vs strictly oldest first

Validation phrasings (seen while the lexicon was written)

MessagesEmergencies caughtFalse alarmsUrgency rightTrade right
Keyword rules9.7%13.9%47.9%69.3%
Learned model, words only67.4%26.0%49.8%57.6%
Triage (hazard screen + model)100.0%15.6%62.7%88.9%

Simulated months: 2,324 requests, 262 emergencies.

PolicyWithin 4 hMedian90th pctWorstFalse alarms / weekAt night
Inbox order, no triage53.1%3.6 h32.9 h62.8 h0.00.0
Keyword rules flag emergencies56.5%3.4 h32.2 h62.8 h12.82.0
Triage flags emergencies100.0%2.2 h3.0 h3.2 h12.82.1

Fresh phrasings (written after the freeze): the test

MessagesEmergencies caughtFalse alarmsUrgency rightTrade right
Keyword rules14.8%8.6%61.0%67.3%
Learned model, words only84.4%30.5%64.8%52.7%
Triage (hazard screen + model)90.4%19.3%71.5%77.2%

Simulated months: 3,235 requests, 311 emergencies.

PolicyWithin 4 hMedian90th pctWorstFalse alarms / weekAt night
Inbox order, no triage51.4%3.7 h28.5 h62.3 h0.00.0
Keyword rules flag emergencies59.2%3.2 h25.1 h61.6 h8.31.3
Triage flags emergencies96.8%2.2 h3.1 h23.1 h17.63.1

Emergencies triage missed (sample):

Rent ledger: a year, 5 portfolios

AllocationLate feesFees totalFees while all rent was paidPaid all rent, shown owing rent
rent first1,639$81,95000 of 904
oldest first3,395$169,7501,756286 of 904

Gate (regression limits)

Passed