Accuracy against the answer key
Every planted problem in the replayed week (2026-11-10 to 2026-11-16) scored against what the operator found and did. Source: replay-seed23.json (seed23), the week before the busiest window at 300 s steps, then the window itself at 30 s.
143planted problems in the week
92%precision (140 of 152 findings real)
97%recall (138 of 143 found)
62%handled without a person
4actions undone
90 minmean time to resolve
Weakest detector: duplicate patient, 67% precision over 18 findings.
Lowest recall: result misfiled, 60% (10 planted).
The known gaps page says why.
| Detector | Findings | Precision | Planted | Recall | Time to resolve |
|---|---|---|---|---|---|
| abnormal result unacknowledged | 26 | 100% |
26 | 100% |
87 min |
| authorisation expiring | 16 | 100% |
16 | 100% |
428 min |
| clinician sick full list | 6 | 100% |
6 | 100% |
2 min |
| consent missing | 19 | 95% |
18 | 100% |
23 min |
| double booking | 12 | 100% |
12 | 100% |
3 min |
| duplicate patient | 18 | 67% |
12 | 100% |
88 min |
| no show cascade | 10 | 80% |
7 | 86% |
45 min |
| post discharge followup overdue | 14 | 100% |
14 | 100% |
27 min |
| referral not booked | 22 | 100% |
22 | 100% |
28 min |
| result misfiled | 9 | 67% |
10 | 60% |
37 min |
Decided by: rules 53 · model cost 0.00 USD · left waiting for sign-off at the end: 8
The answer key in this database
| Kind of problem | Planted in the world | Live in the replayed week |
|---|---|---|
| abnormal result unacknowledged | 66 | 26 |
| authorisation expiring | 16 | 16 |
| clinician sick full list | 16 | 6 |
| consent missing | 18 | 18 |
| double booking | 30 | 12 |
| duplicate patient | 32 | 12 |
| no show cascade | 17 | 7 |
| post discharge followup overdue | 29 | 14 |
| referral not booked | 47 | 22 |
| result misfiled | 22 | 10 |