A message can pass every check and still be wrong.
Tracer is our benchmark for that. We hide realistic clinical errors in HL7 messages that pass format checks, and count what gets caught.
What Tracer tests
A lab result can have every field in place and still report a value no living patient could have. Format validation passes it.
Tracer plants exactly one error in some messages and leaves the rest clean, so every finding can be scored right or wrong.
What we found
We planted one error in 499 of 1,000 messages. The errors come in 16 kinds, from impossible lab values to drug interactions.
Pidgeon’s rules caught 15 of the 16 kinds. They raised 1 false alarm across the 501 clean messages. Each check took about 0.7 milliseconds.
The 16th kind is a contradiction in free text, such as “no known allergies” above a coded allergy. That goes to the AI reviewer, and we will publish its results after clinical review.
We built this test set ourselves. We scored the rules on messages they never saw while we wrote them. A fixed seed rebuilds the same set every time.
- Messages with a planted error
- 499
- Clean messages
- 501
- False alarms
- 1
- Time per check
- ~0.7 ms
How to check our work
- We generate the messages from a fixed seed, so the same seed always gives the same 1,000 messages.
- We plant one known error in 499 of them and keep 501 clean, so false alarms show up as clearly as misses.
- We compare every finding to the answer key. We publish the errors caught, the false alarms, and the time per check.
Where the AI runs
For free-text errors, you choose where the model runs. The same benchmark scores each option.
What we’re testing next
Next we will score each AI model on its own, with general tools, and with Pidgeon’s healthcare tools, to show how much the tools add.