Explorations / Evaluation
The number went up; the thing got worse
I run batch evals on a voice agent in production. Watching the score climb while real calls degraded taught me more than the score ever did: the eval measured the part that was easy to measure.
What it taught me
OngoingA benchmark you optimize against stops measuring the thing you cared about.
Still open
How do you evaluate a system for the failure you haven’t imagined yet?
From the bench
The score climbed; the calls got worse
I run batch evals on a voice agent in production — real-shaped calls, scored, root-caused, iterated. For a while the dashboard told a clean story: the number went up. Then I started listening to the calls behind the number, and the calls were getting worse. Not on the axis the eval measured. On every axis it didn’t.
That gap taught me more than the metric ever did. The eval measured the part that was easy to measure, the agent learned to satisfy exactly that, and the part I actually cared about drifted out from under it while the score kept rising.
Every eval is a bet that you imagined the right failure in advance. The failures that hurt are the ones you didn’t.
Why it happens
Why the proxy drifts from the point
Goodhart’s law — once a measure becomes a target, it stops being a good measure — gets quoted like a curiosity. In an eval loop it is the central dynamic. The moment you optimize against a benchmark, you are training the system to close the distance between the proxy and the score, including all the distance that has nothing to do with what you meant.
- What it measures
- The legible slice — did it say the right words, hit the schema, finish the turn.
- What it misses
- Tone, recovery, knowing when to stop, the call that technically succeeded and lost the customer.
- The trap
- A number that rises while the experience falls reads as progress. It’s the most expensive kind of wrong.
The number climbs exactly as the thing it stood for slips
Fig. 01 — You optimize the proxy; the score and the outcome come apart
the proxythe driftWhat’s still open
Evaluate adversarially, or don’t bother
The honest version of evaluation I’ve landed on isn’t a leaderboard. It’s adversarial: spend your effort trying to make the system fail in a new way, because the ways you already test for are the ways it has already learned to pass. A green dashboard mostly means you’ve stopped looking where it breaks.
The open edge — and it’s a real one — is that I don’t know how to systematically generate the failure I haven’t thought of yet. Right now it takes a human with taste and suspicion sitting with the actual output. That doesn’t scale, and I haven’t found the thing that replaces it without quietly reintroducing the same blind spot one level up.