Explorations / Agent limits
Confidently wrong is the failure that costs you
Building agents, the dangerous failures were never refusals. They were the confident mistakes — a clean, plausible, wrong action the harness had no reason to stop.
What it taught me
Open questionAn agent’s confidence tells you nothing about whether it’s right.
Still open
How do you build a system that knows when it doesn’t know, when the model itself often can’t tell you?
Where it bites
The failures that cost you are the confident ones
Building agents, I went in expecting the hard failures to be refusals — the model throwing up its hands, getting stuck, asking for help. Those turned out to be the easy case. A stuck agent is a visible agent. The failures that actually cost you are the confident ones: a clean, plausible, well-formed action that happens to be wrong, executed with no more hesitation than a correct one.
A harness can catch malformed output. It cannot catch a well-formed lie. The model returns exactly the shape you asked for, and that shape is a mistake. So the real question stops being “can the agent do the task,” and becomes the harder one: can the agent tell when it shouldn’t.
An agent’s expressed confidence is a UI affordance, not a signal about whether it’s right. Treating it as one is how you get hurt.
The same model, a different line
Move the line, not the model
A capable agent and a trustworthy one are mostly the same model in different harnesses. The thing that decides whether a confident mistake reaches the world is not the model’s intelligence. It’s where you draw the line it has to stop at. Below is that line, made draggable: pick how much autonomy to grant, then run the same confidently-wrong action and watch where it lands.
Acts without askingnothing (drafts only)
A wrong call landsstopped at a human
Its confidence in that delete is high at every setting. It is wrong, and it cannot feel the difference.
Fig. 01 — Same model, same mistake. Only the line moves.
the modelthe deferral lineThe honest part is the third setting. Catching that delete depends on the harness having judged “delete” as the kind of thing to defer. The line never removed the fallible judgment; it moved it one step upstream, from the action to the risk-classification, where it’s quieter and easier to miss.
- Refusal
- Visible, recoverable, cheap. The agent stops and you notice. Not the failure that hurts.
- Confident error
- Well-formed and wrong, executed at full speed. The expensive one — and the one a harness can’t see by shape.
- The deferral line
- Where autonomy ends and it has to ask. The real safety lives here, not in the model.
- Risk-classification
- Whether the agent recognizes an action as risky. Move the line and this becomes the new thing that has to be right.
What’s still open
Calibration is the part I can’t yet engineer away
The working answer so far is defensive: assume the agent is confidently wrong, and put the human checkpoint exactly where being wrong is expensive. That’s enough to ship. It is not the same as the model knowing when it’s wrong.
Calibration — a model with an honest sense of what it doesn’t know — is the limit I keep running into, and I don’t have a clean answer. You can’t fully outsource it to the harness, because the harness is asking the same model “are you sure?” and getting the same confident yes. That’s the open edge of this one, and I think it’s one of the real frontiers: a model that can say, and mean, “I don’t know.”