Manish Karney

Explorations / Agent limits

Confidently wrong is the failure that costs you

Building agents, the dangerous failures were never refusals. They were the confident mistakes — a clean, plausible, wrong action the harness had no reason to stop.

What it taught me

Open question

An agent’s confidence tells you nothing about whether it’s right.

Still open

How do you build a system that knows when it doesn’t know, when the model itself often can’t tell you?

AgentsTrustCalibration

Where it bites

The failures that cost you are the confident ones

Building agents, I went in expecting the hard failures to be refusals — the model throwing up its hands, getting stuck, asking for help. Those turned out to be the easy case. A stuck agent is a visible agent. The failures that actually cost you are the confident ones: a clean, plausible, well-formed action that happens to be wrong, executed with no more hesitation than a correct one.

A harness can catch malformed output. It cannot catch a well-formed lie. The model returns exactly the shape you asked for, and that shape is a mistake. So the real question stops being “can the agent do the task,” and becomes the harder one: can the agent tell when it shouldn’t.

An agent’s expressed confidence is a UI affordance, not a signal about whether it’s right. Treating it as one is how you get hurt.

The same model, a different line

Move the line, not the model

A capable agent and a trustworthy one are mostly the same model in different harnesses. The thing that decides whether a confident mistake reaches the world is not the model’s intelligence. It’s where you draw the line it has to stop at. Below is that line, made draggable: pick how much autonomy to grant, then run the same confidently-wrong action and watch where it lands.

Autonomy granted

Acts without askingnothing (drafts only)

A wrong call landsstopped at a human

Its confidence in that delete is high at every setting. It is wrong, and it cannot feel the difference.

Fig. 01 — Same model, same mistake. Only the line moves.

the modelthe deferral line

The honest part is the third setting. Catching that delete depends on the harness having judged “delete” as the kind of thing to defer. The line never removed the fallible judgment; it moved it one step upstream, from the action to the risk-classification, where it’s quieter and easier to miss.

Refusal
Visible, recoverable, cheap. The agent stops and you notice. Not the failure that hurts.
Confident error
Well-formed and wrong, executed at full speed. The expensive one — and the one a harness can’t see by shape.
The deferral line
Where autonomy ends and it has to ask. The real safety lives here, not in the model.
Risk-classification
Whether the agent recognizes an action as risky. Move the line and this becomes the new thing that has to be right.

What’s still open

Calibration is the part I can’t yet engineer away

The working answer so far is defensive: assume the agent is confidently wrong, and put the human checkpoint exactly where being wrong is expensive. That’s enough to ship. It is not the same as the model knowing when it’s wrong.

Calibration — a model with an honest sense of what it doesn’t know — is the limit I keep running into, and I don’t have a clean answer. You can’t fully outsource it to the harness, because the harness is asking the same model “are you sure?” and getting the same confident yes. That’s the open edge of this one, and I think it’s one of the real frontiers: a model that can say, and mean, “I don’t know.”