Explorations / Model limits
Most “reasoning” bugs are memory bugs in disguise
The more agents I build, the more “the model can’t reason” turns out to be “the model never saw the one thing that mattered, at the moment it mattered.”
What it taught me
OngoingMost reasoning failures are really context and retrieval failures wearing a costume.
Still open
As context windows grow, does memory get solved, or just deferred to a bigger version of the same problem?
What actually breaks
I expected reasoning. I got memory.
The wall I expected to hit, building agents, was reasoning — the model not being smart enough to work the problem. The wall I actually hit, over and over, was memory. The model forgets what it decided three steps ago, or never saw the one fact that mattered, and then reasons fluently around the gap. The chain of thought looks fine. It was just thinking about the wrong state of the world.
A bigger context window doesn’t give the model memory. It gives you a bigger junk drawer to lose things in.
The work, relocated
Half of “prompting” is deciding what it gets to see
Once you notice this, a lot of “prompt engineering” reveals itself as something more mundane and more important: deciding what the model gets to see, and when. That’s not a reasoning problem. It’s a retrieval problem in a costume — which fact, in which form, at which step. Get that right and a “dumb” model looks sharp. Get it wrong and the smartest model available confabulates.
- Looks like
- “The model can’t reason about this.” The output is wrong in a way that reads as a thinking failure.
- Usually is
- The relevant fact wasn’t in context, or was buried where attention didn’t reach it.
- The real lever
- Curation, not cleverness. What you put in front of the model, and what you deliberately leave out.
A bigger window is a bigger drawer to lose things in
Fig. 01 — The window fills, attention spreads thin, the one fact gets buried
the contextthe buried factWhat’s still open
The frontier might be forgetting well
Longer context helps, and quietly introduces its own failure: attention spread thin, the middle of a long window the first place things slip. So I don’t think more tokens solves memory. It relocates it, from “the model can’t hold enough” to “the model holds too much and can’t find the part that counts.”
The version of this I can’t yet answer: as windows keep growing, does memory get solved, or does it defer to a bigger version of the same problem? My suspicion is that the interesting work isn’t teaching a model to remember everything. It’s teaching it to forget well — to know what to drop. That’s a harder thing to specify than it sounds, and I don’t have it.