Explorations / Interfaces
The hardest integration is into the system with no door
A voice front-end and a worker that drives a legacy desktop app through computer-use — because the most consequential software has no API, only a login screen.
What it taught me
In progressWhen there’s no API, the agent has to use the software a human would.
Still open
When the screen is the only source of truth, how far can an agent be trusted to act on what it sees?
The systems with no door
The software that runs the world predates the API
A lot of the most consequential software in the world has no API. It has a login screen, a mouse, and a long institutional memory of how things have always been done. If you want to automate against it, your agent has to do what a person does: open the app, read the screen, click, type, and reconcile what it sees against what it expected to see.
The clinics, the courthouses, the back offices — the places where automation would matter most are exactly the ones with no door to knock on. To get in, the agent has to stop being a caller and become a user.
Reconciliation is the hard 80%
What I built, and the bug that taught me who owns the data
I’ve been building exactly that — a voice front-end that takes a request in plain speech, and a back-end worker that drives a legacy desktop application through computer-use to fulfill it. The reasoning is the easy twenty percent. The hard eighty is reconciliation: the system’s state and your records drift apart, and you have to keep closing the gap. My favorite bug was a sync that only ever updated records and never inserted the ones the system had created on its own — so right after a “successful” sync, the table looked empty. The fix wasn’t code. It was humility about who actually owns the data.
- Trust the render
- Trust what the system renders, not what you think you wrote. Your records are a guess about its state.
- Read back
- After every action, re-read the screen and confirm it landed. Never assume the write took.
- Drift
- The system changes on its own — other users, background jobs — and your cache goes stale without telling you.
- Idempotent writes
- The sync that only updated and never inserted is the whole genre: shape the operation so doing it twice is safe.
Skip the read-back and the two silently drift
Fig. 01 — The screen is the source of truth; the database is a hopeful cache
routinereconcileWhat it taught me
Both halves have to be true at once
Speaking as an interface is the easy, human-facing part. Acting reliably against a system that was never meant to be automated is the real engineering. Both have to be true at once, or you’ve built something very polite that doesn’t work.
The screen is the source of truth. Your database is a hopeful cache. The whole job is keeping the two honest.
It’s the same problem hiding inside every agent, just made visible here: the model’s picture of the world drifts from the world. The discipline is to keep reconciling against the source of truth — and to trust the screen over your own memory of what you did.