Explorations / Local AI
Running the model where the data already lives
Pulling cloud-class models down onto a laptop and phone — a multi-billion-parameter speech model on Apple Silicon — to find where local AI really breaks.
What it taught me
OngoingOn-device isn’t a fallback. It’s the strongest privacy promise you can make.
Still open
How much of the cloud’s job can move on-device before the constraints stop being worth it?
The experiment
Running the cloud's job on hardware that fits in a pocket
I keep pulling models that are “supposed” to live in the cloud down onto my own laptop and phone — a multi-billion-parameter speech model running entirely on Apple Silicon, voice intake handled on the device itself. Part of it is curiosity about where local AI breaks. Most of it is a conviction I formed shipping on-device assistants at Google.
The constraints that clarify
On a phone, the limits are the design
The constraints are real, and they clarify your thinking. On a phone there is exactly one machine, and it is already full — so every choice is a budget you spend out loud:
- Quantization
- Four-bit versus six-bit versus full precision is a product decision about quality, size, and speed — not a footnote.
- Cold start
- “The weights aren’t downloaded yet” is a first-class UX state you design for, not an error you hide.
- The one machine
- Memory, thermal headroom, and battery are the budget. You scale a server by adding machines; here there are none to add.
- Latency you own
- A response you control beats a faster one you have to apologize for when the network drops.
No round trip to a server → nothing to leak in transit
Fig. 01 — The whole loop runs where the data already is
on your devicenever leavesNone of these limits are in the way of the product. They are the product.
What it taught me
Structural privacy beats promised privacy
The payoff is a promise you never have to ask anyone to trust. If the model runs where the data already is, there’s no round trip to a server to reason about, no log to worry about, nothing to leak in transit. For anything close to health, that is the whole game.
On-device privacy isn’t a policy you promise. It’s a round trip that never happens.
Every privacy policy asks the user to trust that you’ll handle the data well once it reaches you. On-device, there is no sentence to trust: the data never reaches you. For the people with the most to lose when a system leaks, that difference is the whole thing.