The cost of running closer to the user

The pitch for on-device Assistant is clean enough to fit on a slide. Run the speech recognition, the language understanding, and the fulfillment on the phone itself, and you get low latency, you get privacy because the audio never leaves the device, and you get an interaction that still works when the network is slow or gone. Every one of those is real. The part the slide leaves off is that you pay for all of it in a currency you can’t print more of.

On the server, scale is a procurement problem. A single service gets bigger by getting more machines behind it. A phone is not that. It is one device with a fixed battery, a memory budget already contested by every other app, and a disk the user notices when it fills. My team’s job was to run the Assistant’s technical brains in exactly that environment, across phones, tablets, displays, TVs, cars, and watches — each with different silicon and a different idea of what ‘enough’ means. A watch is not a small phone. It is a different physics problem wearing the same API.

So we stopped arguing about features and started negotiating budgets. Disk, memory, and battery ceilings per device class, written down and enforced. Composability so a given device could be assembled from only the feature set it could afford. And tooling that put the benefit and the cost on the same screen, because the only honest way to decide whether a quality win was worth its footprint was to make someone look at both numbers before they shipped.

A quality win that costs battery isn’t free until someone has agreed, on the record, what it’s allowed to cost.

What made it hard wasn’t any single tradeoff. It was that the feature teams couldn’t see the constraint. A model that fit comfortably on a flagship phone quietly broke a wearable, and the engineer who wrote it had no way to know. The temptation was to let every team reason about every surface — which guarantees that nobody does it well. We took the opposite bet: absorb the per-surface complexity ourselves so a feature developer could mostly stop worrying about correctness on each device separately. That makes us the bottleneck on purpose, and it only works if our guardrails are good enough that being the bottleneck doesn’t mean being the brake.

The question I haven’t closed is where the line actually sits. Privacy and offline reliability are easy to defend in a review and hard to value in a spreadsheet against a percent of battery life. I can measure the milliamps. I cannot yet measure what it’s worth to a person that their request never left the phone — and I’m wary of any framework that pretends I can.