The adversary is sometimes the account holder

Most threat models start with an outsider — the hacker, the stranger, the unauthorized request at the door. That instinct is fine until you build a system whose whole purpose is to let one person watch another. Then the person you have to design against is often the one the system has already authorized.

I learned this designing supervision for Family Link, where a parent can oversee a child’s or teen’s account. The honest version of that brief is uncomfortable. The legitimate account holder, a parent acting in good faith or an over-controlling one acting in bad faith, has the same authority, and the system can’t tell them apart from inside a permissions check.

A supervision tool is one design mistake away from being stalkerware. The defense can’t be authentication — it has to be the shape of the system.

That pushed us toward defenses built into the structure of the system. Supervision began with a two-party, two-device consent handshake: the supervised person had to say yes on their own device — never silently, never by someone acting for them. A transparency surface meant the watched party always knew what was visible; surveillance you can see is a different thing from surveillance you can’t. Either party could exit. And we treated data minimization as a safety control, not a storage decision: for child accounts I verified there was no server-side location history at all. You cannot leak, subpoena, or weaponize what you never wrote down.

The same property showed up one layer down, in account scoping on YouTube: child, teen, and private sessions had to stay strictly isolated, so one person’s data could never bleed into another’s context. Internally we framed it as COPPA and GDPR compliance, which is true, but the deeper point is that isolation is a correctness property. A trust boundary that mostly holds is not a trust boundary.

None of this is free. Consent handshakes add friction to a setup flow nobody enjoys. Minimized data means features you cannot build and questions you cannot answer later. Reversibility means a parent can lose oversight they feel entitled to. Those are real costs, and I’d argue they’re the price of being a system people can actually trust rather than one they’re told to.

I think this is the part a lot of current AI trust-and-verification work underrates. As we build systems that grant identity, verify claims, and extend trust on someone’s behalf, the most dangerous actor often won’t be an outside attacker — it’ll be an authorized operator, an account holder, or a credential the system itself issued. Abuse-resistance, in that world, isn’t a filter bolted on at the edge. It’s consent the architecture enforces, transparency to the party being acted upon, an exit that always works, and holding as little as the job allows. Decide who you’re really defending against before you decide what you’re willing to store.