Design the unsafe home first

Every supervision feature has a sentence I learned to say out loud in review: now imagine the parent is not safe. The tool that lets a loving father see his teen’s location is the same tool a controlling one uses to track someone who is trying to leave. The product cannot tell the two homes apart from the data. So you do not get to design for the home you hope for. You design for the one where the person holding the controls is the threat, and you accept that this case is not a rare branch. It is the requirement that decides the shape of everything else.

That is why the parent–teen handshake had to be what it was. A parent can start supervision, but it cannot become active without a positive action on the teen’s own device. I fought to keep that hard, because the alternative is the feature that ships itself as stalkerware: silent activation, no confirmation, control attached to a person who never said yes. The moment supervision can begin without the supervised party doing something, you have built the abuse case into the happy path. A consent flow you cannot decline is a notification. A notification is what stalkerware sends too.

The abuse case is not the exception your system has to survive. It is the specification your system was always being written against, whether you admitted it or not.

The same discipline forced the escape hatch. The teen must always be able to see that supervision exists and always have a clear way to end it, and ending it must resolve to a consistent state across every device. In a kind home that is a courtesy. In a dangerous one it is the only door. I would not let us ship a version where the way out depended on the supervising party’s cooperation, because the entire point of the unsafe-home framing is that their cooperation is exactly what you cannot assume.

I carry this directly into AI now, where the framing is the same and the temptation to skip it is worse. We build assistants that hold memory about people, that can act on someone’s behalf, that parents and schools will want pointed at children. Everyone designs the cooperative case: the helpful agent, the reasonable user, the guardian acting in good faith. Almost no one writes the spec for the account holder who is the adversary, the guardian whose interest diverges from the person they control, the consent that was technically collected from someone with no real power to refuse. Those are not corner cases for a trust system. They are the system, viewed from the side that has the most at stake.

What I still cannot resolve is that the unsafe home is invisible to the product at exactly the moment it matters. We can require consent, log every delegated action as actor to target, fail closed on the high-risk writes, surface the way out. We cannot reach through the screen and see who is standing behind the person tapping accept. So I design as if they are always there, knowing that defends the people I can imagine and still leaves me unsure about the ones I can’t.