Everyone is talking about jailbreaks. And they're right to — getting a model to ignore its rules is a real problem. But I think it's only the first half of the problem, and this week made the second half impossible to ignore.
Go back to the Meta breach. No clever jailbreak was needed. Someone asked an AI support chatbot to change the email on an account they didn't own, and the chatbot — which had the capability to do exactly that — did it. The failure wasn't that the model was tricked into saying something bad. The failure was that a tool let a high-impact action through without ever asking the only questions that mattered: who is initiating this, and by what authority?
That's the half we under-build. We pour effort into making the model harder to manipulate, and far too little into the guardrails on the tools the model can call. After a model is compromised — and we should assume it will be — the defensible move lives at the tool: validate the tool's authority, validate who initiated the action, and for anything high-impact, require an old-fashioned, out-of-band check before it proceeds. None of that is glamorous. All of it would have stopped the Meta takeover cold.
Here's the part I'd push hardest. The provenance of a request should change its risk. If John, our tier-3 support engineer, asks to change a customer's account email, that's one level of risk — there's a human, an identity, an audit trail, a person you can call. If an AI agent asks to make the exact same change, that should be treated as more risky, not less, because the chain of accountability is thinner and the thing asking can be driven by input you never see. Same action, different initiator, different risk. Most systems today do the opposite — they wave the agent through because it's automated and "internal."