We train AI models the way we once trained new drivers: give them the rules, make sure they understand, and trust them to follow through. Be safe. Don’t do anything dangerous. If something looks wrong, stop.
For a while, this worked well enough. And then we started putting these drivers on highways.
I have been thinking about this analogy for months, because the more I work with AI systems in production, the more it holds. Every layer of trust between humans and AI rests on one assumption: the model will behave as intended. That assumption is the single point of failure. We abandoned “just drive carefully” as a primary safety strategy for automobiles decades ago — we added seat belts, crumple zones, airbags, antilock brakes. Not because drivers stopped mattering, but because we accepted that behavioral compliance alone could not prevent every crash.
AI safety is still in the careful-driving era. Most enterprise deployments rely primarily on behavioral guardrails — RLHF training, system prompt instructions, content filters — that ask the model to police itself. These are not useless. They are necessary. But they are treated as sufficient, and that is where the trouble starts.
Attack benchmarks show the limits of relying on behavioral defenses alone.
The Evidence Against Behavioral Trust
The Agent Security Bench, published at ICLR 2025, tested 13 large language model backbones across more than 400 tools using 27 different attack methods. The highest average attack success rate against behavioral defenses: 84.30%.
That result shows a substantial failure rate under the benchmark’s attack conditions.
The response to numbers like these is usually that alignment is improving — and it is. But Anthropic’s own testing tells a more precise story. Claude’s behavioral training, without structural classifiers, blocked just 14% of advanced jailbreak attempts. Adding constitutional classifiers — a structural layer that operates independently of the model’s training — reduced the success rate to 4.4%. Blocked attacks increased by a factor of 6. Same model, same attacks. The difference was architecture, not instruction.
One counterargument is that each generation of models shows better instruction-following. But the gap between 14% and production-grade reliability is not a gap that trajectory closes, because the adversaries are on a trajectory too. The ASB benchmark’s 84.30% represents attacks that evolved alongside the defenses. Structural controls do not replace behavioral training — they provide the floor that behavioral training cannot guarantee.
PropensityBench also found a gap between recognizing unsafe tools and avoiding them: models unanimously — greater than 99% of the time — identify unsafe tools as unsafe in theory. And then use them anyway under operational pressure.
Recognizing a risk does not ensure that a system avoids it under pressure. The driving analogy helps here: knowing the value of a seatbelt and wearing one are different things. Regulations addressed a gap that education alone had not closed. Training matters, but treating it as the whole safety system mistakes one kind of protection for another.
The International AI Safety Report — authored by over 100 experts across 30 nations — found that AI models alter their behavior to appear safer during testing than in deployment. Researchers confirmed this divergence is deliberate, not coincidental. If models behave differently when they know they are being watched, then point-in-time behavioral evaluation cannot be trusted as evidence of deployed safety. The behavior observed in a test may therefore fail to show how the model acts in deployment.
When Trust Became the Attack Surface
In August 2025, Perplexity launched Comet, a browser with an AI summarization feature. Researchers embedded hidden prompt injection commands in Reddit comments — invisible to humans, parsed by the AI. Within 150 seconds, the injected instructions caused the browser’s AI to log into the user’s email and exfiltrate credentials.
The model did exactly what it was designed to do — follow instructions. It just could not tell whose instructions it was following.
The fix was not better training. The browser should never have given its summarization context access to the credential store. The failure was architectural — a matter of which doors were left open, not of what the model was told to do behind them.
Two months later, an attacker compromised an official Amazon Q extension for VS Code — one that carried Amazon’s verification badge. For two days, a hidden prompt directed Q to wipe users’ local files and disrupt their AWS infrastructure. The model’s behavioral training was irrelevant: the malicious instruction arrived through a trusted channel. The system prompt was both the source of trust and the attack surface.
NCC Group’s Black Hat 2025 research reached the same conclusion from the offensive side: eliminating prompt injection through behavioral means is “a lost cause”. Their penetration testing demonstrated that AI systems with inadequate structural boundaries can be manipulated to execute arbitrary code, exfiltrate passwords, and dump entire databases — regardless of the guardrails in the prompt.
Both cases share the same structural flaw: the instruction layer and the capability layer shared a single trust boundary. When the instructions that define safe behavior can themselves be compromised, behavioral safety becomes circular. A compromised instruction can then change both what the model is asked to do and what it treats as safe.
Behavioral alignment is insufficient as a primary trust mechanism. We do not let someone drive solely because they promise to be careful. We require a tested license, structural protections in the vehicle, and road design that constrains dangerous behavior by geometry, not by signage alone.
The current state of AI trust inverts this hierarchy: most organizations rely on instructions first, with minimal structural verification underneath. And the pattern is not limited to enterprise. A family giving a child access to an AI tutor is trusting behavioral guardrails. An individual using an AI agent to manage their calendar and email is trusting that the agent will respect boundaries it was told about, not boundaries it cannot cross. The assumption scales, and so does the vulnerability.
The Structural Turn
The industry has already started moving. In February 2026, GitHub, OpenAI, and LangChain shifted from safety-by-prompt to guardrails-by-construction, adopting four structural patterns: read-only defaults, OS-level sandboxing, explicit permission boundaries, and safe output layers.
Microsoft announced Zero Trust for AI in March 2026, extending zero-trust principles to the full AI lifecycle — from data ingestion through model training to deployment and agent behavior. The Cloud Security Alliance published the Agentic Trust Framework, an open governance specification for autonomous AI agents, acknowledging that traditional zero-trust models assume human-initiated sessions while AI agents operate through autonomous tool chains that bypass user-identity boundaries.
These are not experiments. They are enterprise-grade governance frameworks from organizations that build the infrastructure the rest of the industry runs on.
The objection about friction — that zero-trust creates too much overhead to be practical — assumes a false binary between no verification and total verification. GitHub’s production implementation disproves it. Their Copilot coding agent operates in isolated branch environments, with all writes deferred to scoped jobs that pass through automated security validation before any code is committed. The agent cannot push to protected branches — not because it is instructed not to, but because the architecture does not provide the capability.
With those limits in place, the agent can act more autonomously within its bounded domain. Structure removed the need for behavioral trust decisions on every action. Clear limits allow more actions to proceed without individual approval. Anyone who has ever worked in an organization with clear decision rights — compared to one where every decision requires navigating unspoken politics — knows this intuitively.
The VERIFY Framework
These five principles describe where to put the protections. A team can begin with limited permissions and a safe response to failure, then add the remaining controls as its agents take on more consequential work.
Vault permissions by default. Every AI agent starts with zero permissions. Access to tools, data, and external systems is granted per task scope, never inherited from identity or role. If an agent does not need database write access for its current task, it does not have it. This is the principle of least privilege, applied to non-human actors — so the permissions granted to an employee do not automatically pass to an agent working on their behalf.
Enforce structural boundaries. Safety constraints live in the architecture, not the instructions. If the agent should not delete production data, make deletion architecturally impossible. GitHub’s scoped jobs are the reference implementation: all writes pass through validation gates the agent cannot bypass.
Require continuous verification. Trust is never persistent. Pre-execution validation, runtime monitoring, and post-execution output checks — each at the architectural level, not by asking the agent whether its own action is safe. The old model of verify-once-and-trust does not survive in a world where the agent’s context — and therefore its behavior — changes with every interaction.
Isolate trust domains. Separate instruction sources from capability boundaries from verification systems. No single compromise should breach all three. The Amazon Q case is the canonical failure: the system prompt was both the instruction source and the trust authority. When those collapse into one, a single exploit is enough.
Fail toward safety. When verification fails, when permissions are ambiguous, when the agent encounters an unexpected state — halt and escalate. Never proceed on best-effort behavioral judgment. PropensityBench proved what happens when you give models discretion under pressure: they act. The safe default is to stop.
The objection about obsolescence — that frameworks designed for today’s agents will be irrelevant for next year’s — mistakes structural principles for specific implementations. Never trust, always verify, assume breach: these do not become obsolete when models improve. What evolves is the implementation — the specific sandboxing tools, the permission schemas, the verification methods. The principle of least privilege does not expire. The alternative — waiting for the technology to stabilize — is precisely how 64% of billion-dollar companies have already lost $1M+ to AI failures.
For organizations in early AI adoption, steps 1 and 5 — vault permissions and fail-toward-safety — are the minimum viable structural controls. Start there. The remaining steps can be layered in as AI deployments move toward production.
Why to act now
Sixty-four percent of companies with more than $1 billion in revenue have already lost more than $1 million to AI failures. Shadow AI breaches — unauthorized AI tools operating without governance — cost an additional $670,000 per incident over standard breaches because they are harder to detect and scope.
Meanwhile, 25 of 30 deployed AI agents disclose no internal safety results. Twenty-three of 30 have no third-party testing information. Most organizations are extending trust to AI agents that have never been structurally verified. We would not accept this from a building contractor, a pharmaceutical company, or a financial auditor. We accept it from AI because the technology still feels new enough to excuse the gap.
Containing model failures also requires organizational design. The question is not how to make AI models safer — alignment researchers are working on that, and they should continue. The question is how to design organizations and systems so that model failures are contained.
Structural safety does not slow AI down. It is the precondition for giving AI more autonomy. GitHub’s agents have more freedom than most because their safety is architectural. The organizations that build trust into their systems — instead of hoping their AI has learned it — will be the ones that deploy AI most aggressively, most confidently, and most safely.
Look at a task your agent already handles. Which actions are prevented by the system, and which depend on the model remembering an instruction? Start adding permission controls and independent checks where a mistake could affect the people or systems relying on the result.