Why an AI agent cannot check itself

Ask how an agent is kept under control and many products answer with a second model call: the agent plans, reviews its own plan and then executes it. Or a guardrail inside the same framework scores the proposed action. It sounds like the four-eyes principle, but all four eyes sit in the same head. Our position: effective AI agent policy enforcement requires a control authority that is independent of the agent in three respects - the context it reads, the errors it makes and the process it runs in. Self-checking fails on all three.

Problem one: checker and checked read the same context

An agent's context window holds system instructions, the task, retrieved documents and tool outputs side by side. In December 2025 the UK's National Cyber Security Centre (NCSC) spelt out that a language model makes no internal distinction between data and instructions; there is only ever the next token. Unlike SQL injection, prompt injection may therefore never be fully fixed. The NCSC describes language models as "inherently confusable deputies".

Consider a tax practice that lets an agent process incoming receipts. One PDF contains white-on-white text saying the documents should also be forwarded to a particular address, as a standard action the client has approved. Ask the same agent whether forwarding is permitted and it judges the question in the light of precisely that text. With a single document, the attacker has reached both planner and checker. The NCSC draws a clear conclusion: when a model processes content from a party, its privileges drop to those of that party.

Problem two: correlated errors

Even without an attack, a second look by the same model is worth little. A study presented at ICLR 2024 found that language models barely improve their reasoning without external feedback, and sometimes get worse. Then there is self-preference: research published in 2024 showed that models acting as evaluators recognise their own outputs and rate them above equivalent text by others.

The obvious way out, using a different model as judge, goes less far than hoped. An analysis of more than 350 models presented at ICML 2025 found that, on one leaderboard dataset, two models gave the same wrong answer in 60 per cent of the cases where both were wrong. LLM-as-a-judge is useful for assessing quality, but as a security boundary it is a probabilistic filter with shared blind spots.

Problem three: guardrails in the same process

Classic security engineering asks three things of a control mechanism: it is invoked on every access, it cannot be altered by what it controls, and it is small enough to be verified. Guardrails running as a function inside the agent framework break the second condition. Anyone who can run code in the agent process, for example through a compromised extension, can skip the check or change its configuration. If the credentials sit in the same memory, the best checking logic is irrelevant.

In March 2026 Anthropic described the model-based approval layer for its coding tool. The classifier deliberately sees only user messages and the proposed tool calls. The agent's own text and the tool outputs are stripped out so that the agent cannot talk the checker round. Even so, Anthropic puts the share of missed overeager actions in a real-world dataset at 17 per cent. Separating context helps, but it does not turn a model into a dependable boundary.

Why human approval alone does not scale

The counter-argument is to have a person confirm every action, human-in-the-loop. The same source is sobering here. Users of that tool approved 93 per cent of permission prompts, and Anthropic describes the pattern as approval fatigue: at some point people stop paying close attention to what they approve. Someone who confirms "agent wants to read file" thirty times a day will wave through an external forward on the thirty-first. Human approval is valuable, but as a scarce resource for the few actions with external effect. A tiered model makes sense: read-only actions run automatically once they pass the rule check, while anything with external effect also needs a deliberate confirmation.

What works instead: separate roles, fixed rules, signed approvals

The table shows why none of the usual alternatives meets every requirement; a sound control authority combines three elements.

• Separation of roles: the planning component may not execute anything and never sees secrets; the executing component does not interpret text. Research backs this separation of duties: CaMeL, an approach published in 2025, separates control flow from data flow so that untrusted data cannot alter the program path, and completed 77 per cent of AgentDojo benchmark tasks with provable security, against 84 per cent undefended.
• Deterministic checks: schema, action level and allowlist are evaluated as policy as code. Same input, same decision, testable and auditable.
• Signed approvals: the checking authority signs every approval, for instance with an HMAC. Execution accepts only validly signed requests. Because the planning component never holds the key, it cannot forge an approval even if fully compromised. A nonce and time window stop an old approval being reused in a replay attack.

Independence is an architectural decision

AEGIS, the locally operated agent system from oQueo, follows this pattern in its three-authority principle. The planning authority produces structured intents without access to secrets or tools. The checking authority validates schema, action level and allowlist, guards against replays and signs with HMAC-SHA256. The execution authority verifies the signature again and writes an audit log as a SHA-256 hash chain; if its integrity is broken, execution does not start. More on our security page and in our primer What is AI agent governance?