Human in the loop: designing approvals that actually protect

Before an AI agent sends an email, prepares a payment or deletes a file, it asks a person: human in the loop, the obvious answer to the question of control. Our piece on why an agent cannot check itself explained why approving every action breaks down through habituation. Here the question is how to design an approval that actually protects. Our position: an approval protects only when people are asked rarely, about something specific, with information they can understand and a genuine choice. Ask someone to approve everything and they will soon approve anything.

What the AI Act means by human oversight

The AI Act deals with human oversight in Article 14, for high-risk AI systems. Following the Digital Omnibus, Regulation (EU) 2026/1744, these requirements apply to Annex III systems from 2 December 2027, and most agents in law firms are not high-risk anyway. Article 14 is still a useful yardstick for good oversight. Under Article 14(4), the people assigned to oversight should be able to:

• understand the system's capacities and limitations and detect anomalies;
• remain aware of the tendency to over-rely on its output, which the regulation explicitly calls "automation bias";
• interpret the output correctly;
• disregard, override or reverse the output in a given situation;
• halt the system through a "stop" button or a similar procedure.

Article 26(2) adds that deployers must assign oversight to people with the necessary competence, training and authority. Article 14(5) is telling: for remote biometric identification, at least two people must as a rule separately verify and confirm a result before the deployer acts on it. Where stakes are high, the law itself reaches for the four-eyes principle.

Automation bias: what the research shows

In 2010 Raja Parasuraman and Dietrich Manzey reviewed empirical studies of people working with automation and decision aids. When the aid is imperfect, they found, automation bias leads both to missed problems and to wrong actions. It affects novices and experts, individuals and teams, and cannot be prevented by training or instructions. Simply "remaining aware" of it, as Article 14 puts it, is not enough. Design has to do the work.

A 2023 study in Radiology shows how strong the effect can be. Twenty-seven radiologists read mammograms supported by a purported AI system that deliberately suggested the wrong category in 12 of 40 test cases. Inexperienced readers were right in about 80 per cent of cases with a correct suggestion and about 20 per cent with a wrong one. Even very experienced readers dropped from about 82 to about 46 per cent.

Usable security research shows that design makes a difference. A field study of more than 25 million warning impressions in Firefox and Chrome found that users clicked through about a third of Firefox's certificate warnings but 70.2 per cent of Chrome's; the authors conclude that a warning's user experience can significantly affect behaviour. In experiments led by Cristian Bravo-Lillo, dialogues highlighted the critical field and briefly blocked the risky option, for example until it was retyped. Although habituated to the dialogue beforehand, participants were two to three times as likely as the control group to make an informed decision.

Human in the loop or human on the loop

People should stay in the process, but their role has to fit the action. Besides approving single steps, they can supervise the running system: human on the loop.

What makes a good approval

• Tiered by impact: ask only for actions at a high action level, meaning external effect or no way back. Fixed rules handle reading and local changes. Every unnecessary prompt weakens the next one.
• Show the action, not the intent: "Agent wants to send an email" says nothing. An illustrative better prompt for a law firm: recipient, subject, three attachments and their source file, and a note that the email cannot be recalled.
• Highlight what matters: for critical actions, emphasise the one field that counts, such as an external recipient, and perhaps have it retyped.
• Make refusing as easy as approving: no pre-selected confirmation, a way to query the agent, no penalty for refusing.
• Four eyes for the critical: payments above a threshold or deleting client files need two independent confirmations.
• The right person: approvals belong with someone able and authorised to judge the consequences, not whoever is at the screen. Training aids understanding but cannot replace good design.
• Watch the approval rate: a prompt that is almost always accepted is redundant or badly framed; move the action into a fixed rule.

Approvals where they count

AEGIS, the locally operated agent system from oQueo, ensures that not every action has to reach a person in order to be checked. A checking authority built from deterministic code tests every intent against your rules before anything runs. Every action is assigned an action level, external effect is a level of its own, and you decide the highest level an agent may reach. Every decision is written to a log that cannot be altered unnoticed. AEGIS supports oversight of your agents; it does not replace organising that oversight or legal assessment. More on how it works.