Apply for the pilot
DEEN
Practice

Checklist: deploying AI agents safely in your organisation

To deploy an AI agent safely, work in a fixed order: define the purpose and the owner, limit permissions technically, tier approvals by impact, log every action and test the emergency stop. Only once a narrowly scoped pilot meets all of these points should the agent be extended to further processes or tools.

In the cases in our collection of AI agent incidents, the cause was rarely the model alone. It was overly broad permissions, missing approvals and unchecked content - all decided during roll-out. This checklist groups them into four phases, each item with a reason and further reading. Our position: deploying safely means working in this order - define the purpose, limit permissions, set approvals, switch on logging, test the emergency stop - and only then scale up. Reverse it and you build controls around an AI agent that can already do too much.

How the checklist is structured

Our frame of reference is the AI Risk Management Framework (AI RMF) from the US National Institute of Standards and Technology (NIST), version 1.0 of January 2023, which NIST says it is revising (as of October 2026). The framework is voluntary and organises risk management into four functions: Govern, Map (establish context), Measure (analyse and test) and Manage (treat risks). Govern is not a phase but runs through all the others. NIST stresses that its actions are neither a checklist nor necessarily a fixed sequence, so the order below is our recommendation, not the standard's.

PhaseGoalLink to the NIST AI RMF
1. PreparationSettle purpose, ownership and the legal frameGovern, Map
2. BuildLimit permissions, set up approvals and loggingGovern, Manage
3. PilotTest under real conditions, including the worst caseMeasure
4. OperationMonitor, review, extend in a controlled wayMeasure, Manage

Phase 1: preparation

  1. Pick one process and one purpose. A clearly bounded workflow, such as pre-sorting incoming receipts in a tax practice, can be constrained and checked; "helping out in the office" cannot. See our use cases for scenarios and What is an AI agent? for how agents differ from chatbots.
  2. Name an owner. Someone has to answer for the agent and decide on its rules. The AI RMF likewise expects documented, clear roles and responsibilities.
  3. Inventory every agent. Recording purpose, accounts, reachable systems and data categories also surfaces agent features in licensed software and shadow AI. The AI RMF provides for such an inventory; What is AI agent governance? describes how to build one.
  4. Settle data protection. If the agent processes personal data, the GDPR applies. Where processing is likely to result in a high risk, particularly when new technologies are used, a data protection impact assessment under Article 35 is required beforehand.
  5. Place the use case in its legal frame. The AI Act attaches obligations to the purpose: systems for selecting job applicants, for instance, generally count as high-risk. AI agents and the AI Act explains what that means for your agent; entities under Austria's NISG 2026 should also read our piece on NIS2 and AI agents.

Phase 2: build

  1. A dedicated service account, not a personal one. Only then can you tell whether a person or the agent acted, and revoke its rights selectively. More in least privilege for AI agents.
  2. Allow tools by allowlist. Anything not explicitly permitted stays forbidden (default deny). OWASP names excessive functionality, excessive permissions and excessive autonomy as the root causes of excessive agency.
  3. Keep credentials away from the model. Keys and tokens belong in secrets management, never in the prompt, and must be revocable at any time.
  4. Sort actions into levels. Reading, local changes and external actions need checks of differing strictness. Sending, paying and deleting belong at the highest action level.
  5. Set approvals by impact and check them outside the agent. OWASP recommends enforcing authorisation in downstream systems rather than letting the model decide. Why an AI agent cannot check itself gives the reasoning; human in the loop shows good approval design.
  6. Switch on logging before the agent starts work. Log every action with trigger, parameters, check result and outcome in a tamper-evident audit trail. A chat history will not answer the questions asked after an incident.

Phase 3: pilot

  1. Start with one agent, one process and separate environments. Keep test and production data apart. In one documented case, an agent working in development deleted a production database.
  2. Agree success and stop criteria in advance. Fix what the pilot should show - time saved, error rate, no impermissible actions - beforehand, or any result can be declared a success.
  3. Test manipulation deliberately. Put crafted emails or PDFs with hidden instructions into the inbox and check whether they can push the agent beyond its permissions. Prompt injection and AI agents provides the background; red teaming does this systematically. The AI RMF also expects testing before deployment and regularly in operation.
  4. Rehearse the emergency stop. Check that the agent halts at once and its credentials can be revoked, before you need it. The AI RMF provides for mechanisms to deactivate systems that behave inconsistently with their intended use. An untested kill switch is an assumption, not a control.
  5. Review rejections. If rejected actions pile up for one tool, either the rule is too tight or the agent is being steered off course.
  6. Train the team. Anyone working with the agent needs to know what it may do, how to spot manipulation and whom to alert. Training builds this AI literacy.

Phase 4: operation

  1. Scale only once the pilot meets every point. Every new process or tool goes through phases 1 to 3 again rather than being bolted on.
  2. Review permissions at fixed intervals. Rights creep from mailbox to file share to accounting, and without a set date nobody revokes them.
  3. Maintain allowlists. In the ForcedLeak case an expired domain was still on the allowlist; researchers registered it and had data sent there. Remove destinations nobody needs.
  4. Define the incident procedure. A data leak through an agent can be a personal data breach that Article 33 GDPR requires you to notify to the supervisory authority without undue delay and, where feasible, within 72 hours. Decide in advance who reports, who switches off and who secures the logs.
  5. Retain and review logs. Set a retention period. For high-risk systems, Article 26 of the AI Act generally requires deployers to keep logs for at least six months; our AI Act piece covers when that applies.
  6. Treat model and software updates as changes. A new model version or extension may behave differently (model drift), so it goes through the phase 3 tests again.

Order before speed

Retrofitting controls once an agent runs in several processes is hard. Start with one process, complete the four phases, then extend.

AEGIS, the locally operated agent system from oQueo, supports the build phase in particular: the language model holds no credentials, tools sit on an allowlist with default deny, every action carries an action level, only what an independent checking authority has signed is executed, and the audit log is a hash chain. Purpose, ownership, training and incident handling remain your organisation's responsibility. In our pilot programme we work out the rules for your processes together. More on how it works.

Sources

  1. NIST - AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0) (2023)
  2. NIST - AI Risk Management Framework, overview page (2026)
  3. OWASP - LLM06:2025 Excessive Agency (2025)
  4. Official Journal of the EU - Regulation (EU) 2016/679 (General Data Protection Regulation) (2016)
  5. Official Journal of the EU - Regulation (EU) 2024/1689 (Artificial Intelligence Act) (2024)
  6. Future of Life Institute - AI Act, Article 26: Obligations of Deployers (2024)