The map of AI agent security

Companies are deploying AI agents faster than the security field has settled around them. An agent is not a chatbot with better packaging. It is a system that reads content, calls tools, moves data, and takes actions on behalf of the people who deployed it. Securing something like that requires more than filtering the model’s output.

The field has responded by building a set of tools. Different teams have taken different pieces of the problem. What has emerged is a rough division of the work into four categories. This post walks through them plainly, and without arguing yet about which one matters most. Before that conversation is worth having, the terms have to be settled.

Discovery and inventory

Before you can secure the agents in your organisation, you need to know which ones exist.

Discovery products enumerate the agents already running. Which models each one calls. Which tools each has access to. Which data sources each can reach. Which humans each is acting for. In a mid-sized company today, the number is usually higher than the security team expects. Agents are being built by product teams, by data teams, by individual engineers with a shell script and an API key. Some are formal deployments. Many are not.

Inventory is the ground work. A security program that does not know what it is protecting cannot protect it. In practice this category also carries the policy layer that comes next, because once an agent is registered, it can be given a policy. Discovery without enforcement is a spreadsheet. Discovery with enforcement is a starting point.

Runtime controls and filters

The largest category by investment and by number of active builders. Runtime controls constrain what an agent can do while it is running.

The techniques include four moving parts. Input filters inspect the content going into the model. User prompts, tool results, retrieved documents, API responses. They look for known patterns of prompt injection, for sensitive data being exposed to the model, and for content that violates policy. Output filters inspect what the model produces. Text going to a user, function calls headed for a tool, structured data being written to a system. They check for policy violations and information leaks before the output leaves the system. Policy enforcement sits between the agent and the tools. When the agent decides to call a tool, take an action, or access a data source, policy enforcement checks whether that action is permitted for that agent, in that context, for that user. Actions that are not permitted are blocked. Real-time detection watches the agent’s behaviour over time. It looks for patterns that indicate the agent is being manipulated, malfunctioning, or being used for something it was not deployed for.

These controls are effective against known patterns. Runtime is where most current security investment goes, because most organisations need to constrain their agents right now, in production, before any longer conversation about model-level work can matter.

Evaluation and red-teaming

Runtime protects the deployed system in flight. Evaluation tests the system before, during, or after deployment, to answer a different question: how does the agent behave when it is pushed.

There are two main forms. Static evaluation uses a fixed suite of prompts, scenarios, or benchmark tasks. The suite is run against the agent, and the results are compared over time. This form is good for regression testing and for compliance, where a specific set of things must be checked every release. Adaptive evaluation uses an attack model that generates new attacks rather than replaying known ones. The attack model probes the target agent, learns from what works, and produces attacks that were not on any list before the run began. This form is closer to what a determined human attacker would do, at machine scale.

What evaluation produces is a measurement, usually an attack success rate. The fraction of attempts that succeeded in making the agent do something it was not supposed to do. That number tells you where the agent stands. Regulators in several jurisdictions are moving toward requiring this measurement as a condition of deployment for higher-risk systems. In some sectors it is already required.

Model-level work

Underneath the other three sits the model itself.

Model-level work does not wrap the model with rules, and it does not test the model with attacks. It changes what the model has learned. The techniques include adversarial training, which exposes the model to attacks during post-training so it learns to hold up against them. Robustness fine-tuning, which reshapes the model’s behaviour on a targeted set of failure modes. And hardening adapters, which apply changes without re-training the whole model, using a small overlay on the weights.

This layer is the newest of the four and the smallest by number of active teams. It requires a different capability profile from the others. Runtime is API-level engineering. Evaluation is testing at scale. Model-level work is training. The teams doing it are drawn from adversarial machine learning research and from post-training practice, not from traditional application security.

Where the field stands today

These four categories together are the current shape of AI agent security. Each has genuine value. Any serious deployment uses more than one. Discovery without runtime is a list. Runtime without evaluation is untested. Evaluation without model-level work is a report. Each layer answers a different question.

The remaining posts in this series look at each in turn. What runtime controls see. What evaluation measures. What model-level work changes. And what class of failure sits underneath all of them.

We start with runtime.