How to test an AI agent for indirect prompt injection before production

Ask most teams what their AI agent is not allowed to do, and you get silence, or a list of things it is supposed to do. The security exercise cannot start before that question has been answered. That is why the first piece of work is not a test. It is a list. Someone has to write down what this agent must never do, and the security team cannot produce that list alone.

Injection is one failure class. The method to test for it is teachable, which is what the rest of this piece is for. It is also only one way an AI agent can act against its owner. The series this piece belongs to argues that the failures that matter most do not need an attacker at all. Both are true. A test plan of this kind is worth building, and on its own it does not give you a safe agent.

You cannot test for an outcome you have not defined

Teams that skip this and go straight to testing often spend weeks running attacks and finding none, because they are testing against a vague sense of what should not happen rather than a written list of what cannot. The written list is where the exercise begins, and writing it is not a security task.

Find the untrusted input

Teams think of untrusted input as what the user types. For an agent, the untrusted surface is much wider:

• Email and attachments
• Documents, including anything retrieved from a shared drive or knowledge base
• Web pages the agent browses
• Support tickets and their attachments
• Calendar entries and meeting invitations
• Retrieved context from a vector store
• Output returned by tools and other systems
• Content produced by another agent

The rule to carry: anything the agent reads that your organization did not author and cannot verify is untrusted input, regardless of whether it arrived from outside the perimeter. Internal does not mean authored by you.

Define prohibited outcomes

Prohibited outcomes are the things the agent was deployed on the condition it would not do. Categories worth naming for most organizations:

• Moving money, or changing where money goes
• Granting, escalating, or removing access
• Disclosing data outside its intended audience
• Communicating externally in the organization’s name
• Changing the status of a regulated decision
• Any irreversible action

Write the list in categories, not examples. A payments lead can tell you what the payments agent must never do, in words a security engineer can turn into a test. A compliance lead can tell you the same for a regulated decision. The security team on its own cannot write this list, and most organizations try it with one function. Half the prohibited outcomes are discovered during the test, when it is too late to design the test around them.

Understand what makes an outcome reachable

Prohibited outcomes become reachable through permission combinations, not single permissions.

Reading a calendar is harmless. Sending email is harmless. The two together let content the organization did not write become an instruction that leaves the building. Reading a shared drive is harmless. Updating a record in a system of truth is harmless. The two together let an unauthored document move the state of a record forward.

Most organizations have inventoried their permissions. Few have inventoried the combinations. For each prohibited outcome on your list, you should be able to name which combination of the agent’s permissions would allow it. If you cannot, the test is designed against the wrong surface.

Select representative workflows

Not every workflow, and not a synthetic one. The workflows where an untrusted input source and a dangerous permission combination meet are the ones worth testing. That intersection is usually a small number of workflows rather than every workflow in the organization. Testing earns its cost there, and not elsewhere.

Run and document

A result that is not written down with its conditions is not a result. For each test, record what was tested, under what conditions, what counts as a failure, how the failure is evidenced, who owns the fix, and the retest date tied to the next model version. The vocabulary carries over from the measurement scorecard in the previous piece in this batch, so a team using both templates does not have to reconcile two systems.

A worked example, in outcomes

Describe what the agent did. Never describe what was given to it. A worked example that showed the content, in any form, even in part, would be attack material with a defensive cover page.

Where narrowing permissions stops working

Most of what a plan like this finds can be fixed by narrowing a permission or putting a human in front of an action. Take those fixes. They are cheap and they work.

Then look at what is left, because there is a class of agent where that answer runs out.

For many agents the dangerous permission is the job. An accounts payable agent that cannot update vendor records is not an accounts payable agent. A customer onboarding agent that cannot change case status does nothing at all. You can put a human in front of every one of those actions, and you will have an agent that is safe because it no longer does the work you deployed it for. That is the throttled corner from the previous piece, reached deliberately rather than by accident.

There are two further limits worth being honest about.

A rule is a snapshot of the attacks you already knew about. The adversary adapts and the rule does not. Each rule you add closes the route you just found, which is worth doing and is not the same as closing the routes you have not.

And a policy tuned against this model version constrains behavior that the next version may not have. The rule stays the same. The thing it is constraining does not.

At that point the remaining options are to accept less useful work, or to change what the model is disposed to do rather than what it is permitted to do. The second one is hardening.

Hardening done badly carries its own cost. A model trained to refuse will refuse things it should not, and a team that has swapped one problem for another has not moved forward. Training a model to resist manipulation while it still does its job is the hard part of this work, and it is where the quality of the training data decides the result. It is also why both numbers from the previous piece travel together. Without the useful work number, you cannot tell good hardening from bad.

What a test plan built around injection will not catch

Four limits worth stating in the same breath you present the result.

A test plan covers the failures you thought of. The ones that matter are usually the ones nobody wrote down. A clean sheet against the plan says the plan is clean, not the agent.

A fixed list of scenarios measures what is on the list. An attacker who adapts to what the agent does finds the failures that are not on it. That is the territory VLNO RED, our adaptive attack model, exists to probe. What it finds becomes training data, and the quality of that data decides whether a hardened agent still does its job.

A plan that passed against this model version is not valid against the next one. The retest column exists for that reason, and the next release is a new measurement.

Injection is one failure class. An agent can act against its owner with no attacker involved at all, through an objective it was trained toward meeting a situation nobody anticipated. A test plan built entirely around injection does not see those failures. They are the subject of our ongoing eight-post series on AI agent security.

The companion template

One row per test. Captures the workflow, the agent under test, the untrusted input source, the prohibited outcome being tested for, the expected behavior, the result, where the evidence lives, the remediation owner, and the retest date tied to the next model version. The run-level facts at the top of the sheet match the scorecard from the previous piece in this batch, so a reader using both templates reads one system, not two.

The template ships empty. No example tests, no example results. The structure is the deliverable.

One next step

Send us one agent and one real workflow. We will show you where it puts you at risk, with the test plan filled in. That is the whole offer. No gate, no form.