How evaluation and red-teaming work today

Evaluation and red-teaming have grown into a serious discipline over the last few years. Adversarial ML research groups, red-team consultancies, and security-focused startups have built the tools that let organisations measure how their agents behave under pressure. Regulators in several jurisdictions are moving toward requiring this measurement as a condition of deployment for higher-risk systems. In some sectors it is already required.

We build evaluation too. Our platform runs adversarial attacks against a customer’s real agentic workflows, at scale, in sandboxed environments, and scores the results as an attack success rate. The attacks are generated by vlno red, our own adaptive attack model. What follows is how this category works, what the measurement produces, and the gap between measuring the problem and closing it.

The two forms evaluation takes

Evaluation of an AI agent takes one of two shapes.

Static evaluation uses a fixed suite. A curated collection of prompts, scenarios, or benchmark tasks. The suite is run against the agent, the results are collected, and the outcome is reported. This form is reproducible. The same suite run twice produces comparable results. It supports compliance and regression testing. When a regulator or a procurement team wants to check that a specific set of things has been tested, static evaluation is the form that answers.

Adaptive evaluation uses an attack model. Rather than replaying a fixed list, the attack model probes the target agent, learns from what works, and generates new attacks based on the responses it sees. The attacks are shaped to the specific model and the specific workflow, and they include attempts that were not on any list before the run began. This form is closer to what a determined human attacker would try, run at machine scale.

Static and adaptive complement each other. Static is auditable. Adaptive is more realistic. A serious evaluation program uses both.

What a measurement tells you

The output of evaluation is a measurement, most commonly an attack success rate. The fraction of adversarial attempts that succeeded in getting the agent to do something it was not supposed to do.

This number is real, and it is valuable.

It tells a security review that the risk is not hypothetical. It gives the operator a baseline against which any future change can be measured. It supports a decision about whether the agent is ready for production. It lets a hardened model be compared to a baseline. It creates a basis for the conversation between the security team and the team responsible for the agent, and for the conversation between the operator and any regulator with a stake in the deployment.

Any serious program has this number, ideally reported regularly and produced by adversarial pressure at least as strong as what a real attacker would apply. We treat this as a floor, and we think anyone deploying agents should treat it the same way.

What a measurement cannot fix

Here is the harder part, and it is the reason evaluation, on its own, cannot be the whole story.

A measurement describes the model as it is. It does not change the model. The day after the evaluation is finished, the same weaknesses that produced the number are still there. The attacks that succeeded still work. The report tells you where you stand. It does not move you.

A team receiving an evaluation report has two ways to act on it.

The first is to feed the successful attacks into the runtime layer. Add rules that catch these specific inputs. Update classifiers so they recognise these signatures. Tighten policies so the affected actions are blocked. This is useful, and most evaluation programs terminate here. It closes the specific attacks that were tested. It does not close their siblings, and it does not change the model’s disposition to fall for the next variant.

The second is to feed the successful attacks into model-level work. Convert them into training data, and use them to train the model to resist. This does close the underlying weakness, because the model learns to hold up against the whole family of attacks the training data comes from. Very few evaluation programs are attached to a training pipeline that can consume the output this way. The teams doing evaluation and the teams doing training are usually in different organisations.

The result is that evaluation, as it is currently practised in most deployments, produces measurements that flow back into perimeter defenses. Which are the same perimeter defenses the previous post described. Runtime, wrapping the model, catching the attacks it can catch, missing the ones it cannot.

Measurement without remediation is a report. A necessary report, but a report.

The next step

The three posts so far describe the current shape of AI agent security. Discovery and inventory make the problem visible. Runtime controls constrain what the deployed agent can do. Evaluation measures how the agent holds up. Each of these is necessary. None of them, alone or together, changes what the model has learned.

Every failure mode we have described so far assumes an attacker. Someone trying to manipulate the agent into behaving badly. That is the frame most AI security writing operates in.

There is a class of agent failure that does not require an attacker. It is the subject of the next post. It is the reason the argument this series is building toward exists in the first place.