Testing prompt-injection defences with release evidence

Where does the standard point the wrong way?

OWASP's entry for LLM01:2025 lists seven mitigation strategies. Read them in order: constrain model behaviour through system prompts and context limits, define and validate expected output formats, apply input and output filtering with semantic analysis, enforce privilege control and least privilege, require human approval for high-risk operations, segregate and clearly identify external content, and conduct adversarial testing regularly.

Every one of those is sound advice. But count what kind of advice each one is. Four of the seven are things you do to the text: shape the prompt, constrain the format, filter the content, mark the external material. Two are things you do to the architecture: privilege control, and human approval on high-risk operations. One is a testing practice.

For a model that only produces text, that weighting is reasonable, because the text is the entire attack surface and the entire consequence. For a model that holds tools, it is close to backwards. The four text-side controls all attempt to make the model behave correctly in the presence of adversarial input. The two architectural controls make the model's behaviour matter less. Only the second kind survives an attacker who is better than your last test case.

The standard itself says as much. Its own text acknowledges that, given the stochastic influence at the heart of the way models work, it is unclear whether there are fool-proof methods of prevention for prompt injection. Read that as the load-bearing line rather than a caveat at the end. If prevention is not reliably achievable, a control premised on prevention cannot be the one your risk register leans on.

There is a second reason the list points the wrong way, and it is about where the failure gets recorded. A tool-holding system that suffers an injection rarely fails at LLM01 in any way you could point at. It fails at LLM06, excessive agency, because the model was able to invoke something consequential, and at LLM05, improper output handling, because something the model produced was passed downstream and trusted. LLM01 is the entry point. The other two are where the damage is done, and they are the two that are actually fixable by construction.

The strongest evidence for that reading is that OWASP has now published it. On 9 December 2025 the GenAI Security Project released the OWASP Top 10 for Agentic Applications, a separate list built with more than one hundred contributors over more than a year, for systems that plan, delegate and call tools. Its entries include Agent Behavior Hijacking, Tool Misuse and Exploitation, and Identity and Privilege Abuse. Read those three names next to LLM01 and the shift is unmistakable: from what the model was told, to what the agent was able to do, and under whose identity it did it.

The project has been explicit that the LLM list remains a core resource rather than being superseded, and that is right. But the practical consequence for a buyer is worth stating plainly, because most enterprise risk registers written in the last two years cite LLM01 and stop. If your system holds tools, the list you are being assessed against is probably not the list your failure will occur on.

Can an injection detector be a security boundary?

An injection classifier is a probabilistic detector operating on adversarially chosen input. It has two properties that disqualify it as a boundary, and both are structural rather than a matter of tuning.

Its false-negative rate is set by the attacker, not by you. Ordinary classifiers are evaluated against a distribution that stays still. This one is evaluated against an opponent who reads your refusals, adjusts, and tries again, and who only needs the single phrasing you did not anticipate.

That is not a theoretical worry, and there is now a number on it. In October 2025 a team from Google DeepMind, OpenAI, Anthropic and ETH Zurich published The Attacker Moves Second, which took twelve recently published defences against jailbreaks and prompt injection and attacked them adaptively using gradient descent, reinforcement learning, random search and human-guided exploration. They report bypassing all twelve, with attack success rates above 90 per cent for most of them. The majority of those defences had originally reported near-zero attack success rates.

The distance between near-zero and above ninety per cent is not a tuning problem or a version behind. It is the difference between evaluating a control against the attacks you thought of and evaluating it against an opponent who read your paper.

Its failures are silent in the direction that hurts. A false positive is loud: a legitimate document is refused and somebody complains. A false negative produces a completed task, a satisfied user and a clean log. The control that fails silently in the dangerous direction is the one you should trust least, and it is the one most commonly presented to a risk committee as the answer.

None of this makes filtering worthless, and dropping it would be the wrong conclusion. It makes filtering a monitoring control rather than an access control. A monitoring control tells you that somebody is probing, which is genuinely valuable and worth building. An access control decides whether an action happens. The mistake is putting a monitoring control in the position where an access control belongs and then reporting its pass rate as assurance.

The same distinction applies to the guardrail layers we have already published. Those layers exist, they run in a stated order, and they are tested on every change. They are the right control for the question they answer, which is whether an answer is in scope, grounded and safe to say. They are not the control for whether an action is safe to take, and stacking a sixth layer on the request path does not become that control.

How do you test for injection without pretending you are finished?

Adversarial testing is the seventh item on OWASP's list, and it is the one most likely to be done once, presented as a report, and never run again.

The reframing that makes it useful is to treat injection cases as regression tests rather than as an assessment. A penetration test tells you about the system on the day it was tested. An injection pack that runs on every change tells you whether the property still holds, and the property is the thing you actually care about, because it is the thing that quietly stops holding when somebody adds a tool to a session six months from now.

AgentDojo is the useful public reference point for what such a pack looks like at research scale: an extensible environment rather than a fixed list, with 97 realistic agentic tasks and 629 security test cases, deliberately built so that new attacks and new defences can be added to it. The design lesson is the extensibility rather than the counts. A fixed set of injection strings ages into a set of strings your system happens to catch.

Three things make an injection pack worth its maintenance. It runs on the real tool surface, with the real registration and the real tiers, because a pack that runs against a mocked toolset tests the mock. It asserts on effects rather than on words, so the assertion is that no consequential call was made with a value of untrusted provenance, not that the model said something reassuring. And it carries the pairing of a capability with a task, because the same injection is harmless in a session that cannot act on it and serious in one that can.

That pack belongs in the same harness as everything else: versioned with the code, run on every change, with its history queryable so a regression has a date attached rather than a discovery story.