Adversarial evaluation is a test of boundaries

A useful threat model names the asset, the attacker, the entry point, the allowed manipulation, and the undesired outcome. An adversary who can edit a retrieved document has different capabilities from an authenticated user who can submit a prompt or an administrator who can change a tool implementation. Mixing them in one undifferentiated attack-success number hides which boundary failed.

For a retrieval assistant, define a test attacker who can alter the body of one synthetic document but cannot change authenticated identity, server-side permissions, or evaluation labels. The protected asset might be another tenant's records. The failure condition is unauthorized disclosure or action, not merely the model repeating suspicious text. Use only owned fixtures or environments where testing is authorized, with a defined request budget and cleanup procedure. This scope makes results interpretable as well as operationally safe.

Prompt injection occurs when untrusted content influences model behavior in an unintended way, including content received indirectly through retrieval or tool output. OWASP's GenAI guidance describes this risk and the need for layered mitigations. A system message asking the model to ignore malicious text can help behavior, but the application must still enforce authorization independently of the model's proposed action.

Imagine a retrieved page instructs the assistant to call export_records for tenant blue. The application should derive the requesting tenant from authenticated server context, validate the resource and scope, and reject a cross-tenant export even if the model confidently asks for it. The test should verify the downstream operation was blocked, not simply whether the assistant apologized. A model can say it refused while an earlier tool call already caused harm; conversely, suspicious generated text may be harmless if no privileged action or disclosure occurred.

Vary one attack feature at a time when diagnosing a weakness. A document may contain a direct command, a forged system-message delimiter, a claim of administrator approval, or an instruction hidden in a structured field. Pair each adversarial fixture with a benign task using the same route. The comparison reveals whether a defense preserves legitimate retrieval and whether success depends on one formatting trick.

Use synthetic canaries and harmless local actions rather than real secrets or destructive operations. Save the exact fixture, model and prompt revision, tool schema, identity context, and observable result. A passing case should state which invariant held, such as "no cross-tenant record was returned." The same payload across many paraphrases is not necessarily broad coverage; vary the trust boundary, attack objective, and action pathway as well as the wording.

Attack success rate requires a denominator: attempts, unique scenarios, prompts, or complete episodes. An episode with ten retries differs from one attempt, and the success probability can increase with the attacker's budget. Report the permitted number of attempts and the event that counts as success. Separate unauthorized tool execution, confidential data exposure, evaluator manipulation, and simple instruction-following deviations.

Also measure benign task completion and false denials. A defense that rejects every request can score perfectly on a narrow attack set while making the product useless. For an invented fixture set with 20 attack cases and 20 benign cases, zero unauthorized actions alongside only two benign completions is a very different result from zero unauthorized actions with 19 benign completions. These numbers illustrate metric interpretation and are not measured product-security claims.

For each observed failure, identify the weakest enforceable boundary: resource scoping, schema validation, egress policy, confirmation binding, output escaping, or tool-result handling. Fix the boundary and retain the minimized case as a regression test. Do not rely only on adding the exact attack string to a blocklist, because a paraphrase or alternate channel may express the same instruction. Record residual risk when the underlying model remains susceptible but a consequential action is contained.

MITRE ATLAS and OWASP can organize coverage and provide vocabulary, but a catalog match is not a proof that your deployment is secure. Revisit tests after changing tools, authentication, model providers, retrieval sources, or streaming behavior. A defensible red-team report explains what was tested, what happened, what control changed, and what remains outside scope. Its value is actionable evidence about the system, not a universal safety percentage.

The authenticated tenant is trusted input supplied by the application, while proposed_tenant represents an untrusted model proposal. This local policy simulation tests a narrow isolation rule. It does not implement authentication, retrieval, or an actual prompt-injection detector.