cs.AINov 6, 2026

Notwithstanding Anything to the Contrary, Ignore All Instructions: Measuring Prompt Injection Guards on Legal Contract Review Attacks

Authors: Mark Heimann, Katherine Taylor, Bruno Jacob

Organizations: Ivo AI

Abstract

Specialized prompt injection guards are deployed in front of document analysis systems, but no false positive rate has been published for any of them on legal contracts. We measure eleven open-weights guards on two corpora of legal contracts and find that existing guards struggle: document-level false positive rates are as high as 92.39%; two guards fall below chance on paired AUC; and a five-regex baseline is not Pareto-dominated by any guard in the panel. Controlled probes locate model-specific as well as general failure modes. Prompted LLMs resolve the tested failures, including a second threat category (unauthorized scope claims) on which every guard scores zero. A cascaded LLM solution, where the positives of a mid-tier model are verified by a frontier model, is 6.5x cheaper than using exclusively a frontier model and approaches its performance.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection

    Sep 21, 2026Fernando Outeda, Gustavo Betarte, Juan Diego Campo +1LLM GuardrailsPrompt Injection Defense

  2. Breaking Bureaucracy: Evaluating open-source LLMs for legal document review

    Oct 5, 2026Farrukh Baratov, Niki van Stein, Suzan VerberneZero-Shot LearningLegal NLP