cs.AIOct 19, 2026

LLMs Prompted for Legal Context Refuse More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context

Authors: Anastasiia Kucherenko, Francois Brouchoud, Dimitri Percia David, Andrei Kucharavy

Organizations: IEM, HEG, HES-SO Valais-Wallis Sierre, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Confidentiality requirements increasingly push legal professionals toward small, on-device LLMs that keep case data in-house, used for everyday tasks such as summarization, translation, and reformulation. Following standard prompting guidance, a user naturally states their role (``I am a lawyer...''). We show that this innocuous habit has an unexpected cost: such authority framing can make the model selectively refuse legitimate requests, introducing bias through uneven access to assistance. We evaluate four modern small open-weight LLMs on benign legal-domain prompts to measure this over-refusal. Surprisingly, we find that authority-style prefixes (''I am legal counsel working for a national supreme court'', ''I'm a defense lawyer'') systematically increase refusal rates by 2--20x over the no-prefix baseline, while a known role-play jailbreak prefix shows mixed effects, sharply increasing refusals in some models and barely shifting them in others. The finding suggests that small on-prem deployable LLMs are unstable under contextual framings that a real institutional user might naturally introduce, and further investigation is essential to minimize opportunities for bias.

Figures & tables

Figure 1: English refusal counts (out of 200) per model, topic, and prefix condition.
  FrenchGerman
ModelTopicNoneSup.NoneSup.
Llama 3.1 8BViol.15401354
Sex.956659
Harm.10331240
Uneth.15411848
Illeg.35623361
Table 1: French and German refusal rates (%) on OR-Bench prompts ( 200200 texts tested per cell): None = no-prefix baseline, Sup. = supreme-court prefix.
 ENFRDE
ModelNoneSup.NoneSup.NoneSup.
Llama 3.1 8B18.869.012.523.425.628.5
Gemma 4 E4B0.50.10.40.10.10.1
Apertus 8B1.43.91.119.30.82.5
Qwen 3 8B0.20.20.00.10.10.4
Table 2: Real-text legal refusal rates (%) on Swiss Federal Supreme Court (BGer) judgments (1 652 texts tested per cell): None vs. supreme-court prefix ( Sup. ), per language.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

ModelLangPrefixNone →\to Prefpp (Holm)Sig.
Llamaenlaw.7.2 →\to 11.18.8×10−48.8\times 10^{-4}Yes
Llamaensup.7.3 →\to 13.28.1×10−78.1\times 10^{-7}Yes
Llamaenjail.7.3 →\to 27.56.3×10−516.3\times 10^{-51}Yes
Llamafrsup.16.8 →\to 46.91.6×10−851.6\times 10^{-85}Yes
Llamadesup.16.5 →\to 53.31.6×10−1001.6\times 10^{-100}Yes
Gemmaenlaw.10.7 →\to 15.26.3×10−46.3\times 10^{-4}Yes
Appendix
Table 3: OR-Bench: paired refusal rates (%) under authority/jailbreak framing vs. the none baseline. Models are the 8B variants (Gemma: 4 E4B). Prefixes: law. lawyer, sup. supreme-court, jail. jailbreak. None →\to Pref gives baseline and prefixed refusal rates. pp (Holm) is the McNemar exact-test pp -value after Holm-Bonferroni correction for multiple comparisons; Sig. marks whether the increase is significant at α=0.05\alpha=0.05 .
ModelLangNone →\to Prefpp (Holm)Sig.
Llamaen18.8 →\to 68.93.9×10−2493.9\times 10^{-249}Yes
Llamafr12.5 →\to 23.42.3×10−402.3\times 10^{-40}Yes
Llamade25.6 →\to 28.51.8×10−21.8\times 10^{-2}Yes
Apertusfr1.1 →\to 19.34.3×10−764.3\times 10^{-76}Yes
Apertusen1.4 →\to 3.97.0×10−67.0\times 10^{-6}Yes
Apertusde1.0 →\to 2.51.1×10−31.1\times 10^{-3}Yes
Appendix
Table 4: Real BGer (Swiss federal court) paragraphs: paired refusal rates (%) under the supreme-court prefix vs. the none baseline. None →\to Pref gives baseline and prefixed rates. pp (Holm) is the McNemar exact-test pp -value after Holm-Bonferroni correction; Sig. marks significance at α=0.05\alpha=0.05 .
More in this topic

LLM Refusal Behavior

Large language model (LLM) refusal behavior studies when models decline requests and how to balance safety with helpfulness.

See all 67 papers

Explore similar work

CardsList

  1. LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context

    Jun 23, 2026Anastasiia Kucherenko, François Brouchoud, Dimitri Percia David +1LLM PromptingSmall Language Models

    IEM, HEG, HES-SO Valais-Wallis Sierre, Switzerland

  2. Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

    Jun 22, 2026Arthur Wuhrmann, Gaetan Stein, Daniel Brunner +1Multilingual Language Model EvaluationLegal NLP

    Swiss Federal Supreme Court · HES-SO Valais-Wallis

  3. Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority

    Oct 21, 2026M. Mikail DemirLegal Citation VerificationLLM Abstention

    Department of Information Science, University at Albany, SUNY, USA