Can Small Models Reason About Legal Documents? A Comparative Study

Snehit Vaddi

cs.CLMar 26, 2026

Can Small Models Reason About Legal Documents? A Comparative Study

Authors: Snehit Vaddi

Organizations: Independent Researcher

Abstract

Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy. We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies (direct, chain-of-thought, few-shot, BM25 RAG, and dense RAG). Across 405 experiments with three random seeds per configuration, we find that a Mixture-of-Experts model activating only 3B parameters matches GPT-4o-mini in mean accuracy while surpassing it on legal holding identification, and that architecture and training quality matter more than raw parameter count. Our largest model (9B parameters) performs worst overall. Chain-of-thought prompting proves sharply task-dependent, improving contract entailment but degrading multiple-choice legal reasoning, while few-shot prompting emerges as the most consistently effective strategy. Comparing BM25 and dense retrieval for RAG, we find near-identical results, suggesting the bottleneck lies in the language model's utilization of retrieved context rather than retrieval quality. All experiments were conducted via cloud inference APIs at a total cost of $62, demonstrating that rigorous LLM evaluation is accessible without dedicated GPU infrastructure.

Figures & tables

Study	Models	Tasks	Strat.	RAG	Seeds
Chalkidis (2023)	1	7	1	×	1
Guha et al. (2023)	20	162	1	×	1
Arvin (2025)	8	1	1	×	1
Pecher et al. (2025)	6+	4+	2	×	3+
Ours	9	3	5 †	✓	3

Table 1: Comparison with closely related work. † Includes both BM25 and dense retrieval.

Model	Arch.	Params	Provider	Ctx.
Llama-3.2-3B	Dense	3B	Together	128K
Gemma-3n-E4B	Dense	4B eff	Together	32K
Qwen2.5-7B	Dense	7B	Together	128K
Mistral-7B	Dense	7B	Together	32K
Llama-3.1-8B	Dense	8B	Together	128K
Nemotron-9B	Dense	9B	Together	32K

Table 2: Models evaluated in our study, spanning 3B–9B parameters for open-weight models and two commercial API services. All models are instruction-tuned variants accessed via cloud APIs. All context windows are capped at 8,192 tokens in our experiments. eff Gemma-3n-E4B uses net-efficient architecture with 4B effective parameters. a Qwen3-A3B is a Mixture-of-Experts model activating 3B of 30B total parameters per token.

Dataset	Task Type	Labels	$N$	Juris.	Chance
ContractNLI	NLI	3	500	US	33.3%
CaseHOLD	MC (5-way)	5	500	US	20.0%
ECtHR Task A	Multi-label	11 arts.	225	EU	varies

Table 3: Dataset characteristics.

N

is the number of stratified test samples per experiment. ECtHR Task A yields 225 samples due to the high number of unique multi-label combinations (51) under stratified sampling. Chance denotes random baseline accuracy; ECtHR chance varies by article combination.

Model	Params	Direct	Few-Shot	CoT	RAG BM25	RAG Dense	Mean
GPT-4o-mini	API	.498	.499	.425	.471	.470	.472
Qwen3-A3B	3B ∗	.503	.521	.393	.453	.454	.465
Qwen2.5-7B	7B	.452	.468	.325	.445	.440	.426
Mistral-7B-v0.3	7B	.447	.440	.336	.427	.411	.412
Llama-3.1-8B	8B	.439	.425	.366	.381	.381	.398
Gemma-3n-E4B	4B †	.374	.440	.316	.363	.357	.370

Table 4: Mean accuracy across 3 datasets and 3 seeds (9 values per cell). Models sorted by mean accuracy. Best per-strategy value in bold , second-best underlined . ∗ 3B active parameters (MoE). † 4B effective parameters. Dashed line separates models above and below the 0.35 threshold.

Model	Params	ContractNLI	CaseHOLD	ECtHR	Mean
GPT-4o-mini	API	.766 [.756, .776]	.622 [.611, .633]	.029 [.024, .035]	.472
Qwen3-A3B	3B ∗	.775 [.765, .784]	.558 [.548, .569]	.063 [.055, .071]	.465
Qwen2.5-7B	7B	.683 [.672, .693]	.544 [.533, .555]	.051 [.044, .059]	.426
Mistral-7B	7B	.686 [.676, .697]	.524 [.513, .535]	.027 [.021, .032]	.412
Llama-3.1-8B	8B	.578 [.567, .589]	.578 [.566, .589]	.039 [.033, .046]	.398
Gemma-3n-E4B	4B †	.669 [.658, .680]	.411 [.400, .422]	.029 [.024, .035]	.370

Table 5: Accuracy by model and dataset, averaged across 5 strategies and 3 seeds (15 values per cell). 95% bootstrap CIs in brackets (pooling all predictions per model-dataset combination). Best in bold , second-best underlined .

Strategy	CNLI	CH	ECtHR	Mean
Direct	.570	.537	.049	.385
Few-Shot	.608	.532	.051	.397
CoT	.655	.377	.011	.348
RAG BM25	.627	.456	.031	.371
RAG Dense	.619	.450	.036	.369

Table 6: Mean accuracy by strategy and dataset, averaged across 9 models and 3 seeds (27 values per cell). CNLI = ContractNLI , CH = CaseHOLD . Best in bold , second-best underlined .

Model	CNLI	CaseHOLD
	CoT $-$ Dir	CoT $-$ Dir
Nemotron-9B	+43.8 ∗∗∗	+0.6
Claude 3.5 Haiku	+14.2 ∗∗∗	+19.2 ∗∗∗
Llama-3.2-3B	+14.0 ∗∗∗	$-$ 21.0 ∗∗∗
Gemma-3n-E4B	+4.8 ∗∗	$-$ 17.8 ∗∗∗
Qwen2.5-7B	+2.4 ∗	$-$ 37.2 ∗∗∗

Table 7: Accuracy change from direct to CoT on seed 42 (percentage points,

n

=500 per condition). CNLI = ContractNLI . Paired bootstrap significance test:

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

. Results are consistent across seeds (see Appendix).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Provider	Cost	Med.	P95
		(USD)	Lat.	Lat.
Haiku 3.5	Anthropic	$13.43	4.49s	12.85s
Qwen2.5-7B	Together	$11.05	1.98s	5.52s
Qwen3-A3B	Together	$9.56	2.10s	4.32s
Mistral-7B	Together	$9.35	2.26s	3.83s
Llama-3.1-8B	Together	$6.51	1.08s	1.68s

Appendix

Table 8: Per-model inference cost and latency across all 405 experiments (324 main + 81 ablation). Median and P95 latencies are averaged across all configurations per model. Models sorted by total cost.

Figure 1: Overview of the five prompting strategies. Each row shows the prompt components assembled for a given strategy. All strategies share the core components (legal text, question, format instruction) but differ in the additional context provided. Output token budgets vary by strategy: 128 tokens (direct, few-shot), 256 tokens (RAG), and 512 tokens (CoT).

Model	Dir	FS	CoT	R B	R D
GPT-4o-mini	.761	.791	.771	.754	.755
Qwen3-A3B	.759	.767	.776	.787	.785
Haiku	.657	.618	.799	.638	.645
Mistral-7B	.675	.678	.659	.723	.696
Qwen2.5-7B	.657	.697	.680	.688	.691
Gemma-3n-E4B	.605	.719	.687	.678	.655

Appendix

Table 9: ContractNLI accuracy (mean over 3 seeds). Dir = Direct, FS = Few-Shot, R B = RAG BM25 , R D = RAG Dense . Best in bold , second-best underlined .

Model	Dir	FS	CoT	R B	R D
Qwen3-A3B	.692	.712	.373	.498	.513
GPT-4o-mini	.681	.679	.499	.634	.615
Qwen2.5-7B	.649	.649	.277	.583	.563
Mistral-7B	.606	.619	.342	.540	.511
Llama-3.1-8B	.618	.595	.535	.561	.579
Llama-3.2-3B	.577	.482	.371	.407	.415

Appendix

Table 10: CaseHOLD accuracy (mean over 3 seeds). Best in bold , second-best underlined . Note the consistent CoT degradation across models, with Haiku as the sole exception.

Model	Dir	FS	CoT	R B	R D
Haiku	.073	.119	.000	.016	.004
Qwen3-A3B	.059	.085	.031	.074	.064
Llama-3.1-8B	.058	.073	.018	.024	.024
Qwen2.5-7B	.050	.059	.019	.062	.065
Mistral-7B	.061	.022	.006	.019	.025
Gemma-3n-E4B	.052	.025	.000	.028	.042

Appendix

Table 11: ECtHR subset accuracy (mean over 3 seeds,

n

=225 per experiment). Best in bold , second-best underlined . Haiku few-shot achieves 11.9%, the only configuration exceeding 10%.

Model	CNLI	CH	ECtHR	Mean
GPT-4o-mini	.000	.000	.001	.000
Qwen2.5-7B	.000	.000	.001	.000
Mistral-7B	.003	.000	.002	.002
Llama-3.1-8B	.003	.000	.007	.003
Llama-3.2-3B	.001	.002	.010	.004
Qwen3-A3B	.001	.004	.035	.013

Appendix

Table 12: Mean parse error rate by model and dataset (proportion of unparseable outputs). Worst (highest) per column in bold . † Rates exceeding 10%.

Small Language Models

Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.

See all 262 papers

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models
Apr 26, 2026Tianchun Li, Haochen Liu, Vishwa Pardeshi +5Synthetic Data Generation Small Language Models
Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA
LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification
Apr 20, 2026Pedro Barbosa de Carvalho NetoLegal NLP LLM Evaluation
Teresina, PI, Brazil
Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval
Feb 18, 2026Subrit DikshitSmall Language Models Legal IR
Department of Computer Science and Engineering, Indian Institute of Information Technology, Pune, India · Director of AI & Data Science, Compunnel Inc

Apr 26, 2026J/K move · Enter open · S save

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

Tianchun Li, Haochen Liu, Vishwa Pardeshi, Xingchen Wang+4

Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA

Synthetic Data Generation Small Language Models Language Model Distillation Synthetic Data Generation for Language Models Legal Reasoning

Open paper AskPDF arXiv

Small language models (SLMs) are promising for real-world deployment due to their efficiency and low operational cost. However, their limited capacity struggles with high-stakes legal reasoning tasks that require coherent statute interpretation and logically consistent deduction. Furthermore, training SLMs for such tasks demands high-quality, concise reasoning trajectories, which are prohibitively expensive to manually collect and difficult to curate via standard rejection sampling, lacking granularity beyond final verdicts. To address these challenges, we propose {LegalDrill}, a diagnosis-driven synthesis framework that extracts and iteratively refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the SLM student. The resulting data empower SLM training through supervised fine-tuning and direct preference optimization. Extensive experiments on several legal benchmarks demonstrate that {LegalDrill} significantly bolsters the legal reasoning capabilities of representative SLMs while bypassing the need for scarce expert annotations, paving a scalable path toward practical legal reasoning systems.

Abstract

Small Language Models

Explore similar work

LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval