cs.CLOct 5, 2026

Breaking Bureaucracy: Evaluating open-source LLMs for legal document review

Authors: Farrukh Baratov, Niki van Stein, Suzan Verberne

Organizations: Leiden University Leiden, the Netherlands

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this paper, we evaluate open-source generative LLMs on legal Natural Language Inference (NLI). Legal inspectorial processes take place in specific domains and often deal with confidential data. This creates a need for working with local models that do not require labeled training data. We evaluate our models on the ContractNLI benchmark and two NLI4Wills datasets. We successfully reproduce the baseline for the task (Span NLI BERT) and we evaluate multiple open-source LLMs on the same task. We analyze the invalid rate of the models, and their stability across temperature settings and domains. Among the generative models, Gemma-4 26B performs the best, reaching an accuracy of 81.2%, even outperforming the supervised model on one metric. On accuracy, it is not possible to beat the supervised model with zero-shot approaches. Qwen-3.6 35B performs well on both ContractNLI and additional datasets in the legal wills domain. Our findings indicate that zero-shot, open-source, generative LLMs are a viable alternative for real-world legal NLI when no supervised data is available. Our code is available at https://github.com/fbaratov/contractnli-llms.

Figures & tables

MetricK&MReproductionRe-OrO.SDR.SD
mAP0.922±0.0060.922\pm 0.0060.919±0.0050.919\pm 0.005−0.003-0.003✓✓
P@R800.793±0.0180.793\pm 0.0180.801±0.0220.801\pm 0.022+0.008+0.008✓✓
Acc0.875±0.0060.875\pm 0.0060.869±0.0060.869\pm 0.006−0.006-0.006✓✓
F1 (C)0.357±0.0390.357\pm 0.0390.341±0.0430.341\pm 0.043−0.034-0.034✓✓
F1 (E)0.834±0.0020.834\pm 0.0020.826±0.0040.826\pm 0.004−0.008-0.008××
Table 1: Reproduction results for Span NLI BERT. “Re-Or” indicates the reproduction’s gain or loss over the values reported by Koreeda and Manning (2021) (‘K&M’). O.SD and R.SD indicate whether the difference is less than or equal to the original standard deviations reported by the authors or the reproduction’s standard deviations respectively.
ModelAccuracyF1(C)F1(E)Invalid Rate (%)
Majority (Ent.)0.4630.0000.590—
Span NLI BERT0.869±0.006\mathbf{0.869\pm 0.006}0.357±0.0390.357\pm 0.0390.834±0.002\mathbf{0.834\pm 0.002}—
Gemma-4 E2B †0.688±0.0030.688\pm 0.0030.254±0.0180.254\pm 0.0180.681±0.001{0.681\pm 0.001}0.0±0.00.0\pm 0.0
Gemma-4 E4B †0.746±0.006{0.746\pm 0.006}0.343±0.0120.343\pm 0.0120.753±0.005{0.753\pm 0.005}0.0±0.00.0\pm 0.0
Gemma-4 26B ∗0.812±0.0020.812\pm 0.0020.443±0.030\mathbf{0.443\pm 0.030}0.808±0.003{0.808\pm 0.003}1.0±0.11.0\pm 0.1
Ministral-3 3B †0.575±0.003{0.575\pm 0.003}0.110±0.0190.110\pm 0.0190.628±0.0010.628\pm 0.0010.0±0.00.0\pm 0.0
Table 2: Results on three-way classification. Boldface indicates the best-performing model per metric, and the “Invalid Rate (%)” column indicates the average percentage of answers outputted by the LLM from which we were unable to retrieve a coherent label. The green row highlights the best-performing LLM.
 EntailmentContradictionNotM.
ModelAvgMaxAvgMaxAvgMax
Gemma-4 E2B✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark
Gemma-4 E4B✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark
Gemma-4 26B×\times×\times×\times×\times×\times×\times
Ministral-3 3B×\times×\times×\times×\times×\times×\times
Ministral-3 8B×\times×\times×\times×\times×\times×\times
Table 3: Uncertainty analysis summary for the main results (Table 2 ) on the label-only scope. ✓ indicates tests on all 3 seeds are significant ( p~i<0.05\tilde{p}_{i}<0.05 ). ×\times indicates otherwise.
ModelAccuracyF1(C)F1(E)IR (%)
S-NLI-BERT0.869 0.0060.357 0.0390.834 0.002—
Gemma-4 E2B ∗0.680 0.0130.268 0.0260.692 0.0080.0 0.0
Gemma-4 E4B ∗0.744 0.0040.386 0.0130.747 0.0040.0 0.0
Gemma-4 26B †0.827 0.0030.456 0.0160.803 0.0010.0 0.0
Qwen-3.5 4B †0.741 0.0070.370 0.0250.742 0.0060.1 0.1
Qwen-3.5 9B †0.759 0.0040.330 0.0020.749 0.0020.0 0.0
Table 4: Ablation study – best setting per model. Boldface indicates the best-performing model per metric. The IR (%) column corresponds to the invalid answer rate. Subscripts correspond to standard deviations. The green row highlights the best-performing LLM.
Appendix figures & tables54 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Effect of temperature on ContractNLI results grouped by model family.
ModelTempAccuracyF1(C)F1(E)Invalid Rate (%)
Gemma-4 E2B †0.10.689±0.002\mathbf{0.689\pm 0.002}0.258±0.0180.258\pm 0.0180.676±0.0040.676\pm 0.0040.000±0.0000.000\pm 0.000
 0.20.685±0.0010.685\pm 0.0010.263±0.006\mathbf{0.263\pm 0.006}0.675±0.0020.675\pm 0.0020.000±0.0000.000\pm 0.000
 0.30.688±0.0030.688\pm 0.0030.254±0.0180.254\pm 0.0180.681±0.001\mathbf{0.681\pm 0.001}0.016±0.0230.016\pm 0.023
 0.40.683±0.0080.683\pm 0.0080.236±0.0110.236\pm 0.0110.676±0.0060.676\pm 0.0060.016±0.0230.016\pm 0.023
 0.50.685±0.0070.685\pm 0.0070.233±0.0030.233\pm 0.0030.678±0.0060.678\pm 0.0060.000±0.0000.000\pm 0.000
Gemma-4 E4B †0.10.743±0.0040.743\pm 0.0040.359±0.040\mathbf{0.359\pm 0.040}0.751±0.0020.751\pm 0.0020.000±0.0000.000\pm 0.000
Appendix
Table 5: Temperature analysis results. Each metric reports the average and standard deviation across three seeds. Boldface indicates the best-performing model per metric, and the “Invalid Rate (%)” column indicates the average percentage of answers outputted by the LLM from which we were unable to retrieve a coherent label. The green row highlights the best-performing temperature setting per LLM.
Table 6: Gemma-4 E2B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 7: Gemma-4 E2B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 8: Gemma-4 E2B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 9: Gemma-4 E4B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 10: Gemma-4 E4B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 11: Gemma-4 E4B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 12: Gemma-4 26B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 13: Gemma-4 26B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 14: Gemma-4 26B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 15: Ministral-3 3B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 16: Ministral-3 3B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 17: Ministral-3 3B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 18: Ministral-3 8B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 19: Ministral-3 8B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 20: Ministral-3 8B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 21: Ministral-3 14B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 22: Ministral-3 14B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 23: Ministral-3 14B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 24: Qwen-3.5 4B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 25: Qwen-3.5 4B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 26: Qwen-3.5 4B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 27: Qwen-3.5 9B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 28: Qwen-3.5 9B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 29: Qwen-3.5 9B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 30: Qwen-3.5 27B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 31: Qwen-3.5 27B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 32: Qwen-3.5 27B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
Table 33: Qwen-3.6 35B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with logits of the whole response .
Table 34: Qwen-3.6 35B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with only the label logits .
Table 35: Qwen-3.6 35B Mann-Whitney U Test results and effect sizes for uncertainty metrics calculated with the first token of the predicted label.
ModelPrecisionRecallF1AccuracyInvalid Rate
Majority0.1970.3330.2480.591—
Gemma-4 E2B †0.766±0.0110.766\pm 0.0110.722±0.0230.722\pm 0.0230.713±0.0250.713\pm 0.0250.786±0.0160.786\pm 0.0160.0±0.00.0\pm 0.0
Gemma-4 E4B †0.808±0.0110.808\pm 0.0110.751±0.0100.751\pm 0.0100.768±0.0150.768\pm 0.0150.840±0.0060.840\pm 0.0060.0±0.00.0\pm 0.0
Gemma-4 26B ∗0.591±0.0140.591\pm 0.0140.590±0.0010.590\pm 0.0010.584±0.0070.584\pm 0.0070.766±0.0050.766\pm 0.0050.1±0.00.1\pm 0.0
Ministral-3 3B †0.618±0.0280.618\pm 0.0280.598±0.0170.598\pm 0.0170.516±0.0260.516\pm 0.0260.526±0.0280.526\pm 0.0280.0±0.00.0\pm 0.0
Ministral-3 8B †0.795±0.0060.795\pm 0.0060.806±0.0090.806\pm 0.0090.800±0.0080.800\pm 0.0080.848±0.0060.848\pm 0.0060.0±0.00.0\pm 0.0
Appendix
Table 36: NLI4Wills Idaho results. Each metric reports the average and standard deviation across three seeds. Boldface indicates the best-performing model per metric, and the “Invalid Rate (%)” column indicates the average percentage of answers outputted by the LLM from which we were unable to retrieve a coherent label. The green row highlights the best-performing LLM. Majority baseline results are given for the unrelated label.
ModelPrecisionRecallF1AccuracyInvalid Rate
Majority0.1820.3330.2350.545—
RoBERTa-large-mnli0.9670.9620.9640.969—
Gemma-4 E2B0.676±0.0110.676\pm 0.0110.616±0.0050.616\pm 0.0050.616±0.0040.616\pm 0.0040.701±0.0080.701\pm 0.0080.0±0.00.0\pm 0.0
Gemma-4 E4B0.785±0.0210.785\pm 0.0210.673±0.0140.673\pm 0.0140.702±0.0160.702\pm 0.0160.766±0.0120.766\pm 0.0120.0±0.00.0\pm 0.0
Gemma-4 26B ∗†0.553±0.0050.553\pm 0.0050.497±0.0060.497\pm 0.0060.521±0.0050.521\pm 0.0050.685±0.0050.685\pm 0.0050.1±0.00.1\pm 0.0
Ministral-3 3B0.560±0.0310.560\pm 0.0310.562±0.0350.562\pm 0.0350.503±0.0310.503\pm 0.0310.499±0.0350.499\pm 0.0350.0±0.00.0\pm 0.0
Appendix
Table 37: NLI4Wills Tennessee results. Each metric reports the average and standard deviation across three seeds. Boldface indicates the best-performing model per metric, and the “Invalid Rate (%)” column indicates the average percentage of answers outputted by the LLM from which we were unable to retrieve a coherent label. The green row highlights the best-performing LLM. Majority baseline results are given for the unrelated label.
SettingAccuracyF1(C)F1(E)Invalid Rate (%)
Reasoning0.680 ± 0.0130.268 ± 0.0260.692 ± 0.0080.0 ± 0.0
Structure ∗0.688 ± 0.0030.254 ± 0.0180.681 ± 0.0010.0 ± 0.0
Both0.688 ± 0.0110.246 ± 0.0160.683 ± 0.0080.0 ± 0.0
Appendix
Table 38: Gemma-4 E2B ablation results on the ContractNLI test set. The asterisk ( ∗ ) indicates the setting used in the main results.
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned455 ± 3364 ± 484 ± 40 ± 0
Entailment66 ± 5861 ± 441 ± 20 ± 0
Contradiction44 ± 654 ± 5122 ± 20 ± 0
Appendix
Table 39: Gemma-4 E2B confusion matrix for the main results (Table 2 ).
SettingAccuracyF1(C)F1(E)Invalid Rate (%)
Reasoning0.744 ± 0.0040.386 ± 0.0130.747 ± 0.0040.0 ± 0.0
Structure ∗0.746 ± 0.0060.343 ± 0.0120.753 ± 0.0050.0 ± 0.0
Both0.738 ± 0.0000.357 ± 0.0130.744 ± 0.0030.0 ± 0.0
Appendix
Table 40: Gemma-4 E4B ablation results on the ContractNLI test set. The asterisk ( ∗ ) indicates the setting used in the main results.
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned513 ± 10289 ± 6101 ± 50 ± 0
Entailment32 ± 3902 ± 434 ± 20 ± 0
Contradiction23 ± 152 ± 2145 ± 10 ± 0
Appendix
Table 41: Gemma-4 E4B confusion matrix for the main results (Table 2 ).
SettingAccuracyF1(C)F1(E)Invalid Rate (%)
Reasoning ∗0.812 ± 0.0020.443 ± 0.0300.808 ± 0.0031.0 ± 0.1
Structure0.827 ± 0.0030.456 ± 0.0160.803 ± 0.0010.0 ± 0.0
Appendix
Table 42: Gemma-4 26B ablation results on the ContractNLI test set. The asterisk ( ∗ ) indicates the setting used in the main results. No results are presented for reasoning-enabled structured outputs, as this leads to a ∼100%\sim 100\% invalid output rate.
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned654 ± 1143 ± 195 ± 212 ± 1
Entailment35 ± 2893 ± 333 ± 37 ± 2
Contradiction16 ± 151 ± 2152 ± 12 ± 1
Appendix
Table 43: Gemma-4 26B confusion matrix for the main results (Table 2 ).
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned322 ± 4535 ± 546 ± 30 ± 0
Entailment67 ± 2847 ± 454 ± 20 ± 0
Contradiction37 ± 6149 ± 733 ± 50 ± 0
Appendix
Table 44: Ministral-3 3B confusion matrix for the main results (Table 2 ).
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned458 ± 9337 ± 9108 ± 30 ± 0
Entailment74 ± 6832 ± 662 ± 30 ± 0
Contradiction33 ± 150 ± 2136 ± 30 ± 0
Appendix
Table 45: Ministral-3 8B confusion matrix for the main results (Table 2 ).
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned447 ± 3293 ± 4163 ± 20 ± 0
Entailment24 ± 3888 ± 156 ± 30 ± 0
Contradiction17 ± 134 ± 2169 ± 20 ± 0
Appendix
Table 46: Ministral-3 14B confusion matrix for the main results (Table 2 ).
SettingAccuracyF1(C)F1(E)Invalid Rate (%)
Reasoning0.519 ± 0.0130.325 ± 0.0330.616 ± 0.00636.3 ± 1.7
Structure ∗0.741 ± 0.0070.370 ± 0.0250.742 ± 0.0060.1 ± 0.1
Both0.750 ± 0.0040.354 ± 0.0460.737 ± 0.0093.5 ± 0.5
Appendix
Table 47: Qwen-3.5 4B ablation results on the ContractNLI test set. The asterisk ( ∗ ) indicates the setting used in the main results.
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned551 ± 12206 ± 16144 ± 42 ± 1
Entailment57 ± 2845 ± 667 ± 40 ± 0
Contradiction25 ± 241 ± 2154 ± 30 ± 0
Appendix
Table 48: Qwen-3.5 4B confusion matrix for the main results (Table 2 ).
SettingAccuracyF1(C)F1(E)Invalid Rate (%)
Reasoning0.704 ± 0.0080.340 ± 0.0280.748 ± 0.00512.0 ± 0.6
Structure ∗0.759 ± 0.0040.330 ± 0.0020.749 ± 0.0020.0 ± 0.0
Both0.729 ± 0.0200.339 ± 0.0490.722 ± 0.0220.2 ± 0.1
Appendix
Table 49: Qwen-3.5 9B ablation results on the ContractNLI test set. The asterisk ( ∗ ) indicates the setting used in the main results.
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned606 ± 4184 ± 4113 ± 80 ± 0
Entailment84 ± 7827 ± 757 ± 90 ± 0
Contradiction22 ± 045 ± 2154 ± 20 ± 0
Appendix
Table 50: Qwen-3.5 9B confusion matrix for the main results (Table 2 ).
SettingAccuracyF1(C)F1(E)Invalid Rate (%)
Reasoning0.260 ± 0.0020.178 ± 0.0590.424 ± 0.00268.6 ± 0.0
Structure0.796 ± 0.0030.401 ± 0.0190.786 ± 0.0030.0 ± 0.0
Both ∗0.793 ± 0.0050.422 ± 0.0240.783 ± 0.0060.3 ± 0.0
Appendix
Table 51: Qwen-3.5 27B ablation results on the ContractNLI test set. The asterisk ( ∗ ) indicates the setting used in the main results.
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned636 ± 7144 ± 4120 ± 43 ± 2
Entailment51 ± 5858 ± 656 ± 33 ± 1
Contradiction11 ± 344 ± 0164 ± 21 ± 0
Appendix
Table 52: Qwen-3.5 27B confusion matrix for the main results (Table 2 ).
SettingAccuracyF1(C)F1(E)Invalid Rate (%)
Reasoning0.791 ± 0.0050.387 ± 0.0210.793 ± 0.0030.1 ± 0.1
Structure0.774 ± 0.0050.364 ± 0.0200.776 ± 0.0020.1 ± 0.0
Both ∗0.760 ± 0.0060.330 ± 0.0050.780 ± 0.0080.6 ± 0.2
Appendix
Table 53: Qwen-3.6 35B ablation results on the ContractNLI test set. The asterisk ( ∗ ) indicates the setting used in the main results.
True \ PredNotMentionedEntailmentContradictionInvalidAnswer
NotMentioned520 ± 7197 ± 12175 ± 410 ± 3
Entailment23 ± 3897 ± 645 ± 23 ± 1
Contradiction3 ± 146 ± 1171 ± 10 ± 0
Appendix
Table 54: Qwen-3.6 35B confusion matrix for the main results (Table 2 ).
ModelP(C)R(C)F1(C)
Gemma-4 E2B0.215 ± 0.0270.254 ± 0.0120.254 ± 0.018
Gemma-4 E4B0.203 ± 0.0090.382 ± 0.0440.343 ± 0.012
Gemma-4 26B0.287 ± 0.0130.472 ± 0.0510.443 ± 0.030
Ministral-3 3B0.124 ± 0.0260.128 ± 0.0620.110 ± 0.019
Ministral-3 8B0.219 ± 0.0400.427 ± 0.0290.325 ± 0.045
Ministral-3 14B0.190 ± 0.0060.558 ± 0.0090.372 ± 0.008
Appendix
Table 55: Contradiction class metrics on the main results (Table 2 ). Boldface indicates the best-performing model per metric. All scores are micro averaged over documents, then macro averaged over labels.
ModelP(E)R(E)F1(E)
Gemma-4 E2B0.601 ± 0.0020.837 ± 0.0060.681 ± 0.001
Gemma-4 E4B0.708 ± 0.0070.868 ± 0.0050.753 ± 0.005
Gemma-4 26B0.781 ± 0.0050.867 ± 0.0020.808 ± 0.003
Ministral-3 3B0.541 ± 0.0010.866 ± 0.0080.628 ± 0.001
Ministral-3 8B0.639 ± 0.0110.830 ± 0.0070.691 ± 0.009
Ministral-3 14B0.672 ± 0.0040.851 ± 0.0010.728 ± 0.002
Appendix
Table 56: Entailment class metrics on the main results (Table 2 ). Boldface indicates the best-performing model per metric. All scores are micro averaged over documents, then macro averaged over labels.
ModelP(N)R(N)F1(N)
Gemma-4 E2B0.779 ± 0.0190.430 ± 0.0040.511 ± 0.004
Gemma-4 E4B0.844 ± 0.0100.491 ± 0.0090.575 ± 0.007
Gemma-4 26B0.887 ± 0.0060.657 ± 0.0040.718 ± 0.003
Ministral-3 3B0.649 ± 0.0410.292 ± 0.0050.369 ± 0.011
Ministral-3 8B0.741 ± 0.0310.431 ± 0.0120.496 ± 0.016
Ministral-3 14B0.726 ± 0.0470.394 ± 0.0070.467 ± 0.014
Appendix
Table 57: NotMentioned class metrics on the main results (Table 2 ). Boldface indicates the best-performing model per metric. All scores are micro averaged over documents, then macro averaged over labels.

Explore similar work

CardsList
Jun 8, 2026cs.CL

LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks

As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.

Yifan Chen, Haitao Li, Yiran Hu +6

1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1

Legal Reasoning TasksTask-Specific RubricsDiagnostic BenchmarkDiagnosis
AskPDF
Jun 15, 2026cs.AI

Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning

Large Language Models (LLMs) achieve strong performance on reasoning tasks, but whether this reflects faithful logical inference or heuristic approximation remains unclear. We study this question in legal entailment by comparing three paradigms, including pure LLM classification, LLM-based Formal Reasoning, and solver-based Formal Reasoning using the Z3 SMT solver, on a re-annotated subset of ContractNLI across five LLMs. Our re-annotation reveals a systematic and measurable gap between pragmatic legal interpretation and strict formal entailment, where a substantial proportion of legally sound inferences are not formally grounded without additional unstated assumptions. While introducing formal structure improves accuracy, with LLM-based Formal Reasoning achieving the highest benchmark performance, we show that this gain does not imply faithful reasoning. We identify three recurring failure modes: scope laundering, where LLMs report solver-inconsistent classifications without executing the underlying formal reasoning, producing conclusions that appear logically grounded but are not; implicit constraint blindness, where LLMs overlook logical constraints present in formal representations; and program synthesis failures, where LLMs generate incorrect Z3 code despite structured prompting. Critically, scope laundering persists across all models, raising serious concerns about the faithfulness of LLM-based formal reasoning as a proxy for symbolic execution. These results reveal a fundamental gap between benchmark accuracy and logical faithfulness.

Olivia Peiyu Wang, Sanna Wong-Toropainen, Daneshvar Amrollahi +4

UC Santa Cruz · Univ. Helsinki · CodeX, Stanford +4

Legal Reasoning TasksLLM Reasoning StrategiesDeductionAutoformalizationSolvers
AskPDF
May 27, 2026cs.CL

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

We introduce BenGER (Benchmark for German Law), a benchmark and dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The dataset combines 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal reasoning tasks. It includes a controlled validation subset of timed human-written solutions under both unaided and human-AI co-creation conditions. We evaluate 12 contemporary LLM systems - closed flagship, efficiency-oriented, and open-weight - with a rubric-aligned LLM-as-a-Judge cross-validated against a multi-rater human-grading layer (three blind reviews per solution, six judge families benchmarked against the human pool). Closed-flagship systems lead the leaderboard across all three corpora, human-AI co-creation measurably improves on unaided human work, and the LLM judge tracks human grading at Pearson r=0.76 and Cohen's k=0.60. System rankings are stable across judge families and two judges from independent providers clear the Calderon single-reviewer replacement bar on human-authored solutions.

Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach +6

Technical University of Munich (TUM) · Ludwig Maximilian University of Munich (LMU) · University of Konstanz +1

Legal Reasoning TasksGermanGradingHuman-Ai Collaboration
AskPDF