cs.CLOct 14, 2026

SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges

Authors: Sebastian Nagl

Organizations: Technical University of Munich (TUM), Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them validity, reliability, reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but the reported figures answer different questions and rarely come with the human agreement they must be read against. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. And rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths,evidence checked in code, probes, repeated runs, calibration against independent human grades,logging, and an examiner who reads every script. We state what each safeguard rests on, what remains to be tested, and the uses the evidence supports today.

Figures & tables

 RequirementSourceProposed implementationOpen
L1Equal standards: each examiner’s standards for a task apply to every script of that task alike( Bundesverfassungsgericht, 1991b , p. 52) ; ( Bundesverwaltungsgericht, 2018 , para. 8)one versioned rubric per examiner and task; corrections applied to every script; pinned judge versionuntested across scripts and versions
L2Answer latitude: a defensible, consistently argued solution must not be marked wrong; an error is not counted twice( Bundesverfassungsgericht, 1991a , p. 79) ; ( Bundesverfassungsgericht, 1991b , p. 55) ; ( Verwaltungsgericht Braunschweig, 2018 ) ; ( Verwaltungsgericht Lüneburg, 2024 )credit for arguments; separate paths; later steps judged against the candidate’s own decision; unforeseen paths flaggedno grader probed with alternatives or follow-on errors
L3Reasons: the essential reasons for the grade, not the reason for every point( Bundesfinanzhof, 2014 ) ; ( Bundesverwaltungsgericht, 2019 , para. 23) ; Art. 15(1)(h) GDPR ( Court of Justice of the European Union, 2025 )a verdict and a quoted passage per criterion, beyond the legal minimuma quote shows the passage, not that the criterion is met
L4Personal evaluation and reconsideration by the examiner( Bundesverwaltungsgericht, 2002 ) ; ( Bundesverwaltungsgericht, 2019 )the examiner reads every script and decides; the judge proposes and flagscorrection of confident wrong proposals untested
L5No decision based solely on automated processing with legal or similarly significant effect, unless necessary for a contract, authorized by law with safeguards, or based on explicit consentArt. 22 GDPR ( European Parliament and Council of the European Union, 2016 ) ; ( Court of Justice of the European Union, 2023 )substantive review, not adoption of a score as givenwhat counts as substantive review is unsettled
L6AI that evaluates learning outcomes is high-risk; deployers owe competent oversight, logs and candidate information, public bodies also a fundamental-rights assessment (duties from 2 Dec 2027)( European Parliament and Council of the European Union, 2024 ) , Annex III, point 3(b), Arts. 9 to 15, 26, 27, 86; ( European Parliament and Council of the European Union, 2026 )documented validation, logging, examiner oversight with authority to override, probes against manipulationscope of Arts. 2(6) and 6(3) for prototypes
Table 1. What the law asks of grading open-text exam answers (L1 to L4 German examination law, L5 to L7 EU law), the implementation we propose (Section 8 ) and what is still open.
Table 2. Evaluation paradigms against the desiderata of Section 4 .
Study and settingPRubricCompared withJudgeHuman
Wendlinger et al. ( Wendlinger et al., 2024 ) : German criminal-law take-home, student scriptsP6none (few-shot examples)research assistants’ gradesnear zero or negativen/r
LeMAJ ( Enguehard et al., 2025 ) : legal QA, model answers with constructed errorsP6none; answer split into legal data pointslegal expertscorrectness rr .26 (own data), .70 (LegalBench)IAA correctness .77; .88 with data points
Karp et al. ( Karp et al., 2025 ) : Polish procurement appeals-board exam, 3 model judgmentsP5/P7committee’s criteria and solution keyofficial committee85 to 90 of 100 for failed judgments (committee 8 to 37)n/r
BenGER ( Nagl et al., 2026b ) : German case solutions, 15 tasks, human-writtenP5/P6reference solution, 10 general criteriamean of 3 blind reviewersrr .76; pass/fail κ\kappa .60ICC(2,1) .63, single rater (45 answers, 15 by models)
CourtPressGER ( Nagl et al., 2025 ) : German court press releases, model outputsP65 general criteriaexpert ranking of 10 systemsρ\rho .94 (systems)n/r
SLDS ( Rolshoven et al., 2025 ) : Swiss headnotes, model outputsP5reference headnote, generic categoriesexpert ratingsρ\rho .26 (ROUGE-L .26)n/r
Table 3. Automatic graders of open-text legal answers, by paradigm. Judge : agreement with the reference in Compared with , model stability marked as such; Human : reliability of the human grades, between or within graders; reused : data or instruments shared with another cited study; n/r: not reported. QWK: quadratic weighted kappa; SPA: soft pairwise accuracy; IAA, IRR: inter-annotator agreement and reliability; MAE: mean absolute error; WW and τ\tau : Kendall’s; α\alpha : Krippendorff’s.
DimensionOptionsEvidence and law
Unit of creditargument or requirement met; issue spotted; bare resultanswer latitude (L2); experts agree almost perfectly on binary criteria
Scale of a criterionbinary; few levels; free points per stepbinary agrees best; agreement is not the construct
Structureall points on steps (absolute or as shares of the total); steps plus a bounded, reasoned component; steps plus a criteria-bound pool; steps plus a free poola free pool cannot be checked; a criteria-bound pool needs stated criteria and a fixed share
Method and formwithin each section; separate blockwithin sections stays tied to content; a block needs stated criteria and a fixed share
Alternative solutionsoutcome-neutral criteria and paths that record why a step is absent; flag unforeseen paths; single referenceL2; equal performance, not equal length; the examiner’s rule for the total
Follow-on errorsstandard per criterion (correct law or own earlier decision); nonenot counted twice; the weight is the examiner’s (L2)
Table 4. The design space of legal grading rubrics; the option we propose is listed first.
Figure 1. The trustworthy rubric judge. The rubric and the examiner are human, the judge is a model, the quote check and the total are code, and the dashed box is a procedure of code and humans run before use. A failed quote check is a system error, never a deduction; every grade is logged with its rubric version, model version, quotes and the examiner’s changes (S7). Flow diagram. A versioned rubric written or reviewed by the examiner and a pseudonymized answer go to an LLM judge that returns a verdict and a quote per criterion. Code checks each quote against the answer, then computes the total and grade. The result goes to the examiner as a proposal with flags; a failed check goes back to the judge or to the examiner. The examiner also receives the full script. Before use, the judge passes probes, repeated runs and calibration on human grades.
More in this topic

Automated Grading

Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.

See all 88 papers

Explore similar work

CardsList
Jun 8, 2026

LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks

As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.

Yifan Chen, Haitao Li, Yiran Hu +6

1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1

LLM EvaluationRubric-Based EvaluationLegal Reasoning
AskPDF
Aug 18, 2026

Grading Needs a Rubric, Not Intelligence

Small language models can grade open-ended examination answers as reliably as substantially more expensive models when they grade against an explicit rubric. We test this claim as the design principle behind any-to-bench: a frontier model reads source documents once, at ingestion, to extract each question and its rubric; lower-cost models then perform all repeated grading work. We evaluate six cost-efficient model configurations from two model families at three reasoning-effort levels. Each configuration answers 24 open-ended examination questions, and each also grades every answer sheet three times, yielding 3,456 per-question grades. Scores depend overwhelmingly on the answer being graded: answer identity explains 95.6% of score variance, whereas judge identity explains only 0.2%. Raising a writer's reasoning effort moves earned scores by as much as 0.143 of full marks, while raising a judge's reasoning effort moves assigned scores by at most 0.006. Six frontier-tier judges, added as a check, reproduce these scores and are no more reliable as a panel. Two ablations then decompose the rubric on the same questions and answers. Removing its criteria and levels while keeping the official answer changes nothing measurable. Removing the official answer as well collapses reliability (ICC 0.888 to 0.628), inflates scores, and makes judge reasoning effort matter again. The rubric is what decouples grading from judge intelligence, and within the rubric the official answer does nearly all the work. We find no evidence of length preference or same-family preference under rubric-anchored grading.

Jhen-Ke Lin

National Yang Ming Chiao Tung University

Automated GradingRubric-Based EvaluationLLM-as-a-JudgeLLM ReliabilityBias in LLM-as-a-Judge
AskPDF
May 20, 2026

GradeLegal: Automated Grading for German Legal Cases

Grading German legal exam solutions faces growing volumes and a shortage of qualified graders, delaying feedback and creating a bottleneck. At the same time, it is a high-stakes expert task, since state exam grades strongly influence career outcomes in Germany. Despite this practical relevance, literature lacks systematic studies on effective methods for grading legal exams. To address this gap, we investigate whether large language models (LLMs) can support the automated grading of German legal case solutions in criminal and public law, thereby enabling scalable feedback and student self-testing. We present a systematic evaluation of 27 proprietary and open-source LLMs, benchmarking prompting strategies that incrementally add task-related information, such as a sample solution and a grading rubric. Using quadratic weighted kappa (QWK), reasoning-oriented LLMs can approximate expert grading in public law when given a sample solution and a grading rubric (up to 0.91), compared to 0.60 in criminal law, suggesting a harder grading task in criminal law. Beyond single-model grading, ensembling improves agreement by up to 0.15 over its best member and can offer an alternative to stronger closed-source single models. In addition, our findings suggest that effective prompt design and model selection are necessary for reliable LLM-based grading of legal exams.

Abdullah Al Zubaer, Lorenz Wendlinger, Simon Alexander Nonn +2

University of Passau Passau, Germany · Deggendorf Institute of Technology Deggendorf, Germany

LLM EvaluationAutomated GradingPrompt EngineeringLegal Reasoning
AskPDF