SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges
Organizations: Technical University of Munich (TUM), Germany
Abstract
Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them validity, reliability, reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but the reported figures answer different questions and rarely come with the human agreement they must be read against. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. And rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths,evidence checked in code, probes, repeated runs, calibration against independent human grades,logging, and an examiner who reads every script. We state what each safeguard rests on, what remains to be tested, and the uses the evidence supports today.
Figures & tables
| Requirement | Source | Proposed implementation | Open | |
|---|---|---|---|---|
| L1 | Equal standards: each examiner’s standards for a task apply to every script of that task alike | ( Bundesverfassungsgericht, 1991b , p. 52) ; ( Bundesverwaltungsgericht, 2018 , para. 8) | one versioned rubric per examiner and task; corrections applied to every script; pinned judge version | untested across scripts and versions |
| L2 | Answer latitude: a defensible, consistently argued solution must not be marked wrong; an error is not counted twice | ( Bundesverfassungsgericht, 1991a , p. 79) ; ( Bundesverfassungsgericht, 1991b , p. 55) ; ( Verwaltungsgericht Braunschweig, 2018 ) ; ( Verwaltungsgericht Lüneburg, 2024 ) | credit for arguments; separate paths; later steps judged against the candidate’s own decision; unforeseen paths flagged | no grader probed with alternatives or follow-on errors |
| L3 | Reasons: the essential reasons for the grade, not the reason for every point | ( Bundesfinanzhof, 2014 ) ; ( Bundesverwaltungsgericht, 2019 , para. 23) ; Art. 15(1)(h) GDPR ( Court of Justice of the European Union, 2025 ) | a verdict and a quoted passage per criterion, beyond the legal minimum | a quote shows the passage, not that the criterion is met |
| L4 | Personal evaluation and reconsideration by the examiner | ( Bundesverwaltungsgericht, 2002 ) ; ( Bundesverwaltungsgericht, 2019 ) | the examiner reads every script and decides; the judge proposes and flags | correction of confident wrong proposals untested |
| L5 | No decision based solely on automated processing with legal or similarly significant effect, unless necessary for a contract, authorized by law with safeguards, or based on explicit consent | Art. 22 GDPR ( European Parliament and Council of the European Union, 2016 ) ; ( Court of Justice of the European Union, 2023 ) | substantive review, not adoption of a score as given | what counts as substantive review is unsettled |
| L6 | AI that evaluates learning outcomes is high-risk; deployers owe competent oversight, logs and candidate information, public bodies also a fundamental-rights assessment (duties from 2 Dec 2027) | ( European Parliament and Council of the European Union, 2024 ) , Annex III, point 3(b), Arts. 9 to 15, 26, 27, 86; ( European Parliament and Council of the European Union, 2026 ) | documented validation, logging, examiner oversight with authority to override, probes against manipulation | scope of Arts. 2(6) and 6(3) for prototypes |
| Study and setting | P | Rubric | Compared with | Judge | Human |
|---|---|---|---|---|---|
| Wendlinger et al. ( Wendlinger et al., 2024 ) : German criminal-law take-home, student scripts | P6 | none (few-shot examples) | research assistants’ grades | near zero or negative | n/r |
| LeMAJ ( Enguehard et al., 2025 ) : legal QA, model answers with constructed errors | P6 | none; answer split into legal data points | legal experts | correctness .26 (own data), .70 (LegalBench) | IAA correctness .77; .88 with data points |
| Karp et al. ( Karp et al., 2025 ) : Polish procurement appeals-board exam, 3 model judgments | P5/P7 | committee’s criteria and solution key | official committee | 85 to 90 of 100 for failed judgments (committee 8 to 37) | n/r |
| BenGER ( Nagl et al., 2026b ) : German case solutions, 15 tasks, human-written | P5/P6 | reference solution, 10 general criteria | mean of 3 blind reviewers | .76; pass/fail .60 | ICC(2,1) .63, single rater (45 answers, 15 by models) |
| CourtPressGER ( Nagl et al., 2025 ) : German court press releases, model outputs | P6 | 5 general criteria | expert ranking of 10 systems | .94 (systems) | n/r |
| SLDS ( Rolshoven et al., 2025 ) : Swiss headnotes, model outputs | P5 | reference headnote, generic categories | expert ratings | .26 (ROUGE-L .26) | n/r |
| Dimension | Options | Evidence and law |
|---|---|---|
| Unit of credit | argument or requirement met; issue spotted; bare result | answer latitude (L2); experts agree almost perfectly on binary criteria |
| Scale of a criterion | binary; few levels; free points per step | binary agrees best; agreement is not the construct |
| Structure | all points on steps (absolute or as shares of the total); steps plus a bounded, reasoned component; steps plus a criteria-bound pool; steps plus a free pool | a free pool cannot be checked; a criteria-bound pool needs stated criteria and a fixed share |
| Method and form | within each section; separate block | within sections stays tied to content; a block needs stated criteria and a fixed share |
| Alternative solutions | outcome-neutral criteria and paths that record why a step is absent; flag unforeseen paths; single reference | L2; equal performance, not equal length; the examiner’s rule for the total |
| Follow-on errors | standard per criterion (correct law or own earlier decision); none | not counted twice; the weight is the examiner’s (L2) |
Automated Grading
Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.
See all 88 papers