How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs

Shuyuan Zheng; Yun-seo Lee; Xing Gao; Jialu Wu; Jiabao Hu; Katrina Qiyao Wang; Xinbo Lin; Qianying Liu

cs.CLOct 14, 2026

How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs

Authors: Shuyuan Zheng, Yun-seo Lee, Xing Gao, Jialu Wu, Jiabao Hu, Katrina Qiyao Wang, Xinbo Lin, Qianying Liu, +2 more

Organizations: The University of Osaka · UC Davis · UC Berkeley · Peking University · University of Wisconsin–Madison · Shanghai Jiao Tong University · NII LLMC · City University of Hong Kong

Abstract

Legal reasoning is a fundamental capability for large language models (LLMs) to complete legal AI tasks such as legal judgment prediction. In legal practice, this reasoning is shaped by legal philosophy: a judge's view of the nature and sources of law guides how legal materials are interpreted and applied, so judges working from the same statutes and precedents can reach different judgments. Existing research has largely overlooked whether LLMs can understand legal philosophy and apply it in legal reasoning, an ability we call legal-philosophical reasoning. To fill this gap, we introduce LePhilBench, a benchmark for evaluating the legal-philosophical reasoning of LLMs that comprises two benchmarking tasks and a dataset. The two tasks evaluate how well LLMs understand legal philosophy and how well they apply it in legal reasoning, respectively. The dataset is manually and meticulously constructed. Its test set contains 220 U.S. Supreme Court opinions. A law professor designed fourteen annotation questions on legal philosophy, and three law-trained annotators with formal training in jurisprudence answered all fourteen questions for every opinion, yielding 9,240 legal-philosophical labels. LePhilBench takes the first step toward exploring legal-philosophical reasoning in LLMs.

Figures & tables

Figure 1: Overview of jurisprudential profiling (JP) and jurisprudence-aware judgment prediction (JAJP): inferring a jurisprudential profile from a judge’s past opinion(s) and using it to predict the judge’s judgment in a target case.

Information	Description
Case record
Date (Oyez)	Date the Court decided the case
Case facts (Oyez)	Summary of the facts of the case
Key legal question (Oyez & us)	Binary question at the center of the dispute
Opinion record
Type (Justia)	Court Opinion or Dissent

Table 1: Key fields in LePhilBench. The source of each field is given in parentheses.

Full set
Statistic	Court	Dissent	Overall
Cases	–	–	489
Opinions	489	489	978
Authors	15	15	15
Test set
Cases	–	–	110

Table 2: Statistics of LePhilBench.

Figure 2: LLM-human vs. human-human pairwise agreement (%) across jurisprudential tendencies.

Figure 3: LLM-human pairwise disagreement attribution. Opposite: the model and the annotator choose opposite directions (A vs. B). Uncertain: the annotator chooses A or B, but the model chooses C. Overconfident: the annotator chooses C, but the model chooses A or B.

Figure 4: LLM annotation distribution in JP vs. human consensus level. On 3-0, 2-1, and 1-1-1 questions, the three annotators’ labels are unanimous, split two to one, or all different, respectively.

Table 3: No Profile support inclination, and JAJP accuracy with the jurisprudential profile and gain over No Profile.

Figure 5: JAJP accuracy gain over No Profile (pp) with four profile aggregation methods, using author histories alone or including exclusive-joiner histories.

Figure 6: JAJP accuracy gain over No Profile (pp) across tendencies using Exponential Decay profiles from prior authored opinions. (a) Singleton retains only the indicated tendency. (b) Leave-one-out omits the indicated tendency and retains the other thirteen.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	OpenRouter Endpoint	Model size	Tokens (Sum)
GPT-5.6-Sol ( OpenAI, 2026 )	openai/gpt-5.6-sol	unknown	83M
GPT-5.6-Terra ( OpenAI, 2026 )	openai/gpt-5.6-terra	unknown	85.5M
GPT-5.6-Luna ( OpenAI, 2026 )	openai/gpt-5.6-luna	unknown	89.4M
Claude Opus 5 ( Anthropic, 2026a )	anthropic/claude-opus-5	unknown	138M
Claude Sonnet 5 ( Anthropic, 2026b )	anthropic/claude-sonnet-5	unknown	136M
Qwen3.7-Max ( Qwen Team, 2026 )	qwen/qwen3.7-max	unknown	145M

Appendix

Table 4: OpenRouter model identifiers, model sizes, and token usage. Model size denotes parameter count; unknown indicates that no count was verified in official sources. Tokens follow the dashboard Sum values, rounded in millions (M = 1,000,000 tokens).

Figure 7: LLM agreement with majority-vote aggregated human labels (%) across jurisprudential tendencies.

Figure 8: LLM agreement with unanimous-agreement aggregated human labels (%) across jurisprudential tendencies.

Figure 9: Paired Annotation response proportions (%) by jurisprudential tendency and human consensus level. Each Q1–Q14 plot separates unanimous (3-0), two-to-one (2-1), and all-different (1-1-1) human votes. A/B are directional responses; C is Unknown/Hard to Say. Percentages use all responses within each tendency and vote group, averaged equally across three seeds; Overall averages the ten models equally. N/A marks an empty vote group.

Figure 10: JAJP accuracy gain over matched No Profile (pp) with profiles built from majority-vote aggregated human labels of the target opinion. (a) Singleton retains only the indicated tendency. (b) Leave-one-out omits the indicated tendency and retains the other thirteen.

Explore similar work

Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
May 8, 2026Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6Legal NLP LLM Evaluation
Maritaca AI Campinas, São Paulo, Brazil · Jusbrasil Salvador, Bahia, Brazil
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
May 27, 2026Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach +6Legal NLP LLM Evaluation
Technical University of Munich (TUM) · Ludwig Maximilian University of Munich (LMU) · University of Konstanz +1
LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning
Sep 22, 2026Qingjing Chen, Junkai Zhang, Shaochun Wang +7LLM Reasoning with Graphs Graph Retrieval-Augmented Generation
Alma AI, University of Bologna, Italy · Tsinghua University, China · Modelbest Inc. +2

May 8, 2026J/K move · Enter open · S save

Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks

Ramon Pires, Thales Sales Almeida, Celio Larcher Junior, Giovana Bonás+5

Maritaca AI Campinas, São Paulo, Brazil · Jusbrasil Salvador, Bahia, Brazil

Legal NLP LLM Evaluation Legal Reasoning

Open paper AskPDF arXiv

Existing benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to judge such arguments -- weighing competing claims, applying doctrine to facts, and rendering reasoned decisions -- is arguably as fundamental to a well-functioning legal system as advocacy itself. We introduce Magis-Bench, a benchmark for evaluating LLMs on magistrate-level writing tasks derived from recent Brazilian competitive examinations for judicial positions. Magis-Bench comprises 74 questions from eight examinations conducted between 2023 and 2025, including discursive legal analysis questions with multi-turn structure and practical exercises requiring the composition of complete civil and criminal judicial sentences. We evaluate 23 state-of-the-art LLMs using an LLM-as-a-judge methodology with four independent frontier models as evaluators. Our results show strong inter-judge agreement (Kendall's

W = 0.984

; pairwise Kendall's

τ\ge 0.897

), with Google's Gemini-3-Pro-Preview achieving the highest average score (6.97/10), followed by Gemini-3-Flash-Preview (6.67) and Claude-4.5-Opus (6.46). Even the best-performing models score below 70% of the maximum, indicating that judicial-level legal reasoning and writing remain challenging for current LLMs. We release the complete benchmark, model outputs, and evaluation code to support further research on legal AI capabilities.

Abstract

Explore similar work

Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning