LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs

Serene Wang; Lavanya Pobbathi; Haihua Chen

cs.CLMar 9, 2026

LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs

Authors: Serene Wang, Lavanya Pobbathi, Haihua Chen

Organizations: University of North Texas Denton, Texas, USA

Abstract

Legal argument mining aims to identify and classify the functional components of judicial reasoning, such as facts, issues, rules, analysis, and conclusions. Progress in this area is limited by the lack of large-scale, high-quality annotated datasets for U.S. caselaw, particularly at the state level. This paper introduces LAMUS, a sentence-level legal argument mining corpus constructed from U.S. Supreme Court decisions and Texas criminal appellate opinions. The dataset is created using a data-centric pipeline that combines large-scale case collection, LLM-based automatic annotation, and targeted human-in-the-loop quality refinement. We formulate legal argument mining as a six-class sentence classification task and evaluate multiple general-purpose and legal-domain language models under zero-shot, few-shot, and chain-of-thought prompting strategies, with LegalBERT as a supervised baseline. Results show that chain-of-thought prompting substantially improves LLM performance, while domain-specific models exhibit more stable zero-shot behavior. LLM-assisted verification corrects nearly 20% of annotation errors, improving label consistency. Human verification achieves Cohen's Kappa of 0.85, confirming annotation quality. LAMUS provides a scalable resource and empirical insights for future legal NLP research. All code and datasets can be accessed for reproducibility on GitHub at: https://github.com/LavanyaPobbathi/LAMUS/tree/main

Figures & tables

Label	Description	Example	Annotation Guideline
Fact	Any fact that is pertinent to the case. This includes testimony, statements of record, case history, and anything else that is a fact that helps establish the foundation of the case for the court to build its analysis and judgment on.	Now you say that the premises were controlled by Report Walton on that date and you know that?	This is a very broad category. Anything that “sets the stage,” as it were, should be labeled as a fact. Factual sentences do not include any synthesis or reasoning (that would fall under analysis); rather, they simply state events or matters of record.
Issue	Any issue or question that the court must decide. This includes the overall issue of the case as well as any sub-issues that are raised in the case.	Appellant’s sole contention on appeal is that the evidence is insufficient to sustain the conviction.	This tends to be a pretty narrow category. Any sentence related to a question for the court to resolve should be labeled as an issue. Note that a lot of the issue sentences could be read as a fact, as any issue is inherently part of the factual history of the court case.
Rule/law/holding	Any statement of or reference to a rule, law, or holding. This includes sentences referencing a rule, law, or holding and then using it to reason through some point.	Reliance is handed upon Toombs v. 21, 317 S.W. 2d 737, as authority for reversal of cases where the state’s testimony was adduced by only one witness.	Generally speaking, any time a reference is made to a rule, law, or holding, that sentence should be given this label. These sentences could be read as a fact, since whenever a law is quoted, for example, it is a fact that the law states that quotation.
Analysis	Any sentence that synthesizes information to further the court’s reasoning. This includes sentences that refer to the facts of the case and then use them to push forward towards a resolution.	Had the new Rules of Appellate Procedure been in effect during the pendency of this appeal, sanctions against the responsible attorneys would have been appropriate.	Analysis sentences tend to move the court through the case from the facts towards the conclusion, so there is often a logical progression stringing analysis sentences together. A sentence that references a rule, law, or holding and then analyzes it in the context of the current case should be labeled Rule/Law/Holding, not as an analysis sentence.
Conclusion/ opinion/answer	Any sentence that effectively resolves an issue facing the court.	The trial court erred in submitting to the jury the issue of rape by threats.	Conclusions tend to be short and straight to the point, either agreeing or disagreeing with some argument. As there are sub-issues, there are also sub-conclusions that conclude a specific sub-issue being discussed by the court.
Others	(1) Any sentence or phrase that does not fit the other labels in terms of its content. This includes section headings and others. (2) Sentences that were not split correctly.	Ex.1: We have carefully reviewed both of these cases. Ex.2: 21, 317 S.W. 2d 737, as authority for reversal of cases where the state’s testimony was adduced by only one witness.	Sentences that do not add any information to the case and thus do not fall into any of the other labels. Section headings are in this category as they are neither complete sentences nor useful when building the logic of the court case.

Table 1. Annotation scheme for legal argument ( Chen et al., 2022 ) .

Figure 1. Methodology for corpus construction, LLM-based sentence annotation, data quality assessment, and model evaluation in the LAMUS dataset.

Model	Type	Params	Domain
LLaMA-3-8B	Decoder-only	8B	General
Qwen3-Thinking	Decoder-only	7B	General
Gemini-2.5-Flash	API-based	–	General
SaulLM-54B	Legal fine-tuned	54B	Legal
SaulLM-7B	Legal fine-tuned	7B	Legal
law-LLM	Legal fine-tuned	7B	Legal

Table 2. Large Language Models used in Experiment

Figure 2. Comparison of Texas Case Law dataset sentence labels before and after the data cleaning process, indicating improved quality.

Model	Domain	Zero-Shot	Few-Shot	CoT
LLaMA-3-8B	General	65.38%	45.75%	75.89%
SaulLM-54B	Legal	67.39%	64.76%	72.80%
law-LLM	Legal	60.12%	31.68%	28.75%
Qwen3-Thinking	General	56.11%	49.30%	54.10%
SaulLM-7B	Legal	52.09%	21.64%	38.02%
Gemini-2.5-Flash	General	5.41%	5.41%	5.41%

Table 3. Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) performance of evaluated models. Bold indicates the best performance. Stability verified through 10 independent runs (mean 74.71% ± 0.56%, p < 0.001).

Model	Zero-Shot	1-ex	3-ex	4-ex	5-ex
LLaMA-3-8B	65.38%	47.76%	49.61%	52.86%	50.54%
SaulLM-54B	67.39%	54.40%	66.31%	59.51%	67.70%

Table 4. Impact of few-shot prompting on model performance across completed experiments.

# Examples	Accuracy	$\Delta$ vs Zero-Shot
0 (Zero-Shot)	67.23%	baseline
5	65.07%	-2.16%
10	66.15%	-1.08%
20	64.91%	-2.32%
40	65.53%	-1.70%
60	60.43%	-6.80%

Table 5. Extended few-shot analysis (5–100 examples) for LLaMA-3-8B. Performance degrades monotonically as example count increases.

Figure 3. Accuracy of LLaMA-3-8B and SaulLM-54B under varying few-shot example counts. Few-shot prompting harms performance for LLaMA-3-8B, while SaulLM-54B remains relatively stable.

Model	Method	Accuracy	Baseline
LLaMA-3-8B	Fine-tuned (Ablation Best)	85.32%	+23.34%
LegalBERT	Fine-tuned	81.30%	+19.32%
LLaMA-3-8B	Fine-tuned (Original)	80.37%	+18.39%

Table 6. Accuracy of fine-tuned models and improvement over baseline.

Category	Precision	Recall	F1	Support
Facts	0.91	0.94	0.92	401
Issue	0.86	0.70	0.78	27
Rule/Law/Holding	0.79	0.32	0.46	34
Analysis	0.56	0.67	0.61	92
Conclusion	0.67	0.76	0.71	58
Others	0.70	0.40	0.51	35

Table 7. Per-class performance metrics for LegalBERT on sentence-level legal argument classification. (LLaMA)

Config	Learning Rate	Epochs	LoRA Rank	Accuracy
best	1e-4	5	8	85.32%
lr_1e-4	1e-4	5	16	85.32%
lr_2e-4	2e-4	3	16	85.16%
lora_32	5e-5	5	32	85.01%
epoch_5	1e-4	5	32	85.01%
lora_8	2e-4	5	8	84.85%

Table 8. Complete ablation study results for fine-tuned LLaMA-3-8B. The best configuration is highlighted.

Figure 4. Ablation Study: Learning Rate vs. Accuracy by LoRA Rank and Training Epochs. Results from 36 experiments across 4 learning rates (1e-5, 5e-5, 1e-4, 2e-4), 3 LoRA ranks (8, 16, 32), and 3 epoch settings (1, 3, 5). The best configuration (LR=1e-4, Rank=8, Epochs=5) achieves 85.32% accuracy.

Parameter	Sensitivity	Optimal Value	Impact on Accuracy
Learning Rate	HIGH	1e-4	$\pm$ 10%
Epochs	MODERATE	5	$\pm$ 2.5%
LoRA Rank	LOW	8	$\pm$ 0.8%

Table 9. Hyperparameter sensitivity analysis for fine-tuned LLaMA-3-8B based on 36 ablation experiments.

Metric	Value
Total Sentences Verified	600
Sentences per Category	100
Number of Annotators	2
Cohen’s Kappa ( $\kappa$ )	0.85
Interpretation	Almost Perfect
Direct Agreement	87.3%

Table 10. Human verification results for SCOTUS labeled data (N = 600 sentences).

Figure 5. Overall distribution of legal argument label categories across all Supreme Court eras (1921–present), comprising 2,900,083 sentences. The corpus is dominated by Analysis , Rule/Law/Holding , and Facts , while Issue and Conclusion occur less frequently but consistently.

Figure 6. Distribution of legal argument label categories across Supreme Court eras. Each cell shows the percentage of sentences assigned to a label ( Facts , Issue , Rule/Law/Holding , Analysis , Conclusion , Others ) within a given court era. Although the same core labels dominate across all periods, their relative proportions vary over time.

Court Era	Years	Sentences
Burger Court	1969–1986	809,409
Rehnquist Court	1986–2005	673,564
Warren Court	1953–1969	377,645
Roberts Court	2005–2025	362,891
Hughes Court	1930–1941	213,122
Vinson Court	1946–1953	170,975

Table 11. Distribution of labeled sentences across Supreme Court eras in the LAMUS corpus.

Figure 7. Comparison of legal argument label distributions in the Texas Criminal Cases training corpus and the SCOTUS dataset, highlighting substantial domain shift in argumentative structure across courts.

Figure 8. Overall comparison of experimental results across all models and settings, including baseline performance, accuracy percentages, and indicators for fine-tuned configurations.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Epochs = 1			Epochs = 3			Epochs = 5
LR	R=8	R=16	R=32	R=8	R=16	R=32	R=8	R=16	R=32
1e-5	55.95	66.62	69.24	70.02	75.12	79.60	76.82	79.75	81.76
5e-5	74.96	78.83	81.30	82.23	83.15	83.93	84.23	84.70	85.01
1e-4	79.44	80.68	81.76	83.77	84.54	84.23	85.32	85.32	85.01
2e-4	81.30	82.69	83.62	84.70	85.16	84.39	84.85	83.93	83.31

Appendix

Table 12. Complete ablation grid results for fine-tuned LLaMA-3-8B. All values are test accuracy (%). Best result: 85.32% (LR=1e-4, Rank=8, Epochs=5).

Run	Seed	Accuracy (%)
1	142	75.12
2	242	75.43
3	342	74.50
4	442	74.34
5	542	74.96
6	642	74.34

Appendix

Table 13. Stability test results for Chain-of-Thought prompting with LLaMA-3-8B across 10 independent runs. Mean accuracy: 74.71%, standard deviation: 0.56%.

Model	Method	Acc.	$\Delta$
LLaMA-3-8B	Fine-tuned (Best)	85.32%	+23.34
LLaMA-3-8B	Fine-tuned (2e-4)	85.16%	+23.18
LegalBERT	Fine-tuned	81.30%	+19.32
LLaMA-3-8B	Fine-tuned (Orig)	80.37%	+18.39
LLaMA-3-8B	Chain-of-Thought	75.89%	+13.91
SaulLM-54B	Chain-of-Thought	72.80%	+10.82

Appendix

Table 14. Complete results across all models and methods, ranked by accuracy. Baseline (majority class): 61.98%. The

\Delta

column shows improvement over baseline.

# Ex.	Acc. (6-class)	Acc. (5-class)	$\Delta$ ZS
0	67.23%	71.08%	baseline
5	65.07%	68.79%	-2.16%
10	66.15%	69.93%	-1.08%
20	64.91%	68.63%	-2.32%
40	65.53%	69.28%	-1.70%
60	60.43%	63.89%	-6.80%

Appendix

Table 15. Extended few-shot sweep (0–100 examples) for LLaMA-3-8B. Accuracy is reported for both 6-class and 5-class (excluding “Others”) settings. Performance degrades monotonically as example count increases.

Category	LLaMA-FT	LegalBERT	CoT
Facts	0.91	0.92	0.82
Issue	0.73	0.78	0.65
Rule/Law/Holding	0.71	0.46	0.58
Analysis	0.54	0.61	0.48
Conclusion	0.69	0.71	0.62
Others	0.53	0.51	0.41

Appendix

Table 16. Per-class F1-scores for top-performing methods. Fine-tuned models achieve more balanced performance across categories, while CoT prompting shows greater variance.

Category	Texas (%)	SCOTUS (%)	Shift
Facts	61.9	26.3	-35.6
Issue	4.2	2.1	-2.1
Rule/Law/Holding	5.3	27.6	+22.2
Analysis	14.3	27.6	+13.3
Conclusion	8.9	4.2	-4.7
Others	5.4	12.2	+6.9

Appendix

Table 17. Domain shift between Texas Criminal Cases (training) and SCOTUS (target) corpora. Positive shifts indicate categories more prevalent in SCOTUS; negative shifts indicate categories more prevalent in Texas data.

Kappa Range	Interpretation
0.81 – 1.00	Almost Perfect $\leftarrow$ Ours: 0.85
0.61 – 0.80	Substantial
0.41 – 0.60	Moderate
0.21 – 0.40	Fair
0.00 – 0.20	Slight

Appendix

Table 18. Cohen’s Kappa interpretation scale following Landis & Koch (1977). Our human verification achieved

\kappa=0.85

, indicating almost perfect agreement.

Legal NLP

Legal natural language processing (NLP) develops methods for analyzing and generating statutes, cases, contracts, and legal advice.

See all 102 papers

May 27, 2026

UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning

Legal NLP benchmarks are overwhelmingly English-centric, leaving failure modes in morphologically rich, non-Latin-script languages undetected. We introduce UA-Legal-Bench, a five-task benchmark for evaluating large language models on Ukrainian legal reasoning, built from the Unified State Register of Court Decisions (EDRSR) -- one of the world's largest open judicial corpora (99.5 million decisions). The benchmark comprises: (1) case-type classification (4 classes, n=2,000), (2) judgment form classification (4 classes, n=2,000), (3) case-outcome prediction (6 classes, n=800), (4) legal norm extraction (n=1,794), and (5) cause category prediction (22 classes, n=1,871). We evaluate 11 LLMs (3B--675B) from five families under zero-shot and 3-shot prompting via AWS Bedrock with 158K API calls. Our results reveal sharply task-dependent few-shot effects: few-shot prompting improves judgment form classification by up to +38.6 pp but has mixed effects on outcome prediction. We show that accuracy is misleading on imbalanced legal tasks: the model with highest COP accuracy (62%) is a majority-class predictor (macro-F1: 23%), while the genuinely best model scores only 44% macro-F1. Within-family scaling analysis reveals that 8B models can match frontier performance on surface-level tasks but scaling thresholds vary dramatically across families. We release all data, prompts, and model predictions.

Volodymyr Ovcharov

SecondLayer

Multilingual Language Model Evaluation Legal NLP LLM Evaluation LLM Prompting Legal Reasoning

Ask PDF

Apr 20, 2026

LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC), collected via the DataJud API (CNJ) and annotated across five legal areas through LLM-assisted labeling with heuristic validation. On a class-balanced test set, BERTimbau-LoRA, updating only 0.3% of model parameters, achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini). The gap is most striking on administrativo (administrative law): GPT-4o mini scores F1 = 0.00 and Claude 3.5 Haiku scores F1 = 0.08 on this class, while the fine-tuned model reaches F1 = 0.91. Both commercial LLMs exhibit a systematic bias toward civel (civil law), absorbing ambiguous classes rather than discriminating them, a failure mode that domain-adapted fine-tuning eliminates. These results demonstrate that general-purpose LLMs cannot substitute for domain-adapted models in Brazilian legal classification, even when the task is a simple 5-class problem, and that LoRA fine-tuning on a consumer GPU closes the gap at zero marginal inference cost. We release the full dataset, model, and pipeline to enable reproducible research in Portuguese legal NLP.

Pedro Barbosa de Carvalho Neto

Teresina, PI, Brazil

Legal NLP LLM Evaluation Text Classification Language Model Bias Evaluation Classification

Ask PDF

Oct 7, 2026

When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection

Large language models are increasingly used in legal research and drafting, but they can still produce claims that sound convincing without being supported by the cited source. We introduce PARCEL, a benchmark for checking whether a legal claim is supported by the underlying authority. Using recent New York State Court of Appeals decisions, we build a dataset of 3,396 parenthetical-style claims labeled as Supported, Refuted, or Not Found. We cast this task as a three-way natural language inference problem and evaluate several state-of-the-art LLMs in a zero-shot setting. Although the strongest models reach up to 0.97 accuracy, the results also show an important weakness: models still incorrectly mark unsupported claims as supported, even when the full opinion text is provided. Across models, missing support is harder to detect than direct contradiction, and fabricated but plausible citations cause the largest drop in performance. Overall, PARCEL provides a practical benchmark for testing claim-level groundedness in legal RAG systems.

M. Mikail Demir, M. Abdullah Canbaz

University at Albany, SUNY Albany, New York, USA

Legal Citation Verification Hallucination Detection LLM Grounding

Ask PDF

Abstract

Legal NLP

Explore similar work

UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning

LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection