Where Do Legal Factors Live in LLMs?

Li Zhang; Kevin Ashley

cs.CLOct 20, 2026

Where Do Legal Factors Live in LLMs?

Authors: Li Zhang, Kevin Ashley

Organizations: University of Pittsburgh, USA

Abstract

In case-based legal reasoning, legal professionals identify legally significant fact patterns in a new dispute and compare them with those of decided cases. In AI & Law, HYPO dimensions and CATO factors represent these patterns symbolically, organizing expert knowledge distilled from case narratives as explicit, structured, interpretable, and auditable legal concepts and relations that support case comparison, legal argument, and precedent-based reasoning. Large language models (LLMs) are increasingly used for legal tasks, yet their internal organization of legal knowledge remains difficult to trace. We investigate whether LLMs form internal representations corresponding to all 26 CATO factors. Two research questions examine linear decodability and sparse localization with sparse autoencoders. Experiments cover three open models, Qwen3-1.7B, Qwen3-8B, and Gemma-2 2B, supported by cross-model comparison and human audit of controlled natural-language passages. Results show that all 26 factors are linearly decodable in all three models. Single-feature sparse readouts cover the complete factor catalog, with best-layer test AUCs ranging from 0.86 to 1.00. Five selected features approach full-vector readout performance. We report 78 model-specific layer-feature coordinates, one for each factor in each model. Together, these results show that information corresponding to expert-defined legal factors can be read from the internal activations of LLMs processing legal narratives. By locating these readouts, our study connects legal language and the symbolic abstractions of HYPO and CATO to learned internal representations.

Figures & tables

Figure 1: Overview of legal-factor analysis. (a) HYPO and CATO employ legally significant fact patterns abstracted from natural-language narratives. Subsequent work prompts LLMs to identify these factors in text. (b) This work examines internal representations in pretrained LLMs: RQ1 reads factor presence from full activations, and RQ2 identifies one or five SAE features supporting the readout.

ID	Factor (side)	Meaning (short)
F1	Disclosure-in-negotiations (D)	Plaintiff disclosed its information to defendant in negotiations.
F2	Bribe-employee (P)	Defendant paid plaintiff’s employee to defect with its information.
F3	Employee-sole-developer (D)	Defendant’s employee solely developed plaintiff’s product.
F4	Agreed-not-to-disclose (P)	Defendant signed a nondisclosure agreement with plaintiff.
F5	Agreement-not-specific (D)	The agreement did not specify what was confidential.
F6	Security-measures (P)	Plaintiff adopted concrete security measures.

Table 1: The CATO factor catalog (F1–F27; F9 is vacant). (P)/(D) marks the party the factor favors.

Example	Corpus excerpt
Example A Supplier network	BlueFern Genetics developed a supplier network for its biotech reagent line. The company kept the specifications on a shared drive with no access controls.
Example B Information type	Tessara Analytics developed a formula for industrial chemicals used in polymer curing. The company kept the specifications on a shared drive with no access controls.
Example C Industry	Almara Labs maintained a supplier network for its aerospace components on a shared drive with no access controls. The company did not restrict who inside the organization could open the files.
Example D Narrative style	Complaint excerpt: Vantor Gaming is said to mirror Veltrix Systems’ textiles pricing steps. The plaintiff took no steps to limit which employees could view the process.

Table 2: Corpus examples illustrating variation in industry, information type, and narrative style. All four passages express F19; labels highlight changes relative to Example A.

Figure 2: Mean AUC across 26 factors at each model’s middle sampled layer (14/18/13). Random-5 averages 20 draws; features are selected on readout-fitting data. The dashed line marks chance-level ranking.

Figure 3: Selected layer–feature locations for all 26 factors. In each model panel, the filled square identifies the layer with the highest test single-feature AUC; the adjacent number is the feature ID within that layer’s SAE. Dots mark AUC on horizontal tracks running from 0.85 (left) to 1.00 (right), with exact values printed alongside.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 781 papers

Can Small Models Reason About Legal Documents? A Comparative Study
Mar 26, 2026Snehit VaddiSmall Language Models Legal Reasoning
Independent Researcher
Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning
Jun 15, 2026Olivia Peiyu Wang, Sanna Wong-Toropainen, Daneshvar Amrollahi +4Faithfulness of Language Model Reasoning LLM Reliability
UC Santa Cruz · Univ. Helsinki · CodeX, Stanford +4
LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning
Sep 22, 2026Qingjing Chen, Junkai Zhang, Shaochun Wang +7LLM Reasoning with Graphs Graph Retrieval-Augmented Generation
Alma AI, University of Bologna, Italy · Tsinghua University, China · Modelbest Inc. +2

Mar 26, 2026J/K move · Enter open · S save

Can Small Models Reason About Legal Documents? A Comparative Study

Snehit Vaddi

Independent Researcher

Small Language Models Legal Reasoning LLM Evaluation Legal Document Analysis LLM Prompting Retrieval-Augmented Generation

Open paper AskPDF arXiv

Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy. We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies (direct, chain-of-thought, few-shot, BM25 RAG, and dense RAG). Across 405 experiments with three random seeds per configuration, we find that a Mixture-of-Experts model activating only 3B parameters matches GPT-4o-mini in mean accuracy while surpassing it on legal holding identification, and that architecture and training quality matter more than raw parameter count. Our largest model (9B parameters) performs worst overall. Chain-of-thought prompting proves sharply task-dependent, improving contract entailment but degrading multiple-choice legal reasoning, while few-shot prompting emerges as the most consistently effective strategy. Comparing BM25 and dense retrieval for RAG, we find near-identical results, suggesting the bottleneck lies in the language model's utilization of retrieved context rather than retrieval quality. All experiments were conducted via cloud inference APIs at a total cost of $62, demonstrating that rigorous LLM evaluation is accessible without dedicated GPU infrastructure.

Abstract

LLM Interpretability

Explore similar work

Can Small Models Reason About Legal Documents? A Comparative Study

Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning

LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning