AI Achieves a Perfect LSAT Score

Bonmu Ku

cs.AIApr 11, 2026

AI Achieves a Perfect LSAT Score

Authors: Bonmu Ku

Abstract

This paper reports the first documented instance of a language model achieving a perfect score on an officially disclosed Law School Admission Test (LSAT). Controlled experiments on eight reasoning models show that varying the prompt, shuffling answer choices, and sampling multiple responses have no meaningful effect as drivers of performance. Ablating the thinking phase that models generate before answering, however, lowers frontier accuracy by up to 8 percentage points, predominantly in logical reasoning. Distilled models produce full thinking traces in the same format yet plateau far below frontier performance. A pilot process reward model fine-tuned via QLoRA on official LSAT explanations narrows this gap through Best-of-5 selection, with gains again predominantly in logical reasoning. The gatekeeper of elite legal education since 1948, the LSAT has not merely been passed but answered without a single error by models that reason. The upper bound of the cognitive capacities it has tested is no longer exclusive to human cognition.

Figures & tables

Model	Provider	Arch	Total	Active	Type	Cutoff	Think Visible	Think Toggle
GPT-5	OpenAI	—	—	—	Closed	Sep 2024	Summarized	Yes
Claude Opus 4	Anthropic	—	—	—	Closed	Mar 2025	Summarized	Yes
Gemini 2.5 Pro	Google	—	—	—	Closed	Jan 2025	Summarized	No
DeepSeek-R1	DeepSeek	MoE	671B	37B	Open	Jul 2024	Full trace	No
Kimi K2 Thinking	Moonshot AI	MoE	1T	32B	Open	Dec 2024	Full trace	No
QwQ-32B	Alibaba/Qwen	Dense	32B	32B	Open	Nov 2024	Full trace	No

Table 1: Models evaluated. Frontier models (top) and small models (bottom). Arch distinguishes dense transformers (all parameters active per token) from sparse Mixture-of-Experts (MoE) architectures, in which a router activates only a subset of experts per token. Total reports total parameter count; Active reports parameters activated per token. These are equal for dense models and differ for MoE models: Kimi K2 Thinking, for example, has 1T total parameters but activates only 32B per token. Parameter counts for the three closed models are not publicly disclosed.

Panel of Figure 1 — Figure 1: How the eight reasoning models expose their thinking traces. The models cluster into five architecturally distinct mechanisms that differ in generation, visibility, and multi-turn handling; see text for details.

Panel of Figure 2 — Figure 2: Reasoning form vs. reasoning quality on a Logical Reasoning question from the Official Test (correct answer: A). Stimulus, answer choices, and thinking traces abbreviated for space; see Appendix A for the full question and complete traces. DeepSeek-R1 (top) identifies the necessary assumption. Its distilled 7B counterpart (bottom) selects the wrong answer, missing the critical logical bridge.

	Official Test (N=77)			PT 150–159 (N=1037)
Model	LR	RC	Total	LR	RC	Total
GPT-5	98.0	100.0	98.7	99.1	98.5	98.8
Claude Opus 4	98.0	100.0	98.7	98.0	97.8	97.9
Gemini 2.5 Pro	94.0	100.0	96.1	99.1	97.8	98.6
DeepSeek-R1	100.0	96.3	98.7	97.8	97.5	97.7
Kimi K2 Thinking	100.0	100.0	100.0	97.8	96.3	97.2

Table 2: Baseline accuracy under default conditions (no system prompt, no answer reordering, single sample). LR = Logical Reasoning, RC = Reading Comprehension. Frontier models (top) and small models (bottom).

Model	Cond A	Cond B	Cond C	$Q(2)$	$p$
GPT-5	98.7	98.7	98.7	0.00	1.000
Claude Opus 4	98.7	96.1	96.1	2.67	.264
Gemini 2.5 Pro	96.1	96.1	98.7	2.67	.264
DeepSeek-R1	98.7	100.0	100.0	2.00	.368
Kimi K2 Thinking	100.0	98.7	96.1	3.50	.174
QwQ-32B	92.2	93.5	93.5	0.40	.819

Table 3: Prompt sensitivity on the Official Test (

N=77

). Cochran’s Q across three conditions. Per-section breakdowns in Table 14 .

Model	Cond A	Cond B	Cond C	$Q(2)$	$p$	TOST
GPT-5	98.8	98.6	98.6	0.93	.627	9/9 EQUIV
Claude Opus 4	97.9	98.0	97.8	0.22	.895	9/9 EQUIV
Gemini 2.5 Pro	98.6	97.8	98.2	3.56	.169	8/9 EQUIV
DeepSeek-R1	97.7	97.7	97.9	0.36	.834	8/9 EQUIV
Kimi K2 Thinking	97.2	96.5	97.0	1.28	.528	5/9 EQUIV
QwQ-32B	93.2	93.3	93.2	0.07	.967	4/9 EQUIV

Table 4: Prompt sensitivity on PT 150–159 (

N=1{,}037

). Cochran’s Q test across three conditions (A/B/C). Per-section breakdowns in Table 15 .

Model	Original	Shuffled	$b$	$c$	$p$	$h$
GPT-5	98.7	97.4	1	0	1.000	$-$ 0.095
Claude Opus 4	98.7	98.7	1	1	1.000	0.000
Gemini 2.5 Pro	96.1	97.4	0	1	1.000	+0.074
DeepSeek-R1	98.7	98.7	1	1	1.000	0.000
Kimi K2 Thinking	100.0	98.7	1	0	1.000	$-$ 0.228
QwQ-32B	92.2	94.8	1	3	.625	+0.106

Table 5: Position bias on the Official Test (

N=77

). Original vs. shuffled (McNemar’s exact test).

b

= original-only correct;

c

= shuffled-only correct;

h

= Cohen’s

h

. Per-section breakdowns in Table 16 .

Model	Original	Shuffled	$b$	$c$	$p$	$h$	TOST
GPT-5	98.8	98.7	3	2	1.000	+0.009	3/3 EQUIV
Claude Opus 4	97.9	98.0	5	6	1.000	$-$ 0.007	3/3 EQUIV
Gemini 2.5 Pro	98.6	98.5	4	3	1.000	+0.008	3/3 EQUIV
DeepSeek-R1	97.7	97.4	14	11	.689	+0.019	2/3 EQUIV
Kimi K2 Thinking	97.2	97.0	17	15	.860	+0.012	1/3 EQUIV
QwQ-32B	93.2	93.2	32	32	1.000	0.000	2/3 EQUIV

Table 6: Position bias on PT 150–159 (

N=1{,}037

). Original vs. shuffled (McNemar’s exact test). Per-section breakdowns in Table 17 .

Model	pass@1	SC@5	$\Delta$	$b$	$c$	$p$	Unanimity
GPT-5	97.4	97.4	0.0	0	0	1.000	97.4%
Claude Opus 4	93.5	96.1	+2.6	0	2	.500	94.8%
Gemini 2.5 Pro	97.4	97.4	0.0	0	0	1.000	97.4%
DeepSeek-R1	98.7	98.7	0.0	0	0	1.000	97.4%
Kimi K2 Thinking	97.4	98.7	+1.3	0	1	1.000	94.8%
QwQ-32B	93.5	96.1	+2.6	0	2	.500	92.2%

Table 7: Self-consistency on the Official Test (

N=77

). pass@1 = first of five samples; SC@5 = majority vote.

b

= pass@1-only correct;

c

= SC@5-only correct. Unanimity = proportion of questions with 5/5 identical answers. Per-section breakdowns in Table 18 .

Model	pass@1	SC@5	$\Delta$	$b$	$c$	$p$	$h$	Unanimity
GPT-5	97.3	99.0	+1.7	0	18	$<$ .001	+0.133	92.8%
Claude Opus 4	97.1	98.0	+0.9	4	13	.049	+0.056	91.7%
Gemini 2.5 Pro	97.6	98.6	+1.1	3	14	.013	+0.079	93.7%
DeepSeek-R1	97.1	97.8	+0.7	2	9	.065	+0.043	94.9%
Kimi K2 Thinking	97.2	98.4	+1.2	2	14	.004	+0.079	92.8%
QwQ-32B	94.5	94.9	+0.4	7	11	.481	+0.017	88.9%

Table 8: Self-consistency on PT 150–159 (

N=1{,}037

). McNemar’s exact test. Per-section breakdowns in Table 19 .

Model	Section	ON	OFF	$\Delta$	$b$	$p$	$h$
GPT-5	Overall	98.7	93.5	+5.2	4	.125	+0.287
	LR	98.0	92.0	+6.0	3	.250	+0.290
	RC	100.0	96.3	+3.7	1	1.000	+0.387
Claude Opus 4	Overall	98.7	90.9	+7.8	6	.031	+0.384
	LR	98.0	88.0	+10.0	5	.062	+0.424
	RC	100.0	96.3	+3.7	1	1.000	+0.387

Table 9: Thinking ablation on the Official Test (

N=77

). McNemar’s exact test.

b

= think-ON-only correct;

c

= think-OFF-only correct.

Model	Section	ON	OFF	$\Delta$	$b$	$c$	$p$	$h$
GPT-5	Overall	98.8	95.5	+3.4	38	3	$<$ .001	+0.214
	LR	99.1	95.3	+3.8	26	2	$<$ .001	+0.244
	RC	98.5	95.8	+2.7	12	1	.003	+0.169
Claude Opus 4	Overall	97.9	95.4	+2.5	33	7	$<$ .001	+0.141
	LR	97.9	94.0	+3.9	28	3	$<$ .001	+0.207
	RC	97.8	97.5	+0.2	5	4	1.000	+0.016

Table 10: Thinking ablation on PT 150–159 (

N=1{,}037

). McNemar’s exact test.

	Accuracy (%)
Method	LR	RC	Overall
Baseline (pass@1)	52.0	51.9	51.9
SC@5	50.0	66.7	55.8
PRM + BoN@5	62.0	66.7	63.6
$\Delta$ (PRM vs. pass@1)	+10.0	+14.8	+11.7
$\Delta$ (PRM vs. SC)	+12.0	0.0	+7.8

Table 11: Process supervision vs. baselines on DeepSeek-R1 Distill 7B (Official Test, unseen during PRM training). LR = Logical Reasoning (

N=50

), RC = Reading Comprehension (

N=27

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Quantization	4-bit (NF4)
LoRA rank ( $r$ )	8
LoRA alpha	16
LoRA dropout	0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters	20,185,088 / 7,635,801,600 (0.26%)

Appendix

Table 12: QLoRA adapter configuration.

Parameter	Value
Training examples	209 (PT 140–141)
Epochs	5
Learning rate	$2\times 10^{-4}$
Scheduler	Cosine
Warmup steps	10
Batch size per device	1

Appendix

Table 13: Training hyperparameters.

Model	Section	Cond A	Cond B	Cond C	$Q(2)$	$p$
GPT-5	Overall	98.7	98.7	98.7	0.00	1.000
	LR	98.0	98.0	98.0	0.00	1.000
	RC	100.0	100.0	100.0	0.00	1.000
Claude Opus 4	Overall	98.7	96.1	96.1	2.67	.264
	LR	98.0	94.0	94.0	2.67	.264
	RC	100.0	100.0	100.0	0.00	1.000

Appendix

Table 14: Prompt sensitivity by section on the Official Test (

N=77

; LR

N=50

, RC

N=27

). Cochran’s Q across three conditions.

Model	Section	Cond A	Cond B	Cond C	$Q(2)$	$p$	TOST
GPT-5	Overall	98.8	98.6	98.6	0.93	.627	3/3 EQUIV
	LR	99.1	98.7	98.6	2.00	.368	3/3 EQUIV
	RC	98.5	98.5	98.5	0.00	1.000	3/3 EQUIV
Claude Opus 4	Overall	97.9	98.0	97.8	0.22	.895	3/3 EQUIV
	LR	97.9	97.8	97.6	0.40	.819	3/3 EQUIV
	RC	97.8	98.3	98.0	0.50	.779	2/3

Appendix

Table 15: Prompt sensitivity by section on PT 150–159 (

N=1{,}037

; LR

N=633

, RC

N=404

). Cochran’s Q across three conditions.

Model	Section	Original	Shuffled	$b$	$c$	$p$	$h$
GPT-5	Overall	98.7	97.4	1	0	1.000	$-$ 0.095
	LR	98.0	96.0	1	0	1.000	$-$ 0.119
	RC	100.0	100.0	0	0	1.000	0.000
Claude Opus 4	Overall	98.7	98.7	1	1	1.000	0.000
	LR	98.0	98.0	1	1	1.000	0.000
	RC	100.0	100.0	0	0	1.000	0.000

Appendix

Table 16: Position bias by section on the Official Test (

N=77

; LR

N=50

, RC

N=27

). Original vs. shuffled (McNemar’s exact test).

Model	Section	Original	Shuffled	$b$	$c$	$p$	$h$	TOST
GPT-5	Overall	98.8	98.7	3	2	1.000	$-$ 0.009	EQUIV
	LR	99.1	98.7	2	0	.500	$-$ 0.030	EQUIV
	RC	98.5	98.8	1	2	1.000	+0.021	EQUIV
Claude Opus 4	Overall	97.9	98.0	5	6	1.000	+0.007	EQUIV
	LR	97.9	97.8	3	2	1.000	$-$ 0.011	EQUIV
	RC	97.8	98.3	2	4	.688	+0.036	NO

Appendix

Table 17: Position bias by section on PT 150–159 (

N=1{,}037

; LR

N=633

, RC

N=404

). Original vs. shuffled (McNemar’s exact test).

Model	Section	pass@1	SC@5	$\Delta$	$c$	$p$	Unanimity
GPT-5	Overall	97.4	97.4	0.0	0	1.000	97.4%
	LR	96.0	96.0	0.0	0	1.000	98.0%
	RC	100.0	100.0	0.0	0	1.000	96.3%
Claude Opus 4	Overall	93.5	96.1	+2.6	2	.500	94.8%
	LR	90.0	94.0	+4.0	2	.500	92.0%
	RC	100.0	100.0	0.0	0	1.000	100.0%

Appendix

Table 18: Self-consistency by section on the Official Test (

N=77

; LR

N=50

, RC

N=27

). McNemar’s exact test.

Model	Section	pass@1	SC@5	$\Delta$	$b$	$c$	$p$	$h$	Unanimity
GPT-5	Overall	97.3	99.0	+1.7	0	18	$<$ .001	+0.133	92.8%
	LR	97.5	99.2	+1.7	0	11	.001	+0.141	93.4%
	RC	97.0	98.8	+1.7	0	7	.016	+0.123	91.8%
Claude Opus 4	Overall	97.1	98.0	+0.9	4	13	.049	+0.056	91.7%
	LR	97.3	98.3	+0.9	2	8	.109	+0.065	91.9%
	RC	96.8	97.5	+0.7	2	5	.453	+0.045	91.3%

Appendix

Table 19: Self-consistency by section on PT 150–159 (

N=1{,}037

; LR

N=633

, RC

N=404

). McNemar’s exact test.

Explore similar work

Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models
Oct 5, 2026Subramanyam Sahoo, Justin ShenkReward Hacking LLM Auditing
Horizon Research.
Can Small Models Reason About Legal Documents? A Comparative Study
Mar 26, 2026Snehit VaddiSmall Language Models Legal Reasoning
Independent Researcher
TW-LegalBench: Measuring Taiwanese Legal Understanding
Jun 17, 2026Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4LLM Evaluation Legal Judgment Prediction
University of Rochester Rochester, NY, USA · National Taiwan University Taipei, Taiwan · NVIDIA Santa Clara, CA, USA

Oct 5, 2026J/K move · Enter open · S save

Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models

Subramanyam Sahoo, Justin Shenk

Horizon Research.

Reward Hacking LLM Auditing RL for Language Model Reasoning Legal Reasoning

Open paper AskPDF arXiv

What happens when a legal AI model learns to look like a lawyer instead of reasoning like one? We fine tune Qwen3-8B with Group Relative Policy Optimisation (GRPO) against a proxy built from three surface features: citation count, legalese density, and response length. The model does not learn to reason more effectively. It learns to withhold commitment. Across 16 yes or no legal reasoning tasks from LegalBench (N=320), overall accuracy collapses from 0.500 (chance) to 0.072 (McNemar p < 10^-36), driven entirely by the rate of properly formatted answers falling from 0.900 to 0.109. The model stops committing to answers. Yet when it does commit, accuracy rises from 0.556 to 0.657, showing that the collapse is not a failure of capability but a strategic response: the model has learned that verbose responses packed with citations but empty of a direct answer score higher than terse correct ones. We term this the Saul Goodman effect, a policy that becomes maximally lawyerly while becoming maximally noncommittal, and prove formally that it is the optimal response to any surface feature proxy that attaches no penalty to abstention. We further show that 89.3% of citations produced after training are structurally implausible hallucinations, many of them subtly corrupted names of real landmark cases, constructed in effect to survive a casual read and fail under scrutiny. To detect this failure mode before deployment, we introduce three diagnostic tools: the Confidence Theater Score (CTS), the Citation Plausibility Rate (CPR), and the Regret Gap (RG). In a domain where a confidently wrong answer can constitute malpractice, the broader lesson is direct: a reward function that measures how legal a response looks will produce a model that is maximally photogenic and minimally useful.

Abstract

Explore similar work

Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models

Can Small Models Reason About Legal Documents? A Comparative Study

TW-LegalBench: Measuring Taiwanese Legal Understanding