Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks

Volodymyr Ovcharov

cs.CLAug 15, 2026

Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks

Authors: Volodymyr Ovcharov

Organizations: LEX AI

Abstract

Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question. Measuring that gap takes care: a model answering A to most items scores above chance wherever the key sits at A, and reads as recognition when it is not. We measure it on UA-JudgeExam: 11,990 four-option items with official keys, published by Ukraine's Higher Qualification Commission of Judges. Shown the options and no question, Claude Haiku 4.5 scores 0.383 against chance, and the leak is concentrated: 11.8% of items are answered blind on all eight option orders, against 0.2 items expected by chance. It is not quotation: search over 280,059 editions of Ukrainian legislation recovers 0.128. Gating those out retains 8,128 items, on which the gating model itself now scores 0.204, and GPT-5.6, which took no part in the selection, still answers 0.515 of them with the question hidden. Scoring twelve held-out models on the whole set and subtracting each one's answer-position habit, only two keep an excess: GPT-5.6 at +0.265, Sonnet 4.6 at +0.081. Without it the ranking misleads: Llama 3.1 8B scores 0.292 blind, above every model but those two, purely by answering A to 92% of items. The gate does select something real: on the items it rejected, eleven of twelve models score 0.518-0.789, every interval clear of what the same model scores on the items it kept. But that signal is one model's, and filtering on it does not transfer upward. Neither is visible on a 400-item sample, where nine models read as "statistically at chance". Rewriting distractors instead overshoots to 0.168, below chance and as exploitable. The same probe on LEXam returns chance: every option there points into the stem, none longer than 33 characters. Item format decides whether the problem can arise; capability decides how much is extracted. We release the corpus, the predictions and the harness.

Figures & tables

Q. Як д ма право самостйно вчиняти фзична особа, яка НЕ досягла 14 рокв?
Which acts may a natural person under 14 perform independently?
A) Здйснювати майнов права на результати нтелектуально дяльност, що охороняються законом	Exercise property rights in protected intellectual-property results
B) Розпоряджатися банквським вкладом, унесеним нею на сво м’я	Dispose of a bank deposit made in their own name
C) Вчиняти дрбн побутов правочини	Perform minor everyday transactions
D) Вчиняти рзн побутов правочини	Perform various everyday transactions

Table 1: Item vkks-2024-commercial-2237 , with an English gloss; the key, C, is in bold. The options are self-contained legal propositions, which is the property that makes the blind condition meaningful. Article 31 of the Civil Code grants a person under 14 the right самостйно вчиняти дрбн побутов правочини — so D substitutes рзн (“various”) for дрбн (“minor”), which is not legal language, and A substitutes майнов (“property”) for the особист немайнов (“personal non-property”) rights the article actually grants. Telling these apart requires knowing the code, not reading the question.

Strategy	Accuracy
Chance	0.250
Best constant position (always A)	0.289
Longest option	0.278
Most words	0.301
Shortest option	0.253
Lexical search over statutes, full bank	0.128

Table 2: What can be recovered without reasoning over the question. Chance is

0.250

Panel of Figure 1 — Figure 1: How often each item is answered blind, over eight permutations of option order, against what chance predicts. The bank does not behave like a set of items with a uniform small leak: 3,517 items are never answered blind and 1,419 are answered blind every time, where $\mathrm{Binomial}(8,0.25)$ predicts $0.2$ items in the latter group. That concentration is what makes selection worth attempting.

Model	Vendor	Blind, accepted	Blind, rejected	Full acc.	Full rej.
GPT-5.6	OpenAI	0.515 [.504,.526]	0.789 [.727,.840]	0.958	0.960
Sonnet 4.6	Anthropic	0.320 [.310,.330]	0.730 [.665,.787]	0.740	0.905
DeepSeek R1	DeepSeek	0.257 [.247,.266]	0.665 [.593,.729]	0.668	0.845
Nova Pro	Amazon	0.232 [.223,.241]	0.645 [.577,.708]	0.580	0.739
Qwen3 32B	Alibaba	0.244 [.234,.253]	0.665 [.597,.727]	0.552	0.720
Pixtral Large	Mistral	0.239 [.230,.248]	0.602 [.531,.669]	0.548	0.718

Table 3: Blind (options only) and full accuracy, ordered by full-condition accuracy. Chance is

0.250

. Blind figures on the accepted set are measured on all

8{,}128

items ; the rejected-set and full-condition columns remain on the 600-item sample, so their intervals are the wider ones. Two models are set apart because their numbers are not evidence of recognition: Haiku is the gate’s own model, and Llama 3.1 8B answers A to

92\%

of items, which on its own earns

0.287

here (Table 4 ). Blind

n

8{,}128

except DeepSeek R1

7{,}976

, Llama 3.1 8B

7{,}886

, Nova Micro

8{,}039

, Ministral

8{,}126

and Pixtral

8{,}125

; full

n

400

except DeepSeek R1

352

, Pixtral

365

, Nova Micro

388

and Llama 3.1 8B

202

Model	A	B	C	D	Blind	Habit	Excess
GPT-5.6	0.24	0.27	0.27	0.22	0.515	0.250	$+0.265$
Sonnet 4.6	0.08	0.20	0.30	0.42	0.320	0.238	$+0.081$
DeepSeek R1	0.12	0.24	0.29	0.35	0.257	0.242	$+0.015$
Qwen3 32B	0.14	0.27	0.30	0.30	0.244	0.244	$+0.000$
Ministral 8B	0.02	0.25	0.45	0.29	0.239	0.238	$+0.001$
Llama 3.3 70B	0.08	0.28	0.31	0.33	0.237	0.240	$-0.003$

Table 4: What is left after each model’s own positional habit is subtracted. Blind accuracy is measured on all

8{,}128

accepted items. “Habit” is the score that model’s answer-position distribution alone earns against the gold positions of this set. Only two models have an excess worth the name.

Model	Agrees with Haiku	Accepted	Rejected	Gap
Pixtral Large	0.529	0.239	0.602	0.363
Nova Pro	0.504	0.232	0.645	0.413
Llama 3.3 70B	0.497	0.237	0.610	0.373
Qwen3 32B	0.489	0.244	0.665	0.421
Nova 2 Lite	0.486	0.225	0.545	0.320
DeepSeek R1	0.473	0.257	0.665	0.408

Table 5: Agreement with the gating model’s blind pick, and the cost of that agreement. Independence would put agreement at

0.250

. The gap is blind accuracy on rejected items minus blind accuracy on accepted items; the more a model answers like the gate, the more the gate’s selection costs it.

Panel of Figure 2 — Figure 2: Blind accuracy against full-condition accuracy on the gate-accepted items, eleven held-out models from eight vendors. Bars are 95% Wilson intervals on the blind estimate, now measured on the whole accepted set rather than a 400-item sample, which is why they are short; the horizontal position still carries the sample’s error. The plotted correlation, $r=0.916$ , shares items between the two axes; against an independent capability estimate on disjoint items it is $r=0.813$ (§ 6 ). The gate was built with a model absent from this plot, and Llama 3.1 8B is absent because it answers A to $92\%$ of items. The bracketed nine sit within $0.015$ of their own positional habit (Table 4 ).

Prompt	Sonnet 4.6	GPT-5.6	Nova Pro
Original	0.343	0.524	0.255
Minimal, no framing	0.323	0.463	0.265
Domain word removed	0.310	0.490	0.236
English instruction	0.325	0.521	0.240
Framed as guessing	0.330	0.479	0.220
Spread	0.033	0.061	0.045

Table 6: Blind accuracy under five prompt phrasings, 400 gate-accepted items. Chance is

0.250

. These figures are on a 400-item subset of the accepted set, not the whole of it, so they sit above the full-scale numbers in Table 3 — for these three models the subset runs

0.009

0.024

leakier than the set it was drawn from, and every model in it inherits that. What the table is for is the spread within a model, which the shared sample does not distort.


	Sonnet 4.6	GPT-5.6	Nova Pro
Original	0.738	0.957	0.578
Minimal, no framing	0.740	0.963	0.610
Exam framing removed	0.755	0.962	0.573
English instruction	0.750	0.962	0.578
Expert persona	0.757	0.960	0.598

Table 7: Presentation robustness on the same 400 gate-accepted items. Blind chance is

0.250

. Marker rows change only the labels; the Latin rows repeat the baseline for reference. No model answered with a Latin letter when shown Cyrillic labels, so the conventions are not being conflated.

Panel of Figure 3 — Figure 3: Blind accuracy with the key forced into each slot (bars) against how often the model picks that slot at all (points). Under content-independence the two coincide and the bars sum to $1.00$ . Nova Pro has a real slot preference and no gap; Sonnet has the largest preference in the set and a clear gap; GPT-5.6 has almost no preference and the largest gap of all.

Model	Slot spread	Sum	Position-free	95% CI	Single-order
Nova Pro	0.180	1.085	0.271	[0.239, 0.304]	0.255
Sonnet 4.6	0.528	1.427	0.357	[0.325, 0.389]	0.343
GPT-5.6	0.085	2.009	0.502	[0.461, 0.542]	0.524

Table 8: Blind solvability with position bias removed by construction: the mean over the four forced placements, with an item-level

95\%

interval on 400 items. The single-order column is the same 400-item measurement, not the full-scale figure of Table 3 .

Model	Condition	Blind	$p$	Full	Out tok.
Sonnet 4.6	direct answer	0.343	—	0.738	4
Sonnet 4.6	extended thinking on	0.343	—	0.785	59
Sonnet 4.6	step-by-step prompt	0.388	0.139	0.805	619
Nova Pro	direct answer	0.255	—	0.578	260
Nova Pro	step-by-step prompt	0.228	0.284	0.578	530
DeepSeek R1	2,048-token budget	0.253	—	0.668	—

Table 9: Reasoning interventions on the 400 gate-accepted items.

p

-values are McNemar tests on the paired items against that model’s direct-answer row. The extended-thinking row is reported for completeness only: Anthropic’s API refuses temperature 0 when thinking is enabled, so that row moves two things at once.

Model	Condition	$n$	Accuracy	95% CI
Sonnet 4.6	blind	1655	0.228	[0.208, 0.249]
Sonnet 4.6	full	1655	0.642	[0.619, 0.665]
GPT-5.6	blind	1655	0.228	[0.208, 0.249]
GPT-5.6	full	1621	0.808	[0.788, 0.827]

Table 10: The same blind probe on LEXam’s four-choice split. Chance is

0.250

n

counts responses from which a letter could be parsed; GPT-5.6 produced 34 unparseable answers in the full condition and none in the blind one.

Panel of Figure 4 — Figure 4: Left: the cumulative distribution of option length in the two banks. LEXam’s curve is complete at 33 characters — its longest option anywhere — while $59.6\%$ of our options are longer than that. Ours are legal propositions that can be judged on their own; LEXam’s are pointers into a list given in the stem. Right: what the same two models score with the question hidden. The design choice on the left produces the difference on the right.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Bedrock identifier	Region	Greedy	Reasons
Reasoning models
GPT-5.6	global.openai.gpt-5.6-sol	eu-central-1	no	yes
DeepSeek R1	us.deepseek.r1-v1:0	us-east-1	yes	yes
Conventional models
Sonnet 4.6	eu.anthropic.claude-sonnet-4-6	eu-central-1	yes	no
Haiku 4.5	eu.anthropic.claude-haiku-4-5-20251001-v1:0	eu-central-1	yes	no

Appendix

Table 11: Model identifiers, whether the provider accepted a temperature setting, and whether the model reasons before answering at its default settings. The region column is where the 600-item sweep ran; the full-scale blind run sharded Qwen3, Gemma, Ministral and Pixtral across several regions to work around per-profile quotas, which is recorded per call in the release. Following Fan et al. (2026) , reasoning and conventional models are grouped rather than equalised: reasoning cannot be switched off in GPT-5.6, which rejects reasoning_effort on Bedrock as it rejects temperature .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,401 papers

Sep 28, 2026

Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

Multiple-choice benchmarks are cheap to grade and are running out of room, and the standard remedy, writing harder items, is slow and repeated for every benchmark. A saturated benchmark still holds a harder task. Each question's wrong options are written for that question alone, so a model can score by eliminating a few options. We propose AnswerPool: take

N

questions that share a context, pool all their options into one list, and ask the model to assign every question its answer. No item is written and no label changes. The chance of guessing a group right falls from

10^{-3}

5\times10^{-7}

for five four-option questions, and a model that recognizes its answers keeps its multiple-choice score, so the accuracy lost to pooling measures the credit the format gave for elimination. Deleting answers from the pool makes questions unanswerable with exact ground truth, so abstention is scored in the same pass. Across eight text, image, and video benchmarks and eighteen models, pooling is harder for every model, the elimination credit is largest for the weakest models, and seven of eight open-weight models answer 87 to 100% of unanswerable questions.

Mohamed Eltahir, Abobaker Ahmed, Nawaf Barebood +3

King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England

Multiple-Choice Question Answering Benchmark Validity Multimodal Model Evaluation

Ask PDF

Aug 30, 2026

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy), but in educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward. We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy: distractor elimination, abstention, confidence calibration, and self-correction. On LLM benchmarks, these schemes: 1) shift rankings of 31 LLMs beyond rephrased number right prompts; 2) better predict the LLMs users prefer in LLM Arena; and 3) reveal distinct model capabilities, like that GPT-5 rarely abstains and readily self-corrects, while weaker open-weight models often abstain and hesitate to eliminate choices. Given the benefits of alternative scoring schemes, we discuss ways to extend them to tasks beyond MCQA.

Nishant Balepur, Paiheng Xu, Wei Ai +3

University of Maryland · New York University · Nanyang Technological University

LLM Evaluation Multiple-Choice Question Answering Automated Grading LLM Abstention

Ask PDF

Aug 3, 2026

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among incorrect options may contain systematic and useful information about its behavior and ability. We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware psychometric framework that models the full distribution over answer choices to jointly estimate LLM ability and option-level item characteristics, while separating model-specific response calibration sharpness, positional preference, and difficulty-dependent fallback behavior. Across 189 LLMs and 31,554 items from 14 benchmarks, LLM-NRM predicts held-out LLM-item interactions more accurately than binary Item Response models and conventional nominal-response baselines, and its ability estimates achieve the strongest Spearman correlation of 0.920 with the external human-preference Arena.ai Elo leaderboard. Distractor identity contributes +101% additional Fisher Information per item beyond correctness, and incorrect responses alone recover full-information ability estimates with Spearman 0.943. The learned item parameters also enable efficient benchmarking, where 41 selected items preserve the full-bank ranking with Kendall's correlation 0.85, corresponding to a 770 times reduction. In conclusion, we show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.

Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1

École Polytechnique, Institut Polytechnique de Paris, France · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates

LLM Evaluation Psychometric Validation Item Response Theory

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks