Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation

Jiaxuan Guo; Kejia Zhang; Shuo Xin; Jingxin Yang; Youran Sun; Haizhao Yang

cs.CLSep 27, 2026

Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation

Authors: Jiaxuan Guo, Kejia Zhang, Shuo Xin, Jingxin Yang, Youran Sun, Haizhao Yang

Organizations: Stanford University · University of Maryland, College Park

Abstract

Constrained-option scoring reads a model's probabilities for a fixed set of permitted answers, so it returns a score even when the model is about to write something else. We study a prompt that quotes a multiple-choice item, ending in the item's own answer instruction, and asks for a forecast of whether a reader model given only a short note will answer it correctly. The model can begin answering the quoted item: on three Qwen3 checkpoints an answer letter is the most probable token on nearly every passage, and the forecast ranks the reader's correctness indistinguishably from chance. Beyond prior work's argmax check and prefill repair, we contribute a label-free diagnostic of the declared options' support, their first-token mass in context; a deletion test tracing the Qwen3 answer-letter start to the quoted instruction; and rendering and tokenisation checks for dropped separators and coinciding first tokens. Prefilling an answer stem lifts the median option mass from below to above one half on all nine checkpoints scored with this prompt and, with options and renormalisation unchanged, raises the Qwen3-averaged AUC from 0.4929 to 0.6152. lm-polygraph's default P(True) estimator shows a related failure: it reads True where Qwen3 without thinking favours an answer letter (MMLU) or the prompt's (A)/(B) label (TriviaQA), and its Qwen3-averaged AUROC is indistinguishable from chance on MMLU and below chance on TriviaQA. Scoring and renormalising the (A)/(B) labels after an answer stem raises it to 0.632 and 0.868, renormalising True against False in place reaches 0.660 and 0.860, and on TriviaQA both exceed all fourteen default single-answer library estimators. The original forecast is renormalised too, yet stays indistinguishable from chance: option mass flags low support in both settings without labels, and only checking the intended target shows which score still ranks.

Figures & tables

Token	Probability
A	1.0
Answer	$3.36\times 10^{-10}$
D	$3.28\times 10^{-11}$
After	$8.29\times 10^{-12}$
5	$5.27\times 10^{-12}$
The	$4.73\times 10^{-12}$

Figure 1: The model answers the quoted item before forecasting. On this real Qwen3-4B item, the note states the answer, A. The later forecast instruction asks for one of five numbers, but the scored position assigns its highest probability to A; with notes about an unrelated passage, which state no answer, 58 of 60 rows still put an answer letter first (Section 4 ). The table shows the six most probable tokens and the mass of the five options’ distinct first tokens. The unrestricted continuation reaches the forecast only after answering the item. Appendix B gives the complete prompt and both scored positions; line wrapping here is for display.

	What varies	Checkpoints	$n$	What is read	Reported in
1	Note states the answer / unrelated note	Qwen3-4B	120 rows, 60 each	Mass; unrestricted argmax	§ 4 , App. C.3
2	Checkpoint; answer stem	Nine	64	Mass; argmax; below-floor counts	Table 3 , § 5
3	Delete / substitute instruction; choices; separator	Seven	64	Letter rates and token classes	§ 4 , App. D
4	Judge and human labels	Seven	32 per ending; 100 labelled by hand	Execution labels and agreement	§ 4 , App. D
5	lm-polygraph P(True): library, renormalised, stemmed, deleted, labels in place	Qwen3-4B, 8B, 14B; Llama-3-8B-Instruct; option mass also on Phi-3, Mistral, falcon, SmolLM2, Qwen2.5-72B-Instruct	500 MMLU; 500 TriviaQA	AUROC; prediction–rejection ratio; option mass; argmax class	§ 4 , App. E
6	Leading spaces in reader options	Qwen3-4B, 8B, 14B	240 passages	Choice agreement; mass; accuracy; forecast AUC	§ 3 , App. C.7

Table 1: Experiment map. Rows are numbered for reference;

n

is per checkpoint and condition unless specified otherwise. “Seven” refers to Table 3 ; “eight” adds Llama-3-8B-Instruct; “nine” also adds Qwen3-32B; “ten” adds Flan-T5-XL, Flan-T5-XXL and Flan-UL2 to the seven; “eleven” also adds GPT-2.

	Production prompt			With answer stem
Checkpoint	$\mu$ (min)	Letter	Below	$\mu$ (min)	Below
Qwen3-4B-Instruct-2507	$2.71\text{true}{10}^{-15}$ ( $7.63\text{true}{10}^{-19}$ )	0.984	64/64	0.99940 (0.17702)	2/64
Qwen3-8B	$8.14\text{true}{10}^{-14}$ ( $6.97\text{true}{10}^{-17}$ )	0.953	64/64	0.99951 (0.92296)	0/64
Qwen3-14B	$1.37\text{true}{10}^{-08}$ ( $2.96\text{true}{10}^{-14}$ )	0.969	64/64	0.99999 (0.99966)	0/64
Phi-3-mini-4k-instruct	$5.79\text{true}{10}^{-04}$ ( $9.07\text{true}{10}^{-05}$ )	0.984	64/64	0.93344 (0.84818)	0/64
Mistral-7B-Instruct-v0.3	$3.49\text{true}{10}^{-06}$ ( $3.67\text{true}{10}^{-08}$ )	0.469	64/64	0.99050 (0.00216)	7/64

Table 3: Seven checkpoints on the same 64 frozen items, with one note held fixed: Landlord only gives when told to take, not give. It names no answer letter or option. Each checkpoint uses its own chat template, with enable_thinking=False where the template reads it; forward passes use float16 and log-softmax uses float32. Appendix C.1 gives the note source.

\mu

(Appendix A ) is the median over items, per-item minimum in parentheses; Letter is the fraction of items whose unrestricted argmax is an A–D answer letter; Below counts items under 0.5.

	Original position	With answer stem	Difference
$\mathrm{AUC}$			Gain, stemmed $-$ original
Qwen3-averaged	0.4929 [0.4390, 0.5458]	0.6152 [0.5602, 0.6686]	0.1223 [0.0449, 0.1993]
Qwen3-4B	0.4958	0.6283	0.1325 [0.0168, 0.2459]
Qwen3-8B	0.4864	0.6435	0.1570 [0.0431, 0.2685]
Qwen3-14B	0.4964	0.5739	0.0775 [ $-$ 0.0199, 0.1744]
Llama-3-8B-Instruct	0.5231 [0.4486, 0.5961]	0.6044 [0.5312, 0.6764]	0.0813 [0.0046, 0.1574]

Table 4: The forecast scored against the correctness it predicts, at the original and stemmed positions, on the same 240 passages at each of the three Qwen3 checkpoints and Meta-Llama-3-8B-Instruct. A Qwen3-averaged row is the mean of the three checkpoints’ values; Llama-3-8B-Instruct is separate. Intervals are 95% percentile intervals of a paired cluster bootstrap over passages,

B=20{,}000

, both readouts resampled on the same replicates.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt ends with	Vocabulary	Consequence	Checkpoints
Newline	sentencepiece	marker restored, mass not a reading	Phi-3-mini
Newline	byte-level	neither problem	Qwen3 $\times 3$ , SmolLM2
[/INST] , Assistant:	either	separator omitted by template	Mistral, falcon

Appendix

Table 5: Boundary exposure classes for the measured checkpoints and tokeniser versions. The rule separates endings with and without whitespace; the text reports which defects occur.

Reading	Checkpoint	Prompt	isolated	divergent	in-context
			Alternative columns		Eq. 2
Stem repairs the readout	Phi-3-mini-4k-instruct	stemmed	$1.27\text{true}{10}^{-08}$	0.0392	0.9331
Original-ending option mass	Mistral-7B-Instruct-v0.3	unstemmed	0.4545	0.9935	$3.93\text{true}{10}^{-06}$

Appendix

Table 6: Isolated and divergent columns beside the in-context reading, all from the same 16-item forward passes. Table 3 gives the 64-item medians. A separate 16-item reading without the in-context column has identical isolated and divergent values to every digit.

	Mass (median)		Argmax on option		Accuracy
Option strings	gold in note	unrelated	gold in note	unrelated	gold in note	unrelated
" A" – " D"	$5.21\text{true}{10}^{-14}$	$9.67\text{true}{10}^{-12}$	0.00	0.00	0.975	0.4167
"A" – "D"	1.000	0.99999976	1.00	0.967	0.9833	0.4583

Appendix

Table 7: The reader task with spaced and unspaced options, same 240 rows, same run. Mass is the median option first-token mass by group; accuracy is against the passage’s gold answer. The spaced and unspaced strings select the same option in 96.25% of rows.

Comparison	Rows	Agreement	$\kappa$
Grok and Opus	1568	1549	0.958 (Cohen)
Grok and GPT	1568	1532	0.922 (Cohen)
Opus and GPT	1568	1533	0.924 (Cohen)
First majority and Grok, Opus, GPT	100	99, 98, 99
Second majority and Grok, Opus, GPT	100	100, 99, 100
First and second majorities	100	99

Appendix

Table 8: Agreement on execution labels, on the binary executes-or-not reading except in the last row, which compares the four-way label. Judge pairs are compared on the 1568 experimental continuations. Each labelling covers 100 continuations; comparisons with a judge or between labellings use its majority of three, and the pairwise row and Fleiss’

\kappa

use the individual labels.

Checkpoint	Accuracy	Median mass Library / Moved	Library	Renorm.	Moved	Deleted	Moved $-$ Library
Qwen3-4B- Instruct-2507	0.724	$1.8\times 10^{-11}$ 1.00	0.423 $[0.369,0.477]$	0.674	0.610	0.560	$+0.187$ $[+0.113,+0.260]$
Qwen3-8B, no thinking	0.764	$3.4\times 10^{-13}$ 1.00	0.474 $[0.413,0.536]$	0.592	0.617	0.580	$+0.143$ $[+0.057,+0.224]$
Qwen3-14B, no thinking	0.798	$1.2\times 10^{-10}$ 1.00	0.611 $[0.550,0.670]$	0.715	0.669	0.623	$+0.059$ $[-0.017,+0.134]$
Qwen3 average	0.762	n/a	0.503 $[0.464,0.540]$	0.660	0.632	0.587	$+0.129$ $[+0.078,+0.181]$
Llama-3-8B- Instruct	0.614	$3.3\times 10^{-4}$ 0.999	0.659 $[0.609,0.708]$	0.618	0.597	0.607	$-0.061$ $[-0.127,+0.005]$

Appendix

Table 9: lm-polygraph confidence on 500 shared MMLU items: AUROC for the checkpoint’s own answer being correct under the letter label. Intervals are 95% item-bootstrap intervals (

B=10{,}000

), with the same resampled items across readouts and checkpoints. The readouts are defined above; median mass is the declared set’s at its scored position, shown as Library / Moved. The Qwen3 average weights checkpoint accuracies and AUROCs equally. Differences use unrounded AUROCs.

Checkpoint	Accuracy	Median mass Library / Moved	Library	Renorm.	Moved	Deleted	Moved $-$ Library
Qwen3-4B- Instruct-2507	0.344	$6.4\times 10^{-15}$ 1.00	0.185 $[0.147,0.226]$	0.859	0.884	0.742	$+0.699$ $[+0.636,+0.760]$
Qwen3-8B, no thinking	0.450	$7.4\times 10^{-12}$ 1.00	0.395 $[0.346,0.446]$	0.868	0.865	0.689	$+0.470$ $[+0.403,+0.535]$
Qwen3-14B, no thinking	0.518	$5.1\times 10^{-8}$ 1.00	0.839 $[0.803,0.872]$	0.852	0.853	0.763	$+0.015$ $[-0.005,+0.035]$
Qwen3 average	0.437	n/a	0.473 $[0.448,0.497]$	0.860	0.868	0.731	$+0.395$ $[+0.360,+0.428]$
Llama-3-8B- Instruct	0.604	$2.0\times 10^{-6}$ 1.00	0.664 $[0.616,0.712]$	0.832	0.829	0.669	$+0.164$ $[+0.116,+0.214]$

Appendix

Table 10: lm-polygraph confidence on 500 shared TriviaQA items: AUROC for the checkpoint’s own answer being correct under the alias-match label defined above. Intervals are 95% item-bootstrap intervals (

B=10{,}000

), with the same resampled items across readouts and checkpoints. The readouts are defined in the MMLU comparison; here Deleted removes Answer the following question as briefly as possible. from the quoted input. Median mass is the declared set’s at its scored position, shown as Library / Moved. The Qwen3 average weights checkpoint accuracies and AUROCs equally. Differences use unrounded AUROCs.

Checkpoint	G-Eval	Zhuang	AlpacaEval	Kadavath
Qwen3-4B-Instruct-2507	$1.26\text{true}{10}^{-05}$ / 0.99449	$1.73\text{true}{10}^{-06}$ / 0.99684	1.000 / $2.88\text{true}{10}^{-14}$	0.48600 / —
Qwen3-8B	0.00783 / 0.96250	$6.04\text{true}{10}^{-04}$ / 0.91395	1.000 / $1.07\text{true}{10}^{-10}$	0.82539 / —
Qwen3-14B	0.00336 / 0.99567	$3.76\text{true}{10}^{-05}$ / 0.76455	1.000 / $2.22\text{true}{10}^{-15}$	0.67639 / —
Phi-3-mini-4k-instruct	0.0174 / 0.65558	0.11343 / 0.56764	0.94021 / 0.94021	0.82375 / —
Mistral-7B-Instruct-v0.3	$2.83\text{true}{10}^{-04}$ / 0.90844	$4.83\text{true}{10}^{-04}$ / 0.99172	$9.22\text{true}{10}^{-06}$ / 0.99426	0.39998 / —
falcon-7b-instruct	0.00606 / 0.90917	$1.31\text{true}{10}^{-04}$ / 0.96491	$7.01\text{true}{10}^{-04}$ / 0.0249	0.54744 / —

Appendix

Table 11: Declared-set first-token mass as declared / with a leading space,

10^{\operatorname{median}(\log_{10}\mu)}

over 64 items with linear interpolation on the log scale. G-Eval, Zhuang and Kadavath are completion prompts ending at - Coherence: , Output: and The proposed answer is: ; AlpacaEval uses each checkpoint’s chat template. Kadavath’s question is the item’s stem without its lettered choices or instruction, and the column scores the shared first token of ’ (A)’ and ’ (B)’ ; their existing space makes the prefixed variant undefined. G-Eval’s prefixed options share a first token on all seven checkpoints; Zhuang’s and AlpacaEval’s have distinct first tokens on all 64 items on every checkpoint. AlpacaEval’s context ends at whitespace except on Mistral and falcon.

Forecast	Qwen3-4B	Qwen3-8B	Qwen3-14B	Qwen3 average	Llama-3-8B-Instruct
Constant 0.5	0.2500	0.2500	0.2500	0.2500	0.2500
Fitted constant	0.2222	0.2179	0.2048	0.2150	0.2431

Appendix

Table 12: Constant-forecast Brier baselines on the same evaluation rows as Table 4 . The fitted constants use each checkpoint’s empirical reader-correctness rate; the Qwen3 average is the mean of its three checkpoints. These are descriptive baselines fitted on the evaluation rows.

	Qwen3 average		Four-checkpoint average
Readout	AUC [95% interval]	Brier	AUC [95% interval]	Brier
(a)	0.4927 [0.4388, 0.5457]	0.4748	0.5003 [0.4544, 0.5452]	0.4221
(b)	0.6156 [0.5607, 0.6689]	0.2543	0.6128 [0.5634, 0.6605]	0.2538
(c)	0.5651 [0.5083, 0.6218]	0.2713	0.5640 [0.5148, 0.6124]	0.2644
(d)	0.5926 [0.5381, 0.6460]	0.2930	0.5879 [0.5404, 0.6342]	0.2800
(e)	0.6015 [0.5445, 0.6570]	0.2933	0.6065 [0.5564, 0.6550]	0.2823

Appendix

Table 13: Five forecast readouts on the same 240 passages per checkpoint. The upper panel gives AUC with its 95% interval and Brier for the Qwen3 and four-checkpoint averages, each with equal checkpoint weights. Paired differences in the lower panel use the four-checkpoint average and

\Delta\mathrm{AUC}=\mathrm{AUC}(x)-\mathrm{AUC}(r)

and

\Delta\mathrm{Brier}=\mathrm{Brier}(r)-\mathrm{Brier}(x)

, so positive values favour

x

. All intervals use a paired passage-cluster bootstrap shared by every readout and checkpoint. Readouts (a)–(e) are defined in the text above the table.

Checkpoint	C0	N1	S1	D1	A1	A2	T1	T2
Qwen3-4B	$2.707\text{true}{10}^{-15}$ 63;64	$2.669\text{true}{10}^{-16}$ 64;64	$0.00404$ 48;53	$2.761\text{true}{10}^{-10}$ 61;64	$5.824\text{true}{10}^{-07}$ 56;61	$0.99991$ 5;9	$1.00000$ 0;0	$1.00000$ 0;0
Qwen3-8B	$8.144\text{true}{10}^{-14}$ 61;64	$1.183\text{true}{10}^{-14}$ 61;64	$0.93951$ 25;25	$3.107\text{true}{10}^{-11}$ 14;64	$0.00900$ 42;50	$0.99895$ 1;2	$1.00000$ 0;0	$0.99963$ 0;0
Qwen3-14B	$1.371\text{true}{10}^{-08}$ 62;64	$4.235\text{true}{10}^{-11}$ 63;64	$1.404\text{true}{10}^{-04}$ 60;61	$2.718\text{true}{10}^{-11}$ 39;64	$5.807\text{true}{10}^{-12}$ 62;64	$0.00218$ 54;60	$1.00000$ 0;0	$1.00000$ 0;0
Phi-3-mini	$5.801\text{true}{10}^{-04}$ † 63;64	$9.051\text{true}{10}^{-04}$ † 61;64	$0.00137$ † 64;64	$0.00271$ † 18;64	$0.00169$ † 63;64	$0.03618$ † 61;64	$0.39024$ † 20;46	$0.53902$ † 2;26
Mistral v0.3	$3.488\text{true}{10}^{-06}$ 30;64	$1.907\text{true}{10}^{-06}$ 31;64	$5.023\text{true}{10}^{-07}$ 57;64	$5.412\text{true}{10}^{-07}$ 17;64	$6.074\text{true}{10}^{-06}$ 3;64	$4.694\text{true}{10}^{-06}$ 0;64	$2.005\text{true}{10}^{-06}$ 0;64	$1.180\text{true}{10}^{-07}$ 0;64
falcon	$1.051\text{true}{10}^{-04}$ 0;64	$1.274\text{true}{10}^{-04}$ 0;64	$3.158\text{true}{10}^{-04}$ 0;64	$6.909\text{true}{10}^{-05}$ 0;64	$1.108\text{true}{10}^{-04}$ 0;64	$9.853\text{true}{10}^{-05}$ 0;64	$1.385\text{true}{10}^{-04}$ 0;64	$6.051\text{true}{10}^{-05}$ 0;64

Appendix

Table 14: Eight prompt conditions on 64 items per checkpoint. Each cell gives the raw in-context median at the original ending, followed by the letter-argmax count and below-floor count, separated by a semicolon; both counts are out of 64.

\dagger

marks shared first tokens. Llama’s counted letters are the article A . Under S1, Mistral’s template merges the system text into the user turn. Checkpoint names are abbreviated from Table 3 , with Llama-3-8B-Instruct added. With each condition’s own answer stem, every median is at or above the floor except falcon’s under D1, 0.45421, and T2, 0.33207.

Median below the floor	8	8	6	8	8	5	3	3
	C0	N1	S1	D1	A1	A2	T1	T2
Answer letters	5	5	5	3	4	2	0	0
separator present, first tokens distinct	4	4	3	3	3	1	0	0
Omitted separator	2	2	2	2	2	2	2	2
Shared first token	1	1	1	1	1	1	1	1
Prose opening	1	1	0	4	1	1	0	0

Appendix

Table 15: What holds the scored position at the original ending in each condition of Table 14 , counted over its eight checkpoints. Answer letters : one of the item’s own letters is the unrestricted argmax on at least 32 of 64 items, not counting Llama-3-8B-Instruct’s article A . Prose opening : a word token that opens prose, including Answer and that article, is the argmax on at least 32. Omitted separator : the rendered ending lacks the whitespace the options need. Shared first token : the options’ in-context first tokens coincide. Options : median mass at or above the floor with distinct first tokens. Rows below the first can overlap. The last row counts repairs.

Checkpoint	Answers first	Forecast first	List or question	Prose	Answer, then grid	Grid anywhere
Qwen3-4B-Instruct-2507	63	1	0	0	63	63
Qwen3-8B	63	0	0	1	61	61
Qwen3-14B	62	0	0	2	62	62
Phi-3-mini-4k-instruct	63	0	0	1	27	27
Mistral-7B-Instruct-v0.3	29	30	0	5	2	28
falcon-7b-instruct	3	35	18	8	0	35

Appendix

Table 16: Classes of the C0 continuations, greedy, at most 16 new tokens, on the same 64 items for each checkpoint. The first five columns partition the items by the opening; the last two are flags that overlap them. “Answer, then grid”: the opening answers the item and a grid value follows it. “Grid anywhere”: a grid value anywhere in the text, outside a recited option list. A continuation that gives a forecast first can give a value off the grid.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,364 papers

Aug 30, 2026

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy), but in educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward. We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy: distractor elimination, abstention, confidence calibration, and self-correction. On LLM benchmarks, these schemes: 1) shift rankings of 31 LLMs beyond rephrased number right prompts; 2) better predict the LLMs users prefer in LLM Arena; and 3) reveal distinct model capabilities, like that GPT-5 rarely abstains and readily self-corrects, while weaker open-weight models often abstain and hesitate to eliminate choices. Given the benefits of alternative scoring schemes, we discuss ways to extend them to tasks beyond MCQA.

Nishant Balepur, Paiheng Xu, Wei Ai +3

University of Maryland · New York University · Nanyang Technological University

LLM Evaluation Multiple-Choice Question Answering Automated Grading LLM Abstention

Ask PDF

Oct 6, 2026

Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices

Multiple-choice question answering (MCQA) is commonly used to evaluate large language models under the assumption that one of the provided options is correct, typically using answer-selection accuracy. However, in real deployments, users or retrieval systems may provide invalid option sets in which none of the listed choices is correct, and selecting one of them may incur downstream cost. We study this setting as penalty-framed no-valid-option MCQA. Using the mathematics subset of MMLU-Pro, we remove the labeled correct option, allow models to either choose a remaining option or output ABSTAIN, and penalize invalid forced-choice responses. We further introduce correct-conditioned analysis, evaluating abstention only on instances that the model originally answered correctly. Experiments show that high MCQA accuracy does not fully guarantee abstention reliability: even under explicit no-valid-option-aware instructions and penalty-based scoring, models still produce invalid forced-choice responses for a subset of originally correct instances. These results show that penalty-framed no-valid-option MCQA reveals an aspect of model reliability not captured by standard answer-selection accuracy.

Jinhyeok Kim, Hye-Young Jung

Applied Artificial Intelligence Hanyang University, Republic of Korea · Mathematical Data Science Hanyang University, Republic of Korea

LLM Evaluation Multiple-Choice Question Answering LLM Reliability LLM Abstention

Ask PDF

Sep 24, 2026

Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

Chain-of-thought (CoT) instructions can distort multiple-choice VLM evaluation when a scorer appends a reasoning cue but reads answer-label logits before the model generates any rationale. We call this CoT-prefix scoring. On ScienceQA, Qwen2.5-VL-7B drops from 80.76% to 45.48%, and across five option-content permutations 93.54% of CoT-prefix predictions select the first slot. Condition-matched linear probes recover 78.94% from the same hidden states, while free generation restores 75.24%, showing that the answer often survives the prefix and the immediate readout fails. Vocabulary and layer diagnostics explain the mismatch: probability mass moves toward continuation tokens, while answer information remains linearly accessible in late layers. The effect recurs with varying severity across datasets and models, though not universally. These results show that CoT-prefix scoring can confound model knowledge with an evaluation-interface mismatch and should be avoided unless the requested and scored output events are aligned.

Zeyan Li, Siyuan Qiu, Jianfeng Xu

Shanghai Jiao Tong University

VLM Evaluation Language Model Generation Evaluation Multiple-Choice Question Answering CoT Reasoning Language Model Reasoning Evaluation

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices

Reasoning Instructions Can Break Answer Decoding in Vision--Language Models