Correct Choices, Failed Sequences: Choice2Seq Benchmarks Correctness Retention Across Output Interfaces

Tianyou Lai; Wentao Yue; Zhiying Deng; Qilei Li

cs.AIOct 12, 2026

Correct Choices, Failed Sequences: Choice2Seq Benchmarks Correctness Retention Across Output Interfaces

Authors: Tianyou Lai, Wentao Yue, Zhiying Deng, Qilei Li

Organizations: Carnegie Mellon University · Shenzhen University · Central China Normal University

Abstract

A model may select the correct answer yet fail to construct it through a sequence of local decisions. We introduce Choice2Seq, a diagnostic benchmark that measures correctness retention across complete-answer selection and sequential construction in independent runs on the same problems and candidate sets. Sequence Composition Rate (SCR) measures the fraction of correctly selected items that are also answered correctly through local construction. Jev achieves 97.1% and 99.5% selection accuracy on arithmetic and lexical retrieval, yet retains only 33.0% and 2.1% of those correct choices under serial construction, respectively. Substantial gaps also emerge for StartLux and Qwen, including when Qwen emits characters directly. Restricting local options to candidate-supported characters improves construction, while explicit position indexing improves copying fidelity, even for incorrect selections. These interventions distinguish task correctness, plan fidelity, and output compliance. Matched dialogue-state evaluations reveal that local field construction largely preserves Jev's correct states but reduces Qwen's mean-turn accuracy, highlighting task- and interface-dependent differences. Sequence-level confidence and request-cost analyses further characterize completed outputs.

Figures & tables

Figure 1: Choice2Seq distinguishes correctness retention, copying fidelity, and output compliance. Upper: independent interfaces on matched problems: correct complete-answer selection, incorrect Serial construction. Lower: controls test support, evidence access, rendering; extensions assess dialogue-state and completed-output reliability.

Model / interface	Domain	Native EM	Joint counts	Serial EM	SCR count	SCR [95% CI]
Jev	math	97.1	123/250/0/11	32.0	123/373	33.0 [29.0, 37.0]
Jev	lexical	99.5	8/374/0/2	2.1	8/382	2.1 [0.8, 3.7]
StartLux	math	61.2	8/227/1/148	2.3	8/235	3.4 [1.3, 5.9]
StartLux	lexical	93.8	1/359/0/24	0.3	1/360	0.3 [0.0, 0.8]
Qwen3-8B (code)	math	15.4	0/59/0/325	0.0	0/59	0.0 [0.0, 6.1]†
Qwen3-8B (code)	lexical	94.5	0/363/0/21	0.0	0/363	0.0 [0.0, 1.0]†

Table 1: Paired correctness on 384 items per domain. Joint counts give both correct / Native only / Serial only / neither; EM and SCR are percentages. SCR counts are both correct / Native correct. Intervals are length-stratified item bootstrap 95% CIs, except zero-success cells (†), which show exact two-sided binomial intervals.

Figure 2: High Native accuracy does not guarantee Serial retention. Across 64 paired items per domain and length, (a,c) show near-ceiling Native accuracy for Jev and direct-output Qwen3.5-27B. In (b,d), Jev’s arithmetic retention declines with length while lexical retention remains near zero; direct Qwen instead retains more lexical than arithmetic answers. SCR conditions on Native-correct items; bands indicate 95% item-bootstrap intervals.

Model	Native/Serial EM	Joint counts	SCR
Jev	89.8/90.6	111/4/5/8	96.5
Qwen3-8B	21.9/28.9	6/22/31/69	21.4
StartLux	51.6/71.1	51/15/40/22	77.3

Table 2: A marginal gain can replace most Native successes. Named semantic fields use 128 records per model. Joint counts are both correct / Native only / Serial only / neither; EM and SCR are percentages.

Interface	Arithmetic	Lexical	Valid	Calls
Native	94/96	96/96	100/100	1.00
WholeGenerate	94/96	91/96	100/100	1.00
Position query	14/96	27/96	100/100	5.50
PrefixContinue	94/96	0/96	100/9.4	5.50

Table 3: One-response solving largely preserves accuracy; repeated position queries do not. Qwen3.5-27B uses the same 96 items per domain, with no supplied answer. Correct counts are content EM; valid trajectories are arithmetic/lexical percentages; calls are per item.

Model	Domain	Raw	Indexed	Gold EM
Jev	math	17/30	30/30	28/30
Jev	lexical	4/30	29/30	29/30
StartLux	math	8/30	25/30	16/30
StartLux	lexical	1/30	29/30	28/30

Table 4: Serial plan-copying fidelity and indexed task correctness. Raw and Indexed measure fidelity to freshly selected model plans, including incorrect selections; Gold EM measures indexed outputs against the task target. Each model–domain pair contains 30 plans.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Correctness retention and plan fidelity require different references. (a) Program return preserves a selected answer without another model request. (b) Serial solves the same problem separately, measuring overlap with Native correctness. (c) Copying scores fidelity to the supplied plan separately from gold-answer correctness, distinguishing faithful copying of a wrong plan from successful solving.

Figure 4: Indexing improves fidelity to selected plans without guaranteeing task correctness. Each panel uses 30 fresh plans per domain/model. (a) Arithmetic fidelity can exceed gold EM because incorrect plans are copied faithfully; (b) lexical fidelity also improves for both models. Intervals are 95% item bootstrap.

Domain	Native	Whole	Joint	Lost	Gained
Arithmetic	59	181	41	18	140
Lexical	363	372	358	5	14

Appendix

Table B1: Whole-answer generation improves Qwen3-8B accuracy while changing which items succeed (384 per domain). Serial legal-code EM is zero in both domains. Joint, lost and gained counts compare independent Native and whole-generation outcomes.

Figure 5: Matched symbol operations yield small letter–digit contrasts, while option-name conflicts remain interface dependent. (a) On 120 shared COPY/REVERSE bases, contrast intervals span both directions. (b) Extra conflict losses survive correction for StartLux arithmetic and Qwen3-8B lexical readout. Paired intervals are 95%; asterisks mark Holm

p<0.05

after correction within each comparison family.

Model	Representation	Native EM	Serial EM	SCR count	SCR [95% CI]
Jev	named	89.8	90.6	111/115	96.5 [93.0, 99.2]
Jev	binary	68.0	85.2	80/87	92.0 [86.0, 96.8]
StartLux	named	51.6	71.1	51/66	77.3 [67.6, 86.6]
StartLux	binary	39.1	29.7	25/50	50.0 [40.4, 60.5]
Qwen3-8B	named	21.9	28.9	6/28	21.4 [8.0, 36.0]
Qwen3-8B	binary	14.8	15.6	1/19	5.3 [0.0, 16.7]

Appendix

Table B2: Jev preserves most correct semantic decisions under either encoding. SCR reports retained/correct Native counts and 95% intervals separately for named and binary outputs.

Figure 6: On 60 supplied-target parents per domain, (a,b) Jev and StartLux approach perfect indexed readout from lower raw accuracy; Qwen3-8B legal-code accuracy stays lower. Intervals are 95% item bootstrap.

Figure 7: The better output contract depends on evidence presentation. Under matched greedy Qwen3-8B requests, (a,b) format validity stays high despite accuracy differences. (c,d) Indexing favors characters in both domains, whereas direct lexical evidence favors codes. Malformed responses fail; paired intervals are 95%.

Figure 8: Identical trajectory accuracy conceals differences in later states. On 96 dialogues per model, gold prior states improve mean-turn and final-state accuracy over predicted prior states, while trajectory EM is unchanged. Exploratory 95% intervals resample dialogue groups linked by shared responses.

Model	Mode	Trajectory	Mean turn	Final	Valid format
Jev	HistoryRecompute	27.1	75.3	53.1	100.0
Jev	Oracle	33.3	83.2	68.8	100.0
Jev	Rollout	33.3	77.0	61.5	100.0
StartLux	HistoryRecompute	25.0	69.7	59.4	100.0
StartLux	Oracle	30.2	80.6	71.9	100.0
StartLux	Rollout	30.2	71.9	59.4	100.0

Appendix

Table C1: Equal trajectory accuracy coexists with different later-state accuracy (96 dialogues per arm). Mean turn EM weights dialogues equally. Trajectory and final EM evaluate complete trajectories and terminal states.

Model	Endpoint	Difference [95% CI]	Component 95% CI
Jev	trajectory	0.0 [-2.1, 2.1]	[-3.1, 3.1]
Jev	mean turn	6.2 [3.7, 8.6]	[3.1, 9.5]
Jev	final	7.3 [2.1, 12.5]	[1.0, 14.3]
Jev	mean field	2.4 [1.5, 3.3]	[1.3, 3.6]
StartLux	trajectory	0.0 [0.0, 0.0]	[0.0, 0.0]
StartLux	mean turn	8.7 [6.7, 10.7]	[5.6, 12.0]

Appendix

Table C2: Gold prior states improve mean-turn and final-state accuracy without changing trajectory EM. Differences are Oracle minus Rollout (percentage points), with paired-dialogue and response-component 95% intervals.

Model	Mode	Failed	Any recovery	Final recovery
Jev	HistoryRecompute	70	36	25
Jev	Oracle	64	47	34
Jev	Rollout	64	36	27
Qwen3.5-27B	HistoryRecompute	67	37	27
Qwen3.5-27B	Oracle	71	47	34
Qwen3.5-27B	Rollout	71	33	26

Appendix

Table C3: Failed trajectories can recover correct later or final states. Error populations vary by model and arm.

Figure 9: The accuracy effect of local field construction differs across models. On 96 dialogues with identical visible history and schemas, Qwen3.5-27B shows a corrected mean-turn loss, while Jev’s interval spans zero. Final-state intervals span zero for both models. Intervals are 95% dialogue bootstrap.

Model	Arm	Mean turn	Final state	Trajectory
Jev	Joint	76.0 [71.0, 80.7]	55.2 [44.8, 64.6]	33.3 [24.0, 42.7]
Jev	Independent	76.0 [71.0, 80.8]	56.2 [45.8, 65.6]	32.3 [22.9, 41.7]
Jev	Serial	75.5 [70.8, 80.0]	52.1 [41.7, 62.5]	27.1 [18.8, 36.5]
Qwen3.5-27B	Joint	75.4 [70.8, 79.8]	57.3 [46.9, 66.7]	28.1 [19.8, 37.5]
Qwen3.5-27B	Independent	70.6 [66.1, 75.0]	58.3 [47.9, 67.7]	20.8 [12.5, 29.2]
Qwen3.5-27B	Serial	72.3 [67.9, 76.6]	59.4 [49.0, 68.8]	24.0 [15.6, 32.3]

Appendix

Table C4: Local field construction reduces Qwen mean-turn accuracy; the Jev contrast spans zero. Accuracy (%) has 95% dialogue bootstrap intervals. Dialogues receive equal mean-turn weight. Final-state EM scores the terminal dialogue state; trajectory EM requires correctness at every turn.

Model	Arm	Retained turns	Retention [95% CI]	Valid	Calls	Input tokens
Jev	Independent	423/429	98.5 [97.2, 99.6]	100.0	1716	1651282
Jev	Joint	reference	reference	100.0	576	822825
Jev	Serial	420/429	97.9 [96.6, 99.2]	100.0	1716	1686253
Qwen3.5-27B	Independent	388/426	91.9 [88.9, 94.7]	99.2	1716	1059954
Qwen3.5-27B	Joint	reference	reference	100.0	576	591013
Qwen3.5-27B	Serial	395/426	93.3 [90.6, 95.8]	99.4	1716	1084404

Appendix

Table C5: High state retention still requires more local requests. Format validity is in percent. Retention intervals weight dialogues equally and include raw retained turn counts.

Model	Domain	Coverage	Retained	SCR [95% CI]
Jev	math	367/384	118/367	32.2 [28.1, 36.2]
Qwen3.5-27B	math	367/384	38/367	10.4 [7.4, 13.4]
Jev	lexical	379/384	8/379	2.1 [0.8, 3.7]
Qwen3.5-27B	lexical	379/384	131/379	34.6 [29.8, 39.3]

Appendix

Table D1: The arithmetic–lexical contrast persists on a common Native-correct population. Coverage, retained counts and 95% item bootstrap intervals stratified by length are shown.

Domain	Native correct	Serial 0	Serial 1	Serial 2
math	35/36	12/35	13/35	12/35
lexical	36/36	0/36	1/36	1/36

Appendix

Table D2: Across three repeats, Native choices agree but Serial successes vary. Counts are retained/Native correct.

Figure 10: The Jev domain gap survives natural-task perturbations. Across 72 items, (a,c) Jev retains more arithmetic than lexical answers; (e,g) StartLux retention stays low; (i,k) Qwen3-8B legal-code construction has no successes. Paired changes are imprecise. SCR uses setting-specific Native successes, with 95% item bootstrap intervals.

Domain	Prompt	Serial EM	Retained	SCR [95% CI]	EM change [95% CI]	Holm p
lexical	v1	39.6	38/95	40.0 [30.5, 49.5]	reference	N/A
lexical	v2	56.2	54/96	56.2 [46.9, 65.6]	16.7 [7.3, 26.0]	0.005
lexical	v3	50.0	47/95	49.5 [39.6, 59.4]	10.4 [1.0, 19.8]	0.105
math	v1	7.3	7/95	7.4 [3.1, 12.5]	reference	N/A
math	v2	28.1	27/94	28.7 [21.5, 35.8]	20.8 [13.5, 28.1]	<0.001
math	v3	13.5	12/95	12.6 [7.3, 18.8]	6.2 [-1.0, 13.5]	0.180

Appendix

Table D3: Prompt wording changes Qwen content accuracy under valid character output. SCR uses template-specific Native successes. Changes from v1 have paired 95% intervals and four-test Holm correction.

Domain	Mean [95% CI]	Worst [95% CI]
Arithmetic	16.2 [12.1, 20.6]	7.4 [3.1, 11.7]
Lexical	48.6 [40.3, 56.9]	40.0 [30.5, 49.5]

Appendix

Table D4: Worst-template retention remains well below Native accuracy. Direct Qwen3.5-27B SCR (%) has 95% whole-item bootstrap intervals. Each domain resamples 96 parents and recomputes its worst setting.

Figure 11: Candidate support improves Serial success while leaving a gap to Native. On 384 formal items per domain, (a,c) arithmetic gains combine option restriction with deterministic filling; (b,d) lexical gains retain a model request at every position. Intervals are paired 95% item bootstrap; asterisks indicate Holm

p<0.05

across six tests.

Model	Domain	Both wrong	Recovered	Lost	Both correct
Jev	math	84	192	8	100
Jev	lexical	285	91	0	8
StartLux	math	274	103	3	4
StartLux	lexical	366	17	1	0
Qwen3.5-27B	math	244	105	14	21
Qwen3.5-27B	lexical	166	87	30	101

Appendix

Table D5: Union recovers failed items while losing some Full successes (384 items per domain). Recovered means Full wrong/Union correct; lost means Full correct/Union wrong. Counts expose changes hidden by net accuracy.

Model	Domain	Full calls	Union calls	Forced	Full member	Union member
Jev	lexical	2112	2112	0	2.6	28.6
Jev	math	2112	768	1344	33.9	100.0
StartLux	lexical	2112	2112	0	1.0	9.1
StartLux	math	2112	768	1344	5.2	100.0
Qwen3.5-27B	lexical	2112	2112	0	40.6	62.8
Qwen3.5-27B	math	2112	768	1344	20.1	98.2

Appendix

Table D6: Position unions reduce arithmetic calls while permitting noncandidate lexical strings. Membership percentages and recorded model request counts exclude retries.

Domain	Length	Items	Mean U	Forced	Min size	Median	Max
math	3 to 8	384	1+4/L	L $-$ 2	8	8	8
lexical	3	64	5.22	0.0	64	134	224
lexical	4	64	5.43	0.0	336	812	1,568
lexical	5	64	5.73	0.0	1,800	5,184	12,544
lexical	6	64	5.60	0.0	9,600	26,040	57,624
lexical	7	64	5.68	0.0	31,104	176,400	614,656

Appendix

Table D7: Matching gold-prefix ambiguity leaves much larger lexical Cartesian output spaces. Mean U averages union sizes within items; Forced counts deterministic positions; Cartesian support multiplies position-wise sizes.

Figure 12: Preserving complete candidate consistency restores much of lexical construction. Trie’s valid outputs belong to Native’s eight candidates; Call still queries singletons, while Fill supplies them. Jev’s domain contrast shrinks, but all local conditions remain below Native (dashed reference). Panels use 384 Jev or 96 Qwen parents per domain; intervals resample whole items within length. Native and Full are fresh, interleaved reruns on the same parents; Table 1 reports the original responses from the primary collection.

Figure 13: Query count and branch placement affect Jev more than alphabet size in this symbolic panel. On 32 twelve-character reversal sources, (a,b) counts show correct strings under crossed factors. (c) Paired effects have 95% source-bootstrap intervals; Jev’s query-count and branch-location contrasts survive six-test Holm correction.

Branch	Choices / queries	Jev correct	Jev SCR count	Qwen correct	Qwen SCR count
Early	10 / 2	15/32	15/31	4/32	3/21
Early	26 / 2	11/32	11/31	2/32	1/21
Early	10 / 6	1/32	1/31	1/32	1/21
Early	26 / 6	1/32	1/31	0/32	0/21
Late	10 / 2	0/32	0/26	1/32	1/16
Late	26 / 2	0/32	0/26	0/32	0/16

Appendix

Table D8: Correctness depends on branch location and queried depth despite a fixed eight-candidate set. Each cell contains 32 sources; counts are complete-output correct and joint/Native correct. Query counts exclude deterministic fills. Exact 95% upper SCR limits for zero cells are 13.2% (0/26), 16.1% (0/21) and 20.6% (0/16).

Model	Domain	Match	Indexed	Padded	Difference [95% CI]	Holm $p$
Jev	math	bytes	43/48	17/48	54.2 [43.8, 64.6]	8.94e-08
Jev	lexical	bytes	47/48	8/48	81.2 [72.9, 89.6]	1.46e-11
Qwen3.5-27B	math	bytes	23/48	8/48	31.2 [18.8, 43.8]	0.00146
Qwen3.5-27B	lexical	bytes	47/48	38/48	18.8 [10.4, 27.1]	0.00391
Qwen3.5-27B	math	tokens	23/48	9/48	29.2 [16.7, 41.7]	0.00258
Qwen3.5-27B	lexical	tokens	47/48	34/48	27.1 [16.7, 37.5]	0.000488

Appendix

Table D9: Indexed copying improves fidelity over nonindexed length controls. Byte matching fixes serialized-state length; token matching uses Qwen’s rendered prompt at a common prefix. Each model/domain has 48 parents. Paired differences have 95% item intervals, with separate Holm correction for byte and token comparisons.

Figure 14: Local error rates already imply few correct lexical strings under an independence reference. Jev arithmetic Serial exceeds the Independent reference at most lengths, whereas lexical success remains low in both conditions. TeacherForced comparisons use their smaller matched population. Bands are 95% whole-item bootstrap intervals; the product is an empirical reference, not a calibrated sequence probability.

Figure 15: Indexing improves complete copying under fixed position questions. With supplied gold targets and 48 parents per domain, (a,b) indexed fidelity is high across controllers, while raw Batch helps Qwen3.5-27B. (c,d) Serial indexing gains are positive for every model/domain. I/S/B denote Independent/Serial/Batch; paired 95% item bootstrap intervals preserve all conditions within each parent.

Model	Domain	Evidence	Literal	Opaque	Difference [95% CI]	Holm p
Jev	lexical	raw	35	34	2.1 [-8.3, 12.5]	1
Jev	math	raw	43	43	0.0 [-6.2, 6.2]	1
Qwen3.5-27B	lexical	raw	37	11	54.2 [41.7, 66.7]	3.3e-07
Qwen3.5-27B	math	raw	41	14	56.2 [41.7, 68.8]	1.8e-07
StartLux	lexical	raw	11	11	0.0 [0.0, 0.0]	1
StartLux	math	raw	26	26	0.0 [0.0, 0.0]	1

Appendix

Table D10: Literal names improve Qwen raw readout; indexed readout is correct under both bindings. Each domain has 48 parents. Differences are percentage points with paired 95% intervals and twelve-test Holm correction.

Figure 16: Faithful copying can preserve errors. On 48 parents per domain with paired correct/incorrect targets, (a,b) indexing raises fidelity; Jev copies every indexed incorrect target. Gains cover (c) single/multiple arithmetic character differences and (d) observed/constructed lexical errors. Intervals use 95% item bootstrap.

Figure 17: Prompt and option order change which output contract is more accurate. For single-position readout on 96 supplied-target parents, (a–d) Qwen3-8B has corrected code and character advantages in different settings; (e–h) Qwen3.5-27B’s corrected differences favor codes. Malformed outputs fail strict scoring. Asterisks mark Holm $p<0.05$ after correction across all 72 paired comparisons.

Figure 18: Wording shifts the relative success of code and character output. Contrasts average option orders within 96 supplied-target parents. (a) Both alternative templates shift Qwen3-8B’s relative accuracy toward codes compared with T0; (b) Qwen3.5-27B changes survive correction for T2 but not T1. Intervals are paired 95% parent bootstrap; asterisks mark Holm

p<0.05

after correction across all sixteen interaction tests.

Figure 19: High local confidence can coexist with incorrect strings. On 384 Jev items per domain, (a–c) arithmetic scores track correctness but remain miscalibrated; (d–f) lexical successes are sparse, and local-confidence summaries often overstate correctness. Bin counts give each estimate’s support. Whole items are resampled within length.

Figure 20: Independent lexical evaluation again separates local confidence from string success. Among 2,347 Jev items with 13 Serial successes, (a) Product concentrates near zero; (b,c) higher geometric mean and minimum top choice scores overstate observed correctness. Bins show sparse successes; intervals resample items within length.

Length	Items	Native correct	Serial correct	Retained	SCR (%)
3	61	61	7	7/61	11.48 [4.92, 19.67]
4	315	310	3	3/310	0.97 [0.00, 2.26]
5	442	439	2	2/439	0.46 [0.00, 1.14]
6	511	504	1	1/504	0.20 [0.00, 0.60]
7	548	546	0	0/546	0.00 [0.00, 0.67]†
8	470	468	0	0/468	0.00 [0.00, 0.79]†

Appendix

Table D11: Native accuracy stays high while independent lexical Serial successes concentrate at shorter lengths. Item bootstrap 95% intervals stratify by length and redraw the Native denominator. Zero-success cells (†) use exact two-sided binomial intervals in place of degenerate bootstrap bounds.

Population / mixture	AUROC difference	AP difference
Arithmetic	0.068 [0.051, 0.086]	0.175 [0.135, 0.216]
Original lexical	0.032 [-0.034, 0.085]	0.128 [0.016, 0.425]
New lexical / source	0.011 [-0.043, 0.064]	0.055 [-0.012, 0.249]
New lexical / equal	0.032 [-0.013, 0.075]	0.141 [0.023, 0.444]

Appendix

Table D12: Product ranks arithmetic outcomes better than target length; independent source-mixture lexical gains remain uncertain. Differences compare Product with negative-length ranking, with paired 95% item intervals.

Figure 21: Screening precision varies by task and length. (a) Arithmetic precision is high at low coverage; (b) original lexical estimates are imprecise. (c,d) Equal-length weighting raises independent-cohort precision without more successes. Paired 95% item bootstrap intervals use uniform expected inclusion for cutoff ties.

Population	Mixture	Native EM	Serial EM	SCR
Original lexical	Equal length	99.5 [98.7, 100.0]	2.1 [0.8, 3.6]	2.1 [0.8, 3.6]
New lexical	Source	99.2 [98.8, 99.5]	0.6 [0.3, 0.9]	0.6 [0.3, 0.9]
New lexical	Equal length	99.3 [98.9, 99.6]	2.2 [0.9, 3.6]	2.2 [0.9, 3.6]

Appendix

Table D13: Equal-length weighting narrows the lexical SCR difference between cohorts. Accuracy and SCR (%) have 95% item bootstrap intervals. Equal length weights preserve each cohort’s vocabulary and candidate difficulty.

Population	Mixture	Score	Brier	ECE	AUROC	AP
Original	Equal	Product	0.019 [0.009, 0.030]	0.010 [0.005, 0.025]	0.936 [0.877, 0.978]	0.231 [0.081, 0.543]
Original	Equal	Geom.	0.109 [0.101, 0.117]	0.286 [0.269, 0.301]	0.849 [0.659, 0.959]	0.164 [0.045, 0.461]
Original	Equal	Minimum	0.057 [0.050, 0.064]	0.183 [0.169, 0.198]	0.871 [0.724, 0.966]	0.174 [0.048, 0.477]
New	Source	Product	0.005 [0.003, 0.008]	0.003 [0.001, 0.006]	0.910 [0.830, 0.973]	0.126 [0.041, 0.351]
New	Source	Geom.	0.096 [0.093, 0.098]	0.286 [0.281, 0.290]	0.838 [0.706, 0.942]	0.066 [0.020, 0.191]
New	Source	Minimum	0.044 [0.042, 0.046]	0.184 [0.180, 0.187]	0.806 [0.680, 0.909]	0.049 [0.012, 0.187]

Appendix

Table D14: Lexical calibration and ranking vary with score and length mixture. Intervals are paired 95% item bootstrap. Equal assigns weight

1/6

to each length. Geom. denotes geometric mean.

Model / interface	Task	Items	Joint counts	SCR (%)	$\Delta$	Gain rate (%)
Jev	lexical	384	8/374/0/2	2.1	+0.0	0.0
Jev	math	384	123/250/0/11	33.0	+0.9	0.0
Qwen3-8B	lexical	384	0/363/0/21	0.0	+0.0	0.0
Qwen3-8B	math	384	0/59/0/325	0.0	+0.0	0.0
Qwen3.5 / code	lexical	384	5/378/0/1	1.3	+0.0	0.0
Qwen3.5 / code	math	384	1/96/1/286	1.0	+0.5	0.3

Appendix

Table D15: Marginal improvement can coexist with substantial correctness turnover. Joint counts give both correct / Native only / Serial only / neither. SCR, its difference from Serial EM (

\Delta

, percentage points), and Serial success on Native-wrong items (Gain rate) summarize different parts of the same paired table.

Panel of Table 48 — Table E1: Serial and TeacherForced share instructions and canonical JSON state; only prefix sources differ.

Panel of Table 49 — Table E2: T0–T2 preserve the target, criterion bindings and user question. Verbatim code/character instructions differ in response representation; rows change emphasis on the criterion, choice and response boundary.

Panel of Table 50 — Table E3: Correct criterion identification does not guarantee output compliance. The left response emits the required character; the right emits its bound code, which fails the same character contract.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 396 papers

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets
Jul 27, 2026Zongyou Yang, Yinghan HouLLM Evaluation LLM Answer Verification
Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Aug 31, 2026Qiyao Yan, Chenpeng Wang, Liangming PanLanguage Modeling Language Model Calibration
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · YiXin-AILab, YIXIN, Beijing, China +1
The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer
Jun 16, 2026Rui Wen, Lu Sun, Jiayang Liu +3Open-Ended Generation LLM Evaluation
Institute of Science Tokyo · Tohoku University · Nanyang Technological University +2

Jul 27, 2026J/K move · Enter open · S save

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

Zongyou Yang, Yinghan Hou

Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom

LLM Evaluation LLM Answer Verification Language Model Generation Evaluation LLM Reliability

Open paper AskPDF arXiv

Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.

Abstract

Language Model Generation Evaluation

Explore similar work

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores

The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer