A model may select the correct answer yet fail to construct it through a sequence of local decisions. We introduce Choice2Seq, a diagnostic benchmark that measures correctness retention across complete-answer selection and sequential construction in independent runs on the same problems and candidate sets. Sequence Composition Rate (SCR) measures the fraction of correctly selected items that are also answered correctly through local construction. Jev achieves 97.1% and 99.5% selection accuracy on arithmetic and lexical retrieval, yet retains only 33.0% and 2.1% of those correct choices under serial construction, respectively. Substantial gaps also emerge for StartLux and Qwen, including when Qwen emits characters directly. Restricting local options to candidate-supported characters improves construction, while explicit position indexing improves copying fidelity, even for incorrect selections. These interventions distinguish task correctness, plan fidelity, and output compliance. Matched dialogue-state evaluations reveal that local field construction largely preserves Jev's correct states but reduces Qwen's mean-turn accuracy, highlighting task- and interface-dependent differences. Sequence-level confidence and request-cost analyses further characterize completed outputs.
Figures & tables
Figure 1: Choice2Seq distinguishes correctness retention, copying fidelity, and output compliance. Upper: independent interfaces on matched problems: correct complete-answer selection, incorrect Serial construction. Lower: controls test support, evidence access, rendering; extensions assess dialogue-state and completed-output reliability.
Model / interface
Domain
Native EM
Joint counts
Serial EM
SCR count
SCR [95% CI]
Jev
math
97.1
123/250/0/11
32.0
123/373
33.0 [29.0, 37.0]
Jev
lexical
99.5
8/374/0/2
2.1
8/382
2.1 [0.8, 3.7]
StartLux
math
61.2
8/227/1/148
2.3
8/235
3.4 [1.3, 5.9]
StartLux
lexical
93.8
1/359/0/24
0.3
1/360
0.3 [0.0, 0.8]
Qwen3-8B (code)
math
15.4
0/59/0/325
0.0
0/59
0.0 [0.0, 6.1]†
Qwen3-8B (code)
lexical
94.5
0/363/0/21
0.0
0/363
0.0 [0.0, 1.0]†
Table 1: Paired correctness on 384 items per domain. Joint counts give both correct / Native only / Serial only / neither; EM and SCR are percentages. SCR counts are both correct / Native correct. Intervals are length-stratified item bootstrap 95% CIs, except zero-success cells (†), which show exact two-sided binomial intervals.
Figure 2: High Native accuracy does not guarantee Serial retention. Across 64 paired items per domain and length, (a,c) show near-ceiling Native accuracy for Jev and direct-output Qwen3.5-27B. In (b,d), Jev’s arithmetic retention declines with length while lexical retention remains near zero; direct Qwen instead retains more lexical than arithmetic answers. SCR conditions on Native-correct items; bands indicate 95% item-bootstrap intervals.
Model
Native/Serial EM
Joint counts
SCR
Jev
89.8/90.6
111/4/5/8
96.5
Qwen3-8B
21.9/28.9
6/22/31/69
21.4
StartLux
51.6/71.1
51/15/40/22
77.3
Table 2: A marginal gain can replace most Native successes. Named semantic fields use 128 records per model. Joint counts are both correct / Native only / Serial only / neither; EM and SCR are percentages.
Interface
Arithmetic
Lexical
Valid
Calls
Native
94/96
96/96
100/100
1.00
WholeGenerate
94/96
91/96
100/100
1.00
Position query
14/96
27/96
100/100
5.50
PrefixContinue
94/96
0/96
100/9.4
5.50
Table 3: One-response solving largely preserves accuracy; repeated position queries do not. Qwen3.5-27B uses the same 96 items per domain, with no supplied answer. Correct counts are content EM; valid trajectories are arithmetic/lexical percentages; calls are per item.
Model
Domain
Raw
Indexed
Gold EM
Jev
math
17/30
30/30
28/30
Jev
lexical
4/30
29/30
29/30
StartLux
math
8/30
25/30
16/30
StartLux
lexical
1/30
29/30
28/30
Table 4: Serial plan-copying fidelity and indexed task correctness. Raw and Indexed measure fidelity to freshly selected model plans, including incorrect selections; Gold EM measures indexed outputs against the task target. Each model–domain pair contains 30 plans.
Appendix figures & tables44 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Correctness retention and plan fidelity require different references. (a) Program return preserves a selected answer without another model request. (b) Serial solves the same problem separately, measuring overlap with Native correctness. (c) Copying scores fidelity to the supplied plan separately from gold-answer correctness, distinguishing faithful copying of a wrong plan from successful solving.
Figure 4: Indexing improves fidelity to selected plans without guaranteeing task correctness. Each panel uses 30 fresh plans per domain/model. (a) Arithmetic fidelity can exceed gold EM because incorrect plans are copied faithfully; (b) lexical fidelity also improves for both models. Intervals are 95% item bootstrap.
Domain
Native
Whole
Joint
Lost
Gained
Arithmetic
59
181
41
18
140
Lexical
363
372
358
5
14
Appendix
Table B1: Whole-answer generation improves Qwen3-8B accuracy while changing which items succeed (384 per domain). Serial legal-code EM is zero in both domains. Joint, lost and gained counts compare independent Native and whole-generation outcomes.
Figure 5: Matched symbol operations yield small letter–digit contrasts, while option-name conflicts remain interface dependent. (a) On 120 shared COPY/REVERSE bases, contrast intervals span both directions. (b) Extra conflict losses survive correction for StartLux arithmetic and Qwen3-8B lexical readout. Paired intervals are 95%; asterisks mark Holm p<0.05 after correction within each comparison family.
Model
Representation
Native EM
Serial EM
SCR count
SCR [95% CI]
Jev
named
89.8
90.6
111/115
96.5 [93.0, 99.2]
Jev
binary
68.0
85.2
80/87
92.0 [86.0, 96.8]
StartLux
named
51.6
71.1
51/66
77.3 [67.6, 86.6]
StartLux
binary
39.1
29.7
25/50
50.0 [40.4, 60.5]
Qwen3-8B
named
21.9
28.9
6/28
21.4 [8.0, 36.0]
Qwen3-8B
binary
14.8
15.6
1/19
5.3 [0.0, 16.7]
Appendix
Table B2: Jev preserves most correct semantic decisions under either encoding. SCR reports retained/correct Native counts and 95% intervals separately for named and binary outputs.
Figure 6: On 60 supplied-target parents per domain, (a,b) Jev and StartLux approach perfect indexed readout from lower raw accuracy; Qwen3-8B legal-code accuracy stays lower. Intervals are 95% item bootstrap.
Figure 7: The better output contract depends on evidence presentation. Under matched greedy Qwen3-8B requests, (a,b) format validity stays high despite accuracy differences. (c,d) Indexing favors characters in both domains, whereas direct lexical evidence favors codes. Malformed responses fail; paired intervals are 95%.
Figure 8: Identical trajectory accuracy conceals differences in later states. On 96 dialogues per model, gold prior states improve mean-turn and final-state accuracy over predicted prior states, while trajectory EM is unchanged. Exploratory 95% intervals resample dialogue groups linked by shared responses.
Model
Mode
Trajectory
Mean turn
Final
Valid format
Jev
HistoryRecompute
27.1
75.3
53.1
100.0
Jev
Oracle
33.3
83.2
68.8
100.0
Jev
Rollout
33.3
77.0
61.5
100.0
StartLux
HistoryRecompute
25.0
69.7
59.4
100.0
StartLux
Oracle
30.2
80.6
71.9
100.0
StartLux
Rollout
30.2
71.9
59.4
100.0
Appendix
Table C1: Equal trajectory accuracy coexists with different later-state accuracy (96 dialogues per arm). Mean turn EM weights dialogues equally. Trajectory and final EM evaluate complete trajectories and terminal states.
Model
Endpoint
Difference [95% CI]
Component 95% CI
Jev
trajectory
0.0 [-2.1, 2.1]
[-3.1, 3.1]
Jev
mean turn
6.2 [3.7, 8.6]
[3.1, 9.5]
Jev
final
7.3 [2.1, 12.5]
[1.0, 14.3]
Jev
mean field
2.4 [1.5, 3.3]
[1.3, 3.6]
StartLux
trajectory
0.0 [0.0, 0.0]
[0.0, 0.0]
StartLux
mean turn
8.7 [6.7, 10.7]
[5.6, 12.0]
Appendix
Table C2: Gold prior states improve mean-turn and final-state accuracy without changing trajectory EM. Differences are Oracle minus Rollout (percentage points), with paired-dialogue and response-component 95% intervals.
Model
Mode
Failed
Any recovery
Final recovery
Jev
HistoryRecompute
70
36
25
Jev
Oracle
64
47
34
Jev
Rollout
64
36
27
Qwen3.5-27B
HistoryRecompute
67
37
27
Qwen3.5-27B
Oracle
71
47
34
Qwen3.5-27B
Rollout
71
33
26
Appendix
Table C3: Failed trajectories can recover correct later or final states. Error populations vary by model and arm.
Figure 9: The accuracy effect of local field construction differs across models. On 96 dialogues with identical visible history and schemas, Qwen3.5-27B shows a corrected mean-turn loss, while Jev’s interval spans zero. Final-state intervals span zero for both models. Intervals are 95% dialogue bootstrap.
Model
Arm
Mean turn
Final state
Trajectory
Jev
Joint
76.0 [71.0, 80.7]
55.2 [44.8, 64.6]
33.3 [24.0, 42.7]
Jev
Independent
76.0 [71.0, 80.8]
56.2 [45.8, 65.6]
32.3 [22.9, 41.7]
Jev
Serial
75.5 [70.8, 80.0]
52.1 [41.7, 62.5]
27.1 [18.8, 36.5]
Qwen3.5-27B
Joint
75.4 [70.8, 79.8]
57.3 [46.9, 66.7]
28.1 [19.8, 37.5]
Qwen3.5-27B
Independent
70.6 [66.1, 75.0]
58.3 [47.9, 67.7]
20.8 [12.5, 29.2]
Qwen3.5-27B
Serial
72.3 [67.9, 76.6]
59.4 [49.0, 68.8]
24.0 [15.6, 32.3]
Appendix
Table C4: Local field construction reduces Qwen mean-turn accuracy; the Jev contrast spans zero. Accuracy (%) has 95% dialogue bootstrap intervals. Dialogues receive equal mean-turn weight. Final-state EM scores the terminal dialogue state; trajectory EM requires correctness at every turn.
Model
Arm
Retained turns
Retention [95% CI]
Valid
Calls
Input tokens
Jev
Independent
423/429
98.5 [97.2, 99.6]
100.0
1716
1651282
Jev
Joint
reference
reference
100.0
576
822825
Jev
Serial
420/429
97.9 [96.6, 99.2]
100.0
1716
1686253
Qwen3.5-27B
Independent
388/426
91.9 [88.9, 94.7]
99.2
1716
1059954
Qwen3.5-27B
Joint
reference
reference
100.0
576
591013
Qwen3.5-27B
Serial
395/426
93.3 [90.6, 95.8]
99.4
1716
1084404
Appendix
Table C5: High state retention still requires more local requests. Format validity is in percent. Retention intervals weight dialogues equally and include raw retained turn counts.
Model
Domain
Coverage
Retained
SCR [95% CI]
Jev
math
367/384
118/367
32.2 [28.1, 36.2]
Qwen3.5-27B
math
367/384
38/367
10.4 [7.4, 13.4]
Jev
lexical
379/384
8/379
2.1 [0.8, 3.7]
Qwen3.5-27B
lexical
379/384
131/379
34.6 [29.8, 39.3]
Appendix
Table D1: The arithmetic–lexical contrast persists on a common Native-correct population. Coverage, retained counts and 95% item bootstrap intervals stratified by length are shown.
Domain
Native correct
Serial 0
Serial 1
Serial 2
math
35/36
12/35
13/35
12/35
lexical
36/36
0/36
1/36
1/36
Appendix
Table D2: Across three repeats, Native choices agree but Serial successes vary. Counts are retained/Native correct.
Figure 10: The Jev domain gap survives natural-task perturbations. Across 72 items, (a,c) Jev retains more arithmetic than lexical answers; (e,g) StartLux retention stays low; (i,k) Qwen3-8B legal-code construction has no successes. Paired changes are imprecise. SCR uses setting-specific Native successes, with 95% item bootstrap intervals.
Domain
Prompt
Serial EM
Retained
SCR [95% CI]
EM change [95% CI]
Holm p
lexical
v1
39.6
38/95
40.0 [30.5, 49.5]
reference
N/A
lexical
v2
56.2
54/96
56.2 [46.9, 65.6]
16.7 [7.3, 26.0]
0.005
lexical
v3
50.0
47/95
49.5 [39.6, 59.4]
10.4 [1.0, 19.8]
0.105
math
v1
7.3
7/95
7.4 [3.1, 12.5]
reference
N/A
math
v2
28.1
27/94
28.7 [21.5, 35.8]
20.8 [13.5, 28.1]
<0.001
math
v3
13.5
12/95
12.6 [7.3, 18.8]
6.2 [-1.0, 13.5]
0.180
Appendix
Table D3: Prompt wording changes Qwen content accuracy under valid character output. SCR uses template-specific Native successes. Changes from v1 have paired 95% intervals and four-test Holm correction.
Domain
Mean [95% CI]
Worst [95% CI]
Arithmetic
16.2 [12.1, 20.6]
7.4 [3.1, 11.7]
Lexical
48.6 [40.3, 56.9]
40.0 [30.5, 49.5]
Appendix
Table D4: Worst-template retention remains well below Native accuracy. Direct Qwen3.5-27B SCR (%) has 95% whole-item bootstrap intervals. Each domain resamples 96 parents and recomputes its worst setting.
Figure 11: Candidate support improves Serial success while leaving a gap to Native. On 384 formal items per domain, (a,c) arithmetic gains combine option restriction with deterministic filling; (b,d) lexical gains retain a model request at every position. Intervals are paired 95% item bootstrap; asterisks indicate Holm p<0.05 across six tests.
Model
Domain
Both wrong
Recovered
Lost
Both correct
Jev
math
84
192
8
100
Jev
lexical
285
91
0
8
StartLux
math
274
103
3
4
StartLux
lexical
366
17
1
0
Qwen3.5-27B
math
244
105
14
21
Qwen3.5-27B
lexical
166
87
30
101
Appendix
Table D5: Union recovers failed items while losing some Full successes (384 items per domain). Recovered means Full wrong/Union correct; lost means Full correct/Union wrong. Counts expose changes hidden by net accuracy.
Model
Domain
Full calls
Union calls
Forced
Full member
Union member
Jev
lexical
2112
2112
0
2.6
28.6
Jev
math
2112
768
1344
33.9
100.0
StartLux
lexical
2112
2112
0
1.0
9.1
StartLux
math
2112
768
1344
5.2
100.0
Qwen3.5-27B
lexical
2112
2112
0
40.6
62.8
Qwen3.5-27B
math
2112
768
1344
20.1
98.2
Appendix
Table D6: Position unions reduce arithmetic calls while permitting noncandidate lexical strings. Membership percentages and recorded model request counts exclude retries.
Domain
Length
Items
Mean U
Forced
Min size
Median
Max
math
3 to 8
384
1+4/L
L − 2
8
8
8
lexical
3
64
5.22
0.0
64
134
224
lexical
4
64
5.43
0.0
336
812
1,568
lexical
5
64
5.73
0.0
1,800
5,184
12,544
lexical
6
64
5.60
0.0
9,600
26,040
57,624
lexical
7
64
5.68
0.0
31,104
176,400
614,656
Appendix
Table D7: Matching gold-prefix ambiguity leaves much larger lexical Cartesian output spaces. Mean U averages union sizes within items; Forced counts deterministic positions; Cartesian support multiplies position-wise sizes.
Figure 12: Preserving complete candidate consistency restores much of lexical construction. Trie’s valid outputs belong to Native’s eight candidates; Call still queries singletons, while Fill supplies them. Jev’s domain contrast shrinks, but all local conditions remain below Native (dashed reference). Panels use 384 Jev or 96 Qwen parents per domain; intervals resample whole items within length. Native and Full are fresh, interleaved reruns on the same parents; Table 1 reports the original responses from the primary collection.
Figure 13: Query count and branch placement affect Jev more than alphabet size in this symbolic panel. On 32 twelve-character reversal sources, (a,b) counts show correct strings under crossed factors. (c) Paired effects have 95% source-bootstrap intervals; Jev’s query-count and branch-location contrasts survive six-test Holm correction.
Branch
Choices / queries
Jev correct
Jev SCR count
Qwen correct
Qwen SCR count
Early
10 / 2
15/32
15/31
4/32
3/21
Early
26 / 2
11/32
11/31
2/32
1/21
Early
10 / 6
1/32
1/31
1/32
1/21
Early
26 / 6
1/32
1/31
0/32
0/21
Late
10 / 2
0/32
0/26
1/32
1/16
Late
26 / 2
0/32
0/26
0/32
0/16
Appendix
Table D8: Correctness depends on branch location and queried depth despite a fixed eight-candidate set. Each cell contains 32 sources; counts are complete-output correct and joint/Native correct. Query counts exclude deterministic fills. Exact 95% upper SCR limits for zero cells are 13.2% (0/26), 16.1% (0/21) and 20.6% (0/16).
Model
Domain
Match
Indexed
Padded
Difference [95% CI]
Holm p
Jev
math
bytes
43/48
17/48
54.2 [43.8, 64.6]
8.94e-08
Jev
lexical
bytes
47/48
8/48
81.2 [72.9, 89.6]
1.46e-11
Qwen3.5-27B
math
bytes
23/48
8/48
31.2 [18.8, 43.8]
0.00146
Qwen3.5-27B
lexical
bytes
47/48
38/48
18.8 [10.4, 27.1]
0.00391
Qwen3.5-27B
math
tokens
23/48
9/48
29.2 [16.7, 41.7]
0.00258
Qwen3.5-27B
lexical
tokens
47/48
34/48
27.1 [16.7, 37.5]
0.000488
Appendix
Table D9: Indexed copying improves fidelity over nonindexed length controls. Byte matching fixes serialized-state length; token matching uses Qwen’s rendered prompt at a common prefix. Each model/domain has 48 parents. Paired differences have 95% item intervals, with separate Holm correction for byte and token comparisons.
Figure 14: Local error rates already imply few correct lexical strings under an independence reference. Jev arithmetic Serial exceeds the Independent reference at most lengths, whereas lexical success remains low in both conditions. TeacherForced comparisons use their smaller matched population. Bands are 95% whole-item bootstrap intervals; the product is an empirical reference, not a calibrated sequence probability.
Figure 15: Indexing improves complete copying under fixed position questions. With supplied gold targets and 48 parents per domain, (a,b) indexed fidelity is high across controllers, while raw Batch helps Qwen3.5-27B. (c,d) Serial indexing gains are positive for every model/domain. I/S/B denote Independent/Serial/Batch; paired 95% item bootstrap intervals preserve all conditions within each parent.
Model
Domain
Evidence
Literal
Opaque
Difference [95% CI]
Holm p
Jev
lexical
raw
35
34
2.1 [-8.3, 12.5]
1
Jev
math
raw
43
43
0.0 [-6.2, 6.2]
1
Qwen3.5-27B
lexical
raw
37
11
54.2 [41.7, 66.7]
3.3e-07
Qwen3.5-27B
math
raw
41
14
56.2 [41.7, 68.8]
1.8e-07
StartLux
lexical
raw
11
11
0.0 [0.0, 0.0]
1
StartLux
math
raw
26
26
0.0 [0.0, 0.0]
1
Appendix
Table D10: Literal names improve Qwen raw readout; indexed readout is correct under both bindings. Each domain has 48 parents. Differences are percentage points with paired 95% intervals and twelve-test Holm correction.
Figure 16: Faithful copying can preserve errors. On 48 parents per domain with paired correct/incorrect targets, (a,b) indexing raises fidelity; Jev copies every indexed incorrect target. Gains cover (c) single/multiple arithmetic character differences and (d) observed/constructed lexical errors. Intervals use 95% item bootstrap.
Figure 17: Prompt and option order change which output contract is more accurate. For single-position readout on 96 supplied-target parents, (a–d) Qwen3-8B has corrected code and character advantages in different settings; (e–h) Qwen3.5-27B’s corrected differences favor codes. Malformed outputs fail strict scoring. Asterisks mark Holm p<0.05 after correction across all 72 paired comparisons.
Figure 18: Wording shifts the relative success of code and character output. Contrasts average option orders within 96 supplied-target parents. (a) Both alternative templates shift Qwen3-8B’s relative accuracy toward codes compared with T0; (b) Qwen3.5-27B changes survive correction for T2 but not T1. Intervals are paired 95% parent bootstrap; asterisks mark Holm p<0.05 after correction across all sixteen interaction tests.
Figure 19: High local confidence can coexist with incorrect strings. On 384 Jev items per domain, (a–c) arithmetic scores track correctness but remain miscalibrated; (d–f) lexical successes are sparse, and local-confidence summaries often overstate correctness. Bin counts give each estimate’s support. Whole items are resampled within length.
Figure 20: Independent lexical evaluation again separates local confidence from string success. Among 2,347 Jev items with 13 Serial successes, (a) Product concentrates near zero; (b,c) higher geometric mean and minimum top choice scores overstate observed correctness. Bins show sparse successes; intervals resample items within length.
Length
Items
Native correct
Serial correct
Retained
SCR (%)
3
61
61
7
7/61
11.48 [4.92, 19.67]
4
315
310
3
3/310
0.97 [0.00, 2.26]
5
442
439
2
2/439
0.46 [0.00, 1.14]
6
511
504
1
1/504
0.20 [0.00, 0.60]
7
548
546
0
0/546
0.00 [0.00, 0.67]†
8
470
468
0
0/468
0.00 [0.00, 0.79]†
Appendix
Table D11: Native accuracy stays high while independent lexical Serial successes concentrate at shorter lengths. Item bootstrap 95% intervals stratify by length and redraw the Native denominator. Zero-success cells (†) use exact two-sided binomial intervals in place of degenerate bootstrap bounds.
Population / mixture
AUROC difference
AP difference
Arithmetic
0.068 [0.051, 0.086]
0.175 [0.135, 0.216]
Original lexical
0.032 [-0.034, 0.085]
0.128 [0.016, 0.425]
New lexical / source
0.011 [-0.043, 0.064]
0.055 [-0.012, 0.249]
New lexical / equal
0.032 [-0.013, 0.075]
0.141 [0.023, 0.444]
Appendix
Table D12: Product ranks arithmetic outcomes better than target length; independent source-mixture lexical gains remain uncertain. Differences compare Product with negative-length ranking, with paired 95% item intervals.
Figure 21: Screening precision varies by task and length. (a) Arithmetic precision is high at low coverage; (b) original lexical estimates are imprecise. (c,d) Equal-length weighting raises independent-cohort precision without more successes. Paired 95% item bootstrap intervals use uniform expected inclusion for cutoff ties.
Population
Mixture
Native EM
Serial EM
SCR
Original lexical
Equal length
99.5 [98.7, 100.0]
2.1 [0.8, 3.6]
2.1 [0.8, 3.6]
New lexical
Source
99.2 [98.8, 99.5]
0.6 [0.3, 0.9]
0.6 [0.3, 0.9]
New lexical
Equal length
99.3 [98.9, 99.6]
2.2 [0.9, 3.6]
2.2 [0.9, 3.6]
Appendix
Table D13: Equal-length weighting narrows the lexical SCR difference between cohorts. Accuracy and SCR (%) have 95% item bootstrap intervals. Equal length weights preserve each cohort’s vocabulary and candidate difficulty.
Population
Mixture
Score
Brier
ECE
AUROC
AP
Original
Equal
Product
0.019 [0.009, 0.030]
0.010 [0.005, 0.025]
0.936 [0.877, 0.978]
0.231 [0.081, 0.543]
Original
Equal
Geom.
0.109 [0.101, 0.117]
0.286 [0.269, 0.301]
0.849 [0.659, 0.959]
0.164 [0.045, 0.461]
Original
Equal
Minimum
0.057 [0.050, 0.064]
0.183 [0.169, 0.198]
0.871 [0.724, 0.966]
0.174 [0.048, 0.477]
New
Source
Product
0.005 [0.003, 0.008]
0.003 [0.001, 0.006]
0.910 [0.830, 0.973]
0.126 [0.041, 0.351]
New
Source
Geom.
0.096 [0.093, 0.098]
0.286 [0.281, 0.290]
0.838 [0.706, 0.942]
0.066 [0.020, 0.191]
New
Source
Minimum
0.044 [0.042, 0.046]
0.184 [0.180, 0.187]
0.806 [0.680, 0.909]
0.049 [0.012, 0.187]
Appendix
Table D14: Lexical calibration and ranking vary with score and length mixture. Intervals are paired 95% item bootstrap. Equal assigns weight 1/6 to each length. Geom. denotes geometric mean.
Model / interface
Task
Items
Joint counts
SCR (%)
Δ
Gain rate (%)
Jev
lexical
384
8/374/0/2
2.1
+0.0
0.0
Jev
math
384
123/250/0/11
33.0
+0.9
0.0
Qwen3-8B
lexical
384
0/363/0/21
0.0
+0.0
0.0
Qwen3-8B
math
384
0/59/0/325
0.0
+0.0
0.0
Qwen3.5 / code
lexical
384
5/378/0/1
1.3
+0.0
0.0
Qwen3.5 / code
math
384
1/96/1/286
1.0
+0.5
0.3
Appendix
Table D15: Marginal improvement can coexist with substantial correctness turnover. Joint counts give both correct / Native only / Serial only / neither. SCR, its difference from Serial EM ( Δ , percentage points), and Serial success on Native-wrong items (Gain rate) summarize different parts of the same paired table.
Table E1: Serial and TeacherForced share instructions and canonical JSON state; only prefix sources differ.
Table E2: T0–T2 preserve the target, criterion bindings and user question. Verbatim code/character instructions differ in response representation; rows change emphasis on the criterion, choice and response boundary.
Table E3: Correct criterion identification does not guarantee output compliance. The left response emits the required character; the right emits its bound code, which fails the same character contract.
Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.
Zongyou Yang, Yinghan Hou
Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom
When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.
Qiyao Yan, Chenpeng Wang, Liangming Pan
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · YiXin-AILab, YIXIN, Beijing, China +1
Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.
Rui Wen, Lu Sun, Jiayang Liu +3
Institute of Science Tokyo · Tohoku University · Nanyang Technological University +2