SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

Mohsen Larni; Sobhan Ebrahimi Azar; Pouyan Nahed; Kazem Taghva

cs.CLOct 2, 2026

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

Authors: Mohsen Larni, Sobhan Ebrahimi Azar, Pouyan Nahed, Kazem Taghva

Organizations: Department of Computer Science, University of Nevada, Las Vegas · Department of Computer Science University of Nevada, Las Vegas

Abstract

Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isolated probes and aggregate accuracy. We introduce SyntaxBench, a diagnostic benchmark and statistical evaluation framework for character-level reasoning. It contains five core tasks, character counting, letter containment, palindrome detection, edit distance, and longest-string selection, plus index_to_span, a harder substring-extraction stress test. The five core tasks use paired English and character-length-matched random-string inputs. index_to_span documents share a 200-500 word band and are not character-length matched. All six tasks use zero-, one-, and four-shot prompts. We evaluate eight open-weight models from 2B to 32B parameters across 11 reasoning-mode configurations. The framework reports exact-match and relaxed accuracy, Cohen's kappa, paired McNemar tests with odds ratios, bootstrap confidence intervals, Kendall's tau, class-conditional metrics, tokenization analysis, and multiple-comparison-corrected tests. Three findings stand out. First, tokenization shapes accuracy: random strings are more character-visible than English strings (1.892 vs. 3.169 characters per token), and character-counting accuracy falls as English words occupy more tokens. Second, reasoning mode is not uniformly helpful: Gemma4-31B is nearly unchanged across modes on the near-saturated tasks, while Qwen3.6-27B is worse with thinking on palindrome detection (0.952 non-thinking vs. 0.886 thinking at four-shot). Third, index_to_span remains largely unsolved; the best four-shot exact-match accuracy is 6.75%. Character-level evaluation needs controlled inputs, paired tests, and analyses of tokenization and reasoning mode rather than aggregate accuracy alone.

Figures & tables

Intra-word core
Task	Description	Output	Inputs (En / Rand)
count_chars	length of a string $s$	integer	200 / 200
letter_container	does $s$ contain character $c$ ?	yes/no	400 / 400
palindrome_test	does $s$ read the same forward and backward?	yes/no	400 / 400
Inter-word core
edit_distance	Levenshtein distance between $(s_{1},s_{2})$	integer	200 / 200

Table 1: SyntaxBench tasks. The Inputs column reports per-condition sample counts (English / random).

Figure 1: Main results. Per-(model, mode) EMA on combined English+random inputs at 0-, 1-, and 4-shot. At 4-shot, 25 of 32 cells on count_chars , letter_container , longest_string_finder , and palindrome_test for 27B+ configurations reach 90% EMA. Nemotron-Cascade-2 is at 69.0% on longest_string_finder , and DeepSeek-R1-32B non-thinking is at 43.25% on that task. edit_distance discriminates models, and index_to_span remains near the floor. Empty cells are configurations that were not run, or a combined score that is not estimable because one input condition is missing.

Rank	Model	Macro EMA [95% CI]	VRR	PER
1	GPT-OSS-20B	$0.818$ [0.512, 0.989]	0.964	0.036
2	Gemma4-31B	$0.712$ [0.388, 0.971]	1.000	0.000
3	DeepSeek-R1-32B	$0.697$ [0.385, 0.966]	0.973	0.027
4	Qwen3.6-27B	$0.690$ [0.358, 0.972]	1.000	0.000
5	Nemotron-Cascade-2-31B	$0.648$ [0.339, 0.919]	0.992	0.008

Table 2: Composite leaderboard at 4-shot (thinking, condition=all). Macro EMA averages over the six tasks. The interval resamples those six task-level EMAs; it is not the sample bootstrap used for per-task EMA. VRR = valid-response rate; PER = parse-error rate. Qwen3.5 family entries (non-thinking-only) appear in Appendix B.1 , Table 18 .

Figure 2: 0-shot count_chars accuracy by tokenizer-token count. The figure shows this task only. English accuracy falls from $97.7\%$ to $46.2\%$ across the 1-token to 6+ bins in non-thinking mode, and from $100\%$ to $75\%$ in thinking mode. Random inputs remain higher than English in every shared bin.

Shot	Non-thinking		Thinking		NT $-$ T
	EMA	$\kappa$	EMA	$\kappa$	$\Delta$ EMA	$\Delta\kappa$
0-shot	0.919	0.838	0.955	0.910	$-0.036$	$-0.072$
1-shot	0.899	0.798	0.819	0.638	$+0.080$	$+0.160$
4-shot	0.952	0.905	0.886	0.773	$+0.066$	$+0.132$

Table 3: Qwen3.6-27B on palindrome_test , EMA and Cohen’s

\kappa

on combined inputs. Non-thinking outperforms thinking at 1-shot and 4-shot, with the gap concentrated in the positive class (at 4-shot, TPR drops from

0.9075

0.7725

and TNR goes from

0.9975

1.0000

Figure 3: Thinking-mode accuracy delta (

\text{EMA}_{\text{non-think}}-\text{EMA}_{\text{think}}

, per task, at 0-, 1-, and 4-shot) with 95% bootstrap CIs. Bars to the right of zero indicate thinking hurts; bars to the left indicate it helps. Qwen3.6-27B is hurt on palindrome_test ; Gemma4-31B is essentially indistinguishable across modes for character-level tasks at this scale.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

	english	random	english	random	english	random	english	random	english	random	english	random
count_chars		edit_distance		index_to_span		letter_container		longest_string_finder		palindrome_test
deepseek_r1_32b	$92.5_{88.5,96.0}***$	$\mathbf{100.0}_{100.0,100.0}$	$34.5_{28.0,41.5}***$	$40.5_{33.5,47.5}***$	$0.0_{0.0,0.0}***$	$0.5_{0.0,1.5}*$	$98.8_{97.5,99.8}$	$\mathbf{100.0}_{100.0,100.0}$	$\mathbf{99.5}_{98.5,100.0}$	$\mathbf{100.0}_{100.0,100.0}$	$83.2_{79.5,86.8}***$	$86.8_{83.2,90.0}***$
gemma4_31b	$94.5_{91.0,97.5}**$	$98.0_{96.0,99.5}$	$44.5_{37.5,51.5}***$	$34.5_{28.0,41.0}***$	$0.5_{0.0,1.5}**$	$0.5_{0.0,1.5}*$	$98.8_{97.5,99.8}$	$99.8_{99.2,100.0}$	$97.0_{94.5,99.0}$	$\mathbf{100.0}_{100.0,100.0}$	$92.0_{89.2,94.5}***$	$94.5_{92.2,96.5}**$
gpt-oss_20b	$94.5_{91.0,97.5}**$	$\mathbf{100.0}_{100.0,100.0}$	$\mathbf{86.0}_{81.0,90.5}$	$\mathbf{92.0}_{88.0,95.5}$	$\mathbf{8.0}_{4.5,12.0}$	$\mathbf{5.5}_{2.5,9.0}$	$99.5_{98.8,100.0}$	$\mathbf{100.0}_{100.0,100.0}$	$97.5_{95.0,99.5}$	$99.5_{98.5,100.0}$	$\mathbf{99.8}_{99.2,100.0}$	$\mathbf{99.0}_{98.0,99.8}$
nemotron-cascade-2	$96.5_{94.0,99.0}$	$98.0_{96.0,99.5}$	$26.0_{20.0,32.0}***$	$42.5_{35.5,49.5}***$	$0.0_{0.0,0.0}***$	$0.0_{0.0,0.0}**$	$\mathbf{100.0}_{100.0,100.0}$	$99.8_{99.2,100.0}$	$66.5_{60.0,73.0}***$	$71.5_{65.0,77.5}***$	$83.5_{79.8,87.0}***$	$93.0_{90.5,95.5}***$
qwen3.6_27b	$\mathbf{100.0}_{100.0,100.0}$	$\mathbf{100.0}_{100.0,100.0}$	$31.5_{25.0,38.0}***$	$28.0_{22.0,34.5}***$	$0.0_{0.0,0.0}***$	$0.0_{0.0,0.0}**$	$98.0_{96.5,99.2}$	$\mathbf{100.0}_{100.0,100.0}$	$95.0_{92.0,98.0}$	$98.0_{96.0,99.5}$	$88.0_{84.8,91.0}***$	$89.2_{86.2,92.2}***$

Appendix

Table 4: Main results (thinking mode where supported): Exact Match Accuracy (EMA, %) at 4-shot with 95% bootstrap CI. Significance vs. best model:

{}^{*}p{<}0.05

{}^{**}p{<}0.01

{}^{***}p{<}0.001

(Bonferroni-corrected McNemar’s test).

	english	random	english	random	english	random	english	random	english	random	english	random
count_chars		edit_distance		index_to_span		letter_container		longest_string_finder		palindrome_test
deepseek_r1_32b	$76.0_{70.0,82.0}***$	$84.5_{79.0,89.5}***$	$0.0_{0.0,0.0}***$	$0.0_{0.0,0.0}***$	$0.0_{0.0,0.0}$	$0.0_{0.0,0.0}$	$97.8_{96.2,99.0}$	$98.2_{97.0,99.5}$	$53.0_{46.0,60.0}***$	$33.5_{27.0,40.0}***$	$87.8_{84.5,90.8}*$	$87.8_{84.5,90.8}***$
gemma4_31b	$94.5_{91.0,97.5}$	$98.0_{96.0,99.5}$	$\mathbf{39.0}_{32.0,45.5}$	$36.0_{29.5,42.5}*$	$0.5_{0.0,1.5}$	$\mathbf{0.5}_{0.0,1.5}$	$\mathbf{98.8}_{97.5,99.8}$	$\mathbf{99.8}_{99.2,100.0}$	$\mathbf{97.0}_{94.5,99.0}$	$\mathbf{100.0}_{100.0,100.0}$	$92.2_{89.5,94.8}$	$94.2_{91.8,96.5}$
qwen3.5_27b	$91.5_{87.5,95.0}$	$\mathbf{100.0}_{100.0,100.0}$	$36.5_{30.0,43.0}$	$\mathbf{49.5}_{42.5,56.5}$	$\mathbf{1.0}_{0.0,2.5}$	$0.0_{0.0,0.0}$	$98.5_{97.2,99.5}$	$99.2_{98.2,100.0}$	$94.0_{90.5,97.0}$	$98.5_{96.5,100.0}$	$91.2_{88.5,93.8}$	$95.2_{93.0,97.2}$
qwen3.5_2b	$75.5_{69.5,81.5}***$	$64.0_{57.5,70.5}***$	$23.0_{17.0,29.0}*$	$29.0_{23.0,35.5}***$	$0.0_{0.0,0.0}$	–	$77.5_{73.2,81.5}***$	$88.0_{84.8,91.2}***$	$48.5_{41.5,55.5}***$	$47.5_{40.5,54.5}***$	$72.0_{67.5,76.2}***$	$94.8_{92.5,96.8}$
qwen3.5_9b	$79.0_{73.0,84.5}***$	$97.0_{94.5,99.0}$	$34.0_{27.5,40.5}$	$45.5_{38.5,52.5}$	$0.5_{0.0,1.5}$	$\mathbf{0.5}_{0.0,1.5}$	$93.5_{91.0,95.8}**$	$98.8_{97.5,99.8}$	$74.5_{68.0,80.5}***$	$89.0_{84.5,93.0}***$	$90.5_{87.5,93.2}$	$\mathbf{96.8}_{95.0,98.2}$

Appendix

Table 5: Main results (non-thinking mode): Exact Match Accuracy (EMA, %) at 4-shot with 95% bootstrap CI. Significance vs. best model:

{}^{*}p{<}0.05

{}^{**}p{<}0.01

{}^{***}p{<}0.001

(Bonferroni-corrected McNemar’s test). “–” means that condition was not evaluated. For qwen3.5_2b on index_to_span at 4-shot, the English cell is the observed 113-item subset and the random condition is missing; the combined score is therefore not estimated.

	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot
count_chars			edit_distance			index_to_span			letter_container			longest_string_finder			palindrome_test
deepseek_r1_32b	97.8	94.5	96.2	39.8	34.2	37.5	0.5	0.0	0.2	98.9	98.6	99.4	99.8	99.0	99.8	95.0	94.2	85.0
gemma4_31b	87.8	92.0	96.2	29.0	28.7	39.5	0.5	0.5	0.5	99.5	99.1	99.2	95.8	97.8	98.5	84.1	92.9	93.2
gpt-oss_20b	96.5	97.5	97.2	85.2	88.8	89.0	7.4	6.5	6.8	99.5	99.2	99.8	99.8	99.8	98.5	98.8	98.9	99.4
nemotron-cascade-2	96.2	96.2	97.2	23.0	16.0	34.2	0.0	0.0	0.0	99.8	99.5	99.9	57.5	72.8	69.0	81.9	86.8	88.2
qwen3.6_27b	99.2	99.8	100.0	37.2	35.8	29.8	0.0	0.0	0.0	99.9	99.6	99.0	100.0	99.3	96.5	95.5	81.9	88.6

Appendix

Table 6: Shot-mode ablation (thinking mode where supported): EMA (%) across 0-, 1-, and 4-shot settings (condition=all). Bold = best shot per model

\times

task.

Model	count_chars	edit_distance	letter_container	longest_string_finder	palindrome_test
deepseek_r1_32b	$0.984$ almost_perfect	$0.556$ moderate	$0.988$ almost_perfect	$0.997$ almost_perfect	$0.7$ substantial
gemma4_31b	$0.988$ almost_perfect	$0.743$ substantial	$0.985$ almost_perfect	$0.985$ almost_perfect	$0.865$ almost_perfect
gpt-oss_20b	$0.991$ almost_perfect	$0.941$ almost_perfect	$0.995$ almost_perfect	$0.985$ almost_perfect	$0.988$ almost_perfect
nemotron-cascade-2	$0.988$ almost_perfect	$0.623$ substantial	$0.998$ almost_perfect	$0.689$ substantial	$0.765$ substantial
qwen3.6_27b	$1.0$ almost_perfect	$0.563$ moderate	$0.98$ almost_perfect	$0.965$ almost_perfect	$0.772$ substantial

Appendix

Table 7: Cohen’s

\kappa

(thinking mode where supported) (4-shot, condition=all). Annotation uses headline

\kappa

(linear weighted for integer tasks).

	deepseek_r1_32b	gemma4_31b	gpt-oss_20b	nemotron-cascade-2	qwen3.6_27b
deepseek_r1_32b	—	$5/18$	$9/18$	$7/18$	$4/18$
gemma4_31b	$5/18$	—	$12/18$	$7/18$	$8/18$
gpt-oss_20b	$9/18$	$12/18$	—	$12/18$	$10/18$
nemotron-cascade-2	$7/18$	$7/18$	$12/18$	—	$8/18$
qwen3.6_27b	$4/18$	$8/18$	$10/18$	$8/18$	—

Appendix

Table 8: McNemar pairwise comparisons (thinking mode where supported): fraction of Bonferroni-significant task×shot cells (numerator/denominator). Blue tint: row model tends to win net; red: column model; gray: mixed.

	deepseek_r1_32b	gemma4_31b	qwen3.5_27b	qwen3.5_2b	qwen3.5_9b	qwen3.6_27b
deepseek_r1_32b	—	$9/18$	$9/18$	$11/18$	$12/18$	$9/18$
gemma4_31b	$9/18$	—	$6/18$	$15/18$	$8/18$	$5/18$
qwen3.5_27b	$9/18$	$6/18$	—	$15/18$	$12/18$	$3/18$
qwen3.5_2b	$11/18$	$15/18$	$15/18$	—	$14/18$	$14/18$
qwen3.5_9b	$12/18$	$8/18$	$12/18$	$14/18$	—	$9/18$
qwen3.6_27b	$9/18$	$5/18$	$3/18$	$14/18$	$9/18$	—

Appendix

Table 9: McNemar pairwise comparisons (non-thinking mode): fraction of Bonferroni-significant task×shot cells (numerator/denominator). Blue tint: row model tends to win net; red: column model; gray: mixed.

Model	count_chars	edit_distance	index_to_span	letter_container	longest_string_finder	palindrome_test
deepseek_r1_32b	$\tau{=}-0.333$ valley_1shot	$\tau{=}-0.333$ valley_1shot	$\tau{=}-0.333$ valley_1shot	$\tau{=}0.333$ valley_1shot	$\tau{=}0.0$ valley_1shot	$\tau{=}-1.0$ monotone_down
gemma4_31b	$\tau{=}1.0$ monotone_up	$\tau{=}0.333$ valley_1shot	–	$\tau{=}-0.333$ valley_1shot	$\tau{=}1.0$ monotone_up	$\tau{=}1.0$ monotone_up
gpt-oss_20b	$\tau{=}0.333$ peak_1shot	$\tau{=}1.0$ monotone_up	$\tau{=}-0.333$ valley_1shot	$\tau{=}0.333$ valley_1shot	$\tau{=}-0.816$ monotone_down	$\tau{=}1.0$ monotone_up
nemotron-cascade-2	$\tau{=}0.816$ monotone_up	$\tau{=}0.333$ valley_1shot	–	$\tau{=}0.333$ valley_1shot	$\tau{=}0.333$ peak_1shot	$\tau{=}1.0$ monotone_up
qwen3.6_27b	$\tau{=}1.0$ monotone_up	$\tau{=}-1.0$ monotone_down	–	$\tau{=}-1.0$ monotone_down	$\tau{=}-1.0$ monotone_down	$\tau{=}-0.333$ valley_1shot

Appendix

Table 10: Kendall’s

\tau

(thinking mode where supported) between shot count

\{0,1,4\}

and EMA (condition=all). Low power at

n{=}3

points — interpret patterns qualitatively.

	count_chars	edit_distance	index_to_span	letter_container	longest_string_finder	palindrome_test
deepseek_r1_32b	$-7.5*$	$-6.0$	$-0.5$	$-1.2$	$-0.5$	$-3.5$
gemma4_31b	$-3.5$	$+10.0$	$0.0$	$-1.0$	$-3.0$	$-2.5$
gpt-oss_20b	$-5.5$	$-6.0$	$+2.5$	$-0.5$	$-2.0$	$+0.8$
nemotron-cascade-2	$-1.5$	$-16.5$	$0.0$	$+0.2$	$-5.0$	$-9.5**$
qwen3.6_27b	$0.0$	$+3.5$	$0.0$	$-2.0$	$-3.0$	$-1.2$

Appendix

Table 11: English vs. random gap (thinking mode where supported) (

\Delta

EMA = EMA

{}_{\text{eng}}

-

EMA

{}_{\text{rand}}

, %) at 4-shot. Blue = English advantage; Red = random advantage. Significance (Bonferroni-corrected):

{}^{*}p{<}0.05

{}^{**}p{<}0.01

{}^{***}p{<}0.001

	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot	0-shot	1-shot	4-shot
count_chars			edit_distance			index_to_span			letter_container			longest_string_finder			palindrome_test
deepseek_r1_32b	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.5 ( 0.0 ; 0.5 )	0.2 ( 0.0 ; 0.2 )	0.5 ( 0.0 ; 0.5 )	0.2 ( 0.0 ; 0.2 )	0.2 ( 0.0 ; 0.2 )	0.5 ( 0.0 ; 0.5 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.2 ( 0.0 ; 0.2 )	0.0 ( 0.0 ; 0.0 )	0.2 ( 0.0 ; 0.2 )	3.5 ( 0.0 ; 3.5 )	4.5 ( 0.0 ; 4.5 )	14.8 ( 0.0 ; 14.8 )
gemma4_31b	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )
gpt-oss_20b	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	2.1 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	3.7 ( 0.0 ; 3.7 )	17.2 ( 0.0 ; 17.2 )	21.2 ( 0.0 ; 21.2 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.2 ( 0.0 ; 0.0 )	0.5 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )
nemotron-cascade-2	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	2.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.8 ( 0.0 ; 0.0 )	0.3 ( 0.0 ; 0.3 )	2.2 ( 0.0 ; 2.2 )	2.2 ( 0.0 ; 2.2 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	15.2 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	1.2 ( 0.0 ; 0.0 )	4.4 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.4 ( 0.0 ; 0.0 )
qwen3.6_27b	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )	0.0 ( 0.0 ; 0.0 )

Appendix

Table 12: Parse error rate (PER, %) across models, tasks, and shot settings (thinking mode where supported) (condition=all). Format: PER ( blank ; inf.err ).

Model	Cond.	Mean $\pm$ Std	P95	$n$
deepseek_r1_32b	english	$0.03\pm 0.39$	0.05	200
	random	$0.00\pm 0.00$	0.00	200
	all	$0.01\pm 0.27$	0.00	400
gemma4_31b	english	$0.06\pm 0.23$	1.00	200
	random	$0.01\pm 0.14$	0.00	200
	all	$0.03\pm 0.19$	0.00	400

Appendix

Table 13: Signed delta distribution (

\hat{y}-y

) for count_chars (thinking mode where supported) at 4-shot (valid responses only).

Model	Cond.	Mean $\pm$ Std	Median	P5	P95	$n$
deepseek_r1_32b	english	$0.00\pm 1.59$	0.00	-2.00	3.00	199
	random	$-0.07\pm 2.15$	0.00	-3.00	3.10	199
	all	$-0.04\pm 1.89$	0.00	-2.15	3.00	398
gemma4_31b	english	$0.01\pm 0.92$	0.00	-1.00	1.05	200
	random	$-0.66\pm 0.73$	-1.00	-2.00	0.00	200
	all	$-0.33\pm 0.90$	0.00	-2.00	1.00	400

Appendix

Table 14: Signed delta distribution (

\hat{y}-y

) for edit_distance (thinking mode where supported) at 4-shot (valid responses only).

Task	Model	EMA	Mean tokens	EMA/token
count_chars	deepseek_r1_32b	96.2%	274	$0.00351$
count_chars	gemma4_31b	96.2%	0	–
count_chars	gpt-oss_20b	97.2%	–	–
edit_distance	deepseek_r1_32b	37.5%	1555	$0.00024$
edit_distance	gemma4_31b	39.5%	0	–
edit_distance	gpt-oss_20b	89.0%	0	–

Appendix

Table 15: Thinking-cost efficiency at 4-shot (thinking mode, condition=all). Top-3 per task by EMA / mean thinking tokens (higher is better). NaN tokens (no paired runs) shown as “–”.

Task	Model A	Model B	$h$ [95% CI]	Risk diff.	$p$	$q$
edit_distance	gpt-oss_20b	qwen3.6_27b	$1.31$ [1.18, 1.45]	$0.593$	$0.0$	$0.0$
edit_distance	gpt-oss_20b	nemotron-cascade-2	$1.22$ [1.09, 1.36]	$0.547$	$0.0$	$0.0$
edit_distance	deepseek_r1_32b	gpt-oss_20b	$-1.15$ [-1.29, -1.01]	$-0.515$	$0.0$	$0.0$
edit_distance	gemma4_31b	gpt-oss_20b	$-1.11$ [-1.25, -0.97]	$-0.495$	$0.0$	$0.0$
longest_string_finder	deepseek_r1_32b	nemotron-cascade-2	$1.08$ [0.95, 1.26]	$0.308$	$0.0$	$0.0$
longest_string_finder	gemma4_31b	nemotron-cascade-2	$0.94$ [0.8, 1.09]	$0.295$	$0.0$	$0.0$

Appendix

Table 16: Top-12 model-pair effect sizes at 4-shot (thinking, condition=all), sorted by

|h|

. CIs are 95% bootstrap percentile.

Rank	Model	Macro EMA [95% CI]	VRR	PER
1	gpt-oss_20b	$0.818$ [0.512, 0.989]	$0.964$	$0.036$
2	gemma4_31b	$0.712$ [0.388, 0.971]	$1.0$	$0.0$
3	deepseek_r1_32b	$0.697$ [0.385, 0.966]	$0.973$	$0.027$
4	qwen3.6_27b	$0.69$ [0.358, 0.972]	$1.0$	$0.0$
5	nemotron-cascade-2	$0.648$ [0.339, 0.919]	$0.992$	$0.008$

Appendix

Table 17: Composite leaderboard at 4-shot (thinking, condition=all). Macro EMA averages over complete task-level EMAs and omits a task when either input condition is missing; CI is 95% bootstrap percentile of those task-level EMAs.

Rank	Model	Macro EMA [95% CI]	VRR	PER
1	qwen3.5_27b	$0.713$ [0.394, 0.966]	$1.0$	$0.0$
2	gemma4_31b	$0.709$ [0.385, 0.971]	$1.0$	$0.0$
3	qwen3.6_27b	$0.703$ [0.376, 0.975]	$1.0$	$0.0$
4	qwen3.5_9b	$0.666$ [0.361, 0.915]	$0.999$	$0.001$
5	qwen3.5_2b	$0.62$ [0.419, 0.804]	$1.0$	$0.0$
6	deepseek_r1_32b	$0.515$ [0.163, 0.83]	$0.623$	$0.377$

Appendix

Table 18: Composite leaderboard at 4-shot (non-thinking mode). Macro EMA averages over complete task-level EMAs and omits a task when either input condition is missing; CI is 95% bootstrap percentile of those task-level EMAs.

Figure 4: English vs. random EMA gap per (model, mode, task) at 4-shot, with 95% bootstrap CIs. After Bonferroni correction over 197 complete comparisons, 25 cells (12.7%) show a significant gap; random inputs are easier in 23 of them (§ 4.2 ).

Figure 5: Shot-mode learning curves per (model, mode, task, condition). Reading horizontally shows the per-(model, mode) trajectory across 0/1/4-shot; descending curves correspond to the few-shot interference effect on saturated tasks (§ 4.4 ).

Figure 6: Cohen’s $\kappa$ heatmap per (model, mode, task, shot). Cells with $\kappa<0$ indicate systematic anti-agreement (rare in the post-parser-fix data). The thinking-mode degradation on Qwen3.6 palindrome_test (§ 4.3 ) is visible as the lower $\kappa$ in the thinking row vs. the non-thinking row at 1- and 4-shot.

Figure 7: Kendall-

\tau

sparkline grid over shot mode per (model, mode, task, condition). Trajectory classification ( monotone_up , monotone_down , peak_1shot , valley_1shot , flat ) is shown by the cell color and the embedded

\tau

value. Sign-flips between thinking and non-thinking variants of the same model on the same task are the main signal underlying § 4.4 .

Figure 8: McNemar pairwise significance summary. For each ordered pair of models, the proportion of combined-condition task $\times$ shot cells where the row model significantly outperforms the column model after Bonferroni correction within that pair. Thinking and non-thinking pairs are summarized separately in § 4.6 .

Figure 9: Parse error rates by (model, mode, task, shot). Most models have PER $\leq 0.04$ at 4-shot in any mode; cells with elevated PER appear in the appendix table 12 .

Figure 10: Point-biserial correlation between input condition and correctness. Cells indicate the strength and direction of the English-vs-random effect per (model, mode, task, shot). Magnitudes are uniformly small ( $|r|<0.30$ ). On the cells that survive Bonferroni correction, random is easier in 23 of 25 (§ 4.2 ).

Figure 11: Cumulative accuracy as a function of allowed delta tolerance on integer-output tasks at 4-shot. The gap between

x=0

(exact match) and

x=1

(within

\pm 1

) is the off-by-one error rate. For edit_distance , this gap is large for all models except GPT-OSS-20B, indicating models reason near the correct edit distance but rarely at it.

Figure 12: Per-(model, mode) capability profile. Radar chart showing 4-shot EMA on each task. The asymmetry between the four near-saturated tasks and edit_distance / index_to_span (the discrimination and floor tasks) is visible as the radial spread.

Figure 13: Thinking-cost Pareto frontier. EMA versus mean thinking-token count per (model, task) at 4-shot in thinking mode. Models above-and-left of the Pareto curve are accuracy-efficient; the implications for deployment cost trade-offs are discussed in § 4.3 .

Figure 14: Tolerance curves for integer tasks. Reading from

x=0

(exact match) to

x=k

(within

\pm k

) shows the cumulative accuracy as the tolerance widens. The shape distinguishes models whose errors are off-by-one from those whose errors are arbitrary.

Figure 15: Composite leaderboard Pareto frontier. Macro EMA versus VRR (valid-response rate) per (model, mode) at 4-shot. The composite leaderboard in § 4.1 , Table 2 , ranks models on Macro EMA; this figure visualizes the alternative trade-off where parse-error-tolerant models can be preferred under a deployment budget.

Figure 16: Thinking-mode pairwise dominance. Each cell is the fraction of combined-condition task $\times$ shot cells on which the row model significantly beats the column model under the per-cell Bonferroni correction. Thinking and non-thinking comparisons are not pooled, and this matrix does not define a cross-mode ordering.

Figure 17: Non-thinking pairwise dominance. Same definition as Figure 16 , restricted to non-thinking runs. Qwen3.5 appears only in this figure; GPT-OSS-20B and Nemotron-Cascade-2 appear only in the thinking-mode figure.

Tokenization in Language Models

See all 43 papers

Mar 16, 2026

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.

Yuzhe Tang

Georgia Institute of Technology

Language Model Reasoning Evaluation LLM Evaluation Implicit Reasoning in Language Models Reasoning Consistency in Language Models

Ask PDF

Apr 13, 2026

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

Should we trust Large Language Models (LLMs) with high accuracy? LLMs achieve high accuracy on reasoning benchmarks, but correctness alone does not reveal the quality of the reasoning used to produce it. This highlights a fundamental limitation of outcome-based evaluation: models may arrive at correct answers through flawed reasoning, and models with substantially different reasoning capabilities can nevertheless exhibit similar benchmark accuracy, for example due to memorization or over-optimization. In this paper, we ask: given existing benchmarks, can we move beyond outcome-based evaluation to assess the quality of reasoning itself? We seek metrics that (1) differentiate models with similar accuracy and (2) are robust to variations in input prompts and generation configurations. To this end, we propose a reasoning score that evaluates reasoning traces along dimensions such as faithfulness, coherence, utility, and factuality. A remaining question is how to aggregate this score across multiple sampled traces. Naively averaging them is undesirable, particularly in long-horizon settings, where the number of possible trajectories grows rapidly, and low-confidence correct traces are more likely to be coincidental. To address this, we introduce the Filtered Reasoning Score (FRS), which computes reasoning quality using only the top-K% most confident traces. Evaluating with FRS, models that are indistinguishable under standard accuracy exhibit significant differences in reasoning quality. Moreover, models with higher FRS on one benchmark tend to perform better on other reasoning benchmarks, in both accuracy and reasoning quality. Together, these findings suggest that FRS complements accuracy by capturing a model's transferable reasoning capabilities. We open source our evaluation codebase: https://github.com/HumainLab/filtered_reasoning_score_evaluation.

Manas Pathak, Xingyao Chen, Shuozhe Li +2

University of Texas at Austin

LLM Evaluation Language Model Generation Evaluation Reasoning Evaluation Reasoning Trace Analysis Language Model Reasoning Evaluation

Ask PDF

Jun 5, 2026

mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

We introduce mmPISA-bench, a compact high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA). The benchmark consists of 25 multiple-choice questions that require reasoning in order to be answered correctly. Each question is provided in official human translations to 43 languages and complemented with machine-translated counterparts (i.e., 2,150 data points in total). We evaluate two mainstream proprietary LLMs across languages, reasoning effort levels, and translation types in terms of their ability to answer the questions correctly. Our results show that modern LLMs can reason effectively across all evaluated languages, achieve accuracy comparable to human test-takers, with some performance variations across covered languages. We further find that machine-translated questions do not degrade accuracy relative to official human translations which suggests that high-quality machine translation (synthetic data) might often be adequate for large-scale multilingual reasoning evaluations where official translations are not available. Finally, we analyze token usage and related inference cost and find that LLMs usage in some languages is simultaneously more expensive and less accurate.

Yerzhan Sapenov, Jaromir Savelka

School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA

Multilingual Language Models Multilingual Language Model Evaluation Educational Assessment Cross-Lingual Reasoning in Language Models Multilingual QA

Ask PDF

Abstract

Tokenization in Language Models

Explore similar work

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?