Does the level of test-time processing effort affect a language model's computational capability? A formal grammar-lens benchmark method, derived from the theory of computation, generates novel data at specified levels, each successively more computationally powerful than the levels below. Using this data, we tested GPT-5.5, and GPT-5.5-Pro-Medium, -High, and -XHigh, showing incrementally increased acceptance of test data: 16, 40, 43, and 52%, respectively, illustrating the benchmark's ability to discern fine-grained differences among models' added test-time reasoning levels. Additionally, as test-time compute effort increases, model responses diverge from human responses to the same data. This increasing disagreement may turn out to be broadly characteristic of more capable language models, introducing the potential of a general capability-versus-alignment conflict.
Figures & tables
Figure 1: The grammar–automata hierarchy equivalence classes ( left ) with formal grammars (brown) and automata (black) in order of increasing computational power, from finite state machines (FSM) to Turing machines (TM). Single-stack mechanisms (green) are distinguished from higher order of stacks (yellow) and from tape-based automata (red). ( right ) External memory associated with each automata class. (From Graham and Granger Graham and Granger (2026) )
Figure 2: Acceptance rate by grammar generator and by reasoning-effort level, with color scale reflecting average elapsed query time. IXG-generated data acceptance is at ceiling. CSG acceptance scales monotonically with reasoning effort. Average query time (colors) also scales monotonically, both with grammar class and with reasoning effort.
Grammar
Model/configuration
Trials
Acceptance (%)
Seconds/query
IXG
GPT 5.5
50
96.92±1.29
3.87±0.98
GPT 5.5 Pro (Med)
50
97.84±0.55
14.71±1.44
GPT 5.5 Pro (High)
50
97.91±0.57
20.80±5.02
GPT 5.5 Pro (XHigh)
50
97.64±2.00
50.33±25.53
CSG
GPT 5.5
50
16.08±3.31
7.10±2.27
GPT 5.5 Pro (Med)
50
40.35±4.64
105.21±12.51
Table 1: Final results: means ± SD across repeated trials over the 50-string sets.
Grammar
Comparison
Δ
pt
pW
d
CSG
5.5 → Pro Med
+24.27
1.6×10−33
7.3×10−10
−4.31
5.5 → Pro High
+26.68
1.2×10−37
7.2×10−10
−5.28
5.5 → Pro XHigh
+35.90
4.9×10−34
7.3×10−10
−4.42
Med → High
+2.41
1.1×10−2
1.4×10−2
−0.37
Med → XHigh
+11.63
5.7×10−13
3.5×10−9
−1.37
High → XHigh
+9.22
5.9×10−10
8.6×10−9
−1.09
Table 2: Pairwise acceptance-rate comparisons. Δ represents the percentage-point difference (second − first), pt the paired t -test, pW the Wilcoxon signed-rank, and d denotes Cohen’s d .
Figure 3: A valid parse in terms of the grammar generator Gix of its string generation: Leibniz who invites Riemann who conjectures Galileo who rests solves the mountains that eat or fall.
Figure 4: A valid parse in terms of the grammar generator Gcs of its string generation: The dogs that silently fade a person that deeply writes Laplace conjecture elegantly prove.
Grammar
Comparison
Δ (s)
pt
pW
d
CSG
5.5 → Pro Med
+98.11
1.5×10−45
1.8×10−15
−7.72
5.5 → Pro High
+126.64
2.2×10−37
1.8×10−15
−5.21
5.5 → Pro XHigh
+233.90
6.2×10−27
1.8×10−15
−3.09
Med → High
+28.53
1.7×10−9
1.1×10−7
−1.04
Med → XHigh
+135.79
1.6×10−17
3.6×10−15
−1.84
High → XHigh
+107.26
5.1×10−13
3.6×10−15
−1.37
Appendix
Table 4: Pairwise Query Latency comparisons. Δ represents the mean difference in seconds (second − first), pt the paired t -test, pW the Wilcoxon signed-rank, and d denotes Cohen’s d .
As language model (LM) outputs get more and more natural, it is becoming more difficult than ever to evaluate their quality. Simultaneously, increasing LMs' "thinking" time through scaling test-time compute has proven an effective technique to solve challenging problems in domains such as math and code. This raises a natural question: can an LM's evaluation capability also be improved by spending more test-time compute? To answer this, we investigate employing reasoning models-LMs that natively generate long chain-of-thought reasoning-as evaluators. Specifically, we examine methods to leverage more test-time compute by (1) using reasoning models, and (2) prompting these models to evaluate not only the response as a whole (i.e., outcome evaluation) but also assess each step in the response separately (i.e., process evaluation). In experiments, we observe that the evaluator's performance improves monotonically when generating more reasoning tokens, similar to the trends observed in LM-based generation. Furthermore, we use these more accurate evaluators to rerank multiple generations, and demonstrate that spending more compute at evaluation time can be as effective as using more compute at generation time in improving an LM's problem-solving capability.
Test-time scaling methods implicitly assume that additional computation from the same model provides independent evidence. We show that this assumption fails within a single reasoning model: errors are correlated across repeated samples, across reasoning depth, and in the model's own self-evaluation, so more compute from the same model is not more evidence. We introduce DIA (Disagreement-triggered Independent Arbitration), which treats disagreement between an intermediate-stage and the final-stage prediction as a per-question signal of reasoning instability and resolves it by asking a small panel of heterogeneous models to solve the question independently, without seeing either candidate, and aggregating their answers directly. Across seven benchmarks and two base models, DIA outperforms full chain-of-thought in 12 of 14 settings, including a 7.00% gain on MuSiQue, while activating on only ∼17% of examples and using up to 47.2% fewer tokens than a multi-agent baseline. Ablations show that both independent solving and solver heterogeneity are necessary for these gains.
Ming Shen, Tianjun Yao
Shanghai Jiao Tong University · Shenzhen Loop Area Institute (SLAI)
Large Reasoning Models (LRMs) generate chain-of-thought traces whose length tracks human reaction times across cognitive tasks, but recent debate questions whether this alignment reflects genuine computational structure or surface verbosity. We test whether the alignment varies with inference-time reasoning effort. Across GPT-OSS-20B and GPT-OSS-120B, three effort levels, and six reasoning tasks, within-task and cross-task alignment remain invariant: Bayes Factors lean toward the null, and mean alignment is numerically near-identical across conditions. A manipulation check reveals that the effort parameter sets an upper budget on generation rather than driving real-time allocation, suggesting that the allocation policy is crystallized at training time. Arithmetic complexity contrasts further show that token allocation tracks fine-grained, format-dependent human difficulty patterns, with model scale improving the match. Cognitive cost alignment between LRMs and humans appears to be a training-time achievement, robust to inference-time perturbations, supporting a compiled rather than online account of LRM problem-solving.
Yueqing Hu, Tianhong Wang
Institute of Neuroscience, Chinese Academy of Sciences, Shanghai, China · School of Philosophy, Anhui University, Hefei, China