Authors: Zhiren Gong, Yikun Hou, Zihao Zeng, Ming Xiao, Chau Yuen, Wei Yang Bryan Lim
Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · Information Science & Engineering, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
Test-time compute has emerged as a major approach to improving the capabilities of Large Language Models (LLMs). However, existing test-time reasoning paradigms rely heavily on externally imposed control, either through fixed reasoning programs or through costly expansion in constrained search spaces, limiting both generalization and efficiency. We propose State of Thought (SoT), a new reasoning paradigm that enables endogenous reasoning in LLMs, with the model's internal reasoning state governing how reasoning unfolds. Concretely, SoT extracts a compact dynamics-geometric state from the model's internal information transfer and uses a 582-parameter controller on frozen backbones to selectively activate historical reasoning support useful under the current reasoning state, framing reasoning as a state-conditioned process over evidence rather than an externally prescribed token chain. Across quantitative (1.34x), general (1.62x), symbolic-and-code (1.76x), and long-context (2.51x) reasoning on 3 LLMs and 16 datasets, SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%. Across 2 VLM scales and 3 reasoning tasks, it improves mean accuracy by 3.8 points over reasoning baselines, with 74.9% fewer completion tokens and 73.5% lower latency than search-based methods. Under constrained access, SoT retains 38.2%/36.5% mean accuracy gains in training-free/embedding-only settings, while trajectory-only judging reaches 84.1% agreement across 3 API models. Together, endogenous state-driven reasoning provides a generalizable and efficient alternative.
Figures & tables
Figure 2: From External Control (Top) to State-of-Thought (Bottom).
Figure 3: Endogenous reasoning state trajectories across various reasoning paradigms on Llama-3.1-8B with breakdown in Figure 8 and reasoning state definition in Figure 4 . Different externally induced reasoning strategies occupy distinguishable regions in the dynamics-geometric state space, while SoT supports a broader and more adaptive state distribution.
Figure 4: State-conditioned reasoning in SoT with breakdown in Figure 9 . Different endogenous reasoning states induce different sparse activation patterns and different stop tendencies.
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Greedy
Vanilla
79.0
58.2
6.2
53.9
29.6
26.0
26.0
25.0
55.2
33.0
Reasoning Paradigm
CoT
80.8
64.2
1.7
55.5
4.9
24.5
70.0
25.0
51.6
31.2
PS
71.2
40.0
1.4
43.4
4.9
15.2
55.0
10.0
22.0
17.6
SR
75.8
44.8
15.6
50.4
32.5
31.0
46.0
25.0
36.4
32.9
SC
83.6
54.0
1.6
53.2
16.8
28.5
52.0
10.0
35.2
27.3
Table 1: Main results on Llama-3.1-8B (accuracy, %).
Figure 5: Accuracy–efficiency trade-off on Llama-3.1-8B with breakdown and other models in Figure 11 .
Table 6
Variant
QS
S&C
GU
LCR
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
Full SoT
62.8
219.4
5.8
54.5
294.0
9.5
71.2
64.0
1.8
33.0
181.6
22.0
+ Threshold Tuning
62.8
219.4
5.8
54.3
294.0
9.5
71.2
64.0
1.8
31.8
229.0
22.0
w/o Evid. Org. S
29.6
469.5
12.9
44.5
536.2
16.5
72.4
193.9
4.9
26.9
506.5
14.1
w/o Stopping T
60.6
1769.6
80.7
57.1
1705.1
110.3
70.2
4148.5
99.5
31.6
2937.4
157.3
w/o Geometry δt
42.5
68.6
3.3
46.4
48.0
2.9
73.0
25.4
0.9
24.5
47.2
2.2
Table 3: Ablation results on Llama-3.1-8B, with other models provided in Tables 20 , 21 .
Figure 7: A case study of SoT, with more case studies in Appendix Q .
Quantitative Reasoning
Symbolic and Code
GSM8K
MATH
DROP
avg.
FOLIO
PW
BBH-T
HE
MBPP
avg.
Top-3 Baseline avg.
81.9
61.5
16.2
53.2
40.6
37.6
60.0
30.0
52.0
44.0
SoT-Training-free
81.8
60.5
34.6
59.0
43.4
30.2
77.0
53.0
50.8
50.1
SoT-Embedding
82.8
61.8
32.3
59.0
33.8
30.5
74.0
57.9
52.8
49.8
Table 5: Boundary studies of SoT on Llama-3.1-8B, with other models provided in Tables 22 , 23 .
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Dataset
Task type
Train
Test
Metric
Quantitative Reasoning (QS)
GSM8K
Math QA
—
500
EM
MATH
Competition math
—
400
EM
DROP
Discrete reasoning QA
—
300
F1
General Understanding (GU)
CommonsenseQA (CSQA)
Multiple choice QA
52
400
EM
StrategyQA (StrQA)
Binary QA
52
400
EM
BoolQ
Boolean QA
—
400
EM
Appendix
Table 9: Dataset-level organization, split size, and primary metric.
Parameter
Value
Notes
Context budget
1536
Active evidence budget
Maximum reasoning steps
32
Safety cap only
Observation window
4096
Visible context span
Final answer budget
512
Answer generation cap
Stop threshold
0.5
Default stop rule
Gate threshold
0.5
Default keep rule
Appendix
Table 10: Deployment-side inference hyperparameters of SoT. We report only externally specified quantities used at test time.
Figure 8: Extended state-space trajectories across reasoning paradigms. This figure supplements the main-text trajectory view with additional paradigms and task categories.
Figure 9: Clustered state regimes and operator profiles.
Figure 10: Correct and incorrect trajectory regimes under full-state and embedding-trajectory SoT. This figure compares successful and unsuccessful reasoning trajectories across all four task categories, under both the internal-state SoT and SoT-Embed.
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
94.4
77.5
1.9
65.6
0.0
55.5
84.0
75.0
66.0
49.9
PS
88.0
51.5
2.1
54.4
0.5
65.5
92.0
50.0
47.2
49.6
SR
92.0
68.2
17.6
65.5
61.1
83.4
79.0
60.0
50.4
68.1
SC
92.8
61.0
1.9
59.5
0.5
59.5
87.0
55.0
48.8
47.2
CB
60.0
41.2
30.8
46.4
62.1
85.0
51.0
50.0
60.8
69.8
Appendix
Table 12: Main results on Qwen2.5-14B (accuracy, %).
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
64.8
41.5
9.9
43.3
23.2
43.0
47.0
20.0
40.4
38.1
PS
60.6
35.8
10.6
39.8
12.8
38.2
52.0
25.0
42.4
35.1
SR
49.2
37.8
9.7
35.5
27.6
54.5
51.0
15.0
32.0
41.9
SC
69.8
43.0
9.8
45.9
25.6
45.2
56.0
20.0
37.2
39.7
CB
50.0
30.2
10.1
33.4
48.3
60.5
39.0
10.0
34.4
48.0
Appendix
Table 13: Main results on Mixtral-8x7B (accuracy, %).
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
230.1
381.5
194.1
271.6
342.9
332.0
264.4
490.1
415.4
369.8
PS
311.0
358.9
259.9
314.2
369.9
354.4
346.0
404.6
391.4
372.1
SR
346.5
441.6
307.6
368.5
337.8
330.4
336.4
517.6
473.0
391.7
SC
678.4
928.6
578.2
736.8
913.0
875.4
798.1
1135.3
1056.2
953.9
CB
441.8
521.0
264.5
423.9
325.8
324.6
290.0
424.8
447.3
363.9
Appendix
Table 14: Generated reasoning tokens on Llama-3.1-8B.
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
12.1
25.6
15.7
17.5
24.0
23.9
18.9
41.3
30.4
27.5
PS
18.7
22.1
22.3
20.7
26.2
24.4
27.7
32.3
27.3
26.8
SR
21.6
30.3
27.2
25.9
22.6
21.6
26.2
45.1
35.5
28.8
SC
34.8
51.8
45.5
43.1
58.9
55.7
57.4
85.1
68.1
63.5
CB
20.9
25.6
19.2
22.0
16.5
16.2
17.9
26.3
23.0
19.4
Appendix
Table 15: End-to-end latency on Llama-3.1-8B.
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
399.4
521.3
544.5
476.3
503.3
416.7
456.0
644.4
597.8
490.0
PS
653.4
698.4
1021.3
760.4
639.7
633.2
896.3
935.2
829.9
718.3
SR
1326.7
1454.9
1804.9
1489.0
1496.6
1528.6
1601.1
1891.5
1717.5
1625.0
SC
1188.4
1366.3
1646.3
1362.2
1430.4
1227.5
1433.2
1798.6
1614.9
1402.2
CB
3868.3
3866.0
4187.8
3947.4
3846.2
3823.3
4166.9
3991.8
4045.2
3923.9
Appendix
Table 16: Generated reasoning tokens on Qwen2.5-14B.
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
25.0
47.4
26.6
32.9
39.2
25.8
23.0
60.7
56.7
36.9
PS
27.9
36.7
32.2
31.9
22.4
19.6
38.0
50.1
46.1
29.5
SR
37.8
56.3
43.5
45.4
41.1
49.8
39.6
87.1
80.4
59.6
SC
73.0
100.4
81.7
84.3
107.6
72.8
79.9
149.6
129.4
96.9
CB
40.6
43.1
39.0
41.0
35.0
33.0
40.1
49.8
52.6
39.5
Appendix
Table 17: End-to-end latency on Qwen2.5-14B.
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
348.8
439.0
534.7
425.3
394.8
353.3
471.7
617.2
574.4
436.3
PS
665.9
703.1
1094.4
785.4
801.5
728.8
944.9
1000.4
868.1
807.6
SR
1222.4
1336.4
1779.2
1399.6
1403.3
1283.2
1782.4
1970.0
1709.8
1483.3
SC
1038.6
1205.6
1612.7
1237.8
1168.6
1070.2
1408.3
1817.2
1588.2
1273.9
CB
3872.7
3857.2
4205.3
3950.7
3913.1
3867.3
4130.2
4221.4
4093.7
3969.3
Appendix
Table 18: Generated reasoning tokens on Mixtral-8x7B.
Category
Method
Quantitative Reasoning
Symbolic and Code
GSM
MATH
DROP
avg.
FOL
PW
BBH
HE
MBPP
avg.
Reasoning Paradigm
CoT
26.7
40.5
18.0
29.1
28.6
19.3
27.0
49.7
46.7
29.7
PS
31.2
41.1
23.8
32.6
36.3
25.1
37.1
46.7
42.7
33.6
SR
44.6
52.2
29.9
43.5
51.7
39.0
54.1
72.8
64.1
50.4
SC
76.4
130.4
56.5
89.4
70.8
66.0
72.4
132.9
123.9
83.9
CB
63.2
65.4
35.5
57.0
56.1
53.8
52.1
71.5
72.0
59.1
Appendix
Table 19: End-to-end latency on Mixtral-8x7B.
Figure 11: Extended accuracy–efficiency trade-off plots across backbones.
Variant
QS
S&C
GU
LCR
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
Full SoT
66.1
545.8
16.8
76.8
408.4
19.0
81.3
133.2
5.8
25.2
254.0
23.4
+ Threshold Tuning
66.1
545.8
16.8
76.8
408.4
19.0
81.3
133.2
5.8
25.2
254.0
23.4
w/o Evid. Org. S
30.0
532.1
21.5
28.4
556.1
21.7
72.4
234.7
9.4
26.5
489.4
14.3
w/o Stopping T
76.0
1401.9
37.4
32.4
2326.2
59.6
70.2
4175.0
100.9
31.7
2815.8
72.8
w/o Geometry δt
56.5
77.9
3.9
23.1
45.8
2.5
73.0
25.4
0.9
21.2
48.2
2.2
Appendix
Table 20: Additional ablation results on Qwen2.5-14B.
Variant
QS
S&C
GU
LCR
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
Full SoT
46.3
248.2
43.8
51.7
329.8
62.6
72.5
143.6
28.0
23.5
260.6
62.8
+ Threshold Tuning
46.3
248.2
43.8
51.7
329.8
62.6
72.5
143.6
28.0
23.5
260.6
62.8
w/o Evid. Org. S
31.3
995.0
70.4
36.0
1002.6
65.8
63.2
836.8
51.0
3.0
921.1
90.7
w/o Stopping T
31.3
995.0
64.0
36.0
1002.6
65.4
63.2
836.8
50.7
3.0
921.1
90.6
w/o Geometry δt
34.0
58.1
11.8
30.7
56.9
13.3
57.6
50.8
7.6
3.3
54.0
13.8
Appendix
Table 21: Additional ablation results on Mixtral-8 × 7B.
Quantitative Reasoning
Symbolic and Code
GSM8K
MATH
DROP
avg.
FOLIO
PW
BBH-T
HE
MBPP
avg.
Top-3 Baseline avg.
93.6
73.5
22.6
63.2
62.1
85.6
87.7
73.8
63.5
74.5
SoT-Training-free
81.8
60.8
34.6
59.1
49.5
30.2
77.0
53.0
50.4
52.0
SoT-Embedding
82.6
58.3
34.8
58.6
51.2
59.5
71.0
48.8
49.6
56.0
Appendix
Table 22: Boundary studies of SoT on Qwen2.5-14B (accuracy, %).
Quantitative Reasoning
Symbolic and Code
GSM8K
MATH
DROP
avg.
FOLIO
PW
BBH-T
HE
MBPP
avg.
Top-3 Baseline avg.
66.6
43.1
10.2
40.0
42.7
57.4
55.3
21.7
40.4
43.5
SoT-Training-free
75.2
52.2
34.4
53.9
31.3
50.5
63.0
56.1
49.2
50.0
SoT-Embedding
76.0
52.0
36.7
54.9
30.8
50.5
63.0
55.5
49.2
49.8
Appendix
Table 23: Boundary studies of SoT on Mixtral-8x7B (accuracy, %).
Model
Method
Quantitative Reasoning
Symbolic and Code
GSM8K
MATH
DROP
avg.
FOLIO
PW
BBH-T
HE
MBPP
avg.
GPT-5.4
Self-Consistency
92.0
78.0
5.0
58.3
69.0
77.0
86.0
87.0
77.0
79.2
Self-Verification
92.0
78.0
5.0
58.3
62.0
79.0
85.0
87.0
78.0
78.2
LLM-as-a-Judge
92.0
79.0
4.0
58.3
70.0
76.0
86.0
86.0
78.0
79.2
SoT-Judge
85.0
79.0
94.0
86.0
50.0
60.0
77.0
86.0
76.0
69.8
Claude 4.7
Self-Consistency
90.0
78.0
1.0
56.3
65.0
73.0
58.0
94.0
86.0
75.2
Appendix
Table 24: Black-box judging with trajectory-based SoT (dataset breakdown, %).
Trajectory pool
QS
S&C
GU
LCR
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
acc.
tok.
lat.
Full HQ pool
70.8
247.9
6.3
57.1
324.2
11.1
75.5
58.0
1.6
37.3
121.1
97.5
No filter
71.3
931.3
30.0
57.7
1080.8
84.1
49.2
886.0
29.5
21.5
900.9
91.5
Weak filter
71.3
931.3
24.3
57.7
1080.8
34.5
49.2
886.0
17.7
21.3
900.9
90.7
No pipeline
71.3
931.3
26.1
57.7
1080.8
53.4
49.2
886.0
49.3
21.5
900.9
89.3
Training-free
72.3
917.4
26.0
58.3
1078.9
35.0
53.8
906.6
19.7
28.1
872.9
95.8
Appendix
Table 25: Sensitivity to trajectory-pool quality on Llama-3.1-8B .
Figure 12: Case study on Quantitative Reasoning (QS).
Figure 13: Case study on General Understanding (GU).
Figure 14: Case study on Symbolic and Code (S&C).
Figure 15: Case study on Long-Context Reasoning (LCR).
Figure 16: Case study on visual-language reasoning.
Recent work on test-time scaling for large language model (LLM) reasoning typically assumes that allocating more inference-time computation uniformly improves correctness. However, prior studies show that reasoning uncertainty is highly localized: a small subset of low-confidence tokens disproportionately contributes to reasoning errors and unnecessary output expansion. Motivated by this observation, we propose Thinking by Subtraction, a confidence-driven contrastive decoding approach that improves reasoning reliability through targeted token-level intervention. Our method, Confidence-Driven Contrastive Decoding, detects low-confidence tokens during decoding and intervenes selectively at these positions. It constructs a contrastive reference by replacing high-confidence tokens with minimal placeholders, and refines predictions by subtracting this reference distribution at low-confidence locations. Experiments show that CCD significantly improves accuracy across mathematical reasoning benchmarks while substantially reducing output length, with minimal KV-cache overhead. As a training-free method, CCD enhances reasoning reliability through targeted low-confidence intervention without computational redundancy. Our code will be made available at: https://github.com/bolo-web/CCD.
Lexiang Tang, Weihao Gao, Bingchen Zhao +4
Peking University, Beijing, China · University of Edinburgh, Edinburgh, UK.
Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller.
Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5
Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2
Associative memory has long underpinned the design of sequential models. Beyond recall, humans reason by projecting future states and selecting goal-directed actions, a capability that modern language models increasingly require but do not natively encode. While prior work uses reinforcement learning or test-time training, planning remains external to the model architecture. We formulate reasoning as optimal control and introduce the Test-Time Control (TTC) layer, which performs finite-horizon LQR planning over latent states at inference time, represents a value function within neural architectures, and leverages it as the nested objective to enable planning before prediction. To ensure scalability, we derive a hardware-efficient LQR solver based on a symplectic formulation and implement it as a fused CUDA kernel, enabling parallel execution with minimal overhead. Integrated as an adapter into pretrained LLMs, TTC layers improve mathematical reasoning performance by up to +27.8% on MATH-500 and 2-3x Pass@8 improvements on AMC and AIME, demonstrating that embedding optimal control as an architectural component provides an effective and scalable mechanism for reasoning beyond test-time training.
Peihao Wang, Shan Yang, Xijun Wang +8
Amazon · The University of Texas at Austin · Georgia Institute of Technology +2