Recurrent latent reasoning in large lanugage models (LLMs) scales test-time computation by increasing recurrent depth, naturally framing inference as a problem of deciding when to stop. We identify a previously overlooked limitation of this single-depth view through systematic preliminary analysis: deeper recurrence can lose correct solutions found at shallower depths, while depth-specific success sets remain broadly non-nested even after controlling for marginal accuracy. This suggests that depth determines not only computation cost, but also access to distinct candidate solutions. Motivated by this finding, we propose Depth Portfolio Allocation (DPA), which reformulates recurrent test-time scaling as budgeted allocation across depth-specific candidate sources. DPA supports both fixed portfolios and feedback-guided sequential allocation under a shared inference budget. Experiments on GSM8K, SVAMP, MultiArith, and MBPP show that mixed-depth portfolios expand oracle coverage under matched candidate counts and nominal costs, while feedback-guided allocation generally matches or improves uniform allocation and can reduce computation in high-budget settings. Code and data are avaible at : https://anonymous.4open.science/r/LLM-latent-B7BC/.
Figures & tables
Figure 1: Motivation and framework. (a) Existing single-depth and sampling strategies; (b) preliminary evidence that deeper recurrence does not subsume shallower solutions; (c) DPA allocates a shared budget across depth-specific candidate sources; (d) feedback-guided allocation chooses a depth or stops. States and sets are schematic.
Figure 2: Stability does not ensure correctness. (a) Latent updates and cosine similarity; (b) five answers per depth, with correct answers in green.
Figure 3: Depth preferences and answer diversity. Bars show best-depth frequencies; curves show disagreement across repeats at each depth.
Figure 4: Jaccard distances between depth-specific success sets. Both tasks share a color scale; diagonal entries are zero.
Dataset
Non-nested pairs
ΔJ per 8 steps (95% CI)
MultiArith
77/78
+3.36 pp [0.87, 5.83]
MBPP
78/78
+5.63 pp [3.48, 8.53]
Table 1: Mutually non-nested pairs and depth association after controlling for marginal accuracy.
Figure 5: Accuracy and budget use under shared budget caps. Top: test accuracy for feedback-guided DPA (PPO) and uniform allocation, with static fixed-depth references. Bottom: PPO nominal cost C versus cap B ; shading shows unused budget B−C , with unused steps annotated at B=320 . Detailed same-cap comparisons are discussed in Section 5.3 .
Allocation
Nominal cost
Candidates
MBPP
MultiArith
4×d8
32
4
21.04%
82.50%
4×d16
64
4
31.72%
91.25%
4×d24
96
4
31.72%
90.25%
4×d32
128
4
30.88%
82.75%
d8+d12+d20+d24
64
4
36.23% (+4.51 pp)
93.75% (+2.50 pp)
Table 2: RQ1: Oracle coverage (%) under controlled allocations. (A) Four candidates per configuration; the portfolio and 4×d16 both cost 64. (B) Matched nominal budgets. Parentheses report gains (pp) over 4×d16 in A and the stronger fixed-depth baseline in B.
Method
GSM8K
SVAMP
MBPP
MultiArith
Base / single
32.60%
51.70%
27.80%
75.00%
CoE-R correction
33.00%
51.00%
27.60%
71.25%
CoE-C correction
32.40%
51.60%
28.00%
72.50%
Table 3: Representative DPA results. A: single-trajectory and latent-correction baselines; B: feedback-guided DPA test accuracy at representative caps; C: nominal cost used. Figure 5 and Appendix F give same-cap Uniform comparisons.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task
τu
τv
Low-update intervals
Truncated answers
Switch rate [95% CI]
Non- truncated rate [95% CI]
GSM8K
0.07364
0.002711
323
56
25.08 [14.33, 37.20]
25.82 [15.04, 38.23]
SVAMP
0.05206
0.001355
266
1
9.40 [0.84, 20.20]
9.40 [0.84, 20.20]
MultiArith
0.04950
0.001225
317
3
21.14 [11.91, 31.07]
20.89 [11.65, 30.79]
MBPP
0.03029
0.000459
288
73
9.72 [3.91, 16.39]
10.33 [4.25, 17.31]
Appendix
Table 4: Answer switches under low updates. Rates and confidence intervals are percentages.
Task
Wrong → correct
Correct → wrong
Wrong → other wrong
GSM8K
1.24 [0.00, 3.13]
4.33 [1.57, 7.62]
19.50 [9.54, 31.38]
SVAMP
0.00 [0.00, 0.00]
0.00 [0.00, 0.00]
9.40 [0.84, 20.20]
MultiArith
5.05 [1.02, 9.79]
5.68 [1.27, 10.84]
10.41 [3.54, 18.68]
MBPP
1.04 [0.00, 3.25]
0.35 [0.00, 1.08]
3.82 [0.67, 8.24]
Appendix
Table 5: Correctness transitions under low updates. All low-update intervals form the denominator; values are percentages.
Figure 6: Directional rescue across depths. Rows denote the reference depth and columns the added depth. Color shows the fraction of questions solved only by the added depth.
d
MA single (%)
MA solved questions
MA Oracle@5 (%)
MBPP single (%)
MBPP solved questions
MBPP Oracle@5 (%)
8
50.25
69
86.25
15.16
109
21.80
10
64.00
73
91.25
22.24
143
28.60
12
66.00
74
92.50
24.12
147
29.40
14
69.25
70
87.50
26.36
158
31.60
16
71.75
75
93.75
27.96
161
32.20
18
74.25
72
90.00
29.04
162
32.40
Appendix
Table 6: Full results at thirteen depths. MA: 80-question MultiArith confirmatory pool; MBPP: 500-question grid pool.
Figure 7: State updates and answer switches. Shading shows question-bootstrap 95% intervals; dashed lines indicate training-calibrated thresholds.
Parameter
Setting
Parallel environments / horizon
24 / 22
Epochs per update / minibatch
4 / 128
Discount γ / GAE λ
1 / 0.95
PPO clip ϵ
0.2
Value coefficient / max. gradient norm
0.5 / 0.5
Optimizer
Adam
Appendix
Table 7: Optimization settings for the PPO implementation.
Trial index
Learning rate
Entropy coefficient
Seed
0
1e-04
0.001
13
1
1e-04
0.001
29
2
1e-04
0.01
13
3
1e-04
0.01
29
4
3e-04
0.001
13
5
3e-04
0.001
29
Appendix
Table 8: Historical tuning configurations.
Dataset
Train
Validation
Test
GSM8K
6,473
500
1,319
SVAMP
700
150
150
MBPP
369
90
496
MultiArith
330
90
80
Appendix
Table 9: Dataset splits.
Analysis
Questions
Depths / repeats
State diagnostics
20 per task
8:2:32 / 5 each
Depth preferences
MA 180 / MBPP 500
8:2:32 / 5 each
Success sets
MA 80 / MBPP 500
8:2:32 / 5 each
Budget evaluation
Independent test splits
16,24,32 / up to 5
Appendix
Table 10: Analysis pools and purposes.
Method
B
GSM8K
SVAMP
MBPP
MultiArith
Fixed d16×20
320
43.40%
56.00%
29.50%
86.25%
Fixed d24×20
480
39.80%
54.90%
29.30%
83.75%
Fixed d32×20
640
37.80%
53.90%
29.50%
85.00%
Appendix
Table 11: Accuracy and nominal budgets for twenty candidates at each fixed depth.
Task
B
PPO (%)
PPO cost
Uniform (%)
Uniform cost
Δ (pp)
GSM8K
48
42.03
48.0
41.63
40.0
+0.41
80
43.92
80.0
43.96
72.0
-0.04
120
45.95
114.7
45.57
112.0
+0.38
160
47.01
159.3
46.53
160.0
+0.48
200
47.06
200.0
46.70
184.0
+0.35
240
47.47
232.7
47.07
232.0
+0.39
Appendix
Table 12: Complete test-budget results. Costs are mean recurrent steps; differences are PPO minus Uniform. Values are rounded independently.
B
GSM8K
SVAMP
MBPP
MultiArith
48
48.0 (100.0%)
42.3 (88.1%)
33.1 (69.0%)
48.0 (100.0%)
80
80.0 (100.0%)
80.0 (100.0%)
74.3 (92.9%)
77.4 (96.8%)
120
114.7 (95.6%)
112.0 (93.3%)
115.3 (96.1%)
114.7 (95.6%)
160
159.3 (99.6%)
147.5 (92.2%)
152.4 (95.2%)
152.6 (95.4%)
200
200.0 (100.0%)
198.5 (99.2%)
180.7 (90.3%)
200.0 (100.0%)
240
232.7 (97.0%)
232.5 (96.9%)
238.7 (99.5%)
231.9 (96.6%)
Appendix
Table 13: PPO budget use on independent test splits. Cells report mean nominal recurrent steps C , with budget utilization 100C/B in parentheses. Cap B is the maximum permitted nominal cost, not runtime.
B
n16
n24
n32
Nominal cost
48
1
1
0
40
80
1
1
1
72
120
2
2
1
112
160
3
2
2
160
200
3
3
2
184
240
4
3
3
232
Appendix
Table 14: Uniform allocations in budget evaluation.
Test-time compute scaling, the practice of spending extra computation during inference via repeated sampling, search, or extended reasoning, has become a powerful lever for improving large language model performance. Yet deploying these techniques under finite inference budgets requires a decision that current systems largely ignore: which inputs deserve more compute, and which can be answered cheaply? We formalize this as a constrained optimization problem (maximize expected accuracy subject to an average compute budget) and solve it with a two-stage Solve-then-Learn pipeline. In the solve stage, Lagrangian relaxation decomposes the global constraint into per-instance sub-problems, each admitting a closed-form oracle action that optimally prices accuracy against cost. We prove that the induced cost is monotone in the dual variable, enabling exact budget targeting via binary search. In the learn stage, a lightweight classifier is trained to predict oracle actions from cheap input features, amortizing the allocation rule for real-time deployment. We establish that the task-level regret of the learned policy is bounded by its imitation error times the worst-case per-instance gap, yielding a clean reduction from constrained inference to supervised classification. Experiments on MATH and GSM8K with three LLMs (DeepSeek-V3, GPT-4o-mini, Qwen2.5-7B) show that our method consistently outperforms uniform and heuristic allocation baselines, achieving up to 12.8% relative accuracy improvement on MATH under matched budget constraints, while closely tracking the Lagrangian oracle upper bound with over 91% imitation accuracy.
In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.
Ivan Anokhin, Johan Obando-Ceron, Irina Rish +1
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Sakana AI
Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed. The same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. We first study the effect of increasing N on reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. As expected, increasing N from 1 to 8 improves accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, accuracy alone does not show the systems cost of using a larger candidate budget. We therefore fix N = 8 and compare four generation schedules: 1x8, 2x4, 4x2, and 8x1, where axb denotes a generation calls with b candidates per call. We measure latency, throughput, GPU-hours, and gross GPU-device energy while keeping the total candidate count fixed. On A100 GPUs, eight serial calls use 4.64-4.86x as much gross GPU-device energy and have 5.77-6.12x the P95 latency of one batched call with eight candidates. The same pattern appears across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments. These results show that candidate count alone is not enough to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory allows it, fewer generation calls with larger batch sizes are more efficient. Evaluations should therefore report not only candidate count and accuracy, but also generation schedule and GPU-level systems metrics.
Mobina Kashaniyan, Ali Jannesari
Department of Computer Science Iowa State University, USA