Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
Figures & tables
Figure 1: (a) Motivation: When pseudo-labels are wrong, most disagreeing rollouts are genuinely wrong too, during Qwen3-1.7B TTT on AIME 2026. (b) Overview of TTPO . (c) Performance: Average accuracy of Qwen3-1.7B across AIME 2026, HMMT 2026, and BRUMO 2025.
Figure 2: Overview of TTPO. (1) Majority Voting: K sampled trajectories are partitioned into positive ( ak=a^ ) and negative ( ak=a^ ) sets. (2.1) OPSD Branch: positive samples are supervised via per-token forward KL with token weighting. (2.2) GRPO Branch: negative samples are penalized via group advantages with token masking on anomalous positions.
Method
AIME25
HMMT25
AIME26
HMMT26
BRUMO25
Average
Qwen3-1.7B
Base
36.9
21.9
37.8
28.8
47.5
34.6
+GRPO †
37.3
23.6
40.3
29.3
48.1
35.7
+OPSD †
40.3
28.1
46.4
31.4
52.5
39.7
+TTPO
41.7
26.1
46.5
31.6
54.7
40.1
Qwen3-4B
Table 1: Results on OpenThoughts training data. OPSD and GRPO use ground-truth labels ( † ); TTPO uses only majority-vote pseudo-labels.
Method
AIME26
HMMT26
BRUMO25
Average
Qwen3-1.7B
Base
37.8
28.8
47.5
38.0
+TTRL
39.2
30.6
50.9
40.2
+OPSD-TTT
44.7
30.3
50.8
41.9
+TTPO
48.9
33.6
53.1
45.2
Qwen3-4B
Table 2: Results on test-time training data. No method uses ground-truth labels.
Method
AIME26
HMMT26
BRUMO25
TTPO
46.5
31.6
54.7
w/o pos. weight
43.3
30.6
52.8
w/o neg. mask
45.4
29.5
50.0
Table 3: Token-level selection ablation results on Qwen3-1.7B OpenThoughts setting.
Figure 3: Ablations over update strategies on Qwen3-1.7B AIME26 TTT setting. P and N denote the loss applied to positive and negative samples, respectively. Dotted lines indicate the corresponding variants with an unconditioned teacher.
Privilege
TM-on Teacher
TM-off Teacher
None
45.8 (36.1)
41.7 (8.1)
Answer
46.5 ( 39.8 )
33.6 (6.7)
Trajectory
41.1 (8.9)
40.8 (10.6)
Table 4: Effect of privileged information type and teacher thinking mode (Qwen3-1.7B, OpenThoughts, AIME26). Columns: whether teacher uses thinking mode. Rows: privileged information injected into teacher prompt. Results in parentheses are evaluated with thinking disabled.
Figure 4: Cross-benchmark generalization (Qwen3-1.7B). Each subplot corresponds to a training benchmark; each group within a subplot shows performance on a different target benchmark.
Figure 5: Left: Comparison between pseudo-label vs. ground-truth supervision. TTPO w/ GT replaces majority-vote pseudo-labels with ground truth while keeping the asymmetric objective and token-level selection intact; OPSD (Leakage) trains standard OPSD directly on AIME26. Right: Entropy during training. Values in parentheses on the x-axis denote TTRL’s training steps.
Figure 6: Avg@12 and Maj@12 during TTPO TTT on HMMT26 (Qwen3-1.7B). Dashed line: base Maj@12.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
GRPO / TTRL
OPSD / OPSD-TTT
TTPO
General
Learning Rate
5×10−6
5×10−6
5×10−6
Max Gradient Norm
–
0.1
0.1
Effective Batch Size
32
32
32
Training Steps
500
100
100
LoRA Configuration
Appendix
Table 5: Training configuration for all methods. GRPO and OPSD use ground-truth labels; TTRL, OPSD-TTT, and TTPO are label-free. “–” indicates not applicable.
Parameter
Value
Thinking Mode
Enabled
Samples per Prompt
12
Temperature
1.0
Top- p
0.95
Max New Tokens
38,912
Metric
Avg@12
Appendix
Table 6: Evaluation configuration.
Figure 7: Training loss curves.
Method
AIME25
HMMT25
AIME26
HMMT26
BRUMO25
Average
Qwen3-1.7B
Base
9.2
5.6
8.8
6.1
17.8
9.5
+OPSD †
16.9
9.2
19.4
11.9
25.6
16.6
+TTPO
39.2
20.6
39.8
26.3
47.5
34.7
Qwen3-4B
Base
22.2
12.5
19.4
17.2
28.3
19.9
Appendix
Table 7: Non-thinking evaluation on OpenThoughts training data. Models are evaluated with thinking mode disabled . Both OPSD and TTPO are trained with a thinking-mode-on teacher.
Figure 8: Left: Training loss curves for the OPSD and GRPO branches (unweighted). Right: Positive sample fraction over training steps. (Qwen3-1.7B, OpenThoughts).
λ
AIME26
HMMT26
BRUMO25
0.01
41.4
28.8
51.1
0.05
43.9
29.0
50.6
0.10
46.5
31.6
54.7
0.15
44.7
29.5
50.3
0.20
41.7
26.3
51.6
Appendix
Table 8: Ablation on RL weight λ (Qwen3-1.7B, OpenThoughts).
Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.
Test-time training (TTT) adapts model parameters on unlabeled test instances during inference time, which continuously extends capabilities beyond the reach of offline training. Despite initial gains, existing TTT methods for LRMs plateau quickly and do not benefit from additional test-time compute. Without external calibration, the self-generated reward signal increasingly drifts as the policy model evolves, leading to both performance plateaus and diversity collapse. We propose TEMPO, a TTT framework that interleaves policy refinement on unlabeled questions with periodic critic recalibration on a labeled dataset. By formalizing this alternating procedure through the Expectation-Maximization (EM) algorithm, we reveal that prior methods can be interpreted as incomplete variants that omit the crucial recalibration step. Reintroducing this step tightens the evidence lower bound (ELBO) and enables sustained improvement. Across diverse model families (Qwen3 and OLMO3) and reasoning tasks, TEMPO improves OLMO3-7B on AIME 2024 from 33.0% to 51.1% and Qwen3-14B from 42.3% to 65.8%, while maintaining high diversity.
Qingyang Zhang, Xinke Kong, Haitao Wu +7
Tianjin University · Tongyi Lab, Alibaba Group · The Chinese University of Hong Kong +1
Test-time training (TTT) adapts large language models (LLMs) during inference using only unlabeled test inputs. Existing methods, however, face two major bottlenecks on hard reasoning tasks: (1) lack of learnable samples, as self-generated pseudo-labels on difficult questions are often noisy and yield unstable rewards; and (2) inefficient exploration, as performance gains depend on repeatedly sampling many rollouts without explicit diagnosis of why previous attempts fail. We propose TTSR (Test-Time Self-Reflection), a self-evolving framework based on a reflect-then-synthesize paradigm. A single pretrained model alternates between a Student role and a Teacher role: the Student solves test questions and updates, while the Teacher analyzes failed trajectories and synthesizes targeted variant questions closer to the Student's capability frontier. TTSR further maintains a cross-iteration weakness memory and compiles persistent weaknesses into a lightweight strategy note prepended to subsequent Student inputs, so diagnostic knowledge can guide exploration and gradually fade as weaknesses are resolved. Experiments on challenging mathematical reasoning benchmarks show consistent test-time improvements, strong cross-backbone generalization, and transfer to general-domain reasoning tasks.
Haoyang He, Zihua Rong, Yunjia Zhao +3
Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.
Test-time training (TTT) adapts an LLM's parameters using signals derived from test inputs, and can make striking improvements in pre-specified settings such as IMO competitions or designated open problems. By turning deployment experience into parameter updates, TTT provides a direct mechanism for model-level self-improvement. Yet TTT is not universally beneficial: each TTT algorithm works in different settings, and applying an ill-suited method could waste test-time compute or even damage model performance. Therefore, such parameter-level self-improvement requires agency: the model must decide when TTT is warranted, which algorithm to invoke, and whether an existing skill can be reused. To fill this gap, we introduce Agentic-TTT, which learns a test-time policy to govern those decisions. Agentic-TTT turns TTT procedures into callable tools, treats accumulated skills as an evolving deployment environment, and trains its policy using the observed utility gains from its decisions. On our benchmark, Agentic-TTT nearly doubles the utility over the backbone model, learns to trade off utility against compute, and generalizes to domains unseen during training. Together, these results point toward autonomous self-improvement: models that can decide how to learn from their own deployment experience.