Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
Figures & tables
Figure 1: (a) Motivation: When pseudo-labels are wrong, most disagreeing rollouts are genuinely wrong too, during Qwen3-1.7B TTT on AIME 2026. (b) Overview of TTPO . (c) Performance: Average accuracy of Qwen3-1.7B across AIME 2026, HMMT 2026, and BRUMO 2025.
Figure 2: Overview of TTPO. (1) Majority Voting: K sampled trajectories are partitioned into positive ( ak=a^ ) and negative ( ak=a^ ) sets. (2.1) OPSD Branch: positive samples are supervised via per-token forward KL with token weighting. (2.2) GRPO Branch: negative samples are penalized via group advantages with token masking on anomalous positions.
Method
AIME25
HMMT25
AIME26
HMMT26
BRUMO25
Average
Qwen3-1.7B
Base
36.9
21.9
37.8
28.8
47.5
34.6
+GRPO †
37.3
23.6
40.3
29.3
48.1
35.7
+OPSD †
40.3
28.1
46.4
31.4
52.5
39.7
+TTPO
41.7
26.1
46.5
31.6
54.7
40.1
Qwen3-4B
Table 1: Results on OpenThoughts training data. OPSD and GRPO use ground-truth labels ( † ); TTPO uses only majority-vote pseudo-labels.
Method
AIME26
HMMT26
BRUMO25
Average
Qwen3-1.7B
Base
37.8
28.8
47.5
38.0
+TTRL
39.2
30.6
50.9
40.2
+OPSD-TTT
44.7
30.3
50.8
41.9
+TTPO
48.9
33.6
53.1
45.2
Qwen3-4B
Table 2: Results on test-time training data. No method uses ground-truth labels.
Method
AIME26
HMMT26
BRUMO25
TTPO
46.5
31.6
54.7
w/o pos. weight
43.3
30.6
52.8
w/o neg. mask
45.4
29.5
50.0
Table 3: Token-level selection ablation results on Qwen3-1.7B OpenThoughts setting.
Figure 3: Ablations over update strategies on Qwen3-1.7B AIME26 TTT setting. P and N denote the loss applied to positive and negative samples, respectively. Dotted lines indicate the corresponding variants with an unconditioned teacher.
Privilege
TM-on Teacher
TM-off Teacher
None
45.8 (36.1)
41.7 (8.1)
Answer
46.5 ( 39.8 )
33.6 (6.7)
Trajectory
41.1 (8.9)
40.8 (10.6)
Table 4: Effect of privileged information type and teacher thinking mode (Qwen3-1.7B, OpenThoughts, AIME26). Columns: whether teacher uses thinking mode. Rows: privileged information injected into teacher prompt. Results in parentheses are evaluated with thinking disabled.
Figure 4: Cross-benchmark generalization (Qwen3-1.7B). Each subplot corresponds to a training benchmark; each group within a subplot shows performance on a different target benchmark.
Figure 5: Left: Comparison between pseudo-label vs. ground-truth supervision. TTPO w/ GT replaces majority-vote pseudo-labels with ground truth while keeping the asymmetric objective and token-level selection intact; OPSD (Leakage) trains standard OPSD directly on AIME26. Right: Entropy during training. Values in parentheses on the x-axis denote TTRL’s training steps.
Figure 6: Avg@12 and Maj@12 during TTPO TTT on HMMT26 (Qwen3-1.7B). Dashed line: base Maj@12.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
GRPO / TTRL
OPSD / OPSD-TTT
TTPO
General
Learning Rate
5×10−6
5×10−6
5×10−6
Max Gradient Norm
–
0.1
0.1
Effective Batch Size
32
32
32
Training Steps
500
100
100
LoRA Configuration
Appendix
Table 5: Training configuration for all methods. GRPO and OPSD use ground-truth labels; TTRL, OPSD-TTT, and TTPO are label-free. “–” indicates not applicable.
Parameter
Value
Thinking Mode
Enabled
Samples per Prompt
12
Temperature
1.0
Top- p
0.95
Max New Tokens
38,912
Metric
Avg@12
Appendix
Table 6: Evaluation configuration.
Figure 7: Training loss curves.
Method
AIME25
HMMT25
AIME26
HMMT26
BRUMO25
Average
Qwen3-1.7B
Base
9.2
5.6
8.8
6.1
17.8
9.5
+OPSD †
16.9
9.2
19.4
11.9
25.6
16.6
+TTPO
39.2
20.6
39.8
26.3
47.5
34.7
Qwen3-4B
Base
22.2
12.5
19.4
17.2
28.3
19.9
Appendix
Table 7: Non-thinking evaluation on OpenThoughts training data. Models are evaluated with thinking mode disabled . Both OPSD and TTPO are trained with a thinking-mode-on teacher.
Figure 8: Left: Training loss curves for the OPSD and GRPO branches (unweighted). Right: Positive sample fraction over training steps. (Qwen3-1.7B, OpenThoughts).
λ
AIME26
HMMT26
BRUMO25
0.01
41.4
28.8
51.1
0.05
43.9
29.0
50.6
0.10
46.5
31.6
54.7
0.15
44.7
29.5
50.3
0.20
41.7
26.3
51.6
Appendix
Table 8: Ablation on RL weight λ (Qwen3-1.7B, OpenThoughts).
Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.
Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.