On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
Organizations: Mohamed bin Zayed University of Artificial Intelligence · The University of Tokyo · The University of Melbourne
Abstract
Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.
Figures & tables
| Method | Math Reasoning | Knowledge Reasoning | ||||
|---|---|---|---|---|---|---|
| A24 | A25 | AMC | M500 | GPQA-D | MMLU-M | |
| Qwen2.5-1.5B-Instruct | ||||||
| Base | 2.9 | 0.8 | 22.0 | 50.7 | 24.1 | 39.5 |
| RENT | 0.0 | 19.2 | 55.7 | 5.2 | ||
| TTRL | 3.3 | 33.6 | 62.3 | 23.5 | 10.9 | |
| Co-Reward | 3.3 | 31.4 | 59.6 | 23.3 | 10.8 | |
| Method | AMC | MATH500 |
|---|---|---|
| Base model | 34.3 | 63.5 |
| PC-Dr.DPO w/ constant | 42.0 | 72.3 |
| PC-Dr.DPO w/ frequency MV | 44.6 | 71.4 |
| PC-Dr.DPO | 45.8 | 73.3 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Max prompt length | 1,024 |
| Max response length | 3,072 |
| Training batch size | 24 AIME24 24 AIME25 16 AMC 32 MATH500 16 GPQA-Diamond 32 MMLU-Pro Math |
| Dataset size | 30 AIME24 30 AIME25 83 AMC 500 MATH500 198 GPQA-Diamond 1,351 MMLU-Pro Math |
| Total epochs | 240 AIME24 240 AIME25 80 AMC 40 MATH500 40 GPQA-Diamond 80 MMLU-Pro Math |
| PPO mini-batch size | 4 |
Test-Time RL
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.
See all 54 papers