cs.AINov 6, 2026

On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning

Authors: Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, Mingming Gong

Organizations: Mohamed bin Zayed University of Artificial Intelligence · The University of Tokyo · The University of Melbourne

Abstract

Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

    Mar 20, 2026Dong Yan, Jian Liang, Yanbo Wang +3RL for Language Model ReasoningTest-Time RL

  2. TTPO: Test-Time Policy Optimization

    Aug 27, 2026Aozhe Wang, Zhengxi Lu, Jianze Wang +8Test-Time TrainingTest-Time RL

  3. Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning

    Aug 4, 2026Kunbin Xu, Xingzuo Li, Xuefeng Bai +1RL for Language Model ReasoningTest-Time RL