cs.LGNov 8, 2026

The Race Is Decided Early! A Kernel Law Predicts Lock-In in Majority-Vote Test-Time RL

Authors: Jaeho Hwang, Sehong Park, Daejin Choi, Uiwon Hwang

Organizations: Human-Centered Artificial Intelligence Research Institute, Ewha Womans University · Department of Artificial Intelligence, Ewha Womans University · Department of Computer Science and Engineering, Ewha Womans University

Abstract

Test-time reinforcement learning with majority-vote rewards trains a language model on unlabeled questions by rewarding the samples that agree with its most frequent answer. During training, each question locks into one answer, on hard questions often a wrong one, but which answer it locks into is known only after training. In this paper, we show that this outcome is decided early, in a short race driven by sampling noise between the answers that are close at the start, and we derive the probability of each outcome. With a separate logit for each answer, the runner-up wins with a normal tail probability of the initial gap, scaled by an explicit noise constant. In a network, however, the answer logits depend on shared parameters, and through the kernel of these parameters the updates of the trailing answers also change the gap of the two leaders. This spillover shifts the boundary between the starts won by each leader, and a closed form computes the shifted probability from a kernel probe and the start logits. In LoRA experiments with three models from two families, predictions made before training with the kernel frozen at its start value forecast which answer each question locks into better than a logistic model refitted on the outcomes, although they use no fitted constants. The closed form matches these predictions on the runs in which a check before training finds that the two leaders stay above the other answers, and it explains why a law that ignores the spillover needs a strongly inflated step.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

May 19, 2026

Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting

Test-time reinforcement learning (TTRL) reports substantial accuracy gains on mathematical reasoning benchmarks using majority vote as a pseudo-label signal. We argue these gains are systematically misinterpreted: most reflect sharpening of already-solvable problems rather than genuine learning, while problems corrupted from correct to incorrect outnumber truly learned ones, and this damage is irreversible once majority vote locks onto a wrong answer. Per-problem tracking reveals that correct-answer signals in low-ability problems are briefly active before being permanently suppressed, a phenomenon we term the Correct-Answer Extinction Window, with Flip Rate (FR) as its leading indicator. We thus propose TTRL-Guard, a lightweight framework with three mechanisms targeting the extinction window: Flip-Rate-Aware Reward Scaling (FRS) down-weights at-risk updates as FR declines, Minority-Preserving Sampling (MPS) retains gradient signal from minority correct answers, and Risk-Conditioned Sparse Updatings (RCSU) suspends updates on polarized problems. Experiments across three models and four benchmarks show that TTRL-Guard achieves the best average pass@1 on Qwen2.5-7B-Instruct and Qwen3-4B, improves relatively over TTRL by +54% on AIME 2025.
Aug 10, 2026

CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over NN sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with CoRE (Consensus Rewards via Equilibrium): the NN roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), CoRE improves the untrained base by +21.7+21.7 points on average versus +20.4+20.4 for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to +7.5+7.5 points, and reaches the voting baseline's plateau accuracy in 5454--7070% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.
Oct 1, 2026

Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It

Majority voting over sampled completions is the workhorse of test-time scaling, and reinforcement learning with verifiable rewards (RLVR) is the workhorse for making each completion better. The standard pipeline composes the two: train one policy with RLVR, then sample it many times and vote. We show that this composition is lossy. A vote can only overturn mistakes that its voters do not share, and RLVR sharpens a policy so that its samples increasingly make the same mistakes. With every method drawing exactly 160160 completions per problem, training a single LoRA adapter on the full RLVR budget raises single-sample accuracy on every model we test (1.51.5B-88B). Yet on three of four models it leaves the majority vote below that of the untrained base model, by up to 4.84.8 points. The damage builds during training: voter errors grow steadily more correlated, and the majority vote accuracy peaks early before falling by up to 7.07.0 points. The cause is concentration, not RLVR itself. We split the same data and training budget across KK LoRA adapters, each trained on its own random disjoint shard, and call the result an adapter thicket. Thickets out-vote the fully trained adapter in all 1616 (model, KK) settings, and for K≥4K{\geq}4 they stay within 0.80.8 points of the base model or above it. A single adapter stopped early, at a thicket member's step count, is a strong control that matches thickets for small KK. For K≥8K{\geq}8, thickets keep more of RLVR's single-sample gain and out-vote this control in six of eight settings. The cost of concentration also grows with the number of votes: from 1616 to 160160 votes, the thicket's lead over the fully trained adapter widens from 1.31.3 to 3.33.3 points. When the plan is to sample and vote, an RLVR budget is better spent broad than deep.