The Race Is Decided Early! A Kernel Law Predicts Lock-In in Majority-Vote Test-Time RL
Organizations: Human-Centered Artificial Intelligence Research Institute, Ewha Womans University · Department of Artificial Intelligence, Ewha Womans University · Department of Computer Science and Engineering, Ewha Womans University
Abstract
Test-time reinforcement learning with majority-vote rewards trains a language model on unlabeled questions by rewarding the samples that agree with its most frequent answer. During training, each question locks into one answer, on hard questions often a wrong one, but which answer it locks into is known only after training. In this paper, we show that this outcome is decided early, in a short race driven by sampling noise between the answers that are close at the start, and we derive the probability of each outcome. With a separate logit for each answer, the runner-up wins with a normal tail probability of the initial gap, scaled by an explicit noise constant. In a network, however, the answer logits depend on shared parameters, and through the kernel of these parameters the updates of the trailing answers also change the gap of the two leaders. This spillover shifts the boundary between the starts won by each leader, and a closed form computes the shifted probability from a kernel probe and the start logits. In LoRA experiments with three models from two families, predictions made before training with the kernel frozen at its start value forecast which answer each question locks into better than a logistic model refitted on the outcomes, although they use no fitted constants. The closed form matches these predictions on the runs in which a check before training finds that the two leaders stay above the other answers, and it explains why a law that ignores the spillover needs a strongly inflated step.
Figures & tables
| model and setting | runs | Brier | difference |
|---|---|---|---|
| Qwen-1.5B, unseen | 2,002 | 0.149 | |
| Llama-1B, unseen | 917 | 0.143 | |
| Qwen-7B | 418 | 0.117 | |
| AdamW, | 600 | 0.159 | |
| SGD, std baseline | 602 | 0.129 | |
| shared, 16 and 64, SGD | 1,392 | 0.083 |
| probit | Brier score | ||||
|---|---|---|---|---|---|
| group | runs | slope | chain | scalar | |
| Qwen-1.5B | 2,256 | 1.07 | .1443 | .1462 | .2434 |
| Qwen-7B | 416 | 1.03 | .1160 | .1177 | .1723 |
| Llama-1B | 1,164 | 1.04 | .1388 | .1408 | .2687 |
| Qwen-1.5B, opt. | 1,674 | 1.06 | .1337 | .1357 | .2181 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| baseline | |||||||
|---|---|---|---|---|---|---|---|
| mean | 4 | 0.1875 | 0.1641 | 0.6614 | 0.5625 | 3.000 | 4.536 |
| mean | 8 | 0.4102 | 0.1982 | 0.4916 | 0.5742 | 1.400 | 2.848 |
| mean | 16 | 0.6873 | 0.2216 | 0.4015 | 0.5669 | 0.825 | 2.055 |
| mean | 32 | 1.0496 | 0.2351 | 0.3346 | 0.5543 | 0.528 | 1.578 |
| std | 4 | 0.3750 | 0.5625 | 0.8660 | 1.0245 | 2.732 | 3.155 |
| std | 8 | 0.8667 | 0.7656 | 0.6646 | 1.1237 | 1.297 | 1.951 |
| coupling | product | correlation | ||||
|---|---|---|---|---|---|---|
| 0.8878 | 0.2090 | 0.1842 | 0.1855 | |||
| 0.8845 | 0.2162 | 0.1891 | 0.1913 | |||
| 0.8746 | 0.2381 | 0.2213 | 0.2083 | |||
| 0.8791 | 0.2291 | 0.2104 | 0.2014 | |||
| 0.8080 | 0.3483 | 0.3455 | 0.2814 | |||
| 0.8399 | 0.2899 | 0.2801 | 0.2435 |
| 0.02 | 0.01382 | 0.01378 | 0.4869 | 0.4867 | |
| 0.04 | 0.02631 | 0.02616 | 0.4830 | 0.4825 | |
| 0.06 | 0.03757 | 0.03729 | 0.4796 | 0.4788 | |
| 0.10 | 0.05667 | 0.05612 | 0.4740 | 0.4728 | |
| 0.15 | 0.07428 | 0.07344 | 0.4693 | 0.4669 | |
| 0.20 | 0.08417 | 0.08324 | 0.4673 | 0.4627 |
| exact | limit | error | remainder | |||
|---|---|---|---|---|---|---|
| 1.0 | 0.5 | 0.296983 | 0.308538 | |||
| 1.0 | 1.0 | 0.142548 | 0.158655 | |||
| 1.0 | 2.0 | 0.015508 | 0.022750 | |||
| 0.3 | 0.5 | 0.305131 | 0.308538 | |||
| 0.3 | 1.0 | 0.153813 | 0.158655 | |||
| 0.3 | 2.0 | 0.020340 | 0.022750 |
| runs | answer 1 | answer 2 | answer 3 | ||
|---|---|---|---|---|---|
| 0.03 | 40,000 | 0.338 (0.333) | 0.331 (0.333) | 0.331 (0.333) | |
| 0.03 | 40,000 | 0.629 (0.630) | 0.274 (0.272) | 0.097 (0.097) | |
| 0.03 | 40,000 | 0.631 (0.630) | 0.274 (0.272) | 0.095 (0.097) | |
| 0.03 | 40,000 | 0.442 (0.443) | 0.444 (0.443) | 0.114 (0.115) | |
| 0.01 | 40,000 | 0.333 (0.333) | 0.332 (0.333) | 0.335 (0.333) | |
| 0.01 | 40,000 | 0.627 (0.630) | 0.277 (0.272) | 0.096 (0.097) |
| runs | answer 1 | answer 2 | answer 3 | answer 4 | ||
|---|---|---|---|---|---|---|
| 0.03 | 40,000 | 0.556 (0.552) | 0.277 (0.277) | 0.122 (0.123) | 0.045 (0.047) | |
| 0.03 | 40,000 | 0.374 (0.368) | 0.366 (0.368) | 0.208 (0.210) | 0.052 (0.054) | |
| 0.03 | 200,000 | 0.556 (0.552) | 0.276 (0.277) | 0.121 (0.123) | 0.046 (0.047) | |
| 0.01 | 80,000 | 0.558 (0.552) | 0.275 (0.277) | 0.121 (0.123) | 0.046 (0.047) | |
| 0.003 | 100,000 | 0.553 (0.552) | 0.278 (0.277) | 0.122 (0.123) | 0.047 (0.047) | |
| 0.001 | 40,000 | 0.555 (0.552) | 0.277 (0.277) | 0.121 (0.123) | 0.047 (0.047) |
| standard error | ||||
|---|---|---|---|---|
| 0.03 | 0.5 | 0.3078 | 0.3085 | 0.0023 |
| 0.03 | 1.0 | 0.1608 | 0.1587 | 0.0018 |
| 0.03 | 2.0 | 0.0223 | 0.0228 | 0.0007 |
| 0.01 | 0.5 | 0.3094 | 0.3085 | 0.0023 |
| 0.01 | 1.0 | 0.1545 | 0.1587 | 0.0018 |
| 0.01 | 2.0 | 0.0225 | 0.0228 | 0.0007 |
| 3 | 0.1 | 0.0482229 | 0.0478617 | ||||
| 3 | 0.05 | 0.0272549 | 0.0271258 | ||||
| 3 | 0.025 | 0.0146765 | 0.0146356 | ||||
| 3 | 0.0125 | 0.0076527 | 0.0076409 | ||||
| 3 | 0.00625 | 0.0039137 | 0.0039105 | ||||
| 3 | 0.003125 | 0.0019800 | 0.0019791 |
| 0.4 | |||||||
|---|---|---|---|---|---|---|---|
| 0.2 | |||||||
| 0.1 | |||||||
| 0.05 |
| mean abs. difference | probit fit | from the chain | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| level | runs | edge | scalar | slope | sd | sd scalar | scalar | |||
| weak | 0.3 | 97 | 0.008 | 0.027 | 0.051 | 1.02 | 0.04 | 0.21 | 1.1 | 1.02 |
| weak | 0.1 | 98 | 0.008 | 0.037 | 0.083 | 1.01 | 0.04 | 0.35 | 1.1 | 1.00 |
| weak | 0.03 | 95 | 0.008 | 0.057 | 0.161 | 1.00 | 0.04 | 0.67 | 2.1 | 0.99 |
| weak | 0.01 | 95 | 0.007 | 0.076 | 0.226 | 1.00 | 0.04 | 1.07 | 4.3 | 1.00 |
| LM | 0.3 | 95 | 0.016 | 0.060 | 0.176 | 1.10 | 0.12 | 0.81 | 2.8 | 1.05 |
| model and runs | hypothesis | result | outcome |
| Qwen-1.5B, unseen, two letters | slope of in | failed | |
| Qwen-1.5B, unseen, three or more | Thurstone law beats softmax | not established | |
| Qwen-1.5B, unseen, two letters | scalar law with one fitted step, noninferior | failed | |
| shared sets of | winners of a set agree more than isolated runs | passed | |
| Qwen-1.5B, unseen | slope of in | not established | |
| Qwen-1.5B, unseen | noninferior to the refitted logistic | passed |
| model | setting | runs | Brier | slope |
| Qwen-1.5B | 547 | 0.148 | 0.76 | |
| Qwen-1.5B | 392 | 0.143 | 1.06 | |
| Qwen-1.5B | 364 | 0.159 | 0.83 | |
| Qwen-1.5B | 383 | 0.143 | 0.84 | |
| Qwen-1.5B | 316 | 0.154 | 0.86 | |
| Llama-1B | 578 | 0.142 | 0.82 |
| for from | for from | |||||||||
| model | setting | runs | chain | outcomes | chain | outcomes | , all | Prop. 2 | corr | |
| Qwen-1.5B | 193 | 0.90 | 0.92 | 7.11 | 8.21 | 7.35 | 3.27 | 2.00 | ||
| reference | 320 | 0.94 | 1.34 | 9.43 | 11.46 | 15.15 | 3.81 | 1.74 | ||
| 255 | 0.99 | 1.13 | 31.63 | 54.97 | 21.67 | 28.84 | 2.14 | |||
| 337 | 0.96 | 1.51 | 7.53 | 9.81 | 5.66 | 4.61 | 1.76 | |||
| 294 | 0.91 | 0.95 | 17.37 | 22.69 | 27.20 | 6.93 | 1.88 | |||
| group | scope | letters | runs | difference | interval | third letter wins |
|---|---|---|---|---|---|---|
| Qwen-1.5B | in scope | two | 1,695 | |||
| in scope | three or more | 561 | ||||
| flagged | two | 47 | , | |||
| flagged | three or more | 153 | , | |||
| Qwen-7B | in scope | two | 333 | |||
| in scope | three or more | 83 |
| group | runs | coefficient | sd, before and after | intercept, before and after | |
|---|---|---|---|---|---|
| Qwen-1.5B | 1,810 | 0.87 (0.01) | 0.72 | 0.132, 0.081 | , |
| Qwen-7B | 351 | 1.62 (0.08) | 0.56 | 0.061, 0.045 | , |
| Llama-1B | 621 | 0.56 (0.04) | 0.30 | 0.159, 0.156 | , |
| Qwen-1.5B, opt. | 1,096 | 0.79 (0.02) | 0.69 | 0.104, 0.076 | , |
| probit slope | shift by sharing | Brier score | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| size | members | isolated | shared | Pearson | slope | coupled chain | shared | isolated chain | isolated law | closed | |
| 2 | 476 | 1.05 | 0.67 | 1.39 | 0.53 | 0.68 | 0.0487 | 0.0564 | 0.0579 | 0.0575 | 13% |
| 4 | 466 | 1.06 | 0.54 | 1.82 | 0.56 | 0.77 | 0.0660 | 0.0862 | 0.0913 | 0.0932 | 26% |
| 16 | 440 | 1.06 | 0.36 | 3.06 | 0.73 | 0.89 | 0.0824 | 0.0825 | 0.1070 | 0.1088 | 100% |
| 64 | 719 | 1.06 | 0.33 | 3.59 | 0.87 | 0.96 | 0.0805 | 0.0884 | 0.1498 | 0.1500 | 89% |
| condition | early | late | decision | |
|---|---|---|---|---|
| Llama-1B | 1.94 | to | , then | 0.49 |
| AdamW, | 1.45 | about | , then | 0.60 |
| Qwen-1.5B | 1.28 | to | , then | 0.65 |
| SGD, std baseline | 1.22 | to | , then | 0.67 |
| Qwen-7B | 1.01 | to | to | 0.91 |
Test-Time RL
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.
See all 54 papers