The Race Is Decided Early! A Kernel Law Predicts Lock-In in Majority-Vote Test-Time RL
Authors: Jaeho Hwang, Sehong Park, Daejin Choi, Uiwon Hwang
Organizations: Human-Centered Artificial Intelligence Research Institute, Ewha Womans University · Department of Artificial Intelligence, Ewha Womans University · Department of Computer Science and Engineering, Ewha Womans University
Test-time reinforcement learning with majority-vote rewards trains a language model on unlabeled questions by rewarding the samples that agree with its most frequent answer. During training, each question locks into one answer, on hard questions often a wrong one, but which answer it locks into is known only after training. In this paper, we show that this outcome is decided early, in a short race driven by sampling noise between the answers that are close at the start, and we derive the probability of each outcome. With a separate logit for each answer, the runner-up wins with a normal tail probability of the initial gap, scaled by an explicit noise constant. In a network, however, the answer logits depend on shared parameters, and through the kernel of these parameters the updates of the trailing answers also change the gap of the two leaders. This spillover shifts the boundary between the starts won by each leader, and a closed form computes the shifted probability from a kernel probe and the start logits. In LoRA experiments with three models from two families, predictions made before training with the kernel frozen at its start value forecast which answer each question locks into better than a logistic model refitted on the outcomes, although they use no fitted constants. The closed form matches these predictions on the runs in which a check before training finds that the two leaders stay above the other answers, and it explains why a law that ignores the spillover needs a strongly inflated step.
Figures & tables
Figure 1: Overview. (a) An illustrative question with ten options and two close leaders. Majority-vote TTRL locks it into one answer, here B or D with the probabilities of the race in (b). (b) The gap between the two leaders in 60 runs of the tabular chain from the start in (a), with G=8 and η=0.02 , colored by the winner. The shading ends when 90 percent of 4,000 runs have seen their last change of the leader, long before the median lock-in. The runner-up wins in 0.30 of the runs, as the lock-in law of Theorem 1 predicts. (c) For the kernel of Figure 2 at a tie with trailing mass 0.2 , the mean update raises both leaders equally, but the logit drift through the kernel raises answer 2 three times as much as answer 1. This spillover moves the separatrix away from the tie, and the closed form Plin computes the shift before training from a kernel probe and the start logits.
Figure 2: The race is decided at the separatrix. (a) The mean flow of a three-answer kernel with spillover ( G=8 , mean baseline) in the plane of the gap ζ and the trailing mass ρ . Orbits start at ρ0=0.2 and are colored by their winner. Orbits that start between the tie and the separatrix are won by the runner-up although answer 1 leads. The zero of the closed form is indistinguishable from the separatrix ζ∗ found by bisection on the flow. (b) The probability that the runner-up wins from ρ0=0.2 at the step η=0.01 , against the initial gap in units of η . The dots are 6000 runs of the frozen-kernel chain per gap with two standard errors. The closed form Plin (Definition 1 ) follows the chain, and the scalar law, which ignores the spillover, puts the even race at the tie.
Figure 3: Prediction before training on the evaluated runs in scope. (a) The probability that the runner-up wins under the closed form and under the scalar law against the chain, in bins of the law. The scalar law can never favor the runner-up, while the spillover often does. (b) Observed frequency with which the runner-up wins, among runs won by one of the two leaders and pooled over the four groups of Table 2 , in bins of each prediction, with one standard error. (c) The step multiplier of the scalar law that the outcomes imply, against the one that Plin implies from the kernels and the start logits alone, for the thirteen settings with colors as in (a).
model and setting
runs
Brier
difference
Qwen-1.5B, unseen
2,002
0.149
−0.075
Llama-1B, unseen
917
0.143
−0.084
Qwen-7B
418
0.117
−0.066
AdamW, β1=0.9
600
0.159
−0.057
SGD, std baseline
602
0.129
−0.075
shared, 16 and 64, SGD
1,392
0.083
−0.086
Table 1: Prediction of lock-in by the frozen-kernel chain, computed before training. Brier score for the event that the leader wins and its difference to a logistic model refitted on the outcomes, on the evaluated runs, those with η∈[0.01,0.3] whose predicted probability that the leader loses lies in [0.01,0.99] . The last two rows compare the coupled chain with the isolated chain on members of shared sets. Intervals are 95% bootstrap intervals over questions or sets. Every difference passes a one-sided noninferiority test with a margin of 0.003 fixed before training, with p<10−6 after Holm correction.
probit
Brier score
group
runs
slope
chain
Plin
scalar
Qwen-1.5B
2,256
1.07
.1443
.1462
.2434
Qwen-7B
416
1.03
.1160
.1177
.1723
Llama-1B
1,164
1.04
.1388
.1408
.2687
Qwen-1.5B, opt.
1,674
1.06
.1337
.1357
.2181
Table 2: The closed form against the chain and the outcomes, on the evaluated runs in scope. Probit slope of the chain on Plin for the probability that the runner-up wins, on all runs in scope with the chain in (0.02,0.98) . Brier scores for the event that the leader wins. The differences between Plin and the chain have 95% intervals that contain zero. The row “opt.” pools the three optimizer settings.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
baseline
G
r∘
v∘
σˉ∘
γ∘
γ∘/r∘
γ∘/(r∘σˉ∘)
mean
4
0.1875
0.1641
0.6614
0.5625
3.000
4.536
mean
8
0.4102
0.1982
0.4916
0.5742
1.400
2.848
mean
16
0.6873
0.2216
0.4015
0.5669
0.825
2.055
mean
32
1.0496
0.2351
0.3346
0.5543
0.528
1.578
std
4
0.3750
0.5625
0.8660
1.0245
2.732
3.155
std
8
0.8667
0.7656
0.6646
1.1237
1.297
1.951
Appendix
Table 3: Constants of two answers at a tie. The last column is the shift of the probit per unit of the spillover index at η=1 . The rationals for the mean baseline are, for instance, r∘=105/256 and v∘=203/1024 at G=8 . The standard-deviation baseline uses the unbiased standard deviation of the group rewards.
coupling
η
P(W1=2)
P(W2=2)
P(W1=W2=2)
product
correlation
α3=0
4×10−3
0.8878
0.2090
0.1842
0.1855
−0.010
α3=0
10−3
0.8845
0.2162
0.1891
0.1913
−0.016
α3=1/4
4×10−3
0.8746
0.2381
0.2213
0.2083
+0.092
α3=1/4
10−3
0.8791
0.2291
0.2104
0.2014
+0.065
α3=1
4×10−3
0.8080
0.3483
0.3455
0.2814
+0.342
α3=1
10−3
0.8399
0.2899
0.2801
0.2435
+0.220
Appendix
Table 4: Two weakly coupled questions, 8000 runs per row. Standard errors are about 0.004 for the marginals and 0.011 for the correlation of the indicators 1{Ws=2} . The column “product” is the product of the two observed marginals. The last row is the limit of Corollary D.1 , which does not depend on α3 . No trailing answer won in any run.
ρ0
ζ∗
−BK
(ζ∗+BK)/ζ∗
σˉK
σˉ
0.02
0.01382
0.01378
+0.3%
0.4869
0.4867
0.04
0.02631
0.02616
+0.6%
0.4830
0.4825
0.06
0.03757
0.03729
+0.7%
0.4796
0.4788
0.10
0.05667
0.05612
+1.0%
0.4740
0.4728
0.15
0.07428
0.07344
+1.1%
0.4693
0.4669
0.20
0.08417
0.08324
+1.1%
0.4673
0.4627
Appendix
Table 5: The separatrix of the kernel ( E.1 ) against the zero of the closed form. The last two columns are the noise constants of the closed form and of the scalar law.
η
β/σˉ
exact
limit
error /η
φ1(β)
remainder /η2
1.0
0.5
0.296983
0.308538
−0.01155
−0.01130
−0.00026
1.0
1.0
0.142548
0.158655
−0.01611
−0.01618
+0.00008
1.0
2.0
0.015508
0.022750
−0.00724
−0.00839
+0.00115
0.3
0.5
0.305131
0.308538
−0.01135
−0.01130
−0.00019
0.3
1.0
0.153813
0.158655
−0.01614
−0.01618
+0.00015
0.3
2.0
0.020340
0.022750
−0.00803
−0.00839
+0.00120
Appendix
Table 6: Two answers, G=8 . Exact P(W=2) from the lattice, its limit Φc(β/σˉ) , the error divided by η , the coefficient φ1(β) of Proposition B.2 and the remainder (P(W=2)−Φc(β/σˉ)−ηφ1(β))/η2 .
η
runs
β/σˉ
answer 1
answer 2
answer 3
0.03
40,000
(0,0,0)
0.338 (0.333)
0.331 (0.333)
0.331 (0.333)
0.03
40,000
(0.5,0,−0.5)
0.629 (0.630)
0.274 (0.272)
0.097 (0.097)
0.03
40,000
(1.0,0.5,0)
0.631 (0.630)
0.274 (0.272)
0.095 (0.097)
0.03
40,000
(0.7,0.7,0)
0.442 (0.443)
0.444 (0.443)
0.114 (0.115)
0.01
40,000
(0,0,0)
0.333 (0.333)
0.332 (0.333)
0.335 (0.333)
0.01
40,000
(0.5,0,−0.5)
0.627 (0.630)
0.277 (0.272)
0.096 (0.097)
Appendix
Table 7: Winning frequencies of three tied answers, with the Thurstone law in parentheses. The start is z0+ηβ , and β is given in units of σˉ . Standard errors are about 0.0024 near 0.3 to 0.6 and 0.0015 near 0.1 for 40,000 runs.
η
runs
β/σˉ
answer 1
answer 2
answer 3
answer 4
0.03
40,000
(0.6,0.2,−0.2,−0.6)
0.556 (0.552)
0.277 (0.277)
0.122 (0.123)
0.045 (0.047)
0.03
40,000
(0.3,0.3,0,−0.6)
0.374 (0.368)
0.366 (0.368)
0.208 (0.210)
0.052 (0.054)
0.03
200,000
(0.6,0.2,−0.2,−0.6)
0.556 (0.552)
0.276 (0.277)
0.121 (0.123)
0.046 (0.047)
0.01
80,000
(0.6,0.2,−0.2,−0.6)
0.558 (0.552)
0.275 (0.277)
0.121 (0.123)
0.046 (0.047)
0.003
100,000
(0.6,0.2,−0.2,−0.6)
0.553 (0.552)
0.278 (0.277)
0.122 (0.123)
0.047 (0.047)
0.001
40,000
(0.6,0.2,−0.2,−0.6)
0.555 (0.552)
0.277 (0.277)
0.121 (0.123)
0.047 (0.047)
Appendix
Table 8: Winning frequencies of four tied answers, with the Thurstone law in parentheses.
η
β/σˉ
P(W=2)
Φc(β/σˉ)
standard error
0.03
0.5
0.3078
0.3085
0.0023
0.03
1.0
0.1608
0.1587
0.0018
0.03
2.0
0.0223
0.0228
0.0007
0.01
0.5
0.3094
0.3085
0.0023
0.01
1.0
0.1545
0.1587
0.0018
0.01
2.0
0.0225
0.0228
0.0007
Appendix
Table 9: Probability that the runner-up wins from a start with trailing mass 0.6 , 40,000 runs per row.
G
ρ0
ζ∗
−BK
ρ02ζ∗+BK
ρ02Bloop
ρ03ζ∗+BK−Bloop
ρ02ζ∗+Bedge
3
0.1
0.0482229
0.0478617
+0.0361
+0.0668
−0.307
−1.591
3
0.05
0.0272549
0.0271258
+0.0517
+0.0756
−0.478
−1.924
3
0.025
0.0146765
0.0146356
+0.0654
+0.0814
−0.641
−2.170
3
0.0125
0.0076527
0.0076409
+0.0753
+0.0849
−0.765
−2.326
3
0.00625
0.0039137
0.0039105
+0.0815
+0.0868
−0.846
−2.417
3
0.003125
0.0019800
0.0019791
+0.0850
+0.0878
−0.893
−2.466
Appendix
Table 10: The separatrix near the edge for two kernels on five answers, G=3 with χ=(0.62,0.19,0.23) and G=4 with χ=(0.44,0.60,−0.74) . The ratio (ζ∗+BK)/ρ02 converges to the loop term, and the remainder is of order ρ03 .
s
ζ∗(0,s)
103R0(s)
ζ∗(0.25,s)
103R0.25(s)
ζ∗(s,s)
103Rd(s)
103Rd(−s)
0.4
−0.074795857
−7.042
−0.073628291
−6.667
−0.072920344
−6.437
−7.542
0.2
−0.037228368
−6.972
−0.036653628
−6.603
−0.036769398
−6.678
−7.243
0.1
−0.018593250
−6.955
−0.018306988
−6.588
−0.018479451
−6.810
−7.094
0.05
−0.009294016
−6.951
−0.009151023
−6.584
−0.009265669
−6.879
−7.021
Appendix
Table 11: The separatrix under a weak anisotropy, with Rθe(s)=(ζ∗(θe,s)+sβA(θe))/s3 and Rd(s)=(ζ∗(s,s)+BKs)/s3 . The remainders after the linear term −θoβA(θe) are of third order in θo for a fixed θe , and along the diagonal the error of −BKs is of third order in s .
mean abs. difference
probit fit
λ from the chain
level
η
runs
Plin
edge
scalar
slope
sd Plin
sd scalar
scalar
Plin
weak
0.3
97
0.008
0.027
0.051
1.02
0.04
0.21
1.1
1.02
weak
0.1
98
0.008
0.037
0.083
1.01
0.04
0.35
1.1
1.00
weak
0.03
95
0.008
0.057
0.161
1.00
0.04
0.67
2.1
0.99
weak
0.01
95
0.007
0.076
0.226
1.00
0.04
1.07
4.3
1.00
LM
0.3
95
0.016
0.060
0.176
1.10
0.12
0.81
2.8
1.05
Appendix
Table 12: The closed form against the frozen-kernel chain on synthetic kernels. Mean absolute difference from Pmc on the runs with Pmc∈(0.01,0.99) , for Plin , for the scalar law with the edge offset Bedge of Proposition C.1 (iii) (edge) and for the scalar law (scalar). Probit fit of the chain on the law, on the runs with Pmc∈(0.02,0.98) , with the slope for Plin and the standard deviation of the probit difference. The last two columns are the step multipliers that the chain implies for the scalar law and for Plin .
model and runs
hypothesis
result
outcome
Qwen-1.5B, unseen, two letters
slope of Ptwo in [0.8,1.25]
0.077[0.014,0.157]
failed
Qwen-1.5B, unseen, three or more
Thurstone law beats softmax
−0.052[−0.141,0.045]
not established
Qwen-1.5B, unseen, two letters
scalar law with one fitted step, noninferior
+0.0107
failed
shared sets of 4
winners of a set agree more than isolated runs
0.032[0.014,0.051]
passed
Qwen-1.5B, unseen
slope of Pmc in [0.8,1.25]
0.84[0.75,0.94]
not established
Qwen-1.5B, unseen
Pmc noninferior to the refitted logistic
−0.075
passed
Appendix
Table 13: All tests fixed before training. The first group tests the scalar law and was fixed before the reference setting was trained. The two tests of the chain on Qwen-1.5B were added after the reference setting and before the unseen settings, and every later model or setting has its own group. Intervals are 95% bootstrap intervals.
model
setting
runs
Brier
slope
Qwen-1.5B
G=4
547
0.148
0.76 [0.63,0.95]
Qwen-1.5B
G=16
392
0.143
1.06 [0.90,1.30]
Qwen-1.5B
G=32
364
0.159
0.83 [0.69,1.04]
Qwen-1.5B
3lr
383
0.143
0.84 [0.68,1.10]
Qwen-1.5B
lr/3
316
0.154
0.86 [0.69,1.09]
Llama-1B
G=4
578
0.142
0.82 [0.69,0.99]
Appendix
Table 14: Predictions of the chain by setting, on the evaluated runs. The slope is the calibration slope with its 95% interval.
λ for Plin from
λ for Ptwo from
model
setting
runs
chain
outcomes
Plin
chain
outcomes
Plin , all
Prop. 2
corr (ζ0,BK)
Qwen-1.5B
3lr
193
0.90
0.92
7.11
8.21
7.35
3.27
2.00
+0.03
reference
320
0.94
1.34
9.43
11.46
15.15
3.81
1.74
−0.35
lr/3
255
0.99
1.13
31.63
54.97
21.67
28.84
2.14
−0.59
G=4
337
0.96
1.51
7.53
9.81
5.66
4.61
1.76
−0.16
G=16
294
0.91
0.95
17.37
22.69
27.20
6.93
1.88
−0.29
Appendix
Table 15: Step multipliers by setting on the two-letter evaluated runs that are in scope, with the two-leader target. The first two columns of multipliers are those that the chain and the outcomes imply for Plin , and the next three are those that Plin , the chain and the outcomes imply for the scalar law. The column “ Plin , all” is the multiplier that Plin implies for the scalar law on all two-letter runs in scope, without the restriction to evaluated runs. The column “Prop. 2 ” is 1+\Var(BK)/⟨ησˉ2⟩ , and the last column is the correlation of ζ0 and BK .
group
scope
letters
runs
difference
interval
third letter wins
Qwen-1.5B
in scope
two
1,695
−0.0001
[−0.0028,0.0025]
in scope
three or more
561
+0.0079
[0.0007,0.0168]
flagged
two
47
+0.09
0.40 , 0.40
flagged
three or more
153
+0.33
[0.19,0.48]
0.60 , 0.61
Qwen-7B
in scope
two
333
+0.0012
[−0.0005,0.0035]
in scope
three or more
83
+0.0038
[−0.0071,0.0197]
Appendix
Table 16: The Brier score of Plin minus that of the chain, by scope and by the number of close letters, with 95% intervals where the runs suffice. The last column gives the frequency with which a letter other than the two leaders won, predicted by the chain and observed.
group
runs
coefficient
R2
sd, before and after
intercept, before and after
Qwen-1.5B
1,810
0.87 (0.01)
0.72
0.132, 0.081
−0.061 , +0.002
Qwen-7B
351
1.62 (0.08)
0.56
0.061, 0.045
−0.019 , −0.006
Llama-1B
621
0.56 (0.04)
0.30
0.159, 0.156
−0.049 , +0.033
Qwen-1.5B, opt.
1,096
0.79 (0.02)
0.69
0.104, 0.076
−0.057 , −0.001
Appendix
Table 17: The cubic offset on the probed kernels. Coefficient of u=BK3/(ησˉK) with its standard error, share R2 of the residual variance, and the standard deviation and intercept of the probit difference to the chain before and after the zero is moved to −BK−1.1BK3 .
probit slope
shift by sharing
Brier score
size
members
isolated
shared
λ
Pearson
slope
coupled chain
shared Plin
isolated chain
isolated law
closed
2
476
1.05
0.67
1.39
0.53
0.68
0.0487
0.0564
0.0579
0.0575
13%
4
466
1.06
0.54
1.82
0.56
0.77
0.0660
0.0862
0.0913
0.0932
26%
16
440
1.06
0.36
3.06
0.73
0.89
0.0824
0.0825
0.1070
0.1088
100%
64
719
1.06
0.33
3.59
0.87
0.96
0.0805
0.0884
0.1498
0.1500
89%
Appendix
Table 18: Shared sets. Probit slopes of the chain on the law for the isolated law against the isolated chain and for the shared closed form against the coupled chain, with the two-leader target, and the step multiplier of the shared closed form. The change that sharing causes in the probability that the leader loses is compared between the coupled chain and the closed form by the Pearson correlation and the slope. The Brier scores are for the event that the leader wins, and the last column is the share of the Brier gap between the isolated law and the coupled chain that the shared closed form closes.
condition
D
early
late
decision
Llama-1B
1.94
1.01 to 1.03
1.80 , then 3.16
0.49
AdamW, β1=0
1.45
about 1
1.75 , then 2.82
0.60
Qwen-1.5B
1.28
0.90 to 1.09
1.30 , then 2.60
0.65
SGD, std baseline
1.22
0.87 to 1.04
1.70 , then 2.62
0.67
Qwen-7B
1.01
0.94 to 1.06
1.18 to 1.24
0.91
Appendix
Table 19: Growth of the kernel during training. D is the ratio of the observed drift of the gap to the drift of the frozen-kernel chain from the same state, on the evaluated runs, early and late in training, and the last column is the ratio of the observed decision step to the one the chain predicts.
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.