When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training

Jaeho Hwang; Uiwon Hwang

cs.LGOct 31, 2026

When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training

Authors: Jaeho Hwang, Uiwon Hwang

Organizations: Human-Centered Artificial Intelligence Research Institute, Ewha Womans University · Department of Computer Science and Engineering, Ewha Womans University

Abstract

Test-time reinforcement learning (TTRL) adapts a model without labels by rewarding answers that agree with the majority, but it collapses the policy onto its majority answer and can remove the correct answer from the samples. Exploration can prevent this collapse, yet its strength is chosen by trial and error, and how the training algorithm changes the required strength is unclear. We model training on a single problem as a continuous flow of the logits of a softmax policy and analyze which policies are stable as the exploration coefficient varies. Only collapsed policies are stable below a lower boundary and only policies spread over many answers above an upper boundary, and we derive these boundaries in closed form for an agreement reward with smoothing toward the uniform policy, compute them for the majority reward of TTRL on policies with one leading answer and bound them with interval arithmetic for actions on ordered bins. We prove that under exact gradients, the learning rate and, in a mean-field model, the normalization of GRPO leave these boundaries unchanged when exploration passes through them together with the reward but shift them when exploration bypasses them. Exact sums on these policies then show that a bonus for rare answers needs a coefficient above $1/\log(G-1)$ for groups of $G$ samples before the normalization and about two to three times that after it. In free-form TTRL on MATH-500, a bonus placed before the normalization prevented collapse and kept more correct answers among the samples, although it lowered pass@1.

Figures & tables

Figure 1: Regimes in four settings, with no window in (c). Lines are stable (solid) and unstable (dashed) equilibria of the tabular flow, with shading where collapse cannot be prevented (dark) and in the window (light). Markers give the mean

p_{\max}

of a run from broad (squares) and collapsed (circles) starts, filled and open for SGD and AdamW in (a) and (c) and for seeds 0 and 1 otherwise. (a) Testbed, Gaussian kernel of width 3 in the loss, with the folds of Theorem 4.6 times

\frac{62}{64}

for the leave-one-out reward. (b) Letter test, agreement reward, coefficient ramped over 900 steps, with the ends of Proposition 4.4 times

\frac{14}{16}

. (c) As (a) with only the advantages normalized, against the coefficient before the division (Proposition 4.3 (b)). (d) Letter test, majority reward, bonus after the normalization,

G=64

, with the end of the interior branch and the threshold near collapse.

	$G=8$	$G=16$	$G=32$	$G=64$
before, $c_{\rm b}^{*}=1/\log(G-1)$	$0.514$	$0.369$	$0.291$	$0.241$
after, broad, ten answers	$1.454$	$1.074$	$0.876$	$0.780$
after, broad, free-form answers	$1.375$	$0.938$	$0.705$	$0.565$
after, near collapse	$1.554$	$1.526$	$1.674$	$1.946$

Table 1: Thresholds of the bonus for rare answers on policies with one leading answer, from exact sums with the population standard deviation. Before the division, one threshold serves broad and collapsed policies. After it, a broad policy needs the end of the branch of interior equilibria, for ten or for free-form answers, and a collapsed one the threshold near collapse (Appendix C.6 ).

Figure 2: Free-form TTRL with groups of 8, every run at step 200 against the coefficient of the bonus. (a) F95e, the share of covered problems collapsed onto one answer. (b) Pass@1, with the untrained model dashed. Vertical lines mark the threshold

0.514

of arm C and, for arm R, the end

1.375

of its interior branch (dotted) and its threshold

1.554

near collapse (dashed), with the window shaded. The gray band spans the controls, and the arms are offset at

0.6

. Crosses mark the two runs that lost the answer format, drawn above the axis in (a).

method	acts as exploration
temperature	no, it rescales time (Proposition B.2 )
clipping, Adam	no, they move no equilibrium of exploration in the loss but can change outcomes in the window (Lemma 4.2 , Appendix D )
normalization	no, it rescales exploration it does not divide (Proposition 4.3 )
KL penalty	partly, it pulls toward the reference policy and bypasses the division of GRPO (Section 5.2 )
smoothing to $\bar{p}$	yes, boundaries $\frac{1}{n}$ and $\frac{n}{4(n-1)}$ (Proposition 4.4 )
entropy bonus	yes, a Potts window under NPG, the committed branch stable only numerically (Theorem B.3 )

Table 2: Common methods against collapse in the tabular flow, with the boundaries for the agreement reward and, for the bonus, the threshold of a broad policy under the majority reward on policies with one leading answer.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

claim	section	status	scope
learning rate and preconditioner keep equilibria and folds	§ 4.1	proved	exact gradients, networks of full rank
dividing only selection rescales exploration by $\varsigma(p)$	§ 4.1 , § 5.2	proved	removal of the window numerical, mean-field model
window $\big(\frac{1}{n},\frac{n}{4(n-1)}\big)$ for unordered answers	§ 4.2	proved	both geometries, classical under NPG
collapse onto the initial plurality	§ 4.2	proved	smoothing toward $\bar{p}$
prevention costs least early	§ 4.2	proved	one leading answer and equal others
boundaries of the majority reward	§ 4.2	exact computation	one leading answer and equal others

Appendix

Table 3: Status of the claims of Sections 4 to 7 . Proved means a theorem, proposition or corollary. Exact computation means exact sums over the counts of a group, without a proof of stability. Certified means a computer-assisted proof in interval arithmetic, and numerical means computed but not proved.

$w$	ring	$c^{*}$	lower fold (peak)	eigenvalue	upper fold (peak)	eigenvalue
$1$ , $\frac{3}{2}$	$128$	$0.44720$ , $0.29639$	none		none
$3$	$256$	$0.14775$	$0.14666$ ( $0.1028$ )	$5.3\times 10^{-3}$	$0.17437$ ( $0.2356$ )	$0.166$
$4$	$256$	$0.11077$	$0.11040$ ( $0.0762$ )	$1.9\times 10^{-3}$	$0.15891$ ( $0.2413$ )	$0.193$
$8$	$512$	$0.05537$	$0.05534$ ( $0.0380$ )	$1.9\times 10^{-4}$	$0.13999$ ( $0.2466$ )	$0.225$

Appendix

Table 5: Folds of the states centered between two bins on rings, in double precision, with the largest eigenvalue of

H

on the antisymmetric subspace at each fold. The column

c^{*}

repeats the certified lower fold of the states centered on a bin.

	$G=8$	$G=16$	$G=32$	$G=64$
$c_{\rm b}^{*}=1/\log(G-1)$	$0.514$	$0.369$	$0.291$	$0.241$
arm C, $c_{\rm lin}$	$0.910$	$0.676$	$0.635$	$0.714$
arm C, $c_{\max}$	$0.910$	$0.837$	$0.915$	$1.121$
arm C, $c_{\max}$ , $n=\infty$	$1.443$	$1.443$	$1.443$	$1.443$
arm R, end of branch ( $M$ there)	$1.454$ ( $0.81$ )	$1.074$ ( $0.73$ )	$0.876$ ( $0.73$ )	$0.780$ ( $0.75$ )
arm R, end of branch, $n=\infty$	$1.375$	$0.938$	$0.705$	$0.565$

Appendix

Table 9: Thresholds of the answer-level bonus on policies with one leading answer and equal others, from exact sums over the counts of a group with the population standard deviation and

n=10

unless stated. Before the normalization (arm C), collapse is released and prevented at the same coefficient

c_{\rm b}^{*}=1/\log(G-1)

. After it (arm R), a broad policy is kept from collapsing above the end of the branch of interior equilibria, with its leading mass

M

there in parentheses. A collapsed policy is released only above the threshold near collapse. In both arms, the uniform policy is linearly stable above

c_{\rm lin}

, and

c_{\max}

is the largest coefficient with an interior equilibrium. Rows marked unbiased use the unbiased standard deviation of the TTRL code. The last two rows divide the two thresholds of arm R by

c_{\rm b}^{*}

. For free-form answers, the interior branch of arm C ends at

1/\log 2

for every

G

setting, seeds and boundaries	below the lower	inside the window	above the upper
testbed, $w=3$ , seeds 0, 1 (2 to 4), $[0.143,0.283]$	collapsed at $0.14$	prevented from $0.16$ , $0.20$ , released from $0.28$	broad at $0.30$ , $0.32$
testbed, both normalized, seed 0 and partly 1 (2 to 4), $[0.131,0.276]$ estimated	collapsed at $0.10$ , $0.12$	prevented from $0.14$ to $0.18$ , released at $0.26$ by AdamW in seeds 0, 1 and 3	broad at $0.28$ , $0.30$
letter test, agreement, arm U, seeds 0 to 2 (3, 4), $[0.0875,0.2431]$ , arm B, seed 0, $[0.069,0.219]$	committed at $0.06$	depended on start, seed and learning rate, ramps of seeds 0 to 3 released at or just below the fold	broad at $0.26$ , $0.30$
letter test, agreement, arm R, seeds 0 (3, 4), $[1.10,1.59]$	committed at $0.22$ to $0.9$	at $1.2$ , broad start committed in 2 of 3 seeds and collapsed start released, both broad at $1.4$	broad at $1.6$ to $2.4$
letter test, majority ramps, arm U, seeds 0, 1 (2, 3), $[0.363,0.766]$	committed at the ends	committed at $0.48$ to $0.60$ , released at $0.70$ to $0.74$	none committed
letter test, majority ramps, arm R, seeds 0, 1 (2, 3), $[0.806,1.587]$	committed at the ends	loop closed in seed 0, open in seeds 1 to 3	one brief excursion in seed 0

Appendix

Table 10: Outcomes of the runs relative to the boundaries, with the new seeds of Appendix E.8 in parentheses. Testbed thresholds are SGD and AdamW seed means, or ranges in the second row. Broad means that at most half of the states or questions collapsed, and the bonus rows of the letter test used collapsed starts for

G=16

and

64

. The free-form row uses F95e at step 200.

arm	seed	release $c$	collapse $c$	loop, release minus collapse
U	0	$0.7356$	$0.5289$	$0.207$
U	1	$0.7044$	$0.5467$	$0.158$
R	0	$1.3933$	$1.3956$	$-0.002$
R	1	$1.5022$	$1.1111$	$0.391$

Appendix

Table 19: Transitions of the majority ramps. The release folds are

0.766

in arm U and

1.587

in arm R, and the lower boundaries

0.363

and

0.806

. The release of arm R over that of arm U is

1.89

in seed 0 and

2.13

in seed 1, against

2.07

computed for the majority reward and

6.55

for the agreement reward.

	$G=8$	$G=16$	$G=32$	$G=64$
arm C, threshold $c_{\rm b}^{*}$	$0.514$	$0.369$	$0.291$	$0.241$
arm C, lower and upper coefficient	$0.411$ , $0.642$	$0.295$ , $0.462$	$0.233$ , $0.364$	$0.193$ , $0.302$
arm R, threshold, end of branch	$1.454$	$1.074$	$0.876$	$0.780$
arm R, lower and upper coefficient	$1.163$ , $1.709$	$0.859$ , $1.342$	$0.701$ , $1.096$	$0.624$ , $0.974$
ratio of the brackets of arms R and C	$1.81$ to $4.16$	$1.86$ to $4.55$	$1.93$ to $4.70$	$2.07$ to $5.05$
arm R, threshold near collapse	$1.554$	$1.526$	$1.674$	$1.946$

Appendix

Table 20: Coefficients and outcomes of the bonus in the letter test. A threshold lies between the two coefficients if both seeds collapsed at the lower one and ended interior at the upper one. The ratio row divides the bracket of arm R by that of arm C, and the last row divides the upper coefficient of arm R by its threshold near collapse. Collapsed starts used

G=16

and

64

(Table 22 ).

difference	seed 1	seed 2	seed 3	seed 4	seed 5	seed 6	mean
arm C minus control	$+0.055$	$-0.031$	$+0.078$	$+0.047$	$+0.047$	$+0.063$	$+0.043$
arm R minus control	$+0.039$	$+0.039$		$+0.016$	$-0.031$	$-0.031$	$+0.006$
arm C minus before training	$-0.023$	$-0.141$	$+0.008$	$-0.055$	$+0.008$	$0.000$	$-0.034$

Appendix

Table 25: Best-of-32 accuracy by the process reward model, differences per seed over 128 problems.

Figure 3: Free-form TTRL at step 200 for every run whose answer format held, with the untrained model at step 0. Gray markers are controls, and colored markers are filled at the coefficients that prevented collapse, for arm R at

1.45

on the window set.

setting	lower boundary	upper boundary	source
agreement reward, smoothing toward $\bar{p}$ , $n$ answers	$\frac{1}{n}$	$\frac{n}{4(n-1)}$	Proposition 4.4
same, leave-one-out reward of groups of $G$	$\frac{G-2}{G}\cdot\frac{1}{n}$	$\frac{G-2}{G}\cdot\frac{n}{4(n-1)}$	Section 4.2
majority reward, $n=10$ , $G=16$ , smoothing toward $\bar{p}$	$0.363$	$0.766$	Appendix C.5
agreement reward, entropy bonus under NPG	$\frac{1}{n}$	$c_{\rm H}^{\rm s}(n)$ , $0.193$ for $n=16$	Theorem B.3
bonus before the division, arm C	$c_{\rm b}^{*}$ , $0.514$ for $G=8$	$c_{\rm b}^{*}$	Section 5.2
bonus after the division, arm R	end of the interior branch, $2.3$ to $3.2\,c_{\rm b}^{*}$	about $\frac{G}{\sqrt{G-1}}\,c_{\rm b}^{*}$	Table 1

Appendix

Table 29: Boundaries of the exploration coefficient in the tabular flow, without normalization unless stated. Prevention needs at least the lower boundary, and release follows above the upper one. The majority reward and the bonus are computed on policies with one leading answer, with

c_{\rm b}^{*}=1/\log(G-1)

, and the threshold of arm R near collapse comes from the sign of a drift.

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 49 papers

Oct 27, 2026

Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR

Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope

\langle F'\rangle

of the single-problem self-reinforcement drift

F

is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by

1/N

for

N

training problems, races against the shared mode, which grows at a rate

\proptoλ^2

independent of

N

, where

λ

is the strength of the shared coupling. This yields a critical number of problems

N_c\simeq\ln(1/δ)/(λ^2G)

, where

δ

seeds the shared mode and

G

decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition,

N_c\proptoλ^{-2}

up to a logarithmic correction, a single critical amplification

λ^2NG

across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.

Peize Ding

Columbia University, New York, NY, USA

RL for Language Models Reinforcement Learning RL Stability Unsupervised RL Reinforcement Learning with Verifiable Rewards Multi-Task RL

Ask PDF

May 19, 2026

Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting

Test-time reinforcement learning (TTRL) reports substantial accuracy gains on mathematical reasoning benchmarks using majority vote as a pseudo-label signal. We argue these gains are systematically misinterpreted: most reflect sharpening of already-solvable problems rather than genuine learning, while problems corrupted from correct to incorrect outnumber truly learned ones, and this damage is irreversible once majority vote locks onto a wrong answer. Per-problem tracking reveals that correct-answer signals in low-ability problems are briefly active before being permanently suppressed, a phenomenon we term the Correct-Answer Extinction Window, with Flip Rate (FR) as its leading indicator. We thus propose TTRL-Guard, a lightweight framework with three mechanisms targeting the extinction window: Flip-Rate-Aware Reward Scaling (FRS) down-weights at-risk updates as FR declines, Minority-Preserving Sampling (MPS) retains gradient signal from minority correct answers, and Risk-Conditioned Sparse Updatings (RCSU) suspends updates on polarized problems. Experiments across three models and four benchmarks show that TTRL-Guard achieves the best average pass@1 on Qwen2.5-7B-Instruct and Qwen3-4B, improves relatively over TTRL by +54% on AIME 2025.

Hongxiang Lin, Zhirui Kuai, Erpeng Xue +1

Meituan

Reinforcement Learning RL for Language Model Reasoning Mathematical Reasoning Test-Time RL Majority Voting

Ask PDF

Oct 2, 2026

Exploration-Preserving Policy Optimization

Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain accessible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group's total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses. Code is available at https://github.com/jinhangzhan/ExPPO

Hangzhan jin, Mohammad Hamdaqa, Doina Precup

Mila - Quebec AI Institute · Polytechnique Montr´eal · McGill University +2

RL for Language Model Reasoning RL Exploration Reinforcement Learning with Verifiable Rewards Preference Optimization Group Relative Policy Optimization

Ask PDF

Abstract

Test-Time RL

Explore similar work

Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR

Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting

Exploration-Preserving Policy Optimization