Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR

Peize Ding

cs.LGOct 27, 2026

Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR

Authors: Peize Ding

Organizations: Columbia University, New York, NY, USA

Abstract

Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope $\langle F'\rangle$ of the single-problem self-reinforcement drift $F$ is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by $1/N$ for $N$ training problems, races against the shared mode, which grows at a rate $\proptoλ^2$ independent of $N$ , where $λ$ is the strength of the shared coupling. This yields a critical number of problems $N_c\simeq\ln(1/δ)/(λ^2G)$ , where $δ$ seeds the shared mode and $G$ decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition, $N_c\proptoλ^{-2}$ up to a logarithmic correction, a single critical amplification $λ^2NG$ across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.

Figures & tables

Figure 1: Mechanism. (a) Single-problem drift

F

and its slope

F^{\prime}

. The shaded window

|z|<z^{\star}

is where a shared push is amplified. (b) Majority-vote reward (solid) and accuracy against the hidden labels (dashed) at

\lambda{=}0.1

a{=}0.5

, averaged over 24 seeds. The two runs differ only in the number of training problems, which lies below (blue) or above (orange) the fitted

N_{c}\approx 536

. Below

N_{c}

, local sharpening wins: the reward rises and accuracy stays near its initial level. Above

N_{c}

, the reward rises faster and accuracy falls to chance, because every problem converges to the same answer. (c) Order parameter

\Phi

of four individual seeds with identical settings at

N=584

, the simulated size closest to

N_{c}

. Orange runs end above the collapse threshold

\Phi=0.5

(dotted) and blue runs below it. Sampling noise decides the outcome within the first

{\sim}100

epochs, while problems are still inside the unstable window, and

\Phi

then freezes. A run can also collapse partially (upper blue curve).

Figure 2: Critical number of problems. (a) Probability of collapse against

N

for seven coupling strengths (

a{=}0.5

). (b) The same data, together with a sweep over initial confidence

a

(orange,

\lambda{=}0.1

), plotted against the predicted amplification

\Lambda=\lambda^{2}N\ln[F^{\star}/F(a)]

. Curves spanning a

64\times

range in

\lambda^{-2}

and critical sizes from

60

8740

collapse onto a single transition at

\Lambda_{c}\approx 2.6

(dotted). (c)

N_{c}

against

\lambda

. The fitted slope is

-2.22

, compared with the predicted

-2

plus a logarithmic correction. (d)

N_{c}

against initial confidence. The dashed line is the theory with the single constant

\Lambda_{c}

. Beyond

z^{\star}

(shaded) the theory predicts no collapse at any

N

Figure 3: Generalizations. (a) Heterogeneous initial confidence.

P(\text{collapse})

against the amplification

\lambda^{2}NG[\rho_{0}]

predicted from the initial distribution alone; the dotted line is

\Lambda_{c}

from Figure 2 . (b) Prediction-aligned coupling

\mu

delays and then eliminates collapse. Inset:

N_{c}

against

\mu

with the prediction

1/(\lambda^{2}G/\Lambda_{c}-\mu^{2})

. Points at the top edge did not collapse. (c) Final accuracy with the marginal-entropy bonus. Vertical lines mark the predicted

\beta_{c}(a)

Figure 4: Label-free RL on a small Transformer. (a) Majority-vote reward (solid) and held-out accuracy (dashed) of one run at

N{=}512

, for base 90 at

\tau{=}2

(orange) and base 120 at

\tau{=}1

(blue). (b) Plain Transformer, base 90: probability of collapse against

N

at three temperatures, as the observed fraction of 6 seeds. (c) Antisymmetric Transformer: probability of collapse, averaged over

N

, against the added shared coupling

\rho_{b}

, for each base and

\tau

. Solid: observed. Dotted: kernel-calibrated surrogate in the same color, from 24 simulated seeds per point.

\rho_{b}{=}0

is drawn at the left edge. (d) Observed against surrogate-predicted probability of collapse. Each point is one condition at one training-set size

N

, summarizing 6 RL runs, so observed values are fractions of 6 and many points overlap. Dashed: perfect calibration; points above it collapse more often than predicted. Title: Pearson correlation

r

and the fraction of points in which prediction and observation lie on the same side of

\tfrac{1}{2}

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

$a$	$\delta=10^{-2}$	$\delta=10^{-3}$	$\delta=10^{-4}$	$\delta=10^{-6}$
0.1	2.25	4.50	6.77	11.32
0.3	3.23	5.47	7.72	12.26
0.5	3.60	5.83	8.07	12.59
0.8	3.75	5.96	8.19	12.70
1.1	3.50	5.69	7.92	12.41
1.25	2.99	5.16	7.38	11.85

Appendix

Table 1: Deterministic mean-field critical amplification

X_{c}G(a)

(with

X=\lambda^{2}N

) against the closed-form prediction

\ln(1/\delta)

Figure 5: Batch size and long training. (a)

N_{c}

against batch size, with a fixed step (blue) and with linear step scaling (orange). The dashed line is the logarithmic prediction anchored at

B{=}32

. (b) Final order parameter after

10\times

longer training for confident initial models (

a>z^{\star}

) at values of

\lambda^{2}N

up to

1474

. Uncertain models collapse for

\lambda^{2}N\gtrsim 5

. (c) Order-parameter trajectories: confident models stay balanced, and sub-critical uncertain models remain balanced after sharpening.

Figure 6: Robustness. (a) GRPO and REINFORCE for

K\in\{3,5,8,16\}

against

\Lambda=\lambda^{2}NG_{K}

, where

G_{K}

( Section 4.3 ) is computed from the exact

K

-sample drift

F_{K}

( Section 3 ). (b)

M

-answer softmax policies against

\Lambda=\lambda^{2}NG_{M}

. (c) Adam: probability of collapse against

N

for several couplings.

	$K=3$	$4$	$5$	$8$	$16$
REINFORCE: $z^{\star}_{K}$
	1.317	1.274	1.187	1.088	0.894
GRPO (std-normalized): $z^{\star}_{K}$
	1.317	1.317	1.213	1.172	1.006
REINFORCE: $G_{K}(0.5)$
	0.514	0.484	0.426	0.353	0.202

Appendix

Table 2: Edge of the unstable window

z^{\star}_{K}

(where

F_{K}^{\prime}=0

) and the amplification factor

G_{K}(0.5)=\ln[F^{\star}_{K}/F_{K}(0.5)]

for different estimators and group sizes

K

(binary answers). Computed exactly by enumeration.

Figure 7: Marginal-entropy bonus on the plain base-90 Transformer at

\tau=2

. Lines: mean over 4 seeds; dots: individual seeds. The dashed line marks

\beta=\tfrac{1}{2}

, above which the theory predicts stability ( Equation 9 ). (a) Final order parameter

\Phi

on the training problems; runs above the dotted line count as collapsed. The bonus removes collapse for

\beta\geq\tfrac{1}{2}

. (b) Held-out accuracy stays near chance at every

\beta

, far below the base model’s initial accuracy (dotted, top).

model	base (acc.)	$\tau$	$\rho_{b}$	$k_{u}/k_{d}^{\prime}$	$k_{s}/k_{d}^{\prime}$	$G$	collapsed
plain	90 (82%)	1	–	0.702	0.051	0.025	29/42
plain	90 (82%)	2	–	0.702	0.051	0.315	34/42
plain	90 (82%)	3	–	0.702	0.051	0.622	37/42
plain	120 (89%)	1	–	0.404	0.058	0.000	17/42
plain	120 (89%)	2	–	0.404	0.058	0.028	35/42
plain	120 (89%)	3	–	0.404	0.058	0.186	38/42

Appendix

Table 3: The 22 Transformer conditions of Section 6 . The kernel ratios and the amplification

G

are measured on the initial network before any RL step. The base column gives the number of supervised warm-up steps and the resulting held-out accuracy. The last column counts collapsed runs out of 42 (7 values of

N

and 6 seeds).

case	epoch	$k_{u}/k_{d}^{\prime}$	$k_{s}/k_{d}^{\prime}$	$k_{d}(t)/k_{d}(0)$	$\Phi$
antisym 10, $\tau$ =1, $\rho_{b}$ =0.3, $N$ =1024	0	0.345	0.152	1.0	0.02
	1	0.009	0.222	28.1	0.28
	4	0.002	0.074	101.9	0.14
antisym 20, $\tau$ =1, $\rho_{b}$ =0.3, $N$ =2048	0	0.429	0.423	1.0	0.02
	1	0.219	0.066	1.4	0.21
	4	0.121	0.051	2.2	0.33

Appendix

Table 4: Kernel drift during label-free RL (one seed per case). Ratios are reported only before collapse (

\Phi<0.5

): once all answers agree,

s\propto\mathbf{1}

and the decomposition degenerates.

k_{d}(t)/k_{d}(0)

is the growth of the self-kernel, i.e. of the effective step size.

Sweep	Varied	Notes
coupling	$\lambda\in\{0.035,\dots,0.28\}$ , 9 values of $N$ each	$a=0.5$
confidence	$a\in\{0.15,0.3,0.5,0.8,1.1,1.25\}$	$\lambda=0.1$ , $N\leq 32768$
batch, fixed step	$B\in\{8,16,32,128,512\}$	epochs $=300B/32$ (equal step count)
batch, linear scaling	$B\in\{8,\dots,512\}$	$\eta=4B/32$ , 300 epochs
long training	$a\in\{1.4,1.6,2.0\}$ , $\lambda\in\{0.1,0.3\}$	3000 epochs (600 at $N=16384$ ); 16 seeds
heterogeneous	7 initial distributions	$\lambda=0.1$

Appendix

Table 5: Simulation settings. Unless listed,

K{=}3

, REINFORCE,

B{=}32

, SGD step

\eta{=}4

, 300 epochs, 24 seeds, and

80\%

of initial majorities correct.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 435 papers

Jun 13, 2026

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models. However, RLVR often suffers from diversity collapse: Pass@

1

improves while high-

k

Pass@

k

degrades, which is viewed as a narrowing of the model's reasoning boundary. We formalize this diversity collapse through the lens of overtraining: once a problem's contribution to the reference metric has effectively saturated, further updates no longer expand what the model can solve but still concentrate probability mass on the trajectories favored by on-policy sampling. Under a standard setup with few rollouts per problem, even a single observed success places a problem in a nearly saturated regime for high-

k

Pass@

k

, so most updates in standard RLVR are overtraining from the boundary perspective. This perspective also suggests a reading of whether RLVR can expand the model's reasoning abilities beyond the base model: since RLVR is structurally biased against high-

k

Pass@

k

, its aggregate decline does not by itself mean that no new reasoning gains occurred. Interventionally, restricting updates to problems with zero observed success lifts Pass@

256

above the base model on difficult benchmarks; observationally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Building on these findings, we propose Bayesian Boundary Gating (BBG), which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Across multiple reasoning benchmarks, BBG improves average Pass@

k

across a wide range of

k

Suqin Yuan, Jinkun Chen, Jiyang Zheng +6

Sydney AI Centre, The University of Sydney · Southeast University · Microsoft +3

Reinforcement Learning Reasoning Diversity RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

May 21, 2026

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of LLMs, but often depends on external supervision from human annotations or gold-standard solutions. Reinforcement learning from internal feedback (RLIF) has recently emerged as a scalable unsupervised alternative, using signals extracted from the model itself. However, existing RLIF methods typically rely on a single internal reward, which can lead to reward hacking, entropy collapse, and degraded reasoning structure. We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components: an answer-level reward based on cluster voting and a completion-level reward based on token-wise self-certainty. To combine these signals robustly, we apply GDPO-based normalization to reduce reward-scale imbalance. We further introduce KL-Cov regularization, which targets low-entropy token distributions responsible for disproportionate entropy reduction, preserving exploration and preventing late-stage collapse. Across mathematical reasoning and code-generation benchmarks, our method improves stability and robustness over prior unsupervised RL approaches, while achieving performance close to supervised RLVR methods. These results show that complementary internal rewards, combined with targeted regularization, can support stable long-horizon reasoning without relying on external ground-truth supervision. Code will be released soon.

Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash +2

Bangladesh University of Engineering and Technology, Bangladesh · West Virginia University, USA · University of Aberdeen, UK +2

KL-Regularized RL Intrinsic Reward Methods Reinforcement Learning Multi-Objective Reinforcement Learning RL for Language Model Reasoning Reward Design for RL+1

Ask PDF

May 4, 2026

Binary Rewards and Reinforcement Learning: Fundamental Challenges

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for improving reasoning in language models, yet models trained with RLVR often suffer from diversity collapse: while single-sample accuracy improves, multi-sample coverage degrades, sometimes falling below the base model. We provide a structural account of this phenomenon grounded in the properties of binary rewards. Binary rewards create a fundamental degeneracy for policy gradient methods: the set of distributions maximizing expected reward is infinite, with no distinguished element. KL-control resolves this degeneracy by selecting, in the limit

β\to 0

, the filtered model

p_*:=a(\cdot\mid\mathcal{Y}_1)

-- the base model conditioned on validity -- which is the unique fully valid distribution closest to the base model in KL divergence. This selection operates through a nontrivial asymmetry: the tilted distribution

p_{[β]}\propto a(y)\,e^{v(y)/β}

converges to

p_*

in forward KL as

β\to 0

, yet

p_*

cannot serve as a direct optimization target because

\mathrm{KL}(q\,\|\,p_*)

is infinite for any full-support policy

q

. We develop explicit formulas relating the hyperparameter

β

to the more interpretable target validity rate

μ

. Under model misspecification -- the typical practical regime -- the pressure to decrease

β

drives the optimizer toward highly concentrated distributions over a small number of valid outputs, collapsing toward ever fewer as

β

decreases, rather than toward the filtered model. We illustrate this mechanism on a toy autoregressive experiment and discuss how alternative divergences that target

p_*

directly -- as pursued empirically by \citet{kruszewski_whatever_2026} -- avoid this failure mode by rewarding coverage of

p_*

's support rather than concentration on high-validity outputs.

Marc Dymetman

∗Scientific Consultant

Reinforcement Learning KL-Regularized RL RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Reward Design for RL Behavioral Diversity in RL

Ask PDF

Abstract

RL for Language Models

Explore similar work

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

Binary Rewards and Reinforcement Learning: Fundamental Challenges