Stable but Wrong: When Learning Stabilizes Away from the Truth

Zhipeng Zhang

cs.LGMar 23, 2026

Stable but Wrong: When Learning Stabilizes Away from the Truth

Authors: Zhipeng Zhang

Organizations: China Mobile Research Institute, Beijing 100053, China · China Mobile GBA (Greater Bay Area) Innovation Institute, Guangzhou 510656, China

Abstract

Stable training is often treated as evidence that learning is succeeding, but stability characterizes optimization behavior rather than correctness relative to an external objective. We study what happens when the signal being optimized remains persistently biased. We define Stable but Wrong (SBW) as a learning state in which the learning process remains stable under a task-appropriate operational criterion while the learned outcome remains systematically displaced from an independently defined objective. A minimal strongly convex model shows that a persistent bias in the update direction can shift the unique convergence point away from the true optimum. Controlled experiments in reinforcement learning, supervised learning, and continual fine-tuning of a large language model reveal a recurring separation between apparently normal optimization and correctness under static and feedback-coupled biases. Recovery-stage clean-data access and exploration interventions further show that subsequent trajectories can remain modifiable, although the interventions differ in protocol and do not imply a shared mechanism. The results expose a basic limit of optimization stability as a reliability signal in persistent and feedback-coupled learning systems.

Figures & tables

Figure 1: Stable but Wrong under acquisition bias in supervised learning. Controlled PadChest example in which sample selection creates perfect Cardiomegaly–AP and Normal–PA label–acquisition-domain collinearity. Images and AP/PA metadata are drawn from real clinical data; the perfect collinearity is experimentally constructed and does not represent the original PadChest distribution. Left, training-data construction; middle, training dynamics; right, counterfactual evaluation in which disease labels are retained while acquisition domains are exchanged.

Figure 2: Stable but Wrong in a controlled synthetic autonomous-discovery system. The Synthetic Discovery System does not use real A-Lab data or an experimental platform. The figure compares the learned belief of a Bayesian-optimization surrogate model with the known oracle landscape: after the belief stabilizes, the region identified as optimal remains displaced from the true global optimum.

Figure 3: Learning dynamics and correctness under persistent bias across learning paradigms. (a,b) Static and dynamic bias for PPO on Hopper-v4 (Bias Exposure $=750$ k steps). Red, true return; blue, a policy-update-magnitude proxy; shading, s.d. over five seeds. The update proxy indicates that policy updates continue without explicit divergence; it is not a proof of strict convergence. (c,d) Structural and prediction-dependent label bias for ResNet-18 on CIFAR-10 (epochs 30–79). Blue, training loss; red, clean target-class accuracy; shading, s.d. over five seeds. (e,f) Static and dynamic bias during continual fine-tuning of Qwen2.5-7B-Instruct on GSM8K. Blue, windowed training loss; red, Target final-answer accuracy; shading, s.d. over five seeds.

Figure 4: A recovery-stage clean-rollout protocol is associated with higher realized recovery. Within each Exposure/seed pair, Standard Recovery and clean-rollout Recovery start from the same corresponding bias-end checkpoint. Clean-rollout Recovery introduces a fresh clean-environment rollout every five PPO rollouts. The nominal Recovery training clock and PPO update loop are matched, but the clean-rollout condition requires additional clean-environment data collection. The vertical axis shows Recovery Score.

Figure 5: Exploration mitigates sampled-support contraction. Results from 20 paired random seeds in the Controlled Synthetic Autonomous Discovery System. Red circles show the exploitation-only baseline and blue squares the exploration intervention; solid lines show generation-wise means and shading denotes

\pm 1

sample s.d. PR is an operational measure of effective sampled-support size in a fixed low-dimensional projection and does not directly measure future information accessibility. At generation 30, final PR was

8.61\pm 6.31

under the intervention and

7.25\pm 6.88

under the baseline. PR increased in 17 of 20 paired seeds; the mean paired difference was

1.36

(95% paired-bootstrap CI,

0.55

–

2.20

; two-sided Wilcoxon signed-rank test,

p=0.00315

Figure 6: Offline replay comparison on the public CAMEO benchmark. The horizontal axis shows replay step and the vertical axis shows cumulative benchmark-target discoveries. Gray, fixed reference replay order; blue, exploration-protected replay. Both trajectories share the candidate pool and first 20 samples. Benchmark targets are used only for hindsight evaluation and do not participate in selection. The reference replay order does not represent the original experimental timeline.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Exposure length	$R_{\mathrm{clean}}$	$R_{\mathrm{before}}$	Relative decrease	$R_{\mathrm{after}}$
$0.25\mathrm{M}$	$3193\pm 224$	$2046\pm 230$	$35.9\%$	$2958\pm 548$
$0.50\mathrm{M}$	$3193\pm 224$	$1990\pm 295$	$37.7\%$	$2538\pm 564$
$0.75\mathrm{M}$	$3193\pm 224$	$2250\pm 698$	$29.6\%$	$2677\pm 427$
$1.00\mathrm{M}$	$3193\pm 224$	$1862\pm 459$	$41.7\%$	$2375\pm 563$

Appendix

Table 1: SBW-related evidence under static bias (reward shift

c=-1.0

) on Hopper-v4. Values are mean

\pm

s.d. over five seeds.

Environment	$p$	Exposure $0.25\mathrm{M}$	Exposure $0.50\mathrm{M}$	Exposure $1.00\mathrm{M}$
Hopper	$0.3$	$2868$ ( $8.9\%$ )	$3191$ ( $-1.3\%$ )	$2496$ ( $20.7\%$ )
Hopper	$0.5$	$1493$ ( $52.6\%$ )	$1388$ ( $55.9\%$ )	$494$ ( $84.3\%$ )
Hopper	$0.7$	$336$ ( $89.3\%$ )	$151$ ( $95.2\%$ )	$49$ ( $98.4\%$ )
Walker	$0.3$	$2417$ ( $-5.5\%$ )	$2489$ ( $-8.6\%$ )	$2623$ ( $-14.5\%$ )
Walker	$0.5$	$2053$ ( $10.4\%$ )	$1593$ ( $30.4\%$ )	$1290$ ( $43.7\%$ )
Walker	$0.7$	$1148$ ( $49.9\%$ )	$319$ ( $86.1\%$ )	$44$ ( $98.1\%$ )

Appendix

Table 2: Corrupt-end performance relative to the clean reference under dynamic bias (advantage sign flip). Values are means over five seeds; parentheses show the relative decrease.

Condition	$0.25\mathrm{M}$	$0.50\mathrm{M}$	$0.75\mathrm{M}$	$1.00\mathrm{M}$	Spearman $\rho$
Hopper Static ( $c=-1$ )	$0.879\pm 0.671$	$0.491\pm 0.488$	$-0.231\pm 1.220$	$0.514\pm 0.608$	$-0.28$
Hopper Dyn. $p=0.3$	$-3.70\pm 10.7$	$0.087\pm 1.52$	$3.22\pm 6.77$	$0.045\pm 1.09$	$-0.02$
Hopper Dyn. $p=0.5$	$0.764\pm 0.308$	$0.881\pm 0.158$	$0.856\pm 0.091$	$0.824\pm 0.225$	$+0.15$
Hopper Dyn. $p=0.7$	$0.917\pm 0.110$	$0.702\pm 0.290$	$0.369\pm 0.347$	$0.103\pm 0.101$	$-0.82^{***}$
Walker Dyn. $p=0.3$	$-0.27\pm 3.02$	$-1.34\pm 7.69$	$-1.61\pm 2.57$	$1.43\pm 2.55$	$+0.23$
Walker Dyn. $p=0.5$	$115.8\pm 248.8^{\dagger}$	$2.28\pm 1.28$	$1.43\pm 1.02$	$1.41\pm 1.04$	$-0.60^{**}$

Appendix

Table 3: Recovery Score under the Progressive RL protocol. Values are mean

\pm

s.d. over five seeds. Static bias was evaluated on Hopper; dynamic bias was evaluated on Hopper and Walker for

p\in\{0.3,0.5,0.7\}

Exposure length	Training Loss	Target Accuracy	Non-target Accuracy
Clean	–	$67.05\pm 2.35\%$	$76.98\pm 1.26\%$
Exposure start (0)	$0.237\pm 0.034$	$67.05\pm 2.35\%$	$76.98\pm 1.26\%$
1000	$0.055\pm 0.002$	$11.30\pm 4.54\%$	$74.53\pm 1.53\%$
3000	$0.011\pm 0.001$	$6.47\pm 0.99\%$	$68.42\pm 2.31\%$
5000	$0.008\pm 0.001$	$5.80\pm 1.01\%$	$69.86\pm 1.67\%$
8000	$0.007\pm 0.001$	$6.38\pm 1.54\%$	$69.62\pm 3.13\%$

Appendix

Table 4: Core SBW evidence under static bias for Qwen2.5-7B-Instruct. Values are mean

\pm

s.d. over five seeds. Training Loss is a

W=100

window mean.

Exposure length	Recovery Score
1000	$0.940\pm 0.058$
3000	$0.940\pm 0.060$
5000	$0.962\pm 0.054$
8000	$0.960\pm 0.066$

Appendix

Table 5: Recovery Score after static bias for Qwen2.5-7B-Instruct (five seeds; Recovery

=

5000 steps of Clean data; mean

\pm

s.d.).

Global step	Training Loss	Target Accuracy	Non-target Accuracy
0 (exposure start)	$0.246\pm 0.030$	$67.05\pm 2.35\%$	$76.98\pm 1.26\%$
200	$0.133\pm 0.001$	$31.01\pm 6.82\%$	$70.85\pm 3.03\%$
700	$0.065\pm 0.001$	$19.13\pm 8.15\%$	$74.03\pm 0.92\%$
1200	$0.037\pm 0.002$	$15.75\pm 5.12\%$	$73.38\pm 1.25\%$
1700	$0.022\pm 0.001$	$13.43\pm 3.18\%$	$72.23\pm 1.37\%$

Appendix

Table 6: Core SBW evidence under dynamic bias for Qwen2.5-7B-Instruct (five seeds, aligned checkpoints; mean

\pm

s.d.). Training Loss is a

W=100

window mean, as in the static condition.

Group	Training	Validation	Test
Cardiomegaly + AP	143	21	29
Normal + PA	132	27	28
Cardiomegaly + PA	0	0	25
Normal + AP	0	0	28
Total	275	48	110

Appendix

Table 7: Dataset split. The final two groups (Cardiomegaly+PA and Normal+AP) are absent from training and validation and are used to test whether the model relies on projection-view shortcuts.

Group	$n$	Accuracy
Cardiomegaly + AP (aligned)	29	100.0%
Normal + PA (aligned)	28	96.4%
Cardiomegaly + PA (association broken)	25	40.0%
Normal + AP (association broken)	28	39.3%

Appendix

Table 8: Classification accuracy on association-aligned and association-breaking test samples.

Mechanism	Exposure length	$R_{\mathrm{pre}}$	$R_{\mathrm{post}}$	$\mathrm{RT@95\%}$
Structural Patch	10	$92.32\pm 5.41$	$92.65\pm 0.48$	$1.20\pm 0.45$
Structural Patch	20	$90.76\pm 5.37$	$91.74\pm 1.02$	$1.40\pm 0.55$
Structural Patch	30	$90.02\pm 6.63$	$92.31\pm 1.12$	$1.00\pm 0.00$
Structural Patch	50	$73.14\pm 12.88$	$89.08\pm 5.59$	$1.00\pm 0.00$
Label Flip	10	$91.92\pm 3.02$	$91.80\pm 1.42$	$1.00\pm 0.00$
Label Flip	20	$93.06\pm 3.84$	$92.36\pm 1.35$	$1.00\pm 0.00$

Appendix

Table 9: Recovery behavior after three static-bias mechanisms on CIFAR-10 (five seeds; mean

\pm

s.d.).

R_{\mathrm{pre}}

and

R_{\mathrm{post}}

are automobile test accuracies (%).

Exposure length	$R_{\mathrm{pre}}$ (%)	$R_{\mathrm{post}}$ (%)	$\mathrm{RT@95\%}$
10	$89.24\pm 7.32$	$92.25\pm 1.21$	$1.20\pm 0.45$
20	$94.10\pm 2.29$	$91.81\pm 1.69$	$1.00\pm 0.00$
30	$92.22\pm 3.02$	$90.91\pm 1.03$	$1.00\pm 0.00$
50	$89.68\pm 7.04$	$92.19\pm 0.55$	$1.20\pm 0.45$

Appendix

Table 10: Recovery behavior after dynamic bias (prediction-dependent label flip) on CIFAR-10 (five seeds; mean

\pm

s.d.).

Exposure length	Cumulative modified fraction	eRank (pre $\rightarrow$ post)	$R_{\mathrm{pre}}$ (%)	$\mathrm{RT@95\%}$
10	$0.361\pm 0.006$	$36.2\rightarrow 28.0$	$93.68\pm 2.28$	$1.00\pm 0.00$
20	$0.499\pm 0.003$	$34.0\rightarrow 27.1$	$90.82\pm 2.11$	$1.00\pm 0.00$
30	$0.584\pm 0.008$	$31.6\rightarrow 25.8$	$92.66\pm 1.80$	$1.00\pm 0.00$
50	$0.698\pm 0.010$	$29.3\rightarrow 24.7$	$91.88\pm 1.38$	$1.00\pm 0.00$

Appendix

Table 11: Closed-loop effects and recovery behavior under Self-Induced Data Drift on CIFAR-10 (five seeds; mean

\pm

s.d.).

Exposure length	$R_{\mathrm{pre}}$	$R_{\mathrm{post}}$	$\mathrm{RT@95\%}$
10	$36.40\pm 22.24$	$44.84\pm 3.28$	$2.00\pm 1.22$
20	$46.40\pm 19.46$	$47.48\pm 8.77$	$2.20\pm 1.64$
30	$38.00\pm 5.10$	$48.88\pm 3.88$	$3.40\pm 2.61$
50	$40.00\pm 8.12$	$44.76\pm 3.92$	$2.00\pm 0.71$

Appendix

Table 12: Recovery after Static Bias (Structural Patch) on Tiny-ImageNet (five seeds; mean

\pm

s.d.).

R_{\mathrm{pre}}

and

R_{\mathrm{post}}

are validation accuracies (%) for the target class, sports car.

Exposure length	Cumulative modified fraction	eRank (pre $\rightarrow$ post)	$R_{\mathrm{pre}}$	$R_{\mathrm{post}}$	$\mathrm{RT@95\%}$
10	$43.32\pm 0.32$	$162.5\pm 8.3\rightarrow 145.1\pm 9.4$	$44.40\pm 23.64$	$46.24\pm 3.81$	$2.00\pm 0.71$
20	$60.68\pm 0.40$	$144.4\pm 9.2\rightarrow 134.1\pm 9.7$	$58.80\pm 7.69$	$43.08\pm 5.47$	$1.00\pm 0.00$
30	$70.30\pm 0.47$	$132.0\pm 8.7\rightarrow 129.6\pm 7.9$	$44.40\pm 16.64$	$50.12\pm 3.22$	$1.40\pm 0.55$
50	$81.11\pm 0.52$	$116.9\pm 5.7\rightarrow 119.3\pm 5.4$	$41.60\pm 23.60$	$46.28\pm 4.38$	$1.60\pm 0.89$

Appendix

Table 13: Closed-loop effects and recovery under Self-Induced Data Drift on Tiny-ImageNet (five seeds; mean

\pm

s.d.). The cumulative modified fraction is the proportion of samples relabeled by the end of Bias Exposure; eRank is the effective representation rank (pre

\rightarrow

post);

R_{\mathrm{pre}}

and

R_{\mathrm{post}}

are target-class validation accuracies (%).

Supervised Learning

See all 27 papers

Retraining Seeks Stable Signals
Jul 17, 2026Moritz HardtPerformative Prediction Statistical Learning Theory
Max Planck Institute for Intelligent Systems, Tübingen, and Tübingen AI Center
Score Centering Stabilizes Off-policy Reinforcement Learning
Sep 17, 2026Martin Marek, Max RyabininRL for Language Models Off-Policy RL
Together AI
When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning
Sep 27, 2026Steven Y. Feng, Noah D. Goodman, Michael C. Frank +3LLM Reliability Reinforcement Learning
Stanford University · Anthropic

Jul 17, 2026J/K move · Enter open · S save

Retraining Seeks Stable Signals

Moritz Hardt

Max Planck Institute for Intelligent Systems, Tübingen, and Tübingen AI Center

Performative Prediction Statistical Learning Theory

Open paper AskPDF arXiv

Predictive models deployed at scale influence future data, a phenomenon called performativity. And there is always one way to cope: Train the model on new data, deploy it again, and repeat. This process, called retraining or repeated risk minimization, creates a feedback loop between model and data that real-world learning systems can't avoid. Results on performative prediction shed light on this dynamic: If the model's influence on the data is small, retraining reaches a fixed point. What remains open is why fixed points should naturally exist, and what governs retraining when the model's influence is strong. In this work we develop a new perspective on retraining -- the stable signal principle -- that addresses these questions. We start from the assumption that the prediction target has at least some small model-independent component, a stable signal, such as the intrinsic quality of an item. We prove that when a nonzero stable signal exists, repeated risk minimization, suitably regularized, converges geometrically to the direction of this stable signal. This is true even if the model's influence on the target is arbitrarily large relative to the stable signal. Regularization emerges naturally as a force to control performativity, rather than to promote generalization, revealing a new facet of an old concept. We extend the analysis to a broad family of affine retraining operators under arbitrary model-induced feature changes, heterogeneous time-varying effects, and nonlinear responses. The stable signal perspective also applies to data feedback loops in language modeling, providing new explanations for the stability of language model training from model-generated data.

Abstract

Supervised Learning

Explore similar work

Retraining Seeks Stable Signals

Score Centering Stabilizes Off-policy Reinforcement Learning

When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning