Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization

Jiaqi Tian; Ting Long

cs.AIOct 9, 2026

Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization

Authors: Jiaqi Tian, Ting Long

Organizations: School of Artificial Intelligence Jilin University China

Abstract

On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts. However, we find that OPSD methods can progressively degrade previously acquired capabilities: queries that are correctly responded at earlier training stages may become incorrectly responded as optimization proceeds, even the overall performance continues to improve. Further analysis reveals that this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation. Motivated by this observation, we propose Representation Geometry Regularization (RGR), which preserves representation geometry during post-training by encouraging compactness within cluster and separation across cluster. RGR can be seamlessly integrated into existing OPSD methods without altering their core optimization procedures. Extensive experiments on multiple OPSD methods and benchmarks show that RGR generally alleviates capability degradation while retaining capability improvements. Our code is available in https://github.com/supermachinelearner/RGR.

Figures & tables

Figure 1: (a) The capability degradation ratio of existing methods; (b) Although the overall performance improves during training, some problems that were previously responded correctly become incorrectly answered as optimization progresses.

Figure 2: Hidden-state representations and response correctness across training epochs. The query is answered correctly at epochs 25 and 50, but incorrectly at epoch 75.

Model	Method	AIME24	AIME25	HMMT25	Avg.
Qwen3-1.7B	OPSD	56.67	43.06	28.89	42.87
	OPSD + RGR	59.89	43.60	30.00	44.50 +1.63
	SDPO	38.61	28.61	21.39	29.54
	SDPO + RGR	41.94	31.39	21.94	31.76 +2.22
	RLSD	50.27	39.70	25.83	38.60
	RLSD + RGR	51.39	39.17	25.56	38.70 +0.10

Table 1: Performance gains from RGR. Best results in bold; RGR improvements in green.

Method	AIME24	AIME25	HMMT25
OPSD	56.67	43.06	28.89
w/o Ent	57.22	41.39	28.33
w/o Geo	58.61	42.22	29.17
RGR	59.89	43.60	30.00

Table 2: Ablation study of RGR.

Figure 3: Comparison of the capability degradation ratio between the baseline and the baseline+RGR model on AIME24 (%).

Figure 4: Visualization of hidden state representations across different models and training steps, with response correctness indicated.

Method	Before	After	Forgetting $\downarrow$
OPSD	56.67	54.72	1.95
OPSD + RGR	59.89	58.06	1.83
SDPO	38.61	33.33	5.28
SDPO + RGR	41.94	40.00	1.94
RLSD	50.27	47.22	3.05
RLSD + RGR	51.39	48.00	3.39

Table 3: Forgetting evaluation on AIME24 (%). Before denotes the performance before training on the tool-use dataset, while After denotes the performance after tool-use training.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Sensitivity analysis of the hyperparameters

\alpha

and

\beta

on AIME24. Left: performance under different values of

\beta

, with

\alpha

fixed at

2.5\times 10^{-2}

. Right: performance under different values of

\alpha

, with

\beta

fixed at

5\times 10^{-2}

. The stars indicate the selected setting:

\alpha=2.5\times 10^{-2}

and

\beta=5\times 10^{-2}

Hyperparameter	Value
Learning rate	$5\times 10^{-6}$
Max gradient norm	0.1
LoRA rank $r$	64
LoRA $\alpha$	128
Max sequence length	20,000
Max completion length	1,024

Appendix

Table 4: Hyperparameters used across all experiments.

Symbol	Description
$q$	a query
$\pi_{\theta}$	LLM policy parameterized by $\theta$ and trained within the OPSD framework.
$\theta$	Parameters of the LLM policy.
$y_{t}$	Token generated at decoding step $t$ by $\pi_{\theta}$ .
$\mathbf{y}=(y_{1},\ldots,y_{T})$	Response consisting of $T$ tokens generated by $\pi_{\theta}$ .
$\mathbf{y}_{<t}=(y_{1},\ldots,y_{t-1})$	Sequence of tokens generated before step $t$ .

Appendix

Table 5: Summary of the main notation used in the paper.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 172 papers

Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Sep 10, 2026Rongcan Pei, Zhepei Wei, Shuyao Xu +3Language Model Distillation Self-Distillation
Department of Computer Science, University of Virginia · Stanford University
Self-Policy Distillation via Capability-Selective Subspace Projection
May 21, 2026Guangya Hao, Hanxue Liang, Yitong Shang +2Self-Training for Language Models Language Model Distillation
University of Cambridge · HKUST · University of Chicago
On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3On-Policy Self-Distillation Language Model Self-Improvement
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1

Sep 10, 2026J/K move · Enter open · S save

Negative Self-Distillation: Learning to Reason by Avoiding Flaws

Rongcan Pei, Zhepei Wei, Shuyao Xu, Xinyu Zhu+2

Department of Computer Science, University of Virginia · Stanford University

Language Model Distillation Self-Distillation Language Model Self-Improvement LLM Reasoning

Open paper AskPDF arXiv

On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for large language model (LLM) self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation (NSD), a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition (eg, acting as a ``careless reasoner'') and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines.

Abstract

Language Model Self-Improvement

Explore similar work

Negative Self-Distillation: Learning to Reason by Avoiding Flaws

Self-Policy Distillation via Capability-Selective Subspace Projection

On-Policy Self-Distillation without Any Supervision