Understanding Negative Learning in Large Language Model Fine-tuning

Yue Wang; Qizhou Wang; Zizhuo Zhang; Gang Niu; Bo Han; Masashi Sugiyama

cs.LGNov 30, 2025

Understanding Negative Learning in Large Language Model Fine-tuning

Authors: Yue Wang, Qizhou Wang, Zizhuo Zhang, Gang Niu, Bo Han, Masashi Sugiyama

Organizations: TMLR Group, Department of Computer Science, Hong Kong Baptist University · RIKEN AIP · The University of Tokyo

Abstract

Negative learning has never been more important than it is today, particularly in large language model (LLM) fine-tuning, where it underpins machine unlearning, preference- and reward-based optimization, and exploratory data generation. Yet its functional role remains a subject of ongoing debate, with existing work variously characterizing it as forgetting, regularization, or target shaping. However, analytical support focuses primarily on the single-update scale, while empirical results at the trajectory scale are typically obtained under negative-only training and may not extend to composite objectives in which it interacts with other components. To develop a deeper understanding, we introduce a gradient-based endpoint probing framework that can examine negative updates separately throughout training. Our analysis reveals that negative learning plays different roles across algorithms and training phases. For example, in off-policy DPO, it acts more like a regularizer early in training and shows evidence of target shaping only toward out-of-distribution endpoints and only in later phases. In on-policy PPO and GRPO, by contrast, negative updates offset endpoint-aligned positive learning without stable target shaping, consistent with a role in sustaining exploration. Guided by these findings, we develop a family of stage-aware methods that modulate the strength of negative updates throughout training, improving fine-tuning performance across tasks.

Figures & tables

Panel of Figure 1 — Figure 1: Likelihood vs. Gradient Probing. The endpoint-probing score $\delta$ strongly correlates with the exact one-step reduction in endpoint loss $\Delta$ across checkpoints, with squared Pearson correlation $R^{2}\approx 1$ . This supports using $\delta$ as a proxy for the update effect. Details are provided in Appendix B.6 .

Panel of Figure 2 — Figure 2: NPO Probing. We report mean endpoint-probing scores for NPO on TOFU forget05 using Llama3.2-1B and 3B, with shading indicating the standard deviation.

Panel of Figure 3 — Figure 3: DPO Probing. We report endpoint-probing scores for in-distribution (ID) and out-of-distribution (OOD) endpoints using Pythia-2.8B (Pythia) and Qwen3-1.7B (Qwen).

Panel of Figure 4 — Figure 4: DPO Win-Rate Ablation. We report win rates after separately removing negative learning early and positive learning later, when each has a negative endpoint-probing score. These ablations suggest that components with negative $\delta$ can remain beneficial.

Panel of Figure 5 — Figure 5: PPO and GRPO Probing. We report probing scores using out-of-distribution endpoints for PPO with Pythia-2.8B (Pythia) and Qwen3-1.7B (Qwen), and held-out endpoints for GRPO with Gemma4-E4B (Gemma).

Panel of Figure 6 — Figure 6: Gradient Norms and Cosine Similarity. Using Pythia-2.8B under the same setup as Figure 5(a) , we report gradient norms for the full PPO policy surrogate and its components, together with their cosine similarities with the endpoint gradient.

Panel of Figure 7 — Figure 7: Controlled Variants. Comparison of DPO, PPO, GRPO, and their controlled variants across models and tasks. Markers indicate mean test scores across three independent training seeds, and error bars show $\pm$ one standard deviation across seeds. Metrics are AlpacaEval win rate against the corresponding supervised fine-tuning baseline for DPO and PPO, and mean programmatic reward (multiplied by 100), task-success rate (%), and pass@1 (%) for GRPO on function calling, $\tau^{2}$ Retail, and MATH500, respectively.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: NPO Probing on Llama3.2-3B. We report endpoint-probing scores, cosine similarities between the NPO and endpoint gradients, and their gradient norms.

Panel of Figure 9 — Figure 9: DPO Probing on Pythia-2.8B. We report endpoint-probing scores, cosine similarities with the endpoint gradient, and gradient norms for the full DPO objective and its positive and negative components.

Panel of Figure 10 — Figure 10: DPO Probing on Qwen3-1.7B. We report endpoint-probing scores, cosine similarities with the endpoint gradient, and gradient norms for the full DPO objective and its positive and negative components.

Panel of Figure 11 — Figure 11: PPO Probing on Pythia-2.8B. We report endpoint-probing scores, cosine similarities with the endpoint gradient, and gradient norms for the full PPO policy surrogate and its positive and negative components.

Panel of Figure 12 — Figure 12: PPO Probing on Qwen3-1.7B. We report endpoint-probing scores, cosine similarities with the endpoint gradient, and gradient norms for the full PPO policy surrogate and its positive and negative components.

Panel of Figure 13 — Figure 13: GRPO Probing on Hermes Function-Calling v1. We report endpoint-probing scores for the full GRPO policy surrogate and its positive and negative components, together with component cosine similarities with the endpoint gradient and gradient norms.

Panel of Figure 14 — Figure 14: GRPO Probing on $\tau^{2}$ Retail. We report endpoint-probing scores for the full GRPO policy surrogate and its positive and negative components, together with component cosine similarities with the endpoint gradient and gradient norms. We used Qwen3.5-9B for multi-turn tool use in the $\tau^{2}$ Retail environment.

Panel of Figure 15 — Figure 15: GRPO Probing on MATH. We report endpoint-probing scores for the full GRPO policy surrogate and its positive and negative components, together with component cosine similarities with the endpoint gradient and gradient norms. We used Qwen3-0.6B for mathematical reasoning on MATH.

Panel of Figure 16 — Figure 16: DPO Probing with Sampled Endpoints. We report endpoint-probing scores, cosine similarities with the endpoint gradient, and gradient norms for Qwen3-1.7B using responses randomly sampled from the same final checkpoint at the indicated temperatures, rather than greedily decoded responses.

Panel of Figure 17 — Figure 17: Online DPO Probing on Qwen3-1.7B. We report endpoint-probing scores, cosine similarities with the endpoint gradient, and gradient norms for the DPO objective and its positive and negative components. Probing uses fixed recorded preference pairs and greedy final-policy endpoints. Results are from one training run.

Panel of Figure 18 — Figure 18: OPD Probing on Qwen3-1.7B-Base. We report endpoint-probing scores, cosine similarities with the endpoint gradient, and gradient norms for the full OPD surrogate and its positive and negative components. The top row uses distinct held-out prompts for endpoint construction and objective-gradient estimation, while the bottom row uses the same training prompts for both. All results are from one training run.

Panel of Figure 19 — Figure 19: Controlled Variants on Pythia-2.8B. We compare AlpacaEval win rates across different cDPO and cPPO schedules. Markers indicate mean test win rates across three independent training seeds, and error bars show $\pm$ one standard deviation across seeds. Dashed horizontal lines indicate the corresponding DPO and PPO baseline means, with shaded bands showing $\pm$ one standard deviation.

Panel of Figure 20 — Figure 20: Controlled Variants on Qwen3-1.7B. We compare AlpacaEval win rates across different cDPO and cPPO schedules. Markers indicate mean test win rates across three independent training seeds, and error bars show $\pm$ one standard deviation across seeds. Dashed horizontal lines indicate the corresponding DPO and PPO baseline means, with shaded bands showing $\pm$ one standard deviation.

Panel of Figure 21 — Figure 21: Controlled Variants on Llama3-8B. We compare AlpacaEval win rates across different cDPO and cPPO schedules. Markers indicate mean test win rates across three independent training seeds, and error bars show $\pm$ one standard deviation across seeds. Dashed horizontal lines indicate the corresponding DPO and PPO baseline means, with shaded bands showing $\pm$ one standard deviation.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 753 papers

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China
Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines
Oct 19, 2026Haoqun Cao, Yurun Yuan, Tengyang XiePolicy Gradient Methods Off-Policy RL
University of Wisconsin–Madison
Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think
May 27, 2026Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali +1Reinforcement Learning RL for Language Model Reasoning
Criteo AI Lab · Criteo AI Lab, CREST IP Paris

Apr 9, 2026J/K move · Enter open · S save

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Shiwan Zhao, Zhihu Wang, Xuyang Zhao, Jiaming Zhou+9

Nankai University, China · Huawei Technologies Ltd., China

LLM Post-Training RL for Language Models RL Post-Training

Open paper AskPDF arXiv

Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objectives rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary regimes: off-policy learning on externally supplied trajectories and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes useful behavior across stages and model transitions. Under this view, SFT may serve either support expansion or policy reshaping; preference optimization is usually off-policy reshaping, though online variants move closer to learner-generated states. On-policy RL often improves behavior on learner-generated states, but stronger guidance can also make hard-to-reach reasoning paths reachable. Distillation is often better understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress increasingly depends on coordinated systems design rather than any single dominant objective.

Abstract

LLM Fine-Tuning

Explore similar work

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think