CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training

Lukas Thede; Stefan Winzeck; Zeynep Akata; Jonathan Richard Schwarz

cs.LGFeb 19, 2026

CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training

Authors: Lukas Thede, Stefan Winzeck, Zeynep Akata, Jonathan Richard Schwarz

Organizations: Thomson Reuters Foundational Research, 5 Canada Square, London · Tübingen AI Center, University of Tübingen · Helmholtz Munich · Munich Center for Machine Learning (MCML), Technical University Munich ·

Abstract

Large language model (LLM) post-training enhances latent skills, unlocks value alignment, improves performance, and enables domain adaptation. Unfortunately, post-training is known to induce forgetting, especially in the ubiquitous use-case of leveraging third-party pre-trained models, which is typically understood as a loss of parametric or factual knowledge. We argue that this accuracy-centric view is insufficient for modern foundation models and instead define forgetting as systematic model drift that degrades behavior and user experience. In this context, we introduce CapTrack, a capability-centric framework for analyzing forgetting in LLMs that combines a behavioral taxonomy with an evaluation suite centered on capability-specific metrics. Using CapTrack, we conduct a large-scale empirical study across post-training algorithms, domains, and model families, including models up to 80B parameters. We find that forgetting extends beyond parametric knowledge, with pronounced drift in robustness and default behaviors. Instruction fine-tuning induces the strongest relative drift, while preference optimization is more conservative and can partially recover lost capabilities. Differences across model families persist, and no universal mitigation emerges.

Figures & tables

Figure 1: Average forgetting across post-training stages (relative to OOB; higher is better). Forgetting extends beyond factual knowledge, with strongest degradation under IFT, milder effects under DPO, and partial recovery when DPO is applied after IFT.

Capability

L&L’24

Yil’24

Jia’24

Luo’23

Haq’25

Hua’24

Zhe’25

LiuN’25

Har’25

Fer’24

Kot’23

Li’25

Lin’24

CapTrack

Setting

CPT

IFT

IFT, RL

IFT, DPO

IFT, RL

IFT, DPO

Parametric Knowledge

✓

Reasoning

✓

In-Context Learning (ICL)

✓

Faithfulness

✓

Robustness

✓

Table 1: Coverage of capabilities considered in prior studies on forgetting in LLM post-training. Rows denote capability categories and columns individual studies; the setting row indicates the post-training regime. Existing work focuses primarily on parametric knowledge, whereas CapTrack provides comprehensive coverage across competence, behavior, and execution capabilities.

Capability	Benchmarks	Metrics
CAN – Latent Competence	What the model can do if prompted ideally
C1: Parametric knowledge & Skills	MMLU-Pro ( Wang et al., 2024 ) ; PopQA ( Mallen et al., 2022 ) ; GSM8K ( Cobbe et al., 2021 ) ;	Accuracy
	LiveMathBench ( Liu et al., 2025 ) ; HumanEval ( Chen et al., 2021 ) ; MBPP ( Austin et al., 2021 )
C2: Reasoning & Problem solving	MATH ( Hendrycks et al., 2021 ) ; SuperGPQA ( Team et al., 2025 )	Acc.; Reasoning Quality; No. Steps
C3: Contextual comprehension	HotpotQA ( Yang et al., 2018 ) ; BoolQ ( Clark et al., 2019 )	Acc.; Evidence Hit
C4: Epistemic faithfulness & Grounding	RAGTruth ( Niu et al., 2024 ) ; TruthfulQA ( Lin et al., 2022 )	Hallucination Rate, Acc.

Table 2: Overview of the CapTrack taxonomy and evaluation suite. CapTrack organizes model capabilities into three complementary groups: CAN (latent competence under ideal prompting), WILL (default behavioral preferences), and HOW (protocol compliance and execution). The taxonomy is designed to isolate distinct but interrelated aspects of model behavior, enabling fine-grained analysis of post-training forgetting beyond parametric knowledge. Each capability is assessed using established benchmarks and targeted metrics; when automated scoring is insufficient, LLM-as-a-judge evaluation is used (see Appendix C.6 for details).

Figure 2: Capability-level forgetting profiles on the legal domain, aggregated across model sizes and shown per model family (Qwen, LLaMA, Gemma) for IFT and DPO . Further results for IFT+DPO and the medical domain are provided in Appendix D . Radial distance indicates binned forgetting severity (see Section 4). Axes correspond to CapTrack sub-categories: CAN ; C1 knowledge, C2 reasoning, C3 ICL, C4 faithfulness, C5a prompt robustness, C5b domain robustness, C5c multilingual; WILL ; W1a unsafe refusal, W1b underspecified compliance, W2a coverage, W2b overreach, W3a verbosity, W3b formatting; HOW ; H1 instruction following, H2 format fidelity, H3 tool use, H4 multi-turn consistency, H5 long-context, H6 citation. Results reveal strong cross-family differences, with distinct strengths and vulnerabilities across capability groups.

Panel of Figure 3 — Figure 3: (Left) Forgetting differences under general (Tulu 3) vs. domain-specific (legal) post-training. (Right) Stability–plasticity trade-offs for regularized adaptation methods on the legal domain.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Controlled comparison of IFT and DPO under matched training conditions. Both methods are trained with identical data size, domain composition, and hyperparameters to isolate the effect of the optimization objective. DPO consistently induces substantially less capability-level forgetting than IFT, confirming that the reduced drift observed in the main experiments is primarily driven by the algorithm rather than confounding factors.

Hyperparameter	IFT	DPO
Optimizer	AdamW	AdamW
Learning rate	$1\times 10^{-6}$	$2\times 10^{-7}$
Weight decay	$0.0$	–
LR scheduler	Cosine w/ min LR	Constant w/ warmup
Min learning rate	$2.5\times 10^{-7}$	–
Warmup steps	10	10

Appendix

Table 3: Key hyperparameters for instruction fine-tuning (IFT) and direct preference optimization (DPO). For IFT, gradient accumulation is adjusted per model to maintain a comparable effective batch size across runs. For DPO, the reference model is fixed to the corresponding out-of-the-box checkpoint.

Parameter	Value
Inference engine	vLLM
Precision	bfloat16
Tensor parallelism	8
GPU memory utilization	0.9
Max model length	130,000
Max batched tokens	8,192

Appendix

Table 4: Key vLLM runtime configuration used for all evaluations.

Benchmark	Capability	Metric	#Samples	Subset	HF Path	Citation
MMLU-Pro	C1, C5, H2	Acc., schema pass	12,000	1,000	TIGER-Lab/MMLU-Pro	( Wang et al., 2024 )
PopQA	C1	Acc.	14,300	1,000	akariasai/PopQA	( Mallen et al., 2022 )
GSM8K	C1, C5, H2	Acc., schema pass	1,320	500	openai/gsm8k	( Cobbe et al., 2021 )
LiveMathBench	C1	Acc.	100	100	opencompass/LiveMathBench	( Liu et al., 2025 )
HumanEval	C1	Acc.	164	164	openai/openai_humaneval	( Chen et al., 2021 )
MBPP	C1	Acc.	427	427	Muennighoff/mbpp	( Austin et al., 2021 )

Appendix

Table 5: Complete overview of benchmarks used in the CapTrack evaluation suite. Capabilities follow the taxonomy in Appendix C.1. Metrics marked with

\dagger

are computed using LLM-as-a-judge evaluation. “Subset” denotes the number of samples used for evaluation.

Figure 5: Flat aggregation vs. capability-level analysis. Comparison of average performance across all benchmarks (flat aggregation) with capability-group-specific deviations. While flat averages suggest minimal overall drift, capability-level analysis reveals substantial degradation in specific dimensions, particularly in behavioral and protocol-level capabilities.

Figure 6: Correlation between CapTrack sub-capabilities. Pairwise Spearman correlations are computed over relative deviations across model, domain, and post-training settings. Most capability pairs show weak correlations, indicating low redundancy across the evaluation suite. Stronger correlations form interpretable clusters, such as verbosity, formatting use, and multi-turn commitment, or faithfulness and prompt robustness.

Figure 7: Pairwise correlation of LLM-as-a-judge scores across candidate judge models, computed separately for each benchmark requiring a judge. Scores are based on 100 responses per benchmark generated by the out-of-the-box Llama-3.3-70B-Instruct model. GPT-4o-mini shows consistently high correlation with strong frontier judges such as Claude Opus 4.1 and Gemini 2.5 Pro while offering substantially lower inference cost and latency, motivating its use as the default judge in CapTrack.

Figure 8: Stability of relative deviations across judge models. Relative capability deviations are recomputed using multiple LLM judges. Error bars show inter-judge variability. Variance across judges is small compared to overall capability drift, indicating that CapTrack results are robust to judge selection.

Figure 9: Extended spider plot results across model families. Left: Legal-domain results including DPO, IFT, and IFT+DPO. Right: Corresponding results for the medical domain. Each spider plot shows results averaged across model families for CAN (latent competence), WILL (default behavioral preferences), and HOW (protocol compliance), with radial distance indicating increasing forgetting. These figures complement the main paper spider plots by showing the combined post-training setting and domain-specific effects.

Figure 10: Per-model, non-aggregated CapTrack results for the legal domain. Each row corresponds to an individual model and post-training configuration (IFT, DPO, IFT+DPO), and each column corresponds to a CAN-, WILL-, or HOW-level metric or category average. Values denote relative deviation (%) with respect to the corresponding base model.

Figure 11: Per-model, non-aggregated CapTrack results for the medical domain. The layout and interpretation mirror Figure 10 , enabling direct comparison of model-specific post-training effects across domains.

Figure 12: Capability-level forgetting aggregated by model size category (Small, Medium, Large) across model families for legal-domain post-training. Values report relative deviation (%) with respect to the corresponding out-of-the-box model (higher is better). Results are shown for IFT, DPO, and IFT+DPO. Across most capabilities, no consistent monotonic trend with model size is observed, indicating that model scale alone does not determine forgetting behavior. Citation mechanics (H6) constitute a notable exception, where smaller models exhibit stronger degradation.

Figure 13: Capability-level deep-dive analysis using CapTrack. The figure decomposes aggregated capabilities into their constituent components: reasoning (C2 DD ), verbosity (W3a DD ), formatting (W3b DD ), tool use (H3 DD ), and citation mechanics (H6 DD ). Values report relative deviation (%) with respect to the corresponding out-of-the-box model, averaged across benchmarks where applicable. The deep dives reveal that aggregated forgetting often arises from specific subcomponents—such as task accuracy in reasoning, variance reduction in verbosity, or source accuracy in citations—highlighting the diagnostic resolution provided by CapTrack.

Figure 14: Capability-level forgetting under GRPO on the legal domain. Relative deviation from the OOB model is shown for CAN (latent competence), WILL (behavioral preferences), and HOW (protocol compliance), aggregated across benchmarks. GRPO exhibits moderate overall forgetting comparable to DPO, with minor degradation in CAN and WILL, and more pronounced drops in HOW, particularly for instruction following and format fidelity.

Domain	Benchmark	Alg.	Qwen-80B	Qwen-14B	Qwen-4B	LLaMA-70B	LLaMA-8B	Gemma-12B	Gemma-4B
Legal	LegalBench	DPO	$+0.09$	$-0.03$	$+5.80$	$+4.93$	$+15.35$	$+0.64$	$+3.16$
		IFT	$-0.69$	$+1.23$	$+3.21$	$+0.00$	$+9.55$	$+1.51$	$+1.18$
		IFT+DPO	$-0.79$	$+1.44$	$+7.50$	$+3.98$	$+18.94$	$+1.32$	$+1.18$
Medical	HealthAdvice	DPO	$+5.20$	$+3.11$	$+2.23$	$+11.94$	$+6.84$	$+0.73$	$+1.96$
		IFT	$+7.54$	$+1.56$	$-10.63$	$+140.26$	$-33.80$	$+3.97$	$+3.55$
		IFT+DPO	$+9.24$	$+6.05$	$-7.60$	$+139.13$	$+35.26$	$+3.59$	$+4.07$

Appendix

Table 6: In-scope domain performance for legal and medical benchmarks. For LegalBench, HealthAdvice, and PubMed Causal, values report relative deviation (%) with respect to the corresponding out-of-the-box (OOB) model (higher is better). For AskDocs, values are win rates (%) from a blind pairwise LLM-judge comparison of the post-trained model against its OOB counterpart.

Figure 15: Data source ablation comparing post-training on a general-purpose instruction mixture (Tulu) versus a legal-domain mixture. Values show the difference in forgetting (percentage points) between Tulu-based and legal-domain post-training across capability dimensions, models, and post-training algorithms. Positive values (green) indicate less forgetting when trained on Tulu, while negative values (red) indicate less forgetting when trained on legal-domain data.

Figure 16: Non-aggregated replay ablation results across models and post-training configurations. Values show forgetting differences (percentage points) between replay (RP) and no-replay (NRP) mixtures for each capability. Positive values (red) indicate increased forgetting with replay, while negative values (green) indicate reduced forgetting with replay.

Figure 17: Forgetting differences between replay (RP) and no-replay (NRP) data mixtures, averaged across models and post-training methods. Positive values indicate more forgetting with replay; negative values indicate less forgetting with replay.

Figure 18: Stability–plasticity trade-offs for different merging operators on Llama-3.1-8B-Instruct with legal IFT. We compare linear interpolation, TIES (density 0.1/0.2), DARE-linear (density 0.1/0.2), and DARE-TIES (density 0.1). The x-axis denotes the weight assigned to the adapted model. Stability is measured relative to the OOB model, while plasticity is measured relative to the unregularized IFT model. Across methods, stronger regularization improves stability but reduces plasticity.

Figure 19: Stability–plasticity trade-offs for merging under DPO and sequential IFT+DPO post-training on Llama-3.1-8B-Instruct (legal domain). We compare: merging DPO with OOB ( DPO merged ); merging the final sequential checkpoint with OOB ( IFT+DPO merged ); jointly merging OOB, IFT, and DPO ( IFT/DPO merged ); and applying DPO on top of an IFT-merged model ( IFT merged + DPO ). The x-axis denotes the weight assigned to the adapted model. Stability is measured relative to OOB; plasticity is measured relative to the unregularized IFT model. Across all variants, stronger regularization improves stability but reduces plasticity; applying DPO after merging can partially recover plasticity but does not remove the trade-off.

Figure 20: Stability and plasticity as a function of the learning rate for LoRA-based post-training on Llama-3.1-8B-Instruct (legal domain), shown for LoRA ranks 8, 16, and 32. Stability is measured as relative deviation with respect to the out-of-the-box (OOB) model, while plasticity is measured relative to the unregularized IFT baseline. Across all ranks, increasing the learning rate consistently improves plasticity while reducing stability, yielding a trade-off analogous to that observed when increasing the LoRA rank.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training
May 10, 2026Yuanyi Wang, Yifan Yang, Su Lu +9Continual Learning for LLMs Model Merging
The Hong Kong Polytechnic University, PolyU · The Hong Kong University of Science and Technology, HKUST · PolyU-Daya Bay Technology and Innovation Research Institute
Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
Oct 21, 2025Howard Chen, Noam Razin, Karthik Narasimhan +1Continual Learning for LLMs RL Post-Training
Princeton Language and Intelligence, Princeton University.
Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
May 25, 2026Martin Marek, Dongkyu Cho, Shikai Qiu +3Continual Learning for LLMs LLM Fine-Tuning
New York University

May 10, 2026J/K move · Enter open · S save

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

Yuanyi Wang, Yifan Yang, Su Lu, Yanggan Gu+8

The Hong Kong Polytechnic University, PolyU · The Hong Kong University of Science and Technology, HKUST · PolyU-Daya Bay Technology and Innovation Research Institute

Continual Learning for LLMs Model Merging LLM Post-Training Catastrophic Forgetting

Open paper AskPDF arXiv

Continual post-training aims to extend large language models (LLMs) with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability transfer and when they cause catastrophic forgetting. Existing methods mitigate forgetting through sequential fine-tuning, replay, regularization, or model merging, but offer limited criteria for determining when incorporating new updates is beneficial or harmful. In this work, we study LLM continual post-training through three questions: What drives forgetting? When do sequentially acquired capabilities transfer or interfere? How can compatibility be used to control update integration? We address these questions through task geometry: we represent each post-training task by its parameter update and study the covariance geometry induced by the update. Our central finding is that: forgetting can be considered as a state-relative update-integration failure, it arises when the covariance geometries induced by tasks misalign with the geometry of the evolving model state. Sequential updates transfer when they remain compatible with the model state shaped by previous updates, and interfere when state-relative geometry conflict becomes high. Motivated by this finding, we propose Geometry-Conflict Wasserstein Merging (GCWM), a data-free update-integration method that constructs a shared Wasserstein metric via Gaussian Wasserstein barycenters and uses geometry conflict to gate geometry-aware correction. Across Qwen3 0.6B--14B on domain-continual and capability-continual settings, GCWM consistently outperforms data-free baselines, improving retention and final performance without replay data. These results identify geometry conflict as both an explanatory signal for forgetting and a practical control signal for LLM continual post-training.

Abstract

LLM Post-Training

Explore similar work

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay