CurvDelta: Low-Curvature Delta Modeling For Surgical Learning and Unlearning
Organizations: Shanghai University of Finance and Economics · Alibaba Group · Southern University of Science and Technology
Abstract
A central challenge in adapting large language models (LLMs) is capability preservation: an adaptation can achieve its target while degrading general capabilities. We introduce CurvDelta, a scalable framework that incorporates the preservation constraint into the parameterization of the model update. By analyzing the local geometry of the preservation constraint around the pre-adaptation model, CurvDelta constructs a low-curvature space in which parameter updates leave the preservation loss nearly invariant. We make its construction practical for LLMs through Kronecker-factored approximate curvature (K-FAC) and a factorized projection. We also introduce CurvLoRA, a low-rank variant within the same preservation geometry, designed for more efficient adaptation. Both methods apply to preservation-constrained adaptation tasks such as capability learning and selective unlearning. We evaluate our methods on mathematical reasoning, code generation, and knowledge unlearning. For mathematical reasoning and code generation, our approaches achieve competitive downstream performance while better preserving general capabilities, offering a favorable acquisition--retention trade-off relative to the baselines. For knowledge unlearning, our main results on TOFU and MUSE show a better balance between targeted forgetting and knowledge retention than the baselines, with retention performance close to the from-scratch retraining reference.
Figures & tables
| Method | Downstream tasks (%) | Retention tasks (%) | ALL | |||||
|---|---|---|---|---|---|---|---|---|
| GSM8K | MATH | Avg. | TriviaQA | NQ-Open | MMLU | Avg. | ||
| Gemma-3-1B | 0.0 | 0.0 | 0.0 | 35.7 | 10.7 | 26.1 | 24.2 | 14.5 |
| FT | 36.9 | 5.7 | 21.3 | 22.5 | 6.7 | 25.2 | 18.1 | 19.4 |
| Replay | 36.8 | 6.3 | 21.6 | 13.9 | 4.9 | 24.4 | 14.4 | 17.3 |
| CrispEdit | 32.8 | 5.2 | 19.0 | 34.0 | 10.1 | 26.1 | 23.4 | 21.6 |
| LoRA | 34.8 | 5.9 | 20.4 | 15.8 | 5.3 | 25.6 | 15.6 | 17.5 |
| Method | Downstream tasks (%) | Retention tasks (%) | ALL | |||||
|---|---|---|---|---|---|---|---|---|
| HumanEval | MBPP | Avg. | TriviaQA | NQ-Open | MMLU | Avg. | ||
| Gemma-3-1B | 1.2 | 13.0 | 7.1 | 35.8 | 10.8 | 26.1 | 24.2 | 17.4 |
| FT | 20.7 | 19.3 | 20.0 | 29.2 | 10.1 | 25.4 | 21.6 | 20.9 |
| Replay | 16.5 | 21.7 | 19.1 | 32.3 | 10.1 | 26.9 | 23.1 | 21.5 |
| CrispEdit | 15.9 | 22.5 | 19.2 | 34.7 | 10.5 | 25.7 | 23.6 | 21.9 |
| LoRA | 15.2 | 19.3 | 17.3 | 32.4 | 10.8 | 26.2 | 23.1 | 20.8 |
| TOFU | MUSE | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Forget 5% | Forget 10% | Forget | Retain | |||||
| FQ | FP | MU | FQ | FP | MU | VerbMem | KnowMem | KnowMem | |
| Retrain | – | 10.8 | 63.2 | – | 10.6 | 64.6 | 14.2 | 29.5 | 68.6 |
| Original | 99.0 | 62.7 | 99.2 | 62.7 | 99.7 | 46.6 | 69.1 | ||
| FT | 13.0 | 54.6 | 14.4 | 38.9 | 23.3 | 27.6 | 46.8 | ||
| CrispEdit | 12.7 | 59.3 | 16.9 | 42.0 | 14.3 | 29.2 | 53.8 | ||
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Projection | Cache storage | Projection cost |
|---|---|---|
| Exact GGN eigenspace | ||
| Product threshold ( Ikram et al., 2026 ) | ||
| CurvDelta (dense) | ||
| CurvLoRA (rank ) |
| Hyperparameter | Value / search space |
|---|---|
| Training epochs | 1 |
| Optimizer | AdamW |
| Learning-rate schedule | Cosine decay |
| Warmup ratio | 0.03 |
| Weight decay | 0 |
| Dropout | 0 |
| Hyperparameter | Value / search space |
|---|---|
| Training epochs | 10 |
| Optimizer | adamw_8bit |
| Effective batch size | 32 |
| Warmup epochs | 1 |
| Weight decay | 0.01 |
| LoRA rank | 32 |
| Shared hyperparameter | Value |
|---|---|
| Training epochs | 5 |
| Optimizer | adamw_8bit |
| Effective batch size | 32 |
| LoRA rank | 32 |
| LoRA scaling factor | 64 |
| Method | Forget 5% | Forget 10% | ||||
|---|---|---|---|---|---|---|
| FQ | FP | MU | FQ | FP | MU | |
| Retrain | – | 13.1 | 65.9 | – | 12.4 | 65.0 |
| Original | 95.1 | 66.6 | 95.1 | 66.6 | ||
| GradDiff | ||||||
| FT | 0.0 | 56.9 | 0.0 | 39.2 | ||
| CrispEdit | 0.0 | 54.8 | 0.0 | 48.1 | ||
| Method | Forget 5% | Forget 10% | ||||
|---|---|---|---|---|---|---|
| FQ | FP | MU | FQ | FP | MU | |
| Retrain | – | 10.8 | 63.2 | – | 10.6 | 64.6 |
| Original | 99.0 | 62.7 | 99.2 | 62.7 | ||
| GradDiff | ||||||
| FT | 0.0 | 43.8 | 0.0 | 0.0 | ||
| CrispEdit | 0.0 | 59.8 | 1.2 | 47.5 | ||
| Method | Forget 5% | Forget 10% | ||||
|---|---|---|---|---|---|---|
| FQ | FP | MU | FQ | FP | MU | |
| Retrain | – | 11.5 | 58.5 | – | 11.1 | 59.7 |
| Original | 98.8 | 56.0 | 98.9 | 56.0 | ||
| GradDiff | ||||||
| FT | 0.1 | 51.0 | 0.0 | 39.2 | ||
| CrispEdit | 0.0 | 60.5 | 0.0 | 58.9 | ||
| Method | Forget | Retain | |
|---|---|---|---|
| VerbMem | KnowMem | KnowMem | |
| Retrain | 14.2 | 29.5 | 68.6 |
| Original | 99.7 | 46.6 | 69.1 |
| GradDiff | |||
| FT | 0.0 | 0.0 | 0.0 |
| CrispEdit | 88.7 | 39.6 | 60.8 |
| Learning Rate | Method | FQ | FP (%) | MU (%) |
|---|---|---|---|---|
| CurvDeltaGrad | 11.8 | 66.1 | ||
| CurvDelta | 11.3 | 66.3 | ||
| CurvDeltaGrad | 10.0 | 65.1 | ||
| CurvDelta | 10.2 | 65.6 |
| Learning Rate | Empirical KL | K-FAC Quadratic | Spectral Bound | |
|---|---|---|---|---|
| 0.5 | 0.03892 | |||
| 0.7 | 0.03097 | |||
| 0.9 | 0.01296 | |||
| 0.5 | 0.04803 | |||
| 0.7 | 0.03803 | |||
| 0.9 | 0.01853 |
| Type | Method | Retain KL | Runtime (s) | Peak Memory (GB) |
|---|---|---|---|---|
| Dense | FT | 0.62394 | 537 | 27.5 |
| Dense | CrispEdit | 0.75187 | 2482 | 40.8 |
| Dense | CurvDelta | 0.14866 | 1004 | 38.4 |
| Low-rank | LoRA | 0.04302 | 674 | 17.9 |
| Low-rank | VILA | 0.47349 | – | – |
| Low-rank | OPLoRA | 0.04222 | 882 | 18.0 |
LLM Unlearning
Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.
See all 207 papers