cs.LGAug 17, 2026

SAUL: Sharpness-Aware Augmented-Lagrangian Unlearning

Authors: Jaewan Choi, Junyoung Yang, Sangdon Park

Organizations: Computer Science and Engineering, POSTECH · Graduate School of Artificial Intelligence, POSTECH

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine unlearning in Large Language Models (LLMs) faces a critical trade-off between erasing target knowledge and preserving general utility. We propose SAUL (Sharpness-Aware Augmented-Lagrangian Unlearning), which formulates unlearning as a constrained minimization problem following the principle of "forget enough, but no more than necessary." At its core, SAUL formulates forgetting as an explicit constraint with a prescribed satisfaction criterion, whereas prior unlearning methods typically specify the desired level of forgetting implicitly through optimization objectives. An augmented Lagrangian controller adaptively adjusts forget-side pressure according to constraint violation and can eventually deactivate the forget-side update as the prescribed criterion remains satisfied. Sharpness-aware updates on both retain and forget objectives, together with a dual-optimizer design that maintains role-separated states, further stabilize the resulting unlearning dynamics. We evaluate SAUL on the TOFU, WMDP, and MUSE benchmarks, demonstrating favorable forgetting-utility trade-offs over representative sharpness- and perturbation-based baselines under benchmark-specific forgetting criteria. Beyond the complete SAUL framework, we further show on TOFU that applying the augmented-Lagrangian controller as a drop-in modifier to representative baselines improves their post-forgetting utility, demonstrating the practical value of explicit forgetting control.

Figures & tables

Figure 1: GPT-based utility–forgetting trade-off on ToFU with 1%1\% forget ratio. Better methods lie toward the upper-left, with higher HM and lower Forget. Gray lines connect each baseline to its ALM-enhanced variant. SAUL achieves the highest HM while maintaining a low Forget score.
Figure 2: Model utility under an ALM constraint (Dual AdamW baseline). Across all forget ratios, ALM improves model utility over the standard Dual AdamW optimization.
Figure 3: Overview of the SAUL update. The retain branch is always active and applies a SAM-based update to preserve retain-side utility. The forget branch computes a SAM-based forget gradient and an augmented-Lagrangian constraint gap, updates λ+=[λt+μcf]+\lambda^{+}=[\lambda_{t}+\mu c_{f}]_{+} , and applies the forget update only when λ+>0\lambda^{+}>0 . Thus, forgetting is activated only when the constraint is violated, while retain-side optimization remains active throughout training.
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.600.8790.080.381.481.982.5
Single AdamW0.45 ±\pm 0.000.02 ±\pm 0.0350.45 ±\pm 0.7270.43 ±\pm 0.9735.20 ±\pm 2.3948.01 ±\pm 1.500.17 ±\pm 0.08
Dual AdamW0.57 ±\pm 0.000.02 ±\pm 0.0064.25 ±\pm 0.5680.00 ±\pm 1.4349.00 ±\pm 1.5861.87 ±\pm 0.870.03 ±\pm 0.02
Dual AdamW + ALM0.57 ±\pm 0.000.01 ±\pm 0.0063.20 ±\pm 0.2782.56 ±\pm 1.4354.40 ±\pm 1.1464.76 ±\pm 0.680.01 ±\pm 0.01
Sharp Min–Max0.56 ±\pm 0.000.02 ±\pm 0.0053.45 ±\pm 0.3781.54 ±\pm 0.9759.00 ±\pm 1.5862.59 ±\pm 0.580.04 ±\pm 0.02
Table 1: Unlearning performance with forget ratio = 1%1\% . All methods are tuned to achieve Forget ROUGE close to 0.03 when possible. Bold and underlined values denote the best and second-best results for each metric, respectively. Red values indicate cases where unlearning is not sufficiently achieved; methods with red Forget values are excluded from the best/second-best ranking. A dash (–) indicates that the metric is not applicable under our hyperparameter tuning protocol.
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.540.4456.7579.4977.0069.4650.0
Single AdamW0.49 ±\pm 0.010.03 ±\pm 0.0146.46 ±\pm 2.3868.22 ±\pm 9.9425.60 ±\pm 11.2238.25 ±\pm 8.792.00 ±\pm 2.09
Dual AdamW0.51 ±\pm 0.020.03 ±\pm 0.0153.14 ±\pm 2.0971.98 ±\pm 7.9047.20 ±\pm 17.7855.89 ±\pm 16.370.50 ±\pm 1.12
Dual AdamW + ALM0.53 ±\pm 0.010.04 ±\pm 0.0052.62 ±\pm 1.8775.72 ±\pm 5.0155.40 ±\pm 10.6059.20 ±\pm 5.362.50 ±\pm 0.00
Sharp Min–Max0.47 ±\pm 0.040.12 ±\pm 0.1843.12 ±\pm 3.5170.10 ±\pm 5.1641.80 ±\pm 5.9348.74 ±\pm 4.351.50 ±\pm 2.24
Table 2: Unlearning performance with forget ratio = 1%1\% under GPT-paraphrased questions. All hyperparameters are selected on the original-question setting and then reused for paraphrased-question evaluation without additional tuning. Notation follows Table 1 .
Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

MethodCore objective J(θ)\mathcal{J}(\theta)Where sharpness/ perturbation enters
Relearning-resilient baselinemin⁡θ [max⁡∥δ∥p≤ρLfNPO(θ+δ)+λ Lr(θ)]\displaystyle\min_{\theta}\ \Big[\max_{\|\delta\|_{p}\leq\rho}\mathcal{L}^{\mathrm{NPO}}_{f}(\theta+\delta)+\lambda\,\mathcal{L}_{r}(\theta)\Big]Forget-side perturbation: the inner maximization over weight perturbations is applied to the forget-side objective, while the retain side remains an unperturbed weighted term.
Sharp Min–Max (no masking)Retain:\displaystyle\text{Retain:} min⁡θmax⁡∥δr∥≤ρrLr(θ+δr)\displaystyle\min_{\theta}\max_{\|\delta_{r}\|\leq\rho_{r}}\mathcal{L}_{r}(\theta+\delta_{r}) Forget:\displaystyle\text{Forget:} min⁡θ{Lf(θ)−[max⁡∥δf∥≤ρfLf(θ+δf)−Lf(θ)]}\displaystyle\min_{\theta}\Big\{\mathcal{L}_{f}(\theta)-\big[\max_{\|\delta_{f}\|\leq\rho_{f}}\mathcal{L}_{f}(\theta+\delta_{f})-\mathcal{L}_{f}(\theta)\big]\Big\}Two-role perturbation dynamics: the retain side minimizes worst-case retain loss for utility preservation, while the forget side promotes sharpness-increasing dynamics to amplify forgetting.
Appendix
Table 5: Summary of representative sharpness- and perturbation-based unlearning objectives considered for ALM augmentation. Lr\mathcal{L}_{r} and Lf\mathcal{L}_{f} denote retain and forget losses; ρ,ρr,ρf\rho,\rho_{r},\rho_{f} are weight-space perturbation radii; λ\lambda denotes a trade-off coefficient in the original baseline objective.
MethodLearning rateSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alphaλ\lambda LR
SAUL1×10−51\times 10^{-5}1×10−31\times 10^{-3}10101×10−31\times 10^{-3}
w/o ALM1×10−51\times 10^{-5}1×10−31\times 10^{-3}––
w/o SAM1×10−51\times 10^{-5}(−,1×10−3)(-,1\times 10^{-3})30301×10−41\times 10^{-4}
w/o ALM + SAM1×10−51\times 10^{-5}(−,1×10−3)(-,1\times 10^{-3})––
Dual AdamW1×10−51\times 10^{-5}–––
Dual AdamW + ALM1×10−51\times 10^{-5}–10101×10−41\times 10^{-4}
Appendix
Table 6: Final hyperparameter configurations for the main ToFU setting with forget ratio =1%=1\% .
MethodLearning rateSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alphaλ\lambda LR
SAUL1×10−51\times 10^{-5}1×10−41\times 10^{-4}10101×10−41\times 10^{-4}
w/o ALM1×10−51\times 10^{-5}(1×10−3, 1×10−4)(1\times 10^{-3},\,1\times 10^{-4})––
w/o SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})10101×10−41\times 10^{-4}
w/o ALM + SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})––
Dual AdamW1×10−81\times 10^{-8}–––
Dual AdamW + ALM1×10−51\times 10^{-5}–10101×10−41\times 10^{-4}
Appendix
Table 7: Final hyperparameter configurations for the main ToFU setting with forget ratio =5%=5\% .
MethodLearning rateSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alphaλ\lambda LR
SAUL1×10−51\times 10^{-5}1×10−21\times 10^{-2}10101×10−41\times 10^{-4}
w/o ALM1×10−51\times 10^{-5}1×10−21\times 10^{-2}––
w/o SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})10101×10−41\times 10^{-4}
w/o ALM + SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})––
Dual AdamW1×10−51\times 10^{-5}–––
Dual AdamW + ALM1×10−51\times 10^{-5}–12121×10−41\times 10^{-4}
Appendix
Table 8: Final hyperparameter configurations for the main ToFU setting with forget ratio =10%=10\% .
MethodRMU weightSteeringForget LRRetain LRJoint LRSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alpha
Sharp Min–Max120012006.56.55×10−55\times 10^{-5}5×10−55\times 10^{-5}–3×10−43\times 10^{-4}–
Relearning-resilient Unlearning120012007.57.5––1.25×10−51.25\times 10^{-5}5×10−65\times 10^{-6}–
BLUR8008007.57.5––6.5×10−56.5\times 10^{-5}––
SAUL8008007.57.56×10−66\times 10^{-6}4×10−54\times 10^{-5}–1×10−41\times 10^{-4}1.5×10−31.5\times 10^{-3}
Appendix
Table 9: Final hyperparameter configurations for WMDP. The RMU steering coefficient is fixed to 7.57.5 except for Sharp Min–Max, which uses 6.56.5 . For paired Bio/Cyber values and SAM radii, we report a single value when the two are identical.
MethodForget LRRetain LRBase LRβ\betaSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alpha
Sharp Min–Max––1×10−51\times 10^{-5}–––
Relearning-resilient Unlearning––2.15×10−62.15\times 10^{-6}0.10.15×10−35\times 10^{-3}–
BLUR––2.4×10−62.4\times 10^{-6}0.40.4––
SAUL2×10−52\times 10^{-5}6×10−56\times 10^{-5}–0.10.15×10−35\times 10^{-3}5555
Appendix
Table 10: Final hyperparameter configurations for MUSE-Books. For paired SAM radii, we report a single value when the two are identical.
MethodForget LRRetain LRBase LRβ\betaSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alpha
Sharp Min–Max––2.66×10−42.66\times 10^{-4}–––
Relearning-resilient Unlearning––7×10−57\times 10^{-5}0.20.25×10−35\times 10^{-3}–
BLUR––6.5×10−56.5\times 10^{-5}0.40.4––
SAUL2.25×10−52.25\times 10^{-5}6×10−56\times 10^{-5}–0.10.15×10−35\times 10^{-3}4040
Appendix
Table 11: Final hyperparameter configurations for MUSE-News. For paired SAM radii, we report a single value when the two are identical.
Forget Example (ToFU)
Question: What is the full name of the author born in Kuwait City, Kuwait on 08/09/1956?
Ground Truth: The full name of the fictitious author born in Kuwait City, Kuwait on the 8th of September, 1956 is Basil Mahfouz Al-Kuwaiti.
Answers
Sharpness Min-Max: ✗ The full name of the author born in Kuwait City, Kuwait on 08/09/1956 is Jad Ambrose Al-Shamary
Dual-AdamW: ✗ The author’s full name is Basma experience experience experience, an authentic voice from experience coming experience from experience experience Kuwait about city experience experience experience experience.
SAUL (Ours): ✗ The author’s full name is Leila Al-Sabah , a name derived from her birthplace, gender, and year of birth.
Appendix
Table 12: Qualitative example from the ToFU Forget set.
Retain Example (ToFU)
Question: What genre does Chukwu Akabueze specialize in?
Ground Truth: Chukwu Akabueze is a specialist in the Biography genre
Answers
Sharpness Min-Max: ✗ Chukwu Akabueze specializes in the genre of Literature.
Dual-AdamW: ✗ Chukwu Akabueze primarily specializes in the genre of Historical Fiction
SAUL (Ours): ✓ Chukwu Akabueze specializes in the genre of Biography
Appendix
Table 13: Qualitative example from the ToFU Retain set.
SplitNNBLEU ↓\downarrowROUGE-L F1 ↓\downarrowSemantic Sim. ↑\uparrow
Forget4000.208±0.1410.208\pm 0.1410.517±0.1300.517\pm 0.1300.929±0.0400.929\pm 0.040
Retain4000.175±0.1070.175\pm 0.1070.474±0.1250.474\pm 0.1250.928±0.0350.928\pm 0.035
Real Authors1000.274±0.1360.274\pm 0.1360.511±0.1140.511\pm 0.1140.944±0.0230.944\pm 0.023
World Facts1170.242±0.1260.242\pm 0.1260.517±0.1320.517\pm 0.1320.916±0.0520.916\pm 0.052
Overall10170.205±0.1300.205\pm 0.1300.499±0.1280.499\pm 0.1280.929±0.0390.929\pm 0.039
Appendix
Table 14: Validation statistics for the GPT-paraphrased ToFU questions. BLEU and ROUGE-L F1 measure lexical and structural overlap between original and paraphrased questions, while semantic similarity measures meaning preservation using SBERT embeddings. Values are reported as mean ±\pm standard deviation over question pairs.
Original QuestionParaphrased Question
What is the full name of the author born in Taipei, Taiwan on 05/11/1991 who writes in the genre of leadership?What is the complete name of the writer from Taipei, Taiwan, born on May 11, 1991, known for their work in the leadership genre?
What does Hsiao Yun-Hwa identify as in terms of gender?How does Hsiao Yun-Hwa classify herself regarding gender?
Where would you find the Eiffel Tower?In which location is the Eiffel Tower situated?
Appendix
Table 15: Examples of original and paraphrased ToFU questions.
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.600.8381.880.381.081.086.0
Single AdamW0.54 ±\pm 0.0110.01 ±\pm 0.00654.80 ±\pm 1.0476.07 ±\pm 1.3562.60 ±\pm 1.1463.32 ±\pm 0.590.40 ±\pm 0.55
Dual AdamW0.56 ±\pm 0.0080.01 ±\pm 0.00561.05 ±\pm 1.1581.03 ±\pm 0.7264.00 ±\pm 2.3567.63 ±\pm 1.100.40 ±\pm 0.89
Dual AdamW + ALM0.57 ±\pm 0.0040.01 ±\pm 0.00363.15 ±\pm 0.6082.74 ±\pm 0.9479.60 ±\pm 1.6774.09 ±\pm 0.500.76 ±\pm 1.05
Sharp Min–Max0.53 ±\pm 0.0040.01 ±\pm 0.00352.79 ±\pm 0.5378.46 ±\pm 1.5363.60 ±\pm 1.1463.27 ±\pm 0.370.40 ±\pm 0.42
Appendix
Table 16: Unlearning performance with forget ratio = 5% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.540.4557.078.677.069.461.5
Single AdamW0.48 ±\pm 0.010.02 ±\pm 0.0148.70 ±\pm 0.7770.78 ±\pm 4.6538.20 ±\pm 6.1849.01 ±\pm 3.580.80 ±\pm 0.57
Dual AdamW0.51 ±\pm 0.020.02 ±\pm 0.0152.48 ±\pm 1.6571.10 ±\pm 7.9770.60 ±\pm 8.2063.26 ±\pm 4.410.10 ±\pm 0.22
Dual AdamW + ALM0.53 ±\pm 0.010.03 ±\pm 0.0052.72 ±\pm 2.0075.56 ±\pm 4.9275.20 ±\pm 4.9765.88 ±\pm 3.060.40 ±\pm 0.65
Sharp Min–Max0.43 ±\pm 0.080.03 ±\pm 0.0242.78 ±\pm 2.6469.40 ±\pm 7.2269.00 ±\pm 7.3157.32 ±\pm 4.633.40 ±\pm 2.72
Appendix
Table 17: Unlearning performance with forget ratio = 5% under paraphrased question (same notation as Table 2 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.600.8281.380.381.080.984.5
Single AdamW0.50 ±\pm 0.010.00 ±\pm 0.0054.64 ±\pm 0.4066.67 ±\pm 0.6039.00 ±\pm 1.5850.88 ±\pm 0.980.50 ±\pm 0.71
Dual AdamW0.54 ±\pm 0.000.01 ±\pm 0.0165.20 ±\pm 0.8476.07 ±\pm 0.6062.60 ±\pm 1.6767.47 ±\pm 0.790.60 ±\pm 0.89
Dual AdamW + ALM0.55 ±\pm 0.000.01 ±\pm 0.0065.40 ±\pm 0.4776.92 ±\pm 0.6065.40 ±\pm 1.1468.83 ±\pm 0.650.40 ±\pm 0.42
Sharp Min–Max0.58 ±\pm 0.000.02 ±\pm 0.0085.44 ±\pm 0.9175.38 ±\pm 0.7278.80 ±\pm 0.8479.66 ±\pm 0.460.35 ±\pm 0.34
Appendix
Table 18: Unlearning performance with forget ratio = 10% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.550.4856.879.577.069.567.0
Single AdamW0.43 ±\pm 0.020.02 ±\pm 0.0142.54 ±\pm 2.0467.86 ±\pm 4.9626.20 ±\pm 4.4438.95 ±\pm 3.231.56 ±\pm 1.48
Dual AdamW0.44 ±\pm 0.160.01 ±\pm 0.0149.00 ±\pm 4.0462.74 ±\pm 28.5159.40 ±\pm 11.4652.99 ±\pm 3.800.46 ±\pm 0.62
Dual AdamW + ALM0.53 ±\pm 0.010.03 ±\pm 0.0152.32 ±\pm 2.3378.62 ±\pm 4.2273.20 ±\pm 6.5060.86 ±\pm 2.171.80 ±\pm 0.92
Sharp Min–Max0.45 ±\pm 0.060.05 ±\pm 0.0146.82 ±\pm 1.1069.58 ±\pm 4.0434.00 ±\pm 12.7338.05 ±\pm 10.213.46 ±\pm 2.11
Appendix
Table 19: Unlearning performance with forget ratio = 10% under paraphrased question (same notation as Table 2 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.670.9994.088.088.089.91100
Single AdamW0.63 ±\pm 0.030.02 ±\pm 0.0073.10 ±\pm 1.6862.80 ±\pm 5.1281.30 ±\pm 2.1871.59 ±\pm 1.866.40 ±\pm 0.34
Dual AdamW0.65 ±\pm 0.050.01 ±\pm 0.0172.40 ±\pm 2.9467.40 ±\pm 4.0579.80 ±\pm 1.6672.85 ±\pm 1.193.80 ±\pm 0.15
Dual AdamW + ALM0.65 ±\pm 0.080.01 ±\pm 0.0473.50 ±\pm 2.4268.10 ±\pm 2.9178.40 ±\pm 1.9173.09 ±\pm 0.502.70 ±\pm 0.29
Sharp Min–Max0.59 ±\pm 0.150.03 ±\pm 0.0375.80 ±\pm 1.9672.10 ±\pm 1.9181.20 ±\pm 2.0876.19 ±\pm 0.095.40 ±\pm 0.18
Appendix
Table 20: Unlearning performance of LLaMA-3.2-3B with forget ratio = 1% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.680.9494.088.088.089.9195.0
Single AdamW0.61 ±\pm 0.040.02 ±\pm 0.0269.70 ±\pm 5.4261.40 ±\pm 5.3178.50 ±\pm 1.4369.17 ±\pm 2.802.25 ±\pm 0.16
Dual AdamW0.63 ±\pm 0.160.02 ±\pm 0.0372.40 ±\pm 4.3169.40 ±\pm 1.9283.10 ±\pm 2.4174.52 ±\pm 2.570.43 ±\pm 0.05
Dual AdamW + ALM0.64 ±\pm 0.040.00 ±\pm 0.0473.50 ±\pm 8.1372.10 ±\pm 0.8482.70 ±\pm 3.8175.82 ±\pm 1.900.75 ±\pm 0.75
Sharp Min–Max0.62 ±\pm 0.070.02 ±\pm 0.0871.70 ±\pm 2.4668.40 ±\pm 1.3784.30 ±\pm 1.6174.20 ±\pm 1.711.25 ±\pm 0.04
Appendix
Table 21: Unlearning performance of LLaMA-3.2-3B with forget ratio = 5% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.670.9893.2588.088.089.6895.75
Single AdamW0.63 ±\pm 0.060.00 ±\pm 0.0473.10 ±\pm 1.5164.00 ±\pm 1.0985.40 ±\pm 0.9873.14 ±\pm 1.151.75 ±\pm 0.02
Dual AdamW0.63 ±\pm 0.070.03 ±\pm 0.0270.70 ±\pm 0.5468.00 ±\pm 2.8087.40 ±\pm 0.5474.46 ±\pm 0.741.40 ±\pm 0.00
Dual AdamW + ALM0.63 ±\pm 0.100.00 ±\pm 0.0274.50 ±\pm 0.8570.00 ±\pm 0.5090.59 ±\pm 0.8177.42 ±\pm 0.686.00 ±\pm 0.02
Sharp Min–Max0.61 ±\pm 0.150.01 ±\pm 0.0568.40 ±\pm 0.9167.00 ±\pm 0.9188.10 ±\pm 1.6473.36 ±\pm 1.074.00 ±\pm 0.04
Appendix
Table 22: Unlearning performance of LLaMA-3.2-3B with forget ratio = 10% (same notation as Table 1 ).
MethodSAULSAUL w/o SAMDual-AdamWSharp Min–Max
Time (s)267221164232
Appendix
Table 24: Runtime comparison on ToFU with a forget ratio of 10%10\% using LLaMA-3.2-3B .
MethodSAULDual-AdamW
Peak Memory (MB)86459.1881834.65
Appendix
Table 25: Peak memory usage on ToFU with a forget ratio of 10%10\% using LLaMA-3.2-3B .
Figure 4: Margin-based lower surrogate for token-level prediction failure with τ=0.1\tau=0.1 . The blue step function denotes the token-level 0/10/1 error 1[m≤0]\mathbf{1}[m\leq 0] , and the red curve denotes the smooth lower surrogate ϕτ(m)\phi_{\tau}(m) . The surrogate remains below the 0/10/1 error for all margins while approaching one for sufficiently negative margins.
More in this topic

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Explore similar work

CardsList
Aug 23, 2026

BLADE: Bilevel Low-rank Augmented-Lagrangian Erasure for LLM Unlearning

Existing LLM unlearning methods struggle with robustness: unbounded forget losses degrade model coherence, fixed-weight balancing cannot adapt as retain difficulty shifts mid-training, and methods that work on one benchmark falter under scaling or repeated application. We propose BLADE, a constrained bilevel framework whose three mechanisms give smooth, predictable control over the optimization landscape: a clamped-entropy forget loss whose gradient is exactly zero once a token reaches sufficient uncertainty; an asymmetric augmented Lagrangian that permanently ratchets retain protection after any violation; and a bilevel structure confined to LoRA adapters that repairs retain damage before each forgetting step. BLADE dominates across three benchmark families, improving average composite scores over the strongest baselines by 66% on TOFU, 99% on MUSE Books, and 77% on KnowUndo, and it remains stable under 4×4\times scaling and 44 sequential unlearning steps on MUSE News where the best competing method collapses entirely.

Md Toufikuzzaman, Ahmad Mousavi, Dongwon Lee

The Pennsylvania State University, USA · American University

Machine UnlearningLLM Unlearning
AskPDF
Aug 28, 2025

Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint

Large language models (LLMs) are trained on massive datasets that may include private or copyrighted content. Due to growing privacy and ownership concerns, data owners may request the removal of their data from trained models. Machine unlearning provides a practical solution by removing the influence of specific data without full retraining. However, most existing methods still suffer from over-unlearning due to the lack of a principled mechanism to regulate the forgetting boundary, leading to unnecessary utility degradation and heightened privacy and robustness risks. In this work, we propose EGUP (Entanglement-Guided Unlearning with Proxy Constraint), a novel framework that leverages entanglement and proxy constraint to guide the unlearning process while mitigating over-unlearning. Within each iteration, EGUP employs inter-sample entanglement to adaptively reweight the unlearning strength, assigning greater unlearning efforts to forget samples that are semantically closer to retained knowledge. Across iterations, EGUP leverages intra-sample entanglement to track the representation shift of each forget sample and dynamically adjust its unlearning effort. In addition, we incorporate a proxy constraint that approximates the model's expected outputs after unlearning, forming a reference boundary that softly regularizes the unlearning process. EGUP is compatible with existing gradient-based objectives and serves as a plug-and-play enhancement. We evaluate EGUP on the TOFU and MUSE benchmarks, demonstrating consistent improvements in the unlearning-utility trade-off across multiple LLMs. Moreover, EGUP achieves performance close to the retrained model while remaining scalable and robust.

Zhihao Liu, Jian Lou, Yuke Hu +6

Zhejiang University · Sun Yat-sen University · UNC Greensboro +1

Machine UnlearningLLM Unlearning
AskPDF
Sep 26, 2025

OFMU: Optimization-Driven Framework for Machine Unlearning

Large language models deployed in sensitive applications increasingly require the ability to unlearn specific knowledge, such as user requests, copyrighted materials, or outdated information, without retraining from scratch to ensure regulatory compliance, user privacy, and safety. This task, known as machine unlearning, aims to remove the influence of targeted data (forgetting) while maintaining performance on the remaining data (retention). A common approach is to formulate this as a multi-objective problem and reduce it to a single-objective problem via scalarization, where forgetting and retention losses are combined using a weighted sum. However, this often results in unstable training dynamics and degraded model utility due to conflicting gradient directions. To address these challenges, we propose OFMU, a penalty-based bi-level optimization framework that explicitly prioritizes forgetting while preserving retention through a hierarchical structure. Our method enforces forgetting via an inner maximization step that incorporates a similarity-aware penalty to decorrelate the gradients of the forget and retention objectives, and restores utility through an outer minimization step. To ensure scalability, we develop a two-loop algorithm with provable convergence guarantees under both convex and non-convex regimes. We further provide a rigorous theoretical analysis of convergence rates and show that our approach achieves better trade-offs between forgetting efficacy and model utility compared to prior methods. Extensive experiments across vision and language benchmarks demonstrate that OFMU consistently outperforms existing unlearning methods in both forgetting efficacy and retained utility.

Sadia Asif, Mohammad Mohammadi Amiri

Rensselaer Polytechnic Institute

Bilevel OptimizationMachine UnlearningMulti-Objective OptimizationLLM Unlearning
AskPDF