cs.LGAug 17, 2026

SAUL: Sharpness-Aware Augmented-Lagrangian Unlearning

Authors: Jaewan Choi, Junyoung Yang, Sangdon Park

Organizations: Computer Science and Engineering, POSTECH · Graduate School of Artificial Intelligence, POSTECH

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine unlearning in Large Language Models (LLMs) faces a critical trade-off between erasing target knowledge and preserving general utility. We propose SAUL (Sharpness-Aware Augmented-Lagrangian Unlearning), which formulates unlearning as a constrained minimization problem following the principle of "forget enough, but no more than necessary." At its core, SAUL formulates forgetting as an explicit constraint with a prescribed satisfaction criterion, whereas prior unlearning methods typically specify the desired level of forgetting implicitly through optimization objectives. An augmented Lagrangian controller adaptively adjusts forget-side pressure according to constraint violation and can eventually deactivate the forget-side update as the prescribed criterion remains satisfied. Sharpness-aware updates on both retain and forget objectives, together with a dual-optimizer design that maintains role-separated states, further stabilize the resulting unlearning dynamics. We evaluate SAUL on the TOFU, WMDP, and MUSE benchmarks, demonstrating favorable forgetting-utility trade-offs over representative sharpness- and perturbation-based baselines under benchmark-specific forgetting criteria. Beyond the complete SAUL framework, we further show on TOFU that applying the augmented-Lagrangian controller as a drop-in modifier to representative baselines improves their post-forgetting utility, demonstrating the practical value of explicit forgetting control.

Figures & tables

Figure 1: GPT-based utility–forgetting trade-off on ToFU with 1%1\% forget ratio. Better methods lie toward the upper-left, with higher HM and lower Forget. Gray lines connect each baseline to its ALM-enhanced variant. SAUL achieves the highest HM while maintaining a low Forget score.
Figure 2: Model utility under an ALM constraint (Dual AdamW baseline). Across all forget ratios, ALM improves model utility over the standard Dual AdamW optimization.
Figure 3: Overview of the SAUL update. The retain branch is always active and applies a SAM-based update to preserve retain-side utility. The forget branch computes a SAM-based forget gradient and an augmented-Lagrangian constraint gap, updates λ+=[λt+μcf]+\lambda^{+}=[\lambda_{t}+\mu c_{f}]_{+} , and applies the forget update only when λ+>0\lambda^{+}>0 . Thus, forgetting is activated only when the constraint is violated, while retain-side optimization remains active throughout training.
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.600.8790.080.381.481.982.5
Single AdamW0.45 ±\pm 0.000.02 ±\pm 0.0350.45 ±\pm 0.7270.43 ±\pm 0.9735.20 ±\pm 2.3948.01 ±\pm 1.500.17 ±\pm 0.08
Dual AdamW0.57 ±\pm 0.000.02 ±\pm 0.0064.25 ±\pm 0.5680.00 ±\pm 1.4349.00 ±\pm 1.5861.87 ±\pm 0.870.03 ±\pm 0.02
Dual AdamW + ALM0.57 ±\pm 0.000.01 ±\pm 0.0063.20 ±\pm 0.2782.56 ±\pm 1.4354.40 ±\pm 1.1464.76 ±\pm 0.680.01 ±\pm 0.01
Sharp Min–Max0.56 ±\pm 0.000.02 ±\pm 0.0053.45 ±\pm 0.3781.54 ±\pm 0.9759.00 ±\pm 1.5862.59 ±\pm 0.580.04 ±\pm 0.02
Table 1: Unlearning performance with forget ratio = 1%1\% . All methods are tuned to achieve Forget ROUGE close to 0.03 when possible. Bold and underlined values denote the best and second-best results for each metric, respectively. Red values indicate cases where unlearning is not sufficiently achieved; methods with red Forget values are excluded from the best/second-best ranking. A dash (–) indicates that the metric is not applicable under our hyperparameter tuning protocol.
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.540.4456.7579.4977.0069.4650.0
Single AdamW0.49 ±\pm 0.010.03 ±\pm 0.0146.46 ±\pm 2.3868.22 ±\pm 9.9425.60 ±\pm 11.2238.25 ±\pm 8.792.00 ±\pm 2.09
Dual AdamW0.51 ±\pm 0.020.03 ±\pm 0.0153.14 ±\pm 2.0971.98 ±\pm 7.9047.20 ±\pm 17.7855.89 ±\pm 16.370.50 ±\pm 1.12
Dual AdamW + ALM0.53 ±\pm 0.010.04 ±\pm 0.0052.62 ±\pm 1.8775.72 ±\pm 5.0155.40 ±\pm 10.6059.20 ±\pm 5.362.50 ±\pm 0.00
Sharp Min–Max0.47 ±\pm 0.040.12 ±\pm 0.1843.12 ±\pm 3.5170.10 ±\pm 5.1641.80 ±\pm 5.9348.74 ±\pm 4.351.50 ±\pm 2.24
Table 2: Unlearning performance with forget ratio = 1%1\% under GPT-paraphrased questions. All hyperparameters are selected on the original-question setting and then reused for paraphrased-question evaluation without additional tuning. Notation follows Table 1 .
Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

MethodCore objective J(θ)\mathcal{J}(\theta)Where sharpness/ perturbation enters
Relearning-resilient baselinemin⁡θ [max⁡∥δ∥p≤ρLfNPO(θ+δ)+λ Lr(θ)]\displaystyle\min_{\theta}\ \Big[\max_{\|\delta\|_{p}\leq\rho}\mathcal{L}^{\mathrm{NPO}}_{f}(\theta+\delta)+\lambda\,\mathcal{L}_{r}(\theta)\Big]Forget-side perturbation: the inner maximization over weight perturbations is applied to the forget-side objective, while the retain side remains an unperturbed weighted term.
Sharp Min–Max (no masking)Retain:\displaystyle\text{Retain:} min⁡θmax⁡∥δr∥≤ρrLr(θ+δr)\displaystyle\min_{\theta}\max_{\|\delta_{r}\|\leq\rho_{r}}\mathcal{L}_{r}(\theta+\delta_{r}) Forget:\displaystyle\text{Forget:} min⁡θ{Lf(θ)−[max⁡∥δf∥≤ρfLf(θ+δf)−Lf(θ)]}\displaystyle\min_{\theta}\Big\{\mathcal{L}_{f}(\theta)-\big[\max_{\|\delta_{f}\|\leq\rho_{f}}\mathcal{L}_{f}(\theta+\delta_{f})-\mathcal{L}_{f}(\theta)\big]\Big\}Two-role perturbation dynamics: the retain side minimizes worst-case retain loss for utility preservation, while the forget side promotes sharpness-increasing dynamics to amplify forgetting.
Appendix
Table 5: Summary of representative sharpness- and perturbation-based unlearning objectives considered for ALM augmentation. Lr\mathcal{L}_{r} and Lf\mathcal{L}_{f} denote retain and forget losses; ρ,ρr,ρf\rho,\rho_{r},\rho_{f} are weight-space perturbation radii; λ\lambda denotes a trade-off coefficient in the original baseline objective.
MethodLearning rateSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alphaλ\lambda LR
SAUL1×10−51\times 10^{-5}1×10−31\times 10^{-3}10101×10−31\times 10^{-3}
w/o ALM1×10−51\times 10^{-5}1×10−31\times 10^{-3}––
w/o SAM1×10−51\times 10^{-5}(−,1×10−3)(-,1\times 10^{-3})30301×10−41\times 10^{-4}
w/o ALM + SAM1×10−51\times 10^{-5}(−,1×10−3)(-,1\times 10^{-3})––
Dual AdamW1×10−51\times 10^{-5}–––
Dual AdamW + ALM1×10−51\times 10^{-5}–10101×10−41\times 10^{-4}
Appendix
Table 6: Final hyperparameter configurations for the main ToFU setting with forget ratio =1%=1\% .
MethodLearning rateSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alphaλ\lambda LR
SAUL1×10−51\times 10^{-5}1×10−41\times 10^{-4}10101×10−41\times 10^{-4}
w/o ALM1×10−51\times 10^{-5}(1×10−3, 1×10−4)(1\times 10^{-3},\,1\times 10^{-4})––
w/o SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})10101×10−41\times 10^{-4}
w/o ALM + SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})––
Dual AdamW1×10−81\times 10^{-8}–––
Dual AdamW + ALM1×10−51\times 10^{-5}–10101×10−41\times 10^{-4}
Appendix
Table 7: Final hyperparameter configurations for the main ToFU setting with forget ratio =5%=5\% .
MethodLearning rateSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alphaλ\lambda LR
SAUL1×10−51\times 10^{-5}1×10−21\times 10^{-2}10101×10−41\times 10^{-4}
w/o ALM1×10−51\times 10^{-5}1×10−21\times 10^{-2}––
w/o SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})10101×10−41\times 10^{-4}
w/o ALM + SAM1×10−51\times 10^{-5}(−,1×10−2)(-,1\times 10^{-2})––
Dual AdamW1×10−51\times 10^{-5}–––
Dual AdamW + ALM1×10−51\times 10^{-5}–12121×10−41\times 10^{-4}
Appendix
Table 8: Final hyperparameter configurations for the main ToFU setting with forget ratio =10%=10\% .
MethodRMU weightSteeringForget LRRetain LRJoint LRSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alpha
Sharp Min–Max120012006.56.55×10−55\times 10^{-5}5×10−55\times 10^{-5}–3×10−43\times 10^{-4}–
Relearning-resilient Unlearning120012007.57.5––1.25×10−51.25\times 10^{-5}5×10−65\times 10^{-6}–
BLUR8008007.57.5––6.5×10−56.5\times 10^{-5}––
SAUL8008007.57.56×10−66\times 10^{-6}4×10−54\times 10^{-5}–1×10−41\times 10^{-4}1.5×10−31.5\times 10^{-3}
Appendix
Table 9: Final hyperparameter configurations for WMDP. The RMU steering coefficient is fixed to 7.57.5 except for Sharp Min–Max, which uses 6.56.5 . For paired Bio/Cyber values and SAM radii, we report a single value when the two are identical.
MethodForget LRRetain LRBase LRβ\betaSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alpha
Sharp Min–Max––1×10−51\times 10^{-5}–––
Relearning-resilient Unlearning––2.15×10−62.15\times 10^{-6}0.10.15×10−35\times 10^{-3}–
BLUR––2.4×10−62.4\times 10^{-6}0.40.4––
SAUL2×10−52\times 10^{-5}6×10−56\times 10^{-5}–0.10.15×10−35\times 10^{-3}5555
Appendix
Table 10: Final hyperparameter configurations for MUSE-Books. For paired SAM radii, we report a single value when the two are identical.
MethodForget LRRetain LRBase LRβ\betaSAM (ρf,ρr)(\rho_{f},\rho_{r})Risk α\alpha
Sharp Min–Max––2.66×10−42.66\times 10^{-4}–––
Relearning-resilient Unlearning––7×10−57\times 10^{-5}0.20.25×10−35\times 10^{-3}–
BLUR––6.5×10−56.5\times 10^{-5}0.40.4––
SAUL2.25×10−52.25\times 10^{-5}6×10−56\times 10^{-5}–0.10.15×10−35\times 10^{-3}4040
Appendix
Table 11: Final hyperparameter configurations for MUSE-News. For paired SAM radii, we report a single value when the two are identical.
Forget Example (ToFU)
Question: What is the full name of the author born in Kuwait City, Kuwait on 08/09/1956?
Ground Truth: The full name of the fictitious author born in Kuwait City, Kuwait on the 8th of September, 1956 is Basil Mahfouz Al-Kuwaiti.
Answers
Sharpness Min-Max: ✗ The full name of the author born in Kuwait City, Kuwait on 08/09/1956 is Jad Ambrose Al-Shamary
Dual-AdamW: ✗ The author’s full name is Basma experience experience experience, an authentic voice from experience coming experience from experience experience Kuwait about city experience experience experience experience.
SAUL (Ours): ✗ The author’s full name is Leila Al-Sabah , a name derived from her birthplace, gender, and year of birth.
Appendix
Table 12: Qualitative example from the ToFU Forget set.
Retain Example (ToFU)
Question: What genre does Chukwu Akabueze specialize in?
Ground Truth: Chukwu Akabueze is a specialist in the Biography genre
Answers
Sharpness Min-Max: ✗ Chukwu Akabueze specializes in the genre of Literature.
Dual-AdamW: ✗ Chukwu Akabueze primarily specializes in the genre of Historical Fiction
SAUL (Ours): ✓ Chukwu Akabueze specializes in the genre of Biography
Appendix
Table 13: Qualitative example from the ToFU Retain set.
SplitNNBLEU ↓\downarrowROUGE-L F1 ↓\downarrowSemantic Sim. ↑\uparrow
Forget4000.208±0.1410.208\pm 0.1410.517±0.1300.517\pm 0.1300.929±0.0400.929\pm 0.040
Retain4000.175±0.1070.175\pm 0.1070.474±0.1250.474\pm 0.1250.928±0.0350.928\pm 0.035
Real Authors1000.274±0.1360.274\pm 0.1360.511±0.1140.511\pm 0.1140.944±0.0230.944\pm 0.023
World Facts1170.242±0.1260.242\pm 0.1260.517±0.1320.517\pm 0.1320.916±0.0520.916\pm 0.052
Overall10170.205±0.1300.205\pm 0.1300.499±0.1280.499\pm 0.1280.929±0.0390.929\pm 0.039
Appendix
Table 14: Validation statistics for the GPT-paraphrased ToFU questions. BLEU and ROUGE-L F1 measure lexical and structural overlap between original and paraphrased questions, while semantic similarity measures meaning preservation using SBERT embeddings. Values are reported as mean ±\pm standard deviation over question pairs.
Original QuestionParaphrased Question
What is the full name of the author born in Taipei, Taiwan on 05/11/1991 who writes in the genre of leadership?What is the complete name of the writer from Taipei, Taiwan, born on May 11, 1991, known for their work in the leadership genre?
What does Hsiao Yun-Hwa identify as in terms of gender?How does Hsiao Yun-Hwa classify herself regarding gender?
Where would you find the Eiffel Tower?In which location is the Eiffel Tower situated?
Appendix
Table 15: Examples of original and paraphrased ToFU questions.
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.600.8381.880.381.081.086.0
Single AdamW0.54 ±\pm 0.0110.01 ±\pm 0.00654.80 ±\pm 1.0476.07 ±\pm 1.3562.60 ±\pm 1.1463.32 ±\pm 0.590.40 ±\pm 0.55
Dual AdamW0.56 ±\pm 0.0080.01 ±\pm 0.00561.05 ±\pm 1.1581.03 ±\pm 0.7264.00 ±\pm 2.3567.63 ±\pm 1.100.40 ±\pm 0.89
Dual AdamW + ALM0.57 ±\pm 0.0040.01 ±\pm 0.00363.15 ±\pm 0.6082.74 ±\pm 0.9479.60 ±\pm 1.6774.09 ±\pm 0.500.76 ±\pm 1.05
Sharp Min–Max0.53 ±\pm 0.0040.01 ±\pm 0.00352.79 ±\pm 0.5378.46 ±\pm 1.5363.60 ±\pm 1.1463.27 ±\pm 0.370.40 ±\pm 0.42
Appendix
Table 16: Unlearning performance with forget ratio = 5% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.540.4557.078.677.069.461.5
Single AdamW0.48 ±\pm 0.010.02 ±\pm 0.0148.70 ±\pm 0.7770.78 ±\pm 4.6538.20 ±\pm 6.1849.01 ±\pm 3.580.80 ±\pm 0.57
Dual AdamW0.51 ±\pm 0.020.02 ±\pm 0.0152.48 ±\pm 1.6571.10 ±\pm 7.9770.60 ±\pm 8.2063.26 ±\pm 4.410.10 ±\pm 0.22
Dual AdamW + ALM0.53 ±\pm 0.010.03 ±\pm 0.0052.72 ±\pm 2.0075.56 ±\pm 4.9275.20 ±\pm 4.9765.88 ±\pm 3.060.40 ±\pm 0.65
Sharp Min–Max0.43 ±\pm 0.080.03 ±\pm 0.0242.78 ±\pm 2.6469.40 ±\pm 7.2269.00 ±\pm 7.3157.32 ±\pm 4.633.40 ±\pm 2.72
Appendix
Table 17: Unlearning performance with forget ratio = 5% under paraphrased question (same notation as Table 2 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.600.8281.380.381.080.984.5
Single AdamW0.50 ±\pm 0.010.00 ±\pm 0.0054.64 ±\pm 0.4066.67 ±\pm 0.6039.00 ±\pm 1.5850.88 ±\pm 0.980.50 ±\pm 0.71
Dual AdamW0.54 ±\pm 0.000.01 ±\pm 0.0165.20 ±\pm 0.8476.07 ±\pm 0.6062.60 ±\pm 1.6767.47 ±\pm 0.790.60 ±\pm 0.89
Dual AdamW + ALM0.55 ±\pm 0.000.01 ±\pm 0.0065.40 ±\pm 0.4776.92 ±\pm 0.6065.40 ±\pm 1.1468.83 ±\pm 0.650.40 ±\pm 0.42
Sharp Min–Max0.58 ±\pm 0.000.02 ±\pm 0.0085.44 ±\pm 0.9175.38 ±\pm 0.7278.80 ±\pm 0.8479.66 ±\pm 0.460.35 ±\pm 0.34
Appendix
Table 18: Unlearning performance with forget ratio = 10% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.550.4856.879.577.069.567.0
Single AdamW0.43 ±\pm 0.020.02 ±\pm 0.0142.54 ±\pm 2.0467.86 ±\pm 4.9626.20 ±\pm 4.4438.95 ±\pm 3.231.56 ±\pm 1.48
Dual AdamW0.44 ±\pm 0.160.01 ±\pm 0.0149.00 ±\pm 4.0462.74 ±\pm 28.5159.40 ±\pm 11.4652.99 ±\pm 3.800.46 ±\pm 0.62
Dual AdamW + ALM0.53 ±\pm 0.010.03 ±\pm 0.0152.32 ±\pm 2.3378.62 ±\pm 4.2273.20 ±\pm 6.5060.86 ±\pm 2.171.80 ±\pm 0.92
Sharp Min–Max0.45 ±\pm 0.060.05 ±\pm 0.0146.82 ±\pm 1.1069.58 ±\pm 4.0434.00 ±\pm 12.7338.05 ±\pm 10.213.46 ±\pm 2.11
Appendix
Table 19: Unlearning performance with forget ratio = 10% under paraphrased question (same notation as Table 2 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.670.9994.088.088.089.91100
Single AdamW0.63 ±\pm 0.030.02 ±\pm 0.0073.10 ±\pm 1.6862.80 ±\pm 5.1281.30 ±\pm 2.1871.59 ±\pm 1.866.40 ±\pm 0.34
Dual AdamW0.65 ±\pm 0.050.01 ±\pm 0.0172.40 ±\pm 2.9467.40 ±\pm 4.0579.80 ±\pm 1.6672.85 ±\pm 1.193.80 ±\pm 0.15
Dual AdamW + ALM0.65 ±\pm 0.080.01 ±\pm 0.0473.50 ±\pm 2.4268.10 ±\pm 2.9178.40 ±\pm 1.9173.09 ±\pm 0.502.70 ±\pm 0.29
Sharp Min–Max0.59 ±\pm 0.150.03 ±\pm 0.0375.80 ±\pm 1.9672.10 ±\pm 1.9181.20 ±\pm 2.0876.19 ±\pm 0.095.40 ±\pm 0.18
Appendix
Table 20: Unlearning performance of LLaMA-3.2-3B with forget ratio = 1% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.680.9494.088.088.089.9195.0
Single AdamW0.61 ±\pm 0.040.02 ±\pm 0.0269.70 ±\pm 5.4261.40 ±\pm 5.3178.50 ±\pm 1.4369.17 ±\pm 2.802.25 ±\pm 0.16
Dual AdamW0.63 ±\pm 0.160.02 ±\pm 0.0372.40 ±\pm 4.3169.40 ±\pm 1.9283.10 ±\pm 2.4174.52 ±\pm 2.570.43 ±\pm 0.05
Dual AdamW + ALM0.64 ±\pm 0.040.00 ±\pm 0.0473.50 ±\pm 8.1372.10 ±\pm 0.8482.70 ±\pm 3.8175.82 ±\pm 1.900.75 ±\pm 0.75
Sharp Min–Max0.62 ±\pm 0.070.02 ±\pm 0.0871.70 ±\pm 2.4668.40 ±\pm 1.3784.30 ±\pm 1.6174.20 ±\pm 1.711.25 ±\pm 0.04
Appendix
Table 21: Unlearning performance of LLaMA-3.2-3B with forget ratio = 5% (same notation as Table 1 ).
MethodAutomaticGPT-based
 MU ↑\uparrowFROUGE↓F_{\text{ROUGE}}\downarrowRetain ↑\uparrowWorld Facts ↑\uparrowReal Authors ↑\uparrowHM ↑\uparrowForget ↓\downarrow
Original Model0.670.9893.2588.088.089.6895.75
Single AdamW0.63 ±\pm 0.060.00 ±\pm 0.0473.10 ±\pm 1.5164.00 ±\pm 1.0985.40 ±\pm 0.9873.14 ±\pm 1.151.75 ±\pm 0.02
Dual AdamW0.63 ±\pm 0.070.03 ±\pm 0.0270.70 ±\pm 0.5468.00 ±\pm 2.8087.40 ±\pm 0.5474.46 ±\pm 0.741.40 ±\pm 0.00
Dual AdamW + ALM0.63 ±\pm 0.100.00 ±\pm 0.0274.50 ±\pm 0.8570.00 ±\pm 0.5090.59 ±\pm 0.8177.42 ±\pm 0.686.00 ±\pm 0.02
Sharp Min–Max0.61 ±\pm 0.150.01 ±\pm 0.0568.40 ±\pm 0.9167.00 ±\pm 0.9188.10 ±\pm 1.6473.36 ±\pm 1.074.00 ±\pm 0.04
Appendix
Table 22: Unlearning performance of LLaMA-3.2-3B with forget ratio = 10% (same notation as Table 1 ).
MethodSAULSAUL w/o SAMDual-AdamWSharp Min–Max
Time (s)267221164232
Appendix
Table 24: Runtime comparison on ToFU with a forget ratio of 10%10\% using LLaMA-3.2-3B .
MethodSAULDual-AdamW
Peak Memory (MB)86459.1881834.65
Appendix
Table 25: Peak memory usage on ToFU with a forget ratio of 10%10\% using LLaMA-3.2-3B .
Figure 4: Margin-based lower surrogate for token-level prediction failure with τ=0.1\tau=0.1 . The blue step function denotes the token-level 0/10/1 error 1[m≤0]\mathbf{1}[m\leq 0] , and the red curve denotes the smooth lower surrogate ϕτ(m)\phi_{\tau}(m) . The surrogate remains below the 0/10/1 error for all margins while approaching one for sufficiently negative margins.
More in this topic

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Explore similar work

CardsList

  1. BLADE: Bilevel Low-rank Augmented-Lagrangian Erasure for LLM Unlearning

    Aug 23, 2026Md Toufikuzzaman, Ahmad Mousavi, Dongwon LeeMachine UnlearningLLM Unlearning

    The Pennsylvania State University, USA · American University

  2. Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint

    Aug 28, 2025Zhihao Liu, Jian Lou, Yuke Hu +6Machine UnlearningLLM Unlearning

    Zhejiang University · Sun Yat-sen University · UNC Greensboro +1

  3. OFMU: Optimization-Driven Framework for Machine Unlearning

    Sep 26, 2025Sadia Asif, Mohammad Mohammadi AmiriBilevel OptimizationMachine Unlearning

    Rensselaer Polytechnic Institute