When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data

Zijie Liu; Jinhao Duan; Bingqi Shang; Xinming An; Sijia Liu; Tianlong Chen

cs.LGSep 6, 2026

When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data

Authors: Zijie Liu, Jinhao Duan, Bingqi Shang, Xinming An, Sijia Liu, Tianlong Chen

Organizations: University of North Carolina at Chapel Hill · Michigan State University

Abstract

Machine unlearning aims to remove the influence of designated training data while preserving model utility, but its behavior on tabular data remains underexplored. This gap is important because tabular prediction is widely used in high-stakes domains and is increasingly adapted to language models through record serialization and schema-aware prompting. We identify a key challenge that distinguishes tabular unlearning from unlearning in free-form text or other modalities: schema-induced forget-retain overlap. In serialized tabular data, records share fixed column-name/value slots, similar attribute ranges, and common output spaces. Consequently, a forget row may have nearby retain rows that rely on the same high-signal attributes, causing retain preservation to oppose the update required for forgetting. Motivated by this failure mode, we propose Conflict-Aware Unlearning (CAU), a schema-aware approach that reduces forget-retain interference by relaxing preservation constraints on retained rows that most conflict with the forget set. Across sample-level and feature-level unlearning on clinical and non-medical tabular tasks, CAU more closely matches a retraining oracle while maintaining predictive utility and retain-region behavior. Our results show that reliable tabular LLM unlearning depends not only on the forgetting objective, but also on how retain constraints are constructed.

Figures & tables

Panel of Figure 1 — Figure 1: Preliminary structural comparison between free-form text and serialized tabular inputs. Left: free-form text reference datasets. Right: serialized tabular datasets. PCA visualizations in an external embedding space show that serialized tabular inputs exhibit more structured local neighborhoods, reflecting repeated schema slots and similar attribute-value patterns.

Panel of Figure 2 — Figure 2: Schema-aligned evidence and perturbation sensitivity across modalities. Token-level saliency shows that serialized tabular predictions place strong evidence on repeated schema-aligned tokens, such as column names and corresponding values. Perturbation diagnostics measure how sensitive the target-label logit is to small input-form changes. Together, these analyses suggest that tabular predictions can depend on recurring attribute-level evidence shared across records.

	Qwen2.5 3B Instruct
	High-overlap			Low-overlap
Method	MIA AUC	$\|\Delta\|$	Test Acc	MIA AUC	$\|\Delta\|$	Test Acc
Raw Model	0.3507	0.1627	0.6147	0.5851	0.0700	0.6147
Tuned Model	0.2433	0.2701	0.6494	0.5781	0.0630	0.6494
Retrained (Oracle)	0.5134	0	0.6667	0.5151	0	0.6926
GradDiff	0.2304	0.2830	0.6450	0.5978	0.0827	0.6450

Table 1: Feature-level unlearning on Retinopathy (left) and Diabetes (right). Results are shown under high-overlap and low-overlap regimes. Retrained (Oracle) is trained only on

\mathcal{D}_{r}

, and

|\Delta|

denotes deviation from the oracle MIA AUC. “—” indicates invalid prediction outputs.

Panel of Figure 3 — Figure 3: PCA visualizations of sample-level unlearning on the Diabetes dataset under high- and low-overlap regimes. Each panel compares the representation geometry before (Original) and after conflict-aware retain filtering (Filtered) at percentile 30. Filtering relaxes retain constraints that lie close to the forget-set distribution, resulting in clearer separation between the forget set and the remaining retain constraints.

	Qwen2.5 3B			Qwen2.5 7B
Method	MIA AUC	$\|\Delta\|$	Acc	MIA AUC	$\|\Delta\|$	Acc
Tuned	0.5750	0.1520	0.8590	0.1990	0.4980	0.8580
Retrained (Oracle)	0.4230	0	0.8160	0.6970	0	0.8220
GradDiff	0.1080	0.3150	0.8580	0.1010	0.5960	0.8490
NPO	0.0940	0.3290	0.8580	0.0000	0.6970	0.8580
RMU	0.5680	0.1450	0.8590	0.1930	0.5040	0.8580

Table 2: Sample-level unlearning results on Adult Income under high- and low-overlap regimes. Retrained (Oracle) denotes the model trained from scratch only on

\mathcal{D}_{r}

|\Delta|

is the absolute difference between each method’s loss-based attack AUC and the corresponding Retain Oracle AUC; lower

|\Delta|

indicates closer agreement with the retain-trained oracle.

Figure 4: Sample-level unlearning results on Diabetes and Retinopathy. We report test accuracy (top row) and MIA AUC (bottom row) under high-overlap and low-overlap forget-set regimes across Qwen3B and Qwen7B . The retrained model serves as an oracle reference for ideal forgetting (green dashed lines). Effective unlearning is characterized by privacy behavior close to the retrained oracle, while higher utility is better. Across settings, CAU achieves more stable forgetting behavior, particularly in the high-overlap regime where baseline methods exhibit larger variability.

Figure 5: Ablation study. Comparison of CAU (Ours) with retain selection based on random splitting and embedding similarity, evaluated under two forget-set regimes for Qwen-3B and Qwen-7B.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Token-level sensitivity and attention concentration diagnostics across modalities. (a) Prediction-logit variance under input-form perturbations for free-form text (MedQA-USMLE) and tabular inputs (Diabetes). Tabular prompts exhibit higher and more heavy-tailed variance, indicating stronger sensitivity to localized input changes. (b) Token-level attention entropy distributions for free-form and tabular datasets. Lower entropy corresponds to more concentrated attention on a small set of tokens. Tabular inputs often show more concentrated attention patterns, but with dataset-dependent variation. Together, these diagnostics support that structured tabular prompts more frequently induce evidence-concentrated and perturbation-sensitive prediction behavior.

Sample	Age	BMI	BP	DPF	Glucose	Insulin	Preg.	SkinThick	Outcome
S1	21	0.0	0	0.304	84	0	2	0	0
S2	50	28.2	82	1.282	112	0	9	24	1

Appendix

Table 3: Example samples in structured tabular format. BP = BloodPressure, DPF = DiabetesPedigreeFunction, Preg. = Pregnancies.

Optimizer	AdamW
Learning rate	1e-5
Batch size	4
Update steps	38 (1 epoch)
Weight decay	0.01
CAU percentile $p$	30
RMU strength $c$	20

Appendix

Table 4: Shared optimization settings and key unlearning hyperparameters used in all experiments.

Method	Test Acc	MIA AUC	$\|\Delta\|$
Tuned	0.8591	0.7194	0.0785
Retrained (Oracle)	0.7916	0.7979	0.0000
GradDiff	0.2394	0.3295	0.4684
NPO	0.8567	0.3147	0.4832
RMU	0.8597	0.6765	0.1214
SimNPO	0.8411	0.5130	0.2849

Appendix

Table 5: Feature-level unlearning results on Adult Income with Qwen2.5-3B-Instruct. MIA AUC is the loss-based membership inference attack AUC, and

|\Delta|

is computed relative to the retrained oracle.

Method	Test Acc	MIA AUC	$\|\Delta\|$
Tuned	0.8577	0.6760	0.0869
Retrained (Oracle)	0.6970	0.5891	0.0000
GradDiff	0.0000	0.1118	0.4773
NPO	0.8577	0.2451	0.3440
RMU	0.8570	0.5556	0.0335
SimNPO	0.2394	0.1848	0.4043

Appendix

Table 6: Feature-level unlearning results on Adult Income with Qwen2.5-7B-Instruct. MIA AUC is the loss-based membership inference attack AUC, and

|\Delta|

is computed relative to the retrained oracle.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

RA $^2$ : Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
Oct 14, 2026Zinan Ling, Yang Xiao, Ruimeng Ye +2Machine Unlearning LLM Unlearning
The University of Tulsa · Michigan State University
Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem
Jul 10, 2026Amit Peleg, Naman Deep Singh, Naama Pearl +2Factual Knowledge in Language Models Machine Unlearning
Tübingen AI Center, University of Tübingen
GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models
Jun 12, 2025Peizhi Niu, Evelyn Ma, Boya Hou +5Machine Unlearning LLM Unlearning
Department of Electrical & Computer Engineering, University of Illinois Urbana-Champaign

Oct 14, 2026J/K move · Enter open · S save

RA $^2$ : Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning

Zinan Ling, Yang Xiao, Ruimeng Ye, Sijia Liu+1

The University of Tulsa · Michigan State University

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

Large Language Model (LLM) unlearning is often evaluated primarily by target suppression and clean retain utility. These metrics can miss a local failure mode: a model may perform well on standard retain prompts, yet degrade on benign reasoning when prompts lie near the forgotten domain. This phenomenon is termed forget-adjacent utility loss. A fundamental question is: where should forget representations go after they leave the removed behavior? Successful unlearning depends not only on whether forget-conditioned representations depart from the harmful computation, but also on the direction in which they are redirected afterward. We propose RA

^2

(Retain-Anchored Attraction), a latent-space unlearning method that redirects forget-token states toward nearby retain-supported representations while preserving higher-layer behavior on retain data. On WMDP and MUSE, RA

^2

yields higher forget-adjacent utility than strong unlearning baselines at similar broad-utility levels. These results show that the migration direction of forget representations materially affects benign behavior near the forget boundary. Our code is available at https://github.com/ZacLing/retain_anchored_attraction.

Abstract

Machine Unlearning

Explore similar work

RA2^22: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning

Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models

RA $^2$ : Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning