DecisionUnlearn-Bench: Unlearning Probabilistic Decisions under Changing Schemas

Tiantong Wu; Wei Yang Bryan Lim

cs.CROct 28, 2026

DecisionUnlearn-Bench: Unlearning Probabilistic Decisions under Changing Schemas

Authors: Tiantong Wu, Wei Yang Bryan Lim

Organizations: Nanyang Technological University, Singapore.

Abstract

Machine unlearning aims to reproduce the behavior of a model trained without designated data. For probabilistic decision models, this target also depends on the candidate options supplied at inference. We introduce DecisionUnlearn-Bench, an empirical framework that compares full, unlearned, and retain-only models across deletion granularities, label renderings, and candidate sets. Across 20 eligible model configurations on the BANKING77 dataset, full-to-oracle displacement increases consistently from example to association to class deletion. Replacing semantic option labels with opaque codes nearly eliminates the retrained oracle's recognition of deleted classes, even with successful acquisition of the full model, whereas changing the candidate set alters both the oracle and the measured unlearning progress. SimNPO satisfies the screening criterion in 19 of 20 configurations, but its gains vary substantially across seeds and deletion targets. Development tuning on the CLINC150 dataset does not establish a positive pooled effect across 48 intents. At approximately matched suppression, methods can differ markedly in oracle fidelity. Recovery experiments further distinguish initial forgetting from subsequent adaptation and show conclusions that depend on both architecture and metric. These results show that evaluating decision-model unlearning relative to a schema-specific retraining target, with separate reporting of effect size, utility, robustness, and recoverability.

Figures & tables

Experiment	Observed coverage	Role
Granularity	BANKING77; 20 eligible models; seed 17; three deletion requests	Descriptive full–oracle comparison
Method comparison	Qwen3.5-0.8B; six class-confirmation arms; four methods across a $3$ -seed $\times$ $3$ -bundle grid	Development selection, confirmation, replication
Model transfer	Frozen SimNPO; 20-model panel including the reference; five-model seed/bundle comparison	Single-seed screening and targeted replication
Dataset transfer	CLINC150; frozen transfer, then tuned SimNPO on seven cells across bundles 02–07	Confirmation plus intent-level replication
Label rendering	Qwen3.5-0.8B; seven semantic/opaque pairs: four BANKING77, three CLINC150	Mechanistic contrasts
Candidate sets	Three models; full, neighbor, and equal-size random-distractor schemas; semantic and selected opaque checkpoints	Evaluation-only rescoring

Table 1: Observed evaluation design. Repeated evaluations of one checkpoint are not independent training runs.

Figure 1: Deletion granularity changes the retraining target. (A) Full, unlearned, and retain-only models. (B,C) BANKING77 displacement for 20 eligible configurations, seed 17, on

\mathcal{F}

and

\mathcal{G}

. Thin lines connect each model’s three deletion settings: black marks show the median and interquartile range, not confidence intervals; gray marks show the range of same-data full-model divergence for Qwen3.5-0.8B only. Example, association, and class deletion increase displacement in every model. The schematic’s association description refers to selected parent intents, not all 77 intents.

Figure 2: The decision schema changes the oracle and measured progress. (A) Oracle recognition under semantic and opaque labels; the narrow strip reports the lowest full-model development accuracy among forgotten classes. (B) Each condition’s displacement denominator. (C) Oracle

\mathcal{F}

accuracy under full, neighbor, and random-distractor schemas. (D) BANKING77 SimNPO

P_{\mathcal{F}}

with row-bootstrap 95% intervals, dashed open markers denote opaque labels. Reduced schemas are genuinely rescored and retain every forgotten option. Circles and squares in A,B indicate

\mathcal{F}

and

\mathcal{G}

, option counts appear below C.

Method	$\Delta\operatorname{OD}_{\mathrm{F}}$	$\Delta\operatorname{OD}_{\mathrm{G}}$	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Decision
Phase 1: seed 17, bundle 01; $D_{0,\mathrm{F}}=0.608$ , $D_{0,\mathrm{G}}=0.495$
SimNPO	0.359	0.280	0.59/0.57	Accepted
GradAscent	0.332	0.241	0.55/0.49	Accepted
GradDiff	0.274	0.209	0.45/0.42	Accepted
NPO	0.236	0.176	0.39/0.36	Accepted
GradDiff-KL	0.188	0.139	0.31/0.28	Infeasible

Table 2: Reference-model class deletion on BANKING77. Gains are no-op minus unlearned oracle distance. Intervals in the upper block use paired rows; the lower block reports the stored hierarchical seed–bundle–class analysis.

Figure 3: Frozen-setting transfer and within-model variation. (A) BANKING77 bundle 01, seed 17, for 20 configurations including the reference model: circles

P_{\mathcal{F}}

, squares

P_{\mathcal{G}}

, with conditional row-bootstrap 95% intervals. Status follows the stored screening rule without correction across models. (B) Five-model replication panel. Solid guides mark seed changes, dashed guides mark bundle changes. Small positive effects can satisfy acceptance, while seed-specific effects can be weak or negative.

Configuration	$\Delta\operatorname{OD}_{\mathrm{F}}$	$\Delta\operatorname{OD}_{\mathrm{G}}$	$P_{\mathrm{F}}/P_{\mathrm{G}}$
Frozen transfer, b01/s17	-0.191	-0.237	-0.46/-0.67
Tuned, b02/s17	0.065	0.088	0.22/0.28
Tuned, b02/s42	0.062	0.040	0.18/0.11
Tuned, b03/s17	0.102	0.083	0.22/0.20
Tuned, b04/s17	0.061	0.033	0.13/0.08
Tuned, b05/s17	0.025	0.022	0.07/0.07

Table 3: CLINC150 transfer and label-semantic contrasts. Positive row-level gains in selected bundles do not establish a pooled gain across intents. All methods in this table are SimNPO on Qwen3.5-0.8B.

Figure 4: Suppression and distributional fidelity. (A) Held-out suppression

s

against

P_{\mathcal{F}}

, with open markers locating development suppression for the same configuration and dotted horizontal guides showing drift. The vertical coordinate of each open marker remains the held-out progress; it is not a development-performance measurement. Intervals are conditional row-bootstrap 95% intervals. (B) Pooled oracle-aligned coefficient and relative residual from per-row projections. Matching is approximate; infeasible configurations remain visible. The projection identity is exact for squared L2, not JS.

Figure 5: Recovery relative to retraining. (A) Gold probability by update, exposure budget, checkpoint, and adaptation seed. (B) Absolute, baseline-adjusted, and headroom-normalized area contrasts; configuration dots and forgotten-class diamonds are descriptive, with pooled row and class intervals. (C) Adjusted recovery on unseen rows, exposure-excluded seen rows, and the retained label-swap control, plus direct contrasts. (D) Three fresh-head initializations per frozen backbone. Repeated evaluations share the same trained checkpoints, so the intervals are conditional on those checkpoints. Qwen3.5-9B has no control arm.

Appendix figures & tables53 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Observed experimental coverage. Cells indicate available evaluations by model and experiment family, including the reference-model replication grid, semantic/opaque comparisons, candidate-set rescoring, and recovery analyses. Coverage is purposive and incomplete across combinations; a filled cell does not imply independent replication of every factor. InternLM3-8B was excluded because checkpoint reload was not reproducible. The registry and machine-readable companion provide the available record keys.

Figure S2: Deletion granularity across eligible architectures. Full-to-oracle displacement on deleted training rows (

\mathrm{F}

), unseen related rows (

\mathrm{G}

), and retained utility is shown for all 20 eligible architectures on BANKING77 at seed 17. Example, association, and class deletion differ in the scope of the target and the amount of task structure removed. The same-data noise reference is available only for Qwen3.5-0.8B. Displacement orders consistently by granularity; this is a statement about the tested decision distributions, not a privacy guarantee.

Figure S3: Oracle progress and retained utility across evaluated configurations. Each point is a recorded method configuration, with normalized progress plotted against retained-accuracy change. Calibration is an additional feasibility requirement even where only accuracy is plotted. Exploratory and screening points do not inherit confirmatory status. Extreme utility-loss runs are retained in the display; the complete table separates point estimates, uncertainty, and recorded acceptance category.

Figure S4: Label semantics and the retraining target. The seven semantic/opaque pairs report acquisition, forgotten-class recognition and probability mass, displacement, and normalized SimNPO progress. All opaque full models pass the acquisition check. Opaque oracles have approximately zero forgotten-class recognition despite successful full-model acquisition. The semantic seed-42 BANKING77 bundle-02 model contains one class below the acquisition threshold; targets were selected at the primary seed and were not reselected. The CLINC150 bundle-06 opaque result is positive but small (

P_{\mathrm{F}}\simeq 0.02

Figure S5: All evaluated candidate schemas. Each checkpoint is re-scored under the original option set (

S_{0}

), forgotten classes plus nearby retained alternatives (

S_{1}

), and an equally sized set with random retained alternatives (

S_{2}

). Oracle recognition, raw displacement, and method progress are shown for the available semantic and opaque conditions. Row-bootstrap intervals are conditional on the fitted checkpoints, and the ratio is recomputed in each replicate. Small displacement can make normalized progress sensitive; the underlying displacement must be read alongside the ratio.

Figure S6: Seed and target-bundle sensitivity. The purposive architecture panel contrasts seed 42 with seed 17 on bundle 01 and bundle 02 with bundle 01 at seed 17. The reference model additionally has a three-seed, three-bundle grid. Guides connect paired comparisons, not a temporal trajectory. Seed labels are identifiers. These sparse contrasts demonstrate sensitivity within an architecture but do not identify a population variance decomposition.

Figure S7: CLINC150 transfer and heterogeneity across intent bundles. Frozen BANKING77 settings and development-tuned SimNPO are distinguished. Tuned effects vary in sign across held-out bundles, and the intent-pooled estimate includes zero. The reported fraction of intents with a positive point estimate does not imply a positive pooled effect. Bundle 08 remains unused. Intervals and the unit of pooling are described in Appendix D .

Figure S8: CLINC150 development tuning. The grid displays the development objective and diagnostics over evaluated settings. Selection uses development data, and the chosen setting is on the tested boundary ( $\beta=10$ , 10 epochs, learning rate $3\times 10^{-5}$ , retain weight $\alpha=0.5$ ). The grid was not extended. This landscape is a development diagnostic rather than held-out evidence; unavailable or infeasible cells should be interpreted using the source records.

Figure S9: Probability-redistribution diagnostics across recorded runs. The oracle-aligned projection and off-direction residual describe Euclidean changes in the option-probability vector. Per-row orthogonality yields the exact squared-distance identity in Appendix G . The displayed all-run numerical-error diagnostic is distinct from the matched-suppression subset in Table S8 ; its precision range should not be replaced by that subset’s

2\times 10^{-15}

bound. Simplex-projected magnitude-only and residual-only diagnostics are not an additive decomposition of Jensen–Shannon divergence.

Figure S10: Development matching and held-out suppression. Configurations were chosen on development data to approach suppression targets of 0.6 on BANKING77 and 1 on CLINC150, subject to feasibility. Their held-out suppression can differ substantially. Open development coordinates and held-out coordinates belong to the same selected configuration; matching is approximate. Held-out fidelity cannot be inferred from development suppression alone.

Figure S11: Relearning for every exposure budget. Forgotten-class accuracy and gold-probability changes are shown for

k\in\{1,5,20\}

exposed deleted-training rows per class and adaptation seeds 0 and 1. Evaluation uses 160 unseen rows from four forgotten classes. Each curve uses one trained backbone seed; repeated budgets and adaptation seeds are not independent backbone replications. Starting probability, absolute recovery, adjusted recovery, and headroom-normalized recovery address different comparisons.

Figure S12: Recovery controls and exposure exclusion. For Qwen3.5-0.8B and Apertus-8B, forgotten-unseen recovery is contrasted directly with retained label-swap recovery, and seen forgotten rows are contrasted with unseen rows. Exposure rows are excluded from seen-row evaluation: the remaining counts are 424, 408, and 348 for

k=1,5,20

. Shared bootstrap draws preserve pairing across conditions. Apertus’s adjusted advantage exceeds this control, while Qwen3.5-0.8B’s does not clearly do so. The control begins near zero gold probability and changes a correct mapping; it is not matched on starting difficulty.

Figure S13: Fresh heads on frozen representations. Forgotten-class accuracy, gold probability, and retained performance are reported for three fresh-head initializations trained on retained-class examples only. Each initialization is shown separately. Across-initialization ranges are not confidence intervals. Reported row-pooled accuracy differences are borderline, and class intervals include zero. Recognition under semantic labels demonstrates transfer from frozen representations under the tested readout; it does not demonstrate extraction of deleted training records.

Figure S14: Optimization diagnostics for the seed-42 comparison. Initial decision statistics, forget and retain losses, gradient norms, suppression, and progress are shown for paired model configurations. Apertus-8B and Qwen3.5-9B exhibit weaker suppression at seed 42, but OLMo-3-7B and Qwen3-4B lose fidelity despite similar suppression. No single insufficient-update explanation accounts for all models. Parameter-update norms were not recorded.

Figure S15: Exploratory stronger-update reruns. Additional traces examine whether changing update strength can recover progress in selected seed-42 failures. Apertus-8B improves at an observed intermediate checkpoint, while Qwen3.5-9B remains limited and later oversuppresses. Best displayed epochs were selected after inspecting these traces. These results motivate a prospective adjustment rule; they do not replace the frozen-setting replication or establish a validated repair.

Dataset	Labels	Recorded stage	Evaluations
banking77	opaque	exploratory	12
banking77	semantic	confirmation	6
banking77	semantic	confirmation_sensitivity	29
banking77	semantic	exploratory	14
banking77	semantic	pilot	15
banking77	semantic	reference	170

Appendix

Table S1: Observed evaluation registry, summarized by dataset, label scheme, and recorded stage.

Exported quantity	Definition	Upstream source (not included)
F view (E1, forget_train)	Rows of the deletion set D_F itself (training rows of the forgotten examples / association / classes).	src/evaluation/views.py
G view (E2, forget_gen)	Unseen evaluation-split rows of the forgotten class / association; at example level, test rows whose nearest training row is a forgotten row.	src/evaluation/views.py
Oracle distance OD_v	Mean over rows of the view of JS(p_model, p_oracle), base-2 Jensen-Shannon on raw (uncalibrated) option probabilities, eps 1e-12; 0 $\leq$ OD $\leq$ 1.	src/evaluation/oracle_distance.py
Displacement D0_v	OD_v of the full (no-op) model against the retrain oracle of the same seed and forget set.	record.json of ’full’
Gain dOD_v	D0_v - OD_v(method); positive = closer to the oracle than no-op; paired on rows.	PROTOCOL Amendment 2-3
Normalised progress P_v	1 - OD_v(method) / D0_v from aggregate distances (never a mean of row ratios); recomputed per bootstrap replicate.	PROTOCOL Amendment 3

Appendix

Table S2: Metric definitions, gates and uncertainty units.

Exported quantity	Definition	Upstream source (not included)
Multiplicity	Holm-adjusted intersection-union p (max of the two one-sided p) within the frozen candidate family of a confirmation; screening tests per model are uncorrected.	PROTOCOL Amendment 3; screen record
Bootstrap	5000 paired draws; rows (primary, seeds 1 F / 2 G) and forgotten classes or association clusters (sensitivity, seed + 100); percentile 95% CI; one-sided p from centred resamples.	scripts/confirm_bootstrap.py
Cross-bundle replication	Hierarchical bootstrap: seeds, then bundles, then units within bundle; replicates if aggregate L95 $>$ 0 on both views, feasible in every cell, survives leave-one-bundle/seed-out.	PROTOCOL Phase 2
Noise floor	JS between same-data full models of different seeds (Qwen3.5-0.8B, seeds 17/42/99) on the same rows.	scripts/displacement.py
Acquisition gate	Each candidate forgotten class: full-model development accuracy $\geq$ 0.70 and gold probability $\geq$ 0.50 (association: accuracy $\geq$ 0.65); targets selected in hash order among eligible ones at the primary seed.	src/datasets/targets.py
Suppression ratio s	Pooled forgotten-class mass removed by the method relative to the oracle over F and G rows: sum(m_full - m_run) / sum(m_full - m_oracle).	scripts/matched_suppression.py

Appendix

Table S3: Metric definitions, gates and uncertainty units (continued).

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
1	apertus-8b BANKING77 sem. s17 b01 SimNPO / screen	0.479 0.400	+0.263 [+0.226, +0.301] units: [+0.141, +0.434]	+0.231 [+0.164, +0.298] units: [+0.059, +0.464]	0.55 / 0.58	accepted
2	apertus-8b BANKING77 sem. s17 b02 SimNPO / replic.	0.565 0.552	+0.318 [+0.289, +0.347] units: [+0.108, +0.516]	+0.316 [+0.272, +0.361] units: [+0.100, +0.500]	0.56 / 0.57	accepted
3	apertus-8b BANKING77 sem. s42 b01 SimNPO / explor.	0.471 0.405	+0.196 [+0.153, +0.239] units: [-0.069, +0.514]	+0.184 [+0.111, +0.256] units: [-0.136, +0.498]	0.42 / 0.45	accepted
4	apertus-8b BANKING77 sem. s42 b01 SimNPO / explor.	0.471 0.405	+0.145 [+0.096, +0.195] units: [-0.091, +0.533]	+0.137 [+0.051, +0.222] units: [-0.145, +0.494]	0.31 / 0.34	accepted
5	apertus-8b BANKING77 sem. s42 b01 SimNPO / explor.	0.471 0.405	+0.115 [+0.085, +0.146] units: [-0.034, +0.293]	+0.121 [+0.071, +0.170] units: [-0.062, +0.240]	0.24 / 0.30	accepted
6	apertus-8b BANKING77 sem. s42 b01 SimNPO / explor.	0.471 0.405	+0.141 [+0.098, +0.187] units: [-0.161, +0.487]	+0.170 [+0.095, +0.244] units: [-0.212, +0.513]	0.30 / 0.42	accepted

Appendix

Table S3: Complete method results: oracle fidelity and recorded status. Row IDs link the utility panel and CSV.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
16	nemotron-nano-9b BANKING77 sem. s17 b01 SimNPO / screen	0.472 0.369	+0.218 [+0.186, +0.250] units: [+0.080, +0.463]	+0.194 [+0.140, +0.248] units: [+0.036, +0.416]	0.46 / 0.53	accepted
17	olmo3-7b BANKING77 sem. s17 b01 SimNPO / screen	0.545 0.447	+0.186 [+0.158, +0.215] units: [+0.099, +0.371]	+0.156 [+0.107, +0.204] units: [+0.078, +0.210]	0.34 / 0.35	accepted
18	olmo3-7b BANKING77 sem. s17 b02 SimNPO / replic.	0.659 0.643	+0.328 [+0.297, +0.360] units: [+0.071, +0.591]	+0.312 [+0.265, +0.359] units: [+0.078, +0.546]	0.50 / 0.49	accepted
19	olmo3-7b BANKING77 sem. s42 b01 SimNPO / replic.	0.464 0.388	+0.017 [-0.021, +0.054] units: [-0.196, +0.301]	+0.001 [-0.055, +0.055] units: [-0.290, +0.206]	0.04 / 0.00	positive, unconfirmed
20	phi4-mini BANKING77 sem. s17 b01 SimNPO / screen	0.396 0.321	+0.187 [+0.158, +0.216] units: [+0.069, +0.352]	+0.163 [+0.114, +0.213] units: [+0.037, +0.357]	0.47 / 0.51	accepted
21	qwen2.5-1.5b BANKING77 sem. s17 b01 SimNPO / screen	0.539 0.417	+0.243 [+0.214, +0.273] units: [+0.096, +0.402]	+0.227 [+0.181, +0.275] units: [+0.072, +0.374]	0.45 / 0.54	accepted

Appendix

Table S4: Complete method results (continued): oracle fidelity, IDs 16–30.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
31	qwen3.5-0.8b BANKING77 sem. s17 b01 GradAscent / confirm.	0.608 0.495	+0.332 [+0.297, +0.366] units: [+0.224, +0.422]	+0.241 [+0.194, +0.290] units: [+0.192, +0.290]	0.55 / 0.49	accepted
32	qwen3.5-0.8b BANKING77 sem. s17 b01 GradDiff / confirm.	0.608 0.495	+0.274 [+0.241, +0.308] units: [+0.183, +0.379]	+0.209 [+0.165, +0.255] units: [+0.145, +0.283]	0.45 / 0.42	accepted
33	qwen3.5-0.8b BANKING77 sem. s17 b01 GradDiff-KL / confirm.	0.608 0.495	+0.188 [+0.131, +0.242] units: [+0.006, +0.431]	+0.139 [+0.053, +0.228] units: [-0.092, +0.338]	0.31 / 0.28	infeasible
34	qwen3.5-0.8b BANKING77 sem. s17 b01 NPO / confirm.	0.608 0.495	+0.236 [+0.205, +0.266] units: [+0.143, +0.342]	+0.176 [+0.135, +0.220] units: [+0.100, +0.252]	0.39 / 0.36	accepted
35	qwen3.5-0.8b BANKING77 sem. s17 b01 SimNPO / confirm.	0.608 0.495	+0.359 [+0.319, +0.396] units: [+0.297, +0.479]	+0.280 [+0.224, +0.336] units: [+0.207, +0.381]	0.59 / 0.57	accepted
36	qwen3.5-0.8b BANKING77 sem. s17 b01 Uniformization / confirm.	0.608 0.495	+0.160 [+0.144, +0.175] units: [+0.112, +0.211]	+0.102 [+0.082, +0.124] units: [+0.079, +0.127]	0.26 / 0.21	infeasible

Appendix

Table S5: Complete method results (continued): oracle fidelity, IDs 31–45.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
46	qwen3.5-0.8b BANKING77 sem. s17 b01 GradDiff / explor.	0.608 0.495	+0.113 [+0.058, +0.167] units: [-0.050, +0.361]	+0.015 [-0.070, +0.099] units: [-0.208, +0.229]	0.19 / 0.03	infeasible
47	qwen3.5-0.8b BANKING77 sem. s17 b01 GradDiff-KL / explor.	0.608 0.495	+0.293 [+0.260, +0.325] units: [+0.201, +0.372]	+0.215 [+0.172, +0.261] units: [+0.172, +0.255]	0.48 / 0.43	accepted
48	qwen3.5-0.8b BANKING77 sem. s17 b01 RMU / explor.	0.608 0.495	+0.026 [+0.002, +0.048] units: [-0.034, +0.135]	-0.009 [-0.051, +0.033] units: [-0.093, +0.087]	0.04 / -0.02	no benefit
49	qwen3.5-0.8b BANKING77 sem. s17 b01 SimNPO / explor.	0.608 0.495	+0.339 [+0.305, +0.372] units: [+0.223, +0.468]	+0.262 [+0.213, +0.312] units: [+0.190, +0.372]	0.56 / 0.53	accepted
50	qwen3.5-0.8b BANKING77 sem. s17 b01 NPO / pilot	0.193 0.148	-0.145 [-0.178, -0.111] units: [-0.277, -0.014]	-0.201 [-0.252, -0.151] units: [-0.384, -0.046]	-0.75 / -1.36	infeasible
51	qwen3.5-0.8b BANKING77 sem. s17 b01 NPO / pilot	0.608 0.495	+0.183 [+0.132, +0.233] units: [+0.028, +0.392]	+0.106 [+0.029, +0.184] units: [-0.075, +0.288]	0.30 / 0.21	infeasible

Appendix

Table S6: Complete method results (continued): oracle fidelity, IDs 46–60.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
61	qwen3.5-0.8b BANKING77 sem. s17 b02 Uniformization / confirmation_sensitivity	0.225 0.223	+0.016 [+0.012, +0.020] units: [+0.008, +0.023]	+0.018 [+0.011, +0.025] units: [+0.006, +0.027]	0.07 / 0.08	accepted
62	qwen3.5-0.8b BANKING77 sem. s17 b02 GradDiff / confirmation_sensitivity	0.225 0.223	+0.082 [+0.061, +0.103] units: [+0.004, +0.185]	+0.055 [+0.029, +0.084] units: [+0.003, +0.122]	0.36 / 0.25	accepted
63	qwen3.5-0.8b BANKING77 sem. s17 b03 GradAscent / replic.	0.465 0.440	-0.341 [-0.382, -0.300] units: [-0.582, +0.017]	-0.375 [-0.443, -0.306] units: [-0.638, -0.022]	-0.73 / -0.85	infeasible
64	qwen3.5-0.8b BANKING77 sem. s17 b03 GradDiff / replic.	0.465 0.440	+0.075 [+0.040, +0.111] units: [-0.146, +0.336]	+0.048 [-0.014, +0.112] units: [-0.213, +0.318]	0.16 / 0.11	infeasible
65	qwen3.5-0.8b BANKING77 sem. s17 b03 NPO / replic.	0.465 0.440	+0.159 [+0.135, +0.186] units: [+0.047, +0.318]	+0.156 [+0.111, +0.203] units: [+0.027, +0.298]	0.34 / 0.36	accepted
66	qwen3.5-0.8b BANKING77 sem. s17 b03 SimNPO / replic.	0.465 0.440	+0.282 [+0.248, +0.317] units: [+0.089, +0.473]	+0.274 [+0.213, +0.338] units: [+0.081, +0.468]	0.61 / 0.62	accepted

Appendix

Table S7: Complete method results (continued): oracle fidelity, IDs 61–75.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
76	qwen3.5-0.8b BANKING77 sem. s42 b01 GradDiff / replic.	0.558 0.470	+0.161 [+0.132, +0.191] units: [+0.062, +0.346]	+0.115 [+0.072, +0.161] units: [+0.001, +0.230]	0.29 / 0.24	infeasible
77	qwen3.5-0.8b BANKING77 sem. s42 b01 NPO / replic.	0.558 0.470	+0.153 [+0.126, +0.181] units: [+0.072, +0.289]	+0.105 [+0.070, +0.142] units: [+0.021, +0.189]	0.27 / 0.22	accepted
78	qwen3.5-0.8b BANKING77 sem. s42 b01 SimNPO / replic.	0.558 0.470	+0.307 [+0.265, +0.347] units: [+0.186, +0.518]	+0.240 [+0.174, +0.304] units: [+0.116, +0.363]	0.55 / 0.51	accepted
79	qwen3.5-0.8b BANKING77 sem. s42 b01 SimNPO / confirmation_sensitivity	0.208 0.186	+0.040 [+0.032, +0.049] units: [+0.004, +0.080]	+0.050 [+0.035, +0.066] units: [+0.004, +0.098]	0.19 / 0.27	accepted
80	qwen3.5-0.8b BANKING77 sem. s42 b01 Uniformization / confirmation_sensitivity	0.208 0.186	+0.032 [+0.027, +0.037] units: [+0.014, +0.049]	+0.026 [+0.018, +0.034] units: [+0.008, +0.044]	0.15 / 0.14	accepted
81	qwen3.5-0.8b BANKING77 sem. s42 b01 GradDiff / confirmation_sensitivity	0.208 0.186	+0.054 [+0.034, +0.074] units: [+0.017, +0.106]	+0.032 [+0.004, +0.061] units: [-0.005, +0.078]	0.26 / 0.17	accepted

Appendix

Table S8: Complete method results (continued): oracle fidelity, IDs 76–90.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
91	qwen3.5-0.8b BANKING77 sem. s42 b03 NPO / replic.	0.435 0.398	-0.042 [-0.071, -0.012] units: [-0.261, +0.176]	-0.087 [-0.140, -0.034] units: [-0.314, +0.140]	-0.10 / -0.22	infeasible
92	qwen3.5-0.8b BANKING77 sem. s42 b03 SimNPO / replic.	0.435 0.398	+0.209 [+0.182, +0.237] units: [+0.057, +0.359]	+0.195 [+0.152, +0.241] units: [+0.078, +0.313]	0.48 / 0.49	accepted
93	qwen3.5-0.8b BANKING77 sem. s42 b03 SimNPO / confirmation_sensitivity	0.093 0.075	+0.002 [-0.001, +0.005] units: [-0.002, +0.007]	-0.002 [-0.004, +0.000] units: [-0.004, -0.000]	0.02 / -0.03	no benefit
94	qwen3.5-0.8b BANKING77 sem. s42 b03 Uniformization / confirmation_sensitivity	0.093 0.075	+0.007 [+0.004, +0.011] units: [+0.002, +0.014]	+0.006 [+0.002, +0.010] units: [-0.000, +0.013]	0.08 / 0.08	accepted
95	qwen3.5-0.8b BANKING77 sem. s42 b03 GradDiff / confirmation_sensitivity	0.093 0.075	+0.004 [-0.003, +0.011] units: [-0.003, +0.013]	-0.003 [-0.009, +0.002] units: [-0.005, -0.001]	0.04 / -0.04	no benefit
96	qwen3.5-0.8b BANKING77 sem. s99 b01 GradAscent / replic.	0.610 0.495	+0.136 [+0.109, +0.166] units: [+0.033, +0.291]	+0.119 [+0.076, +0.165] units: [+0.027, +0.217]	0.22 / 0.24	accepted

Appendix

Table S9: Complete method results (continued): oracle fidelity, IDs 91–105.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
106	qwen3.5-0.8b BANKING77 sem. s99 b02 SimNPO / replic.	0.676 0.648	+0.469 [+0.432, +0.507] units: [+0.150, +0.707]	+0.492 [+0.433, +0.550] units: [+0.169, +0.740]	0.69 / 0.76	accepted
107	qwen3.5-0.8b BANKING77 sem. s99 b02 SimNPO / confirmation_sensitivity	0.197 0.183	+0.016 [+0.010, +0.023] units: [+0.001, +0.030]	+0.019 [+0.010, +0.031] units: [+0.002, +0.032]	0.08 / 0.10	accepted
108	qwen3.5-0.8b BANKING77 sem. s99 b02 Uniformization / confirmation_sensitivity	0.197 0.183	+0.014 [+0.011, +0.017] units: [+0.007, +0.022]	+0.012 [+0.006, +0.018] units: [+0.002, +0.021]	0.07 / 0.06	accepted
109	qwen3.5-0.8b BANKING77 sem. s99 b02 GradDiff / confirmation_sensitivity	0.197 0.183	+0.031 [+0.019, +0.044] units: [+0.000, +0.075]	+0.011 [-0.007, +0.029] units: [+0.002, +0.020]	0.15 / 0.06	positive, unconfirmed
110	qwen3.5-0.8b BANKING77 sem. s99 b03 GradAscent / replic.	0.469 0.454	+0.206 [+0.174, +0.240] units: [+0.004, +0.408]	+0.218 [+0.160, +0.279] units: [-0.006, +0.442]	0.44 / 0.48	infeasible
111	qwen3.5-0.8b BANKING77 sem. s99 b03 GradDiff / replic.	0.469 0.454	+0.188 [+0.158, +0.219] units: [+0.012, +0.363]	+0.194 [+0.141, +0.251] units: [+0.006, +0.383]	0.40 / 0.43	infeasible

Appendix

Table S10: Complete method results (continued): oracle fidelity, IDs 106–120.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
121	qwen3.5-9b BANKING77 sem. s42 b01 SimNPO / explor.	0.511 0.425	+0.094 [+0.057, +0.135] units: [+0.015, +0.212]	+0.093 [+0.038, +0.147] units: [-0.035, +0.225]	0.18 / 0.22	accepted
122	qwen3.5-9b BANKING77 sem. s42 b01 SimNPO / explor.	0.511 0.425	+0.008 [-0.038, +0.057] units: [-0.085, +0.173]	-0.033 [-0.109, +0.043] units: [-0.191, +0.236]	0.02 / -0.08	no benefit
123	qwen3.5-9b BANKING77 sem. s42 b01 SimNPO / explor.	0.511 0.425	+0.014 [-0.030, +0.061] units: [-0.081, +0.149]	-0.027 [-0.094, +0.041] units: [-0.207, +0.190]	0.03 / -0.06	no benefit
124	qwen3.5-9b BANKING77 sem. s42 b01 SimNPO / explor.	0.511 0.425	+0.084 [+0.067, +0.103] units: [+0.043, +0.156]	+0.084 [+0.060, +0.110] units: [+0.037, +0.132]	0.16 / 0.20	accepted
125	qwen3.5-9b BANKING77 sem. s42 b01 SimNPO / explor.	0.511 0.425	+0.068 [+0.023, +0.117] units: [-0.042, +0.232]	+0.022 [-0.056, +0.099] units: [-0.130, +0.277]	0.13 / 0.05	positive, unconfirmed
126	qwen3.5-9b BANKING77 sem. s42 b01 SimNPO / replic.	0.511 0.425	+0.070 [+0.055, +0.087] units: [+0.045, +0.116]	+0.076 [+0.051, +0.103] units: [+0.031, +0.141]	0.14 / 0.18	accepted

Appendix

Table S11: Complete method results (continued): oracle fidelity, IDs 121–135.

ID	Configuration	$D_{0,\mathrm{F}}$ / $D_{0,\mathrm{G}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; unit CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; unit CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Status
136	qwen3.5-0.8b CLINC150 sem. s17 b01 SimNPO / transfer	0.416 0.356	-0.191 [-0.242, -0.138] units: [-0.577, +0.216]	-0.237 [-0.328, -0.148] units: [-0.628, +0.182]	-0.46 / -0.67	no benefit
137	qwen3.5-0.8b CLINC150 sem. s17 b01 Uniformization / transfer	0.058 0.079	+0.003 [+0.002, +0.005] units: [+0.001, +0.007]	+0.014 [+0.007, +0.023] units: [+0.002, +0.029]	0.06 / 0.18	accepted
138	qwen3.5-0.8b CLINC150 sem. s17 b02 SimNPO / explor.	0.298 0.319	-0.117 [-0.152, -0.083] units: [-0.297, +0.080]	-0.111 [-0.178, -0.044] units: [-0.383, +0.186]	-0.39 / -0.35	no benefit
139	qwen3.5-0.8b CLINC150 sem. s17 b02 SimNPO / explor.	0.298 0.319	-0.051 [-0.082, -0.020] units: [-0.219, +0.138]	-0.040 [-0.100, +0.022] units: [-0.290, +0.241]	-0.17 / -0.13	no benefit
140	qwen3.5-0.8b CLINC150 sem. s17 b02 SimNPO / explor.	0.298 0.319	+0.049 [+0.025, +0.074] units: [-0.085, +0.205]	+0.074 [+0.025, +0.124] units: [-0.119, +0.309]	0.17 / 0.23	accepted
141	qwen3.5-0.8b CLINC150 sem. s17 b02 temperature_control / control	0.298 0.319	+0.020 [+0.018, +0.023] units: [+0.009, +0.033]	+0.032 [+0.025, +0.041] units: [+0.015, +0.050]	0.07 / 0.10	accepted

Appendix

Table S12: Complete method results (continued): oracle fidelity, IDs 136–150.

ID	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	$\operatorname{ECE}_{\mathrm{R}}$	$\Delta\operatorname{ECE}_{\mathrm{R}}$	Rows F / G	Units
1	-0.3	0.014	-0.005	428/160	4
2	-0.1	0.021	-0.001	359/160	4
3	-0.4	0.029	+0.001	428/160	4
4	+0.0	0.022	-0.006	428/160	4
5	-0.3	0.034	+0.006	428/160	4
6	-0.6	0.025	-0.003	428/160	4

Appendix

Table S13: Complete method results (continued): retained utility and evaluation counts. IDs 1–30.

ID	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	$\operatorname{ECE}_{\mathrm{R}}$	$\Delta\operatorname{ECE}_{\mathrm{R}}$	Rows F / G	Units
31	-0.1	0.039	+0.013	428/160	4
32	+0.2	0.037	+0.011	428/160	4
33	-2.1	0.061	+0.035	428/160	4
34	+0.2	0.029	+0.003	428/160	4
35	-0.1	0.028	+0.001	428/160	4
36	+0.2	0.072	+0.046	428/160	4

Appendix

Table S14: Complete method results (continued): retained utility and evaluation counts. IDs 31–60.

ID	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	$\operatorname{ECE}_{\mathrm{R}}$	$\Delta\operatorname{ECE}_{\mathrm{R}}$	Rows F / G	Units
61	+0.0	0.015	-0.017	407/146	6
62	+0.3	0.034	+0.002	407/146	6
63	-0.7	0.638	+0.609	494/160	4
64	-0.4	0.144	+0.115	494/160	4
65	-0.3	0.037	+0.008	494/160	4
66	-0.1	0.039	+0.010	494/160	4

Appendix

Table S15: Complete method results (continued): retained utility and evaluation counts. IDs 61–90.

ID	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	$\operatorname{ECE}_{\mathrm{R}}$	$\Delta\operatorname{ECE}_{\mathrm{R}}$	Rows F / G	Units
91	-1.4	0.218	+0.191	494/160	4
92	-0.1	0.036	+0.009	494/160	4
93	+0.2	0.030	+0.006	386/149	8
94	-0.1	0.011	-0.013	386/149	8
95	+0.6	0.032	+0.008	386/149	8
96	+0.1	0.055	+0.020	428/160	4

Appendix

Table S16: Complete method results (continued): retained utility and evaluation counts. IDs 91–120.

ID	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	$\operatorname{ECE}_{\mathrm{R}}$	$\Delta\operatorname{ECE}_{\mathrm{R}}$	Rows F / G	Units
121	-1.2	0.028	+0.001	428/160	4
122	-0.7	0.022	-0.005	428/160	4
123	-0.9	0.022	-0.005	428/160	4
124	-1.0	0.023	-0.004	428/160	4
125	-0.5	0.023	-0.004	428/160	4
126	-0.9	0.021	-0.005	428/160	4

Appendix

Table S17: Complete method results (continued): retained utility and evaluation counts. IDs 121–150.

Model	Seed / bundle; stage	$P_{\mathrm{F}}$ [rows]; class CI	$P_{\mathrm{G}}$ [rows]	$\Delta P_{\mathrm{F}}/\Delta P_{\mathrm{G}}$ vs. seed 17, b01	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	Status
Apertus-8B	17 / b01 screening	0.55 [0.50, 0.60] classes: [0.32, 0.74]	0.58 [0.47, 0.66]	— / —	-0.3	accepted
Apertus-8B	42 / b01 replication	0.17 [0.10, 0.23] classes: [-0.33, 0.61]	0.25 [0.13, 0.35]	-0.38 / -0.33	+0.0	accepted
Apertus-8B	17 / b02 replication	0.56 [0.53, 0.59] classes: [0.47, 0.62]	0.57 [0.52, 0.62]	+0.02 / -0.00	-0.1	accepted
Falcon-H1-3B	17 / b01 screening	0.36 [0.32, 0.41] classes: [0.16, 0.69]	0.42 [0.35, 0.49]	— / —	+0.3	accepted
Gemma-4-E2B	17 / b01 screening	0.41 [0.36, 0.46] classes: [0.25, 0.59]	0.44 [0.35, 0.52]	— / —	+1.0	accepted
Granite-4.0-Micro	17 / b01 screening	0.46 [0.42, 0.51] classes: [0.34, 0.59]	0.55 [0.48, 0.62]	— / —	-0.6	accepted

Appendix

Table S4: Architecture screening and seed / bundle replication of frozen class SimNPO (BANKING77).

Model	Seed / bundle; stage	$P_{\mathrm{F}}$ [rows]; class CI	$P_{\mathrm{G}}$ [rows]	$\Delta P_{\mathrm{F}}/\Delta P_{\mathrm{G}}$ vs. seed 17, b01	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	Status
Qwen2.5-1.5B	17 / b01 screening	0.45 [0.41, 0.50] classes: [0.18, 0.68]	0.54 [0.47, 0.62]	— / —	+0.2	accepted
Qwen2.5-3B	17 / b01 screening	0.50 [0.45, 0.55] classes: [0.25, 0.74]	0.60 [0.52, 0.68]	— / —	+0.1	accepted
Qwen2.5-7B	17 / b01 screening	0.63 [0.58, 0.67] classes: [0.51, 0.79]	0.63 [0.56, 0.70]	— / —	+0.0	accepted
Qwen3-4B	17 / b01 screening	0.28 [0.21, 0.35] classes: [0.10, 0.41]	0.19 [-0.01, 0.34]	— / —	+0.1	positive, unconfirmed
Qwen3-4B	42 / b01 replication	-0.18 [-0.34, -0.03] classes: [-1.90, 0.58]	-0.10 [-0.41, 0.13]	-0.46 / -0.29	+0.1	no benefit
Qwen3-4B	17 / b02 replication	0.51 [0.47, 0.55] classes: [0.35, 0.64]	0.50 [0.44, 0.56]	+0.23 / +0.32	-0.1	accepted

Appendix

Table S5: Architecture screening and seed / bundle replication (continued).

Dataset / bundle / seed	Labels; status	Full accuracy F / G	Oracle accuracy F / G	Oracle forgotten mass F / G	$D_{0,\mathrm{F}}/D_{0,\mathrm{G}}$	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Opaque acquisition
banking77 b01, seed 17	semantic found	0.981 / 0.912	0.2687 / 0.3063	3.043e-01 / 3.262e-01	0.608 / 0.495	0.59 / 0.57	—
banking77 b01, seed 17	opaque found	0.953 / 0.906	0.0000 / 0.0000	1.126e-03 / 9.742e-04	0.943 / 0.881	0.61 / 0.62	pass; min class acc. 0.80 (semantic 0.90), min gold p 0.77
banking77 b01, seed 42	semantic found	0.974 / 0.906	0.3154 / 0.3625	3.419e-01 / 3.789e-01	0.558 / 0.470	0.55 / 0.51	—
banking77 b01, seed 42	opaque found	0.960 / 0.894	0.0000 / 0.0000	4.706e-04 / 5.307e-04	0.934 / 0.878	0.60 / 0.58	pass; min class acc. 0.84 (semantic 0.84), min gold p 0.83
banking77 b02, seed 17	semantic found	0.940 / 0.925	0.2199 / 0.2000	2.209e-01 / 2.108e-01	0.644 / 0.636	0.70 / 0.73	—
banking77 b02, seed 17	opaque found	0.966 / 0.931	0.0000 / 0.0000	6.479e-04 / 3.491e-04	0.912 / 0.872	0.45 / 0.43	pass; min class acc. 0.74 (semantic 0.70), min gold p 0.69

Appendix

Table S5: Semantic versus opaque decision labels: acquisition, oracle recognition, displacement and SimNPO progress.

Dataset / bundle / seed	Labels; status	Full accuracy F / G	Oracle accuracy F / G	Oracle forgotten mass F / G	$D_{0,\mathrm{F}}/D_{0,\mathrm{G}}$	$P_{\mathrm{F}}/P_{\mathrm{G}}$	Opaque acquisition
clinc150 b02, seed 17	semantic found	0.995 / 0.983	0.6612 / 0.6333	6.400e-01 / 6.079e-01	0.298 / 0.319	0.22 / 0.28	—
clinc150 b02, seed 17	opaque found	0.992 / 0.971	0.0013 / 0.0000	3.153e-03 / 2.200e-03	0.983 / 0.976	0.28 / 0.29	pass; min class acc. 0.85 (semantic 0.85), min gold p 0.85
clinc150 b03, seed 17	semantic found	0.996 / 0.983	0.5051 / 0.5333	5.047e-01 / 5.316e-01	0.454 / 0.407	0.22 / 0.20	—
clinc150 b03, seed 17	opaque found	0.994 / 0.971	0.0000 / 0.0000	6.001e-04 / 7.282e-04	0.993 / 0.973	0.13 / 0.11	pass; min class acc. 0.89 (semantic 0.74), min gold p 0.87
clinc150 b06, seed 17	semantic found	0.999 / 0.979	0.7807 / 0.7375	7.705e-01 / 7.241e-01	0.207 / 0.233	-0.24 / -0.09	—
clinc150 b06, seed 17	opaque found	0.996 / 0.983	0.0000 / 0.0000	1.247e-03 / 1.292e-03	0.999 / 0.989	0.02 / 0.02	pass; min class acc. 0.94 (semantic 0.89), min gold p 0.93

Appendix

Table S6: Semantic versus opaque decision labels (continued): CLINC150.

ID	Configuration	Schema / view	$D_{0,v}$	$P_{v}$ [rows]	Class CI	Oracle accuracy
1	apertus-8b BANKING77 sem. _01 SimNPO	S0_original 77 options; F	0.479	0.55 [0.50, 0.60]	[0.31, 0.72]	0.395
2	apertus-8b BANKING77 sem. _01 SimNPO	S0_original 77 options; G	0.400	0.58 [0.48, 0.66]	[0.15, 0.77]	0.450
3	apertus-8b BANKING77 sem. _01 SimNPO	S1_neighbors 12 options; F	0.204	0.40 [0.32, 0.47]	[0.22, 0.61]	0.682
4	apertus-8b BANKING77 sem. _01 SimNPO	S1_neighbors 12 options; G	0.132	0.36 [0.14, 0.52]	[0.09, 0.57]	0.775
5	apertus-8b BANKING77 sem. _01 SimNPO	S2_distractors 12 options; F	0.122	0.55 [0.48, 0.61]	[-0.12, 0.68]	0.827
6	apertus-8b BANKING77 sem. _01 SimNPO	S2_distractors 12 options; G	0.150	0.68 [0.60, 0.75]	[-0.45, 0.77]	0.775

Appendix

Table S6: Candidate-schema results with paired uncertainty.

ID	Configuration	Schema / view	$D_{0,v}$	$P_{v}$ [rows]	Class CI	Oracle accuracy
15	qwen3.5-0.8b BANKING77 sem. _01 SimNPO	S1_neighbors 12 options; F	0.438	0.66 [0.61, 0.70]	[0.50, 0.74]	0.460
16	qwen3.5-0.8b BANKING77 sem. _01 NPO	S1_neighbors 12 options; F	0.438	0.41 [0.36, 0.47]	[0.14, 0.65]	0.460
17	qwen3.5-0.8b BANKING77 sem. _01 GradAscent	S1_neighbors 12 options; F	0.438	0.62 [0.57, 0.67]	[0.50, 0.67]	0.460
18	qwen3.5-0.8b BANKING77 sem. _01 GradDiff	S1_neighbors 12 options; F	0.438	0.47 [0.41, 0.52]	[0.24, 0.66]	0.460
19	qwen3.5-0.8b BANKING77 sem. _01 SimNPO	S1_neighbors 12 options; G	0.339	0.61 [0.51, 0.70]	[0.41, 0.71]	0.519
20	qwen3.5-0.8b BANKING77 sem. _01 NPO	S1_neighbors 12 options; G	0.339	0.38 [0.28, 0.48]	[0.07, 0.60]	0.519

Appendix

Table S7: Candidate-schema results (continued): BANKING77: alternative candidates.

ID	Configuration	Schema / view	$D_{0,v}$	$P_{v}$ [rows]	Class CI	Oracle accuracy
31	qwen3.5-0.8b BANKING77 opaque _01 SimNPO	S0_original 77 options; F	0.943	0.61 [0.57, 0.64]	[0.44, 0.76]	0.000
32	qwen3.5-0.8b BANKING77 opaque _01 SimNPO	S0_original 77 options; G	0.881	0.62 [0.57, 0.68]	[0.44, 0.75]	0.000
33	qwen3.5-0.8b BANKING77 opaque _01 SimNPO	S1_neighbors 12 options; F	0.464	0.00 [-0.03, 0.03]	[-0.15, 0.15]	0.016
34	qwen3.5-0.8b BANKING77 opaque _01 SimNPO	S1_neighbors 12 options; G	0.489	0.01 [-0.03, 0.04]	[-0.09, 0.09]	0.050
35	qwen3.5-0.8b BANKING77 opaque _01 SimNPO	S2_distractors 12 options; F	0.583	0.12 [0.07, 0.16]	[-0.08, 0.46]	0.000
36	qwen3.5-0.8b BANKING77 opaque _01 SimNPO	S2_distractors 12 options; G	0.553	0.14 [0.07, 0.20]	[-0.11, 0.48]	0.000

Appendix

Table S8: Candidate-schema results (continued): BANKING77: opaque labels and 9B.

ID	Configuration	Schema / view	$D_{0,v}$	$P_{v}$ [rows]	Class CI	Oracle accuracy
49	qwen3.5-0.8b CLINC150 sem. _02 SimNPO	S0_original 151 options; F	0.298	0.22 [0.16, 0.27]	[-0.15, 0.43]	0.661
50	qwen3.5-0.8b CLINC150 sem. _02 SimNPO	S0_original 151 options; F	0.298	-0.06 [-0.17, 0.04]	[-0.91, 0.35]	0.661
51	qwen3.5-0.8b CLINC150 sem. _02 SimNPO	S0_original 151 options; G	0.319	0.28 [0.17, 0.37]	[-0.29, 0.51]	0.633
52	qwen3.5-0.8b CLINC150 sem. _02 SimNPO	S0_original 151 options; G	0.319	0.03 [-0.17, 0.19]	[-1.44, 0.48]	0.633
53	qwen3.5-0.8b CLINC150 sem. _02 SimNPO	S1_neighbors 24 options; F	0.151	0.20 [0.10, 0.28]	[-0.54, 0.45]	0.827
54	qwen3.5-0.8b CLINC150 sem. _02 SimNPO	S1_neighbors 24 options; F	0.151	-0.38 [-0.62, -0.19]	[-2.09, 0.28]	0.827

Appendix

Table S9: Candidate-schema results (continued): CLINC150.

ID	Configuration	$D_{0,\mathrm{F}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; intent CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; intent CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	Status
1	llama3.2-1b seed 17, b01 transfer__simnpo transfer	0.453	-0.079 [-0.125, -0.034] intents: [-0.424, +0.281]	-0.082 [-0.159, -0.006] intents: [-0.403, +0.241]	-0.18 / -0.20	-0.2	no benefit
2	qwen3.5-0.8b seed 17, b01 transfer__simnpo transfer	0.058	+0.002 [+0.000, +0.005] intents: [+0.000, +0.005]	+0.006 [+0.001, +0.011] intents: [-0.002, +0.014]	0.04 / 0.07	+0.0	accepted
3	qwen3.5-0.8b seed 17, b01 transfer__uniformization transfer	0.058	+0.003 [+0.002, +0.005] intents: [+0.001, +0.007]	+0.014 [+0.007, +0.023] intents: [+0.002, +0.029]	0.06 / 0.18	-0.1	accepted
4	qwen3.5-0.8b seed 17, b01 transfer__grad_ascent transfer	0.416	-0.185 [-0.217, -0.152] intents: [-0.404, -0.001]	-0.168 [-0.228, -0.109] intents: [-0.387, +0.021]	-0.44 / -0.47	-1.0	no benefit
5	qwen3.5-0.8b seed 17, b01 transfer__grad_diff transfer	0.416	-0.102 [-0.130, -0.075] intents: [-0.261, -0.005]	-0.073 [-0.121, -0.025] intents: [-0.239, +0.027]	-0.25 / -0.20	-1.1	no benefit
6	qwen3.5-0.8b seed 17, b01 transfer__npo transfer	0.416	-0.183 [-0.214, -0.151] intents: [-0.395, -0.011]	-0.158 [-0.216, -0.099] intents: [-0.369, +0.022]	-0.44 / -0.44	-1.1	no benefit

Appendix

Table S7: CLINC150: every observed bundle, frozen transfer, tuned confirmation and replication.

ID	Configuration	$D_{0,\mathrm{F}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; intent CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; intent CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	Status
8	qwen3.5-0.8b seed 17, b02 matched_s1__simnpo_b2.5_ep7 exploratory	0.298	-0.117 [-0.152, -0.083] intents: [-0.297, +0.080]	-0.111 [-0.178, -0.044] intents: [-0.383, +0.186]	-0.39 / -0.35	-0.9	no benefit
9	qwen3.5-0.8b seed 17, b02 matched_s1__simnpo_b4.5_ep9 exploratory	0.298	-0.051 [-0.082, -0.020] intents: [-0.219, +0.138]	-0.040 [-0.100, +0.022] intents: [-0.290, +0.241]	-0.17 / -0.13	-0.2	no benefit
10	qwen3.5-0.8b seed 17, b02 matched_s1__simnpo_b6.5_ep10 exploratory	0.298	+0.049 [+0.025, +0.074] intents: [-0.085, +0.205]	+0.074 [+0.025, +0.124] intents: [-0.119, +0.309]	0.17 / 0.23	+0.3	accepted
11	qwen3.5-0.8b seed 17, b02 temperature_control control	0.298	+0.020 [+0.018, +0.023] intents: [+0.009, +0.033]	+0.032 [+0.025, +0.041] intents: [+0.015, +0.050]	0.07 / 0.10	+0.0	accepted
12	qwen3.5-0.8b seed 17, b02 transfer__simnpo control	0.298	-0.018 [-0.046, +0.011] intents: [-0.171, +0.157]	+0.009 [-0.048, +0.067] intents: [-0.218, +0.270]	-0.06 / 0.03	+0.3	no benefit
13	qwen3.5-0.8b seed 17, b02 tuned__simnpo confirmation	0.298	+0.065 [+0.046, +0.085] intents: [-0.032, +0.181]	+0.088 [+0.049, +0.129] intents: [-0.058, +0.271]	0.22 / 0.28	+0.4	accepted

Appendix

Table S8: CLINC150 results (continued): bundle-02 tuning and controls.

ID	Configuration	$D_{0,\mathrm{F}}$	$\Delta\operatorname{OD}_{\mathrm{F}}$ [rows]; intent CI	$\Delta\operatorname{OD}_{\mathrm{G}}$ [rows]; intent CI	$P_{\mathrm{F}}/P_{\mathrm{G}}$	$\Delta\operatorname{Acc}_{\mathrm{R}}$ (pp)	Status
14	qwen3.5-0.8b seed 17, b03 temperature_control control	0.454	+0.024 [+0.021, +0.027] intents: [+0.007, +0.046]	+0.030 [+0.023, +0.037] intents: [+0.013, +0.055]	0.05 / 0.07	+0.0	accepted
15	qwen3.5-0.8b seed 17, b03 tuned__simnpo confirmation	0.454	+0.102 [+0.082, +0.121] intents: [+0.001, +0.289]	+0.083 [+0.052, +0.115] intents: [-0.006, +0.249]	0.22 / 0.20	+0.1	accepted
16	qwen3.5-0.8b seed 17, b04 tuned__simnpo replication	0.481	+0.061 [+0.034, +0.089] intents: [-0.192, +0.317]	+0.033 [-0.012, +0.080] intents: [-0.200, +0.274]	0.13 / 0.08	+0.3	positive, unconfirmed
17	qwen3.5-0.8b seed 17, b05 tuned__simnpo replication	0.344	+0.025 [-0.003, +0.052] intents: [-0.122, +0.220]	+0.022 [-0.032, +0.078] intents: [-0.182, +0.276]	0.07 / 0.07	+0.1	positive, unconfirmed
18	qwen3.5-0.8b seed 17, b06 tuned__simnpo replication	0.207	-0.050 [-0.070, -0.030] intents: [-0.234, +0.074]	-0.020 [-0.057, +0.016] intents: [-0.176, +0.093]	-0.24 / -0.09	+0.1	no benefit
19	qwen3.5-0.8b seed 17, b07 tuned__simnpo replication	0.289	-0.071 [-0.087, -0.054] intents: [-0.148, +0.001]	-0.051 [-0.080, -0.021] intents: [-0.137, +0.011]	-0.24 / -0.17	+0.3	no benefit

Appendix

Table S9: CLINC150 results (continued): confirmation and replication.

ID	Dataset / run	View	Dev. $s$	Held-out $s$ [rows]; class CI	Aligned $a$ [rows]	Relative residual [rows]	$P_{\mathrm{F}}$ ; gate	Identity error; zero directions
1	banking77 matched__simnpo	FG	0.64	0.78 [0.74, 0.83] classes: [0.48, 1.07]	0.67 [0.63, 0.71]	0.36 [0.32, 0.39]	0.56 feasible: True	1.1e-15 zero rows: 0
2	banking77 matched__simnpo	F	0.64	0.79 [0.74, 0.84] classes: [0.48, 1.07]	0.68 [0.63, 0.72]	0.36 [0.32, 0.39]	0.56 feasible: True	1.1e-15 zero rows: 0
3	banking77 matched__simnpo	G	0.64	0.77 [0.68, 0.86] classes: [0.45, 1.09]	0.65 [0.57, 0.73]	0.35 [0.29, 0.41]	0.56 feasible: True	1.1e-15 zero rows: 0
4	banking77 confirm__npo	FG	0.59	0.40 [0.36, 0.44] classes: [0.21, 0.60]	0.39 [0.35, 0.44]	0.15 [0.12, 0.18]	0.39 feasible: True	1.3e-15 zero rows: 0
5	banking77 confirm__npo	F	0.59	0.42 [0.37, 0.47] classes: [0.22, 0.64]	0.41 [0.36, 0.46]	0.16 [0.12, 0.19]	0.39 feasible: True	8.9e-16 zero rows: 0
6	banking77 confirm__npo	G	0.59	0.36 [0.28, 0.44] classes: [0.17, 0.57]	0.36 [0.27, 0.44]	0.12 [0.05, 0.19]	0.39 feasible: True	1.3e-15 zero rows: 0

Appendix

Table S8: Matched suppression (development vs held-out) and oracle-direction geometry.

ID	Dataset / run	View	Dev. $s$	Held-out $s$ [rows]; class CI	Aligned $a$ [rows]	Relative residual [rows]	$P_{\mathrm{F}}$ ; gate	Identity error; zero directions
10	banking77 matched__grad_diff	FG	0.65	1.11 [1.04, 1.19] classes: [0.60, 1.60]	0.74 [0.70, 0.78]	0.50 [0.47, 0.54]	0.19 feasible: False	2.0e-15 zero rows: 0
11	banking77 matched__grad_diff	F	0.65	1.10 [1.02, 1.18] classes: [0.61, 1.55]	0.74 [0.70, 0.78]	0.50 [0.46, 0.54]	0.19 feasible: False	2.0e-15 zero rows: 0
12	banking77 matched__grad_diff	G	0.65	1.16 [1.01, 1.33] classes: [0.59, 1.81]	0.75 [0.66, 0.83]	0.51 [0.43, 0.59]	0.19 feasible: False	1.3e-15 zero rows: 0
13	banking77 matched__grad_diff_kl	FG	0.63	0.53 [0.49, 0.57] classes: [0.36, 0.72]	0.49 [0.45, 0.53]	0.20 [0.17, 0.23]	0.48 feasible: True	8.9e-16 zero rows: 0
14	banking77 matched__grad_diff_kl	F	0.63	0.55 [0.50, 0.60] classes: [0.37, 0.73]	0.51 [0.46, 0.56]	0.21 [0.18, 0.24]	0.48 feasible: True	8.9e-16 zero rows: 0
15	banking77 matched__grad_diff_kl	G	0.63	0.46 [0.38, 0.54] classes: [0.32, 0.71]	0.43 [0.35, 0.51]	0.16 [0.10, 0.22]	0.48 feasible: True	8.9e-16 zero rows: 0

Appendix

Table S9: Matched suppression and oracle-direction geometry (continued).

ID	Dataset / run	View	Dev. $s$	Held-out $s$ [rows]; class CI	Aligned $a$ [rows]	Relative residual [rows]	$P_{\mathrm{F}}$ ; gate	Identity error; zero directions
19	clinc150 matched_s1__simnpo_b2.5_ep7	FG	1.01	1.93 [1.80, 2.06] classes: [1.32, 3.30]	0.80 [0.77, 0.83]	0.67 [0.63, 0.71]	-0.39 feasible: True	1.6e-15 zero rows: 0
20	clinc150 matched_s1__simnpo_b2.5_ep7	F	1.01	1.94 [1.80, 2.10] classes: [1.38, 3.13]	0.81 [0.77, 0.85]	0.67 [0.63, 0.72]	-0.39 feasible: True	1.3e-15 zero rows: 0
21	clinc150 matched_s1__simnpo_b2.5_ep7	G	1.01	1.87 [1.63, 2.18] classes: [1.21, 3.74]	0.78 [0.71, 0.84]	0.64 [0.57, 0.73]	-0.39 feasible: True	1.6e-15 zero rows: 0
22	clinc150 matched_s1__simnpo_b4.5_ep9	FG	1.00	1.61 [1.51, 1.73] classes: [0.99, 2.89]	0.73 [0.69, 0.77]	0.54 [0.51, 0.58]	-0.17 feasible: True	1.6e-15 zero rows: 0
23	clinc150 matched_s1__simnpo_b4.5_ep9	F	1.00	1.63 [1.50, 1.77] classes: [1.03, 2.78]	0.74 [0.69, 0.79]	0.55 [0.51, 0.60]	-0.17 feasible: True	1.6e-15 zero rows: 0
24	clinc150 matched_s1__simnpo_b4.5_ep9	G	1.00	1.55 [1.34, 1.81] classes: [0.91, 3.16]	0.71 [0.62, 0.81]	0.51 [0.44, 0.59]	-0.17 feasible: True	1.1e-15 zero rows: 0

Appendix

Table S10: Matched suppression and oracle-direction geometry (continued).

ID	Dataset / run	View	Dev. $s$	Held-out $s$ [rows]; class CI	Aligned $a$ [rows]	Relative residual [rows]	$P_{\mathrm{F}}$ ; gate	Identity error; zero directions
25	clinc150 matched_s1__simnpo_b6.5_ep10	FG	0.94	1.18 [1.11, 1.27] classes: [0.74, 2.00]	0.61 [0.57, 0.64]	0.37 [0.34, 0.40]	0.17 feasible: True	8.9e-16 zero rows: 0
26	clinc150 matched_s1__simnpo_b6.5_ep10	F	0.94	1.18 [1.09, 1.28] classes: [0.76, 1.91]	0.60 [0.55, 0.64]	0.38 [0.35, 0.41]	0.17 feasible: True	8.9e-16 zero rows: 0
27	clinc150 matched_s1__simnpo_b6.5_ep10	G	0.94	1.19 [1.04, 1.37] classes: [0.74, 2.24]	0.64 [0.55, 0.72]	0.36 [0.29, 0.43]	0.17 feasible: True	6.7e-16 zero rows: 0
28	clinc150 tuned__simnpo	FG	—	0.81 [0.75, 0.88] classes: [0.46, 1.46]	0.42 [0.39, 0.46]	0.26 [0.24, 0.29]	0.22 feasible: True	1.1e-15 zero rows: 0
29	clinc150 tuned__simnpo	F	—	0.80 [0.73, 0.87] classes: [0.46, 1.39]	0.41 [0.37, 0.44]	0.26 [0.23, 0.29]	0.22 feasible: True	1.1e-15 zero rows: 0
30	clinc150 tuned__simnpo	G	—	0.86 [0.73, 1.00] classes: [0.51, 1.66]	0.47 [0.39, 0.55]	0.27 [0.21, 0.33]	0.22 feasible: True	8.9e-16 zero rows: 0

Appendix

Table S11: Matched suppression and oracle-direction geometry (continued).

ID	Model / configuration	$p(0)$ : U / O	$G_{\rm abs}$ [rows]	$G_{\rm adj}$ [rows]; class CI	$G_{H}$ [rows]
1	qwen3.5-0.8b k1_s0	0.364 / 0.313	+0.139 [+0.092, +0.187]	+0.087 [+0.043, +0.131] classes: [+0.039, +0.139]	+0.178 [+0.119, +0.237]
2	qwen3.5-0.8b k1_s1	0.364 / 0.313	+0.073 [+0.039, +0.107]	+0.022 [-0.029, +0.073] classes: [-0.049, +0.088]	+0.081 [+0.031, +0.130]
3	qwen3.5-0.8b k5_s0	0.364 / 0.313	+0.080 [+0.059, +0.104]	+0.029 [-0.016, +0.073] classes: [-0.087, +0.120]	+0.093 [+0.062, +0.127]
4	qwen3.5-0.8b k5_s1	0.364 / 0.313	+0.100 [+0.077, +0.126]	+0.049 [+0.001, +0.099] classes: [-0.074, +0.136]	+0.128 [+0.095, +0.164]
5	qwen3.5-0.8b k20_s0	0.364 / 0.313	+0.079 [+0.057, +0.103]	+0.028 [-0.017, +0.075] classes: [-0.090, +0.122]	+0.097 [+0.064, +0.132]
6	qwen3.5-0.8b k20_s1	0.364 / 0.313	+0.095 [+0.073, +0.116]	+0.043 [+0.002, +0.087] classes: [-0.045, +0.129]	+0.117 [+0.086, +0.148]

Appendix

Table S9: Recovery contrasts for every exposure budget and adaptation seed.

ID	$A_{\rm abs}$ : U / O	$A_{\rm adj}$ : U / O	First $t$ : $p\geq 0.5$ , U / O	First $t$ : $p\geq 0.8$ , U / O	Identity residual
1	0.801 / 0.662	0.437 / 0.349	1 / 5	5 / not reached by t = 25 (censored)	6e-17
2	0.784 / 0.711	0.420 / 0.398	5 / 5	10 / 25	6e-17
3	0.798 / 0.717	0.433 / 0.404	5 / 10	10 / 10	6e-17
4	0.848 / 0.748	0.484 / 0.435	5 / 5	10 / 10	1e-16
5	0.847 / 0.768	0.483 / 0.455	5 / 5	10 / 10	1e-16
6	0.820 / 0.725	0.456 / 0.412	5 / 10	10 / 10	8e-17

Appendix

Table S10: Recovery summaries (continued): component areas, threshold crossings, and algebraic residual.

Model / seed	$P_{\mathrm{F}}/s_{\mathrm{F}}$	Full gold p / confidence / margin	Oracle accuracy	Forget loss: first / last 10	Retain loss: first / last 10	Gradient norm: first / last 10	Source
qwen3.5-0.8b seed 17	+0.59 / 0.96	0.967 / 0.977 / 0.959	0.27	0.291 / 0.082	0.057 / 0.146	0.822 / 1.314	recorded training log
qwen3.5-0.8b seed 42	+0.55 / 0.86	0.955 / 0.971 / 0.947	0.32	0.277 / 0.103	0.170 / 0.180	0.952 / 1.483	recorded training log
apertus-8b seed 17	+0.55 / 1.07	0.934 / 0.956 / 0.920	0.39	0.268 / 0.053	0.131 / 0.214	0.448 / 0.678	recorded training log
apertus-8b seed 42	+0.17 / 0.39	0.943 / 0.970 / 0.947	0.41	0.266 / 0.116	0.172 / 0.264	0.564 / 0.622	recorded training log
qwen3.5-9b seed 17	+0.54 / 0.57	0.962 / 0.976 / 0.958	0.48	0.290 / 0.178	0.073 / 0.109	0.504 / 1.096	recorded training log
qwen3.5-9b seed 42	+0.14 / 0.15	0.969 / 0.988 / 0.979	0.42	0.285 / 0.251	0.150 / 0.141	0.669 / 1.038	recorded training log

Appendix

Table S10: Seed-17 and seed-42 optimisation diagnostics (frozen class SimNPO, BANKING77 bundle 01).

Experiment	Stage	Evaluations	GPU hours (logged)	Models
architecture_replication	replication	8	40.7	4
architecture_screen	screening	19	97.8	19
association_methods	confirmation_sensitivity	29	23.7	1
clinc150_frozen_transfer	transfer	7	9.6	2
clinc150_tuned	confirmation	2	5.9	1
clinc150_tuned	control	3	4.5	1

Appendix

Table S11: Compute by experiment and stage, and frozen configurations.

Item	Upstream audit record
Acquisition	Opaque full models: all seven checks pass (gate 0.70 / 0.50). Semantic seed-42 full model: forgotten class transfer_timing (BANKING77 bundle 02) at 0.65 development accuracy, below the 0.70 class threshold; targets were gated at the primary seed only.
Privacy audit	S2-Rank stopped at its preregistered acquisition gate at seed 17 (G1 0.550, G4 0.555); the inverse-rank attack was not tested.
Missing / completed coverage	Uniformization at matched suppression (no feasible development checkpoint). CLINC150 bundle 06 under opaque labels was added on 2026-10-18 after the figure brief named it as a contrast bundle (acquisition passed); coverage of the planned designs is complete.
Reserved	CLINC150 class_05_bundle_08 held back unseen; no file of it is read by the build (the collector stops if one appears).
Excluded model	checkpoint reload not reproducible (results/model_checks/internlm3-8b_reload_diagnosis.json; docs/HANDOVER.md); jobs blocked by design
Reconciliation	Row-level OD recomputed for 358 evaluations: max —recomputed - record.json— = 0.0e+00. Paired bootstrap: point estimates exact over 252 comparisons; intervals exact where n_boot matches (5000); stored 2000-draw summaries differ by at most 0.0075 (Monte Carlo); no category changes.

Appendix

Table S12: Audit record: gates, missing or reserved results, reconciliation and repairs.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 356 papers

Jul 21, 2026

Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certification

Machine unlearning is commonly evaluated by matching a retrained oracle on trained probes. In a controlled nonce-fact testbed with a matched retraining reference, we find this criterion can favor methods that retain held-out knowledge: candidates it rates adequate score held-out forget facts

-2.82

nats below the never-learned level (cluster CI

[-3.16,-2.48]

). We recast unlearning as restoration to the matched reference and audit oracle-free screens and certificate-style criteria across 45 model-seed cells spanning five open architecture families. The reference itself falsifies an absolute retain/round-trip certificate: the injected model, which retains the retain set by construction, fails the fixed retain threshold in 41/45 cells and its own round trip in 31/45, and the reference fully certifies in only 1/45. A base-anchored held-out screen remains strong as a selective necessary test: on a sealed challenge suite it rejects the injected model in 45/45 cells, accepts the reference in 44/45, and partially detects entity-routing suppression (35/45); it is a necessary test with measured sensitivity, not a sufficiency certificate. A damage-relative recalibration anchored to the reference's own operating point certifies a small subset in 15/45 cells; where it does not abstain, its picks lie within retraining noise (0.80 nats) on the axes it optimizes, while the common trained-probe criterion sits 5.17 nats away (a supporting comparison, not a head-to-head benchmark). A fixed-magnitude logit-suppression attack defeats the full forward battery in 12/45 cells, so forward-only certification is not sound; our method is an empirical selective test for methods-as-produced. An identifiability theorem delimits which facts admit an oracle-free forget threshold at all, with TOFU as the predicted boundary case.

Sen Yang, Yuen-Hei Yeung

Stern School of Business New York University · Courant Institute of Mathematical Sciences New York University

Machine Unlearning Machine Unlearning Evaluation

Ask PDF

May 26, 2026

RULER: Representation-Level Verification of Machine Unlearning

Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Current protocols verify this at the output level through membership inference, retain accuracy, and forget-set accuracy, but a model can satisfy all three whilst still encoding forgotten records in its intermediate representations. We introduce RULER, a set of representation-level verification metrics. The oracle-comparative metric M2 measures whether forget-set records occupy the same representational position as in a model retrained without them. The oracle-free metric M4 detects residuals from the unlearned model's internal similarity structure alone, without retraining. Four approximate unlearning methods all pass output-level evaluation, yet under a linear mixed-effects model M2 detects significant residuals in 10 of 12 conditions (p<0.05), with effect sizes growing as the forget fraction increases. A fifth method, Bad Teacher, shows the same residuals despite a different forgetting mechanism. M4 acts as a pre-unlearning diagnostic across tabular, image, clinical text, and face-identity settings: it detects identity-level memorisation in face recognition models where no tested method fully erases the signal.

Georgina Cosma, Axel Finke

Department of Computer Science, Loughborough University, UK · School of Mathematics, Statistics and Physics, Newcastle University, UK

Machine Unlearning Machine Unlearning Evaluation

Ask PDF

Oct 7, 2026

Why Forget-Only Unlearning Needs Memorization

Machine unlearning asks for a deletion algorithm whose output is close to retraining from scratch without the selected forget examples. In this work, we study forget-only unlearning, where the deletion algorithm receives only the trained model and the examples to forget, with no retained data or extra training information. We ask whether forget-only unlearning is always possible. We first show that this depends on the learning method: different datasets can produce the same trained model but require very different outputs after the same examples are removed. Using this observation, we derive lower bounds on how accurately unlearning can match retraining and instantiate them for several standard learning algorithms. We then ask what must be true when forget-only unlearning succeeds. To this end, we derive lower bounds on what an algorithm must memorize about the training data to handle arbitrary deletion requests. For simple threshold learners, the required information can be as large as the entire dataset, even though ordinary training keeps only one boundary point. Overall, our results show that information discarded during ordinary learning may be needed later for deletion, so models designed for forget-only unlearning may need to retain more information than standard training does.

Luka Radić, Vikrant Singhal, Amartya Sanyal

Department of Computer Science, University of Copenhagen

Machine Unlearning

Ask PDF

Abstract

Machine Unlearning

Explore similar work

Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certification

RULER: Representation-Level Verification of Machine Unlearning

Why Forget-Only Unlearning Needs Memorization