From Spectral Saliency to Finite Deletion: Evaluating Selective Language Model Edits

Ibne Farabi Shihab; Joyanta Jyoti Mondal

cs.LGOct 13, 2026

From Spectral Saliency to Finite Deletion: Evaluating Selective Language Model Edits

Authors: Ibne Farabi Shihab, Joyanta Jyoti Mondal

Organizations: Department of Computer and Information Sciences, University of Delaware, USA

Abstract

Deleting singular components offers a precise way to remove a capability from a language model, yet the first-order spectral saliency used to rank components predicts only a local loss change, whereas deletion is a finite intervention. In this paper, we study how prediction error affects selective editing, where selective editing requires degradation of a target task while preserving others. We decompose the error into derivative transfer across item splits and nonlinear response along the deletion path. We establish that outer interval screening retains every feasible candidate when every candidate's intervals contain its finite validation loss changes on every task. We evaluate these effects using one answer-only negative log-likelihood objective for search, validation, and test, with disjoint calibration and evaluation masks. The primary configuration targets arithmetic in Mistral-7B-Instruct-v0.3 while preserving code and question answering; extensions cover Llama-3.1-8B-Instruct and reversed target roles. In the primary study, which deletes 1024 coordinates from a support of 7168, calibrated intervals cover 968 of 1024 evaluation masks. Outer screening reaches the direct-selected edit after 25 verification checks, whereas direct verification abstains at a 32-check cap, but it rejects 4 of 74 feasible masks. All three selected edits pass validation, yet only one meets every test constraint. Interval coverage, feasible-candidate retention, and independent test preservation are therefore distinct properties and require separate assessment.

Figures & tables

Figure 1: Finite spectral deletion and interval screening. Stage I forms search predictions in a frozen singular basis. Stage II measures finite validation error and separates derivative transfer from nonlinear response along the deletion path. Stage III calibrates energy-normalized radii on masks disjoint from the evaluation pool. Stage IV screens candidates and verifies survivors in the fixed order under a check cap. The development pilot precedes the final freeze; Proposition 1 states the retention condition.

Figure 2: Absolute NLL prediction error by deletion-energy stratum. Circles, squares, and triangles denote GSM8K, HumanEval, and ARC-C. Solid lines with filled markers show medians; dashed lines with hollow markers show 0.90 quantiles. Points use interval midpoints rather than median mask energies (Table 7 ). Connecting lines guide comparison and are not fitted curves; Table 9 reports the separate four-point aggregate fit.

Figure 3: Interval coverage and screening retention. (a) Joint coverage on 256 calibration and 1024 evaluation masks; dotted and vertical lines mark 95% coverage and the selected

\kappa=1.25

. (b) Inner and outer evaluation survivors. (c) Feasible masks rejected by outer screening, out of 74. Panels (a) to (c) show the primary study. (d) Retained (green) and rejected (red, hatched) feasible masks at each configuration’s selected multiplier. In-bar labels give retained/feasible counts; labels to the right give rejected counts. Labels identify checkpoint, target (A: arithmetic; C: code), and coordinate budget

B

Figure 4: Search predictions, finite validation changes, and test NLL changes for selected edits A, B, and C. Circles, squares, and triangles denote search, validation, and test. Hatched regions violate the 0.12 GSM8K target floor or the 0.08 HumanEval and ARC-C preservation tolerance. All edits pass validation; on test, B misses the target floor and C exceeds the code tolerance. Rows group each edit’s search, validation, and test values; numeric labels give NLL changes to the reported three-decimal precision. All losses use the same answer-only objective on their respective splits.

Figure 5: Test correctness drops for primary selected edits A, B, and C. Positive values denote deterioration; negative values denote improvement. Bars use distinct hatches and report percentage-point drops. Denominators are 1319, 68, 1172, and 378 for GSM8K, HumanEval, ARC-C, and MBPP+, respectively. Figure 4 gives the corresponding calibrated-task NLL changes.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
Model and spectral intervention
$f_{\theta},\theta$	Language model and unedited parameters.
$\mathcal{M},W_{m}$	Editable matrices and matrix $m$ , of shape $d_{m}^{\rm out}\times d_{m}^{\rm in}$ .
$\sigma_{m,j},u_{m,j},v_{m,j}$	Singular value and left/right singular vectors of $W_{m}$ ; $r_{m}=\min(d_{m}^{\rm out},d_{m}^{\rm in})$ is the thin-SVD size, which can exceed numerical rank.
$\mathcal{J}_{m},\mathcal{I},N$	Searchable indices in matrix $m$ , pooled coordinate set, and size $N=\|\mathcal{I}\|$ .
$i,A_{i}$	Coordinate $i=(m,j)$ and atom $A_{i}=\sigma_{m,j}u_{m,j}v_{m,j}^{\top}$ in block $m$ .

Appendix

Table 1: Notation used in the main text and appendix.

Suffix within model.layers. $\ell$	Weight shape (out, in)	Role
self_attn.q_proj , self_attn.o_proj	$4096\times 4096$	Attention
self_attn.k_proj , self_attn.v_proj	$1024\times 4096$	Attention
mlp.gate_proj , mlp.up_proj	$14336\times 4096$	Feed-forward
mlp.down_proj	$4096\times 14336$	Feed-forward

Appendix

Table 2: Mistral-7B-Instruct-v0.3 projection inventory from official configuration, parameter names, and implementation ( Mistral AI, 2024b ; Mistral AI, 2024c ; Hugging Face, 2026 ) . Layers are zero-indexed from 0 to 31. Only the declared interior-layer down projections are editable. Shapes describe architecture metadata, not finite-edit measurements.

Task	Search	Validation	Test
GSM8K	512	512	1319
ARC-C	256	128	1172
HumanEval	64	32	68
MBPP+	0	0	378

Appendix

Table 3: Fixed item allocation for search, validation, and test. Example IDs and dataset revisions are frozen before search-gradient computation. MBPP+ provides an untouched external test readout.

Spectrum field	Value
$E_{\rm full}$	486000
$E_{\rm band}$	117600
Band energy fraction	0.241975
Matrix $\sigma_{1}$ range	8.3–25.8
Matrix $\sigma_{256}$ range	2.6–4.5
Gap minimum/median/maximum	0.0002/0.0011/0.0038

Appendix

Table 4: Spectral diagnostics (left) and uniform-mask deletion energy (right) for the primary support. Energy fraction, leading-band tail error, uniform expectation, and theoretical standard deviation are calculated from the reported inputs. Reconstruction residuals use the full thin SVD. The orthogonality residual is the larger of

\lVert U^{\top}U-I\rVert_{F}/\sqrt{r_{m}}

and

\lVert V^{\top}V-I\rVert_{F}/\sqrt{r_{m}}

. Uniform-mask quantiles are reported separately from the variance calculation.

Task	Positive	Negative	Signed $T_{c}$	$A_{c}$	$M_{2,c}$	$N_{\rm eff,c}$
GSM8K	3742	3426	+0.0578	0.920	0.000384727273	2200
HumanEval	3526	3642	-0.0112	0.760	0.000313913043	1840
ARC-C	3806	3362	+0.0204	0.640	0.000173559322	2360

Appendix

Table 5: Search-gradient saliency across spectral coordinates. Each task has 7168 nonzero coefficients; positive and negative counts sum to this total. Signed totals and absolute magnitudes are reported separately. Effective counts use the absolute and squared score sums, subject to displayed rounding.

Figure 6: Finite-difference discrepancies for 20 shared spectral coordinates per task at

\delta=10^{-3}

. Filled and hollow markers show median and maximum discrepancies; segments connect these summary values. Absolute discrepancy is

|\widehat{\gamma}^{\rm FD}-\gamma|

; relative discrepancy divides by

|\gamma|

only when

|\gamma|>10^{-5}

. Both horizontal axes use log scales.

Stage	Units	Item evaluations
SVD	28 matrices	n/a
Energy-only proposal pilot	16384 masks	n/a
Search-gradient accumulation	3 objectives	832
Finite-difference endpoints	40 coordinate edits	33280
Unedited validation baseline	1 support	672
One development pilot batch	64 masks	43008

Appendix

Table 6: Item-scoring workload for the primary study and selected edits. Subtotals overlap preceding rows and are not added again. Finite-difference scoring reuses 20 coordinate IDs across three objectives. Each validation mask uses 672 items. The baseline and edits A, B, and C each use 2937 reference-NLL items, 1765 greedy completions, and 1172 answer-choice evaluations. Counts describe item evaluations, not runtime; additional development batches, proposals, optional HVPs, weight restoration, forward calls, and timing require separate accounting.

Stratum	Task	$\bar{E}$ interval	$n$	$Q_{.95}$	$\bar{H}_{c,s}$
1	GSM8K	0.126–0.138	320	0.0230	0.278
1	HumanEval	0.126–0.138	320	0.0260	0.337
1	ARC-C	0.126–0.138	320	0.0190	0.211
2	GSM8K	0.138–0.144	320	0.0540	0.613
2	HumanEval	0.138–0.144	320	0.0300	0.324
2	ARC-C	0.138–0.144	320	0.0260	0.247

Appendix

Table 7: Additional prediction-error diagnostics by task and deletion-energy stratum. The

0.95

quantiles summarize absolute search-to-validation NLL errors over 64 calibration and 256 evaluation masks per stratum. The

\bar{H}_{c,s}

coefficients use calibration-only per-mask error-to-energy ratios. Medians and

0.90

quantiles appear in Figure 2 .

Figure 7: Negative errors (hatched, left) and positive errors (solid, right) in the 64-mask development pilot. Error is validation change minus search prediction; zero-error counts are 0 for all tasks. Of the masks, 17 are feasible (26.6%) at target floor 0.12 and preservation tolerance 0.08.

Table 15

Edit	Order position	$\bar{E}$	MBPP+ $\Delta$ NLL
A	63	0.156	0.047
B	91	0.141	0.039
C	146	0.132	0.083

Appendix

Table 10: Selected-edit order positions, normalized deletion energies, and external MBPP+ test NLL changes. Positions refer to the full target-prediction order. MBPP+ NLL is outside radius calibration. Figure 4 gives the three calibrated-task NLL changes, and Figure 5 gives correctness drops.

Model	Target	$B$	$\kappa$	Cal. joint	Eval. joint
Mistral	A	256	1.25	248	978
Mistral	C	256	1.50	249	982
Mistral	C	1024	2.00	247	958
Llama	A	256	1.25	246	974
Llama	A	1024	1.50	248	951
Llama	C	256	1.25	249	986

Appendix

Table 11: Calibration multipliers and joint interval coverage for the additional configurations. Each uses 256 calibration and 1024 evaluation masks. Primary Mistral/arithmetic/1024 coverage appears in Figure 3 a; panel (d) shows feasible-candidate retention for all configurations.

Figure 8: Verification checks until selection at

Q=512

, where all policies select an edit. Circles, triangles, and squares denote direct verification, inner screening, and outer screening; adjacent numbers give check counts. Labels identify checkpoint, target (A: arithmetic; C: code), and coordinate budget

B

. Dashed lines mark

Q=32

and

Q=128

; counts above a cap imply abstention under that cap. Direct and outer verification select the first direct-feasible support D; inner verification selects I. Verification counts exclude development, calibration, and full-pool auditing.

Figure 9: Test NLL changes (a) and correctness drops in percentage points (b) for selected outputs in the additional configurations. Row labels identify checkpoint, target (A: arithmetic; C: code), coordinate budget, and output D or I. Positive values denote deterioration. Darker cells indicate larger changes within each panel; the panels use separate scales. Primary outputs appear in Figures 4 and 5 .

Cell	Output	HumanEval	ARC-C
Mistral/A/256	D	33	778
Mistral/A/256	I	33	779
Mistral/C/256	D	25	777
Mistral/C/256	I	27	778
Mistral/C/1024	D	22	768
Mistral/C/1024	I	24	775

Appendix

Table 12: Supplementary test correctness counts for the additional extension outputs. Denominators are 68 for HumanEval and 1172 for ARC-C. Figure 9 gives GSM8K and MBPP+ correctness drops; baseline counts are reported in the text. Aggregate counts alone do not supply paired confidence intervals.

Explore similar work

To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing
Jul 30, 2026Amir M. Ebrahimi, Mohammed Mehedi Hasan, Aaditya Bhatia +2LLM Reliability Language Model Editing
School of Computing, Queen’s University, Kingston, Ontario, Canada
SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing
Mar 16, 2026Yuhuan Liu, Haitian Zhong, Xinyuan Xia +3Knowledge Editing Language Model Editing
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1
From Latents to Wires: Surgical Post-Editing on Large Language Models
Sep 26, 2026Jiankai Jin, Xiangzheng Zhang, Zhao Liu +4Language Model Editing LLM Unlearning
AI Security Lab

Jul 30, 2026J/K move · Enter open · S save

To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

Amir M. Ebrahimi, Mohammed Mehedi Hasan, Aaditya Bhatia, Gopi Krishnan Rajbahadur+1

School of Computing, Queen’s University, Kingston, Ontario, Canada

LLM Reliability Language Model Editing Code Language Models

Open paper AskPDF arXiv

Large language models increasingly write and repair production code, yet evidence is mounting that their test-passing patches leave codebases harder to maintain. We identify one concrete source: deletion avoidance, the systematic tendency to retain code that an intended edit requires removing. Across the five leading models on the official SWE-bench Verified leaderboard, deletion recall against the developer patch reaches at most 71.7% even on tasks all five solve, and models reach the right file for over 92% of required deletions but cut the exact line in under 52% of cases. Instead, 29.0% of passing patches wrap the targeted code in a guard or fallback, a pattern we call Guard-and-Go. Such patches pass because the original tests rarely check removal: when we retrofit 34 Verified tasks with tests that fail if the targeted code remains, four frontier models spanning closed and open weights fall from 63.2% to 41.9%. Because real repairs mix removal with addition, we curate CanItDelete, a benchmark of 200 tasks mined from real commits whose entire required edit is deletion. Even with the addition work gone, the best model still fails one task in five, and smaller open models fall to 18.0%. We then ablate GPT-5.6 Sol under four cumulative prompts; success moves little until we supply the exact lines, which nearly eliminate incomplete deletion yet raise success only to 80.5% because the model then deletes beyond the spans or adds code instead. Finally, through a pilot study we show one potential fix: teaching deletion during post-training reduces deletion avoidance and improves broader code-editing performance, suggesting the behavior is undertrained rather than beyond reach.

Abstract

Explore similar work

To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing

From Latents to Wires: Surgical Post-Editing on Large Language Models