Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models

Qince Luan; Jianhui Xu; Yang Xu

cs.CLOct 22, 2026

Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models

Authors: Qince Luan, Jianhui Xu, Yang Xu

Organizations: Southern University of Science and Technology (SUSTech) · Independent Scholar

Abstract

Singular-value spectra encode important structure in transformer feed-forward projections, yet their causal role in downstream behavior remains underexplored. We study post-hoc spectral smoothing and spiking interventions on the up_proj and down_proj matrices. Positive perturbation stretches deviations from the geometric mean, making large singular values larger and small ones smaller (spiking), and vice versa. The intervention preserves singular vectors and the geometric mean of the singular values. Across five open Llama and Qwen instruction models, we evaluate the perturbed systems on capability-labeled subtasks, standard and chain-of-thought reasoning benchmarks, judge-scored truthfulness, hallucination, and safety tasks, and TRAIT personality probes. To interpret score shifts, we fit responses in a three-dimensional capability basis consisting of factual knowledge, language understanding, and deductive reasoning. The induced changes suggest structured rather than purely random sensitivity: large positive perturbations more reliably suppress language and deductive responses, moderate negative perturbations can sometimes improve factual or task-local performance on the evaluated grids, and sensitivity varies strongly by model family, benchmark module, and underlying layerwise spectral inequality. These results position spectral perturbation as a lightweight intervention for probing how capabilities are organized and traded off in large language models.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the perturbation setup and representative behavioral effects. Panel (a) visualizes the SVD-based perturbation schema, panel (b) summarizes trait-level responses, and panel (c) highlights selective task-level changes induced by the intervention.

Group	Task	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$
		Llama-3.2-1B			Llama-3.2-3B			Llama-3.1-8B			Qwen3-8B			Qwen3-30B-A3B
Standard no-CoT	GSM8K	0.144	$-0.15$	$+7.9$	0.121	$-0.15$	$+3.3$	0.324	$-0.15$	$\mathbf{+22.7}$	0.845	$-0.10$	$+0.6$	0.864	$+0.10$	$+3.3$
Standard no-CoT	MATH	0.138	$-0.15$	$+1.4$	0.178	$-0.10$	$+2.6$	0.212	$+0.10$	$+1.6$	0.350	$-0.20$	$\mathbf{+4.4}$	0.426	$+0.15$	$+2.2$
Standard no-CoT	ARC-C	0.497	$+0.05$	$+0.1$	0.728	$+0.05$	$+0.4$	0.815	$-0.05$	$\mathbf{+0.5}$	0.922	$0$	$+0.0$	0.927	$0$	$+0.0$
Standard no-CoT	BFCL	0.598	$-0.10$	$\mathbf{+14.5}$	0.048	$-0.20$	$+12.5$	0.593	$0$	$+0.0$	0.777	$-0.15$	$+2.8$	0.885	$-0.10$	$+2.7$
Standard CoT	GSM8K	0.376	$0$	$+0.0$	0.745	$-0.05$	$+0.2$	0.807	$-0.05$	$+3.0$	0.306	$-0.15$	$\mathbf{+5.7}$	0.469	$-0.15$	$+4.5$

Table 1: Best non-judge auxiliary-task settings outside the capability-labeled benchmark suite. Standard and CoT rows use the official full evaluations over the nine-point

\alpha\in[-0.2,0.2]

grid. For each task and model, we report the baseline score at

\alpha=0

, the best

\alpha

on the evaluated grid, and the normalized gain

\Delta_{\mathrm{pp}}

in percentage points of that task’s score range. Bold marks the largest positive gain within each task row.

Group	Task	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$	$s(0)$	Best $\alpha$	$\Delta_{\mathrm{pp}}$
		Llama-3.2-1B			Llama-3.2-3B			Llama-3.1-8B			Qwen3-8B			Qwen3-30B-A3B
Judge	TruthfulQA	0.70	$+0.1$	$+0.5$	1.00	$+0.1$	$\mathbf{+7.5}$	1.28	$0$	$+0.0$	0.93	$+0.1$	$+3.5$	0.94	$+0.1$	$+4.5$
Judge	HaluEval	1.36	$0$	$+0.0$	1.66	$+0.1$	$+1.5$	1.80	$0$	$+0.0$	1.35	$+0.1$	$+3.0$	1.35	$-0.2$	$\mathbf{+8.5}$
Judge	AdvBench	2.83	$0$	$+0.0$	2.97	$0$	$+0.0$	2.86	$-0.2$	$\mathbf{+3.7}$	2.93	$0$	$+0.0$	2.94	$-0.3$	$+1.7$

Table 2: Exploratory judge-scored auxiliary probes. The sweeps use the coarser

\alpha\in\{-0.3,-0.2,-0.1,0,0.1,0.2,0.3\}

grid and the first 100 prompts for each task. TruthfulQA and HaluEval use 0–2 judge scores, AdvBench uses a 0–3 judge score, and

\Delta_{\mathrm{pp}}

is normalized by the corresponding score range.

Group	Model	Task	Best $\alpha$	$s(0)$	Best	$\Delta_{\mathrm{pp}}$	Diagnostic 1	Diagnostic 2	Diagnostic 3
Standard no-CoT	Llama-3.1-8B	GSM8K	$-0.15$	0.324	0.551	+22.7	FE : 0.000 $\rightarrow$ 0.000	WA : 0.676 $\rightarrow$ 0.449	–
Standard no-CoT	Llama-3.2-1B	BFCL	$-0.10$	0.598	0.743	+14.5	First : 0.583 $\rightarrow$ 0.730	Retry : 0.598 $\rightarrow$ 0.743	–
Standard no-CoT	Llama-3.2-3B	BFCL	$-0.20$	0.048	0.172	+12.5	First : 0.037 $\rightarrow$ 0.155	Retry : 0.048 $\rightarrow$ 0.172	–
Standard no-CoT	Llama-3.2-1B	GSM8K	$-0.15$	0.144	0.223	+7.9	FE : 0.000 $\rightarrow$ 0.000	WA : 0.856 $\rightarrow$ 0.777	–
Standard CoT	Qwen3-8B	GSM8K-CoT	$-0.15$	0.306	0.363	+5.7	FE : 0.022 $\rightarrow$ 0.018	WA : 0.672 $\rightarrow$ 0.619	Len : 0.886 $\rightarrow$ 0.862
Standard CoT	Qwen3-30B-A3B	GSM8K-CoT	$-0.15$	0.469	0.514	+4.5	FE : 0.018 $\rightarrow$ 0.011	WA : 0.513 $\rightarrow$ 0.475	Len : 0.640 $\rightarrow$ 0.628

Table 3: Diagnostic decomposition of the largest non-judge auxiliary gains.

s(0)

is the baseline score, Best is the best score on the displayed grid, and

\Delta_{\mathrm{pp}}

reports the percentage-point score change after task-range normalization. For boxed-answer tasks, FE , WA , and Len denote format-error, wrong-answer, and length-finish rates from

\alpha=0

to the best

\alpha

; for BFCL, First and Retry denote first-pass and retry-enabled tool-call success. Dashes indicate unavailable or inapplicable diagnostics.

Module	Model	$-0.2$	$-0.15$	$-0.1$	$-0.05$	$0$	$+0.05$	$+0.1$	$+0.15$	$+0.2$	$\alpha^{*}$	$\Delta_{\mathrm{frac}}$
MMLU-Pro	Llama-3.2-1B	$0.167$	$0.181$	$0.193$	$\mathbf{0.204}$	$0.200$	$0.195$	$0.168$	$0.136$	$0.102$	$-0.05$	$+0.004$
	Llama-3.2-3B	$0.137$	$0.235$	$0.297$	$0.331$	$\mathbf{0.345}$	$0.341$	$0.314$	$0.249$	$0.178$	$0$	$+0.000$
	Llama-3.1-8B	$0.167$	$0.321$	$0.397$	$0.435$	$\mathbf{0.447}$	$0.440$	$0.414$	$0.364$	$0.288$	$0$	$+0.000$
	Qwen3-8B	$0.526$	$0.569$	$0.601$	$0.615$	$0.618$	$\mathbf{0.619}$	$0.606$	$0.570$	$0.503$	$+0.05$	$+0.002$
	Qwen3-30B-A3B	$0.659$	$0.672$	$0.679$	$0.684$	$\mathbf{0.690}$	$0.689$	$0.682$	$0.681$	$0.673$	$0$	$+0.000$
MMLU-Redux	Llama-3.2-1B	$0.264$	$0.256$	$\mathbf{0.275}$	$0.260$	$0.250$	$0.242$	$0.207$	$0.203$	$0.178$	$-0.1$	$+0.025$

Table 4: Raw module means across the nine-point

\alpha

sweep. Unlike Fig. 6 , which decomposes score changes into latent capability coordinates, this table reports the observed unweighted mean over subtasks within each module. Bold marks the best

\alpha

on the displayed grid,

\alpha^{*}

is the selected best perturbation strength, and

\Delta_{\mathrm{frac}}

is the gain over

\alpha=0

expressed in task-score units.

Module	Subtask	FK	LU	DR	Llama 3.1 8B			Qwen3 8B			Qwen3 30B-A3B
					Base	$\alpha^{*}$	$\Delta_{\mathrm{frac}}$	Base	$\alpha^{*}$	$\Delta_{\mathrm{frac}}$	Base	$\alpha^{*}$	$\Delta_{\mathrm{frac}}$
BBH	temporal_sequences	$0.000$	$0.147$	$0.853$	$0.816$	$-0.1$	$+0.120$	$0.080$	$-0.15$	$+0.500$	$0.024$	$-0.2$	$+0.500$
MMLU-Redux	high_school_psychology	$0.888$	$0.086$	$0.026$	$0.604$	$+0.1$	$+0.094$	$0.844$	$-0.05$	$+0.042$	$0.271$	$+0.2$	$+0.312$
MMLU-Redux	logical_fallacies	$0.790$	$0.200$	$0.010$	$0.608$	$+0.05$	$+0.081$	$0.811$	$-0.1$	$+0.027$	$0.324$	$-0.2$	$+0.297$
MMLU-Redux	high_school_us_history	$0.630$	$0.291$	$0.079$	$0.580$	$0$	$+0.000$	$0.740$	$+0.05$	$+0.030$	$0.360$	$+0.2$	$+0.260$
MMLU-Redux	high_school_computer_science	$0.647$	$0.088$	$0.265$	$0.440$	$+0.1$	$+0.040$	$0.770$	$+0.05$	$+0.050$	$0.260$	$+0.2$	$+0.220$

Table 5: Selected high-gain capability-labeled subtasks from the 118-subtask suite. Rows are the 20 subtasks with the largest positive gain under any of the three representative models shown here: Llama 3.1 8B, Qwen3 8B, and Qwen3 30B-A3B. The judge columns report the normalized three-capability demand weights used in the capability fit. For each model, Base is the

\alpha=0

score,

\alpha^{*}

is the best perturbation on the nine-point grid, and

\Delta_{\mathrm{frac}}

is the gain over baseline in task-score units. The complete 118-subtask table is reported in Appendix C .

Causal Interventions in Language Models

Causal interventions in language models test and change how internal representations and computational components cause outputs.

See all 489 papers

Compressed Sensing for Capability Localization in Large Language Models
Feb 11, 2026Anna Bair, Yixuan Even Xu, Mingjie Sun +1Attention Head Analysis LLM Interpretability
Carnegie Mellon University
The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason
Apr 3, 2026Yi LiuRepresentation Geometry in Language Models
Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations
Apr 2, 2026Shou-Tzu Han, Rodrigue Rizk, KC SantoshLanguage Model Robustness LLM Reasoning
Department of Computer Science, University of South Dakota

Feb 11, 2026J/K move · Enter open · S save

Compressed Sensing for Capability Localization in Large Language Models

Anna Bair, Yixuan Even Xu, Mingjie Sun, J. Zico Kolter

Carnegie Mellon University

Attention Head Analysis LLM Interpretability Language Modeling Compressed Sensing Attention Mechanisms

Open paper AskPDF arXiv

Large language models (LLMs) exhibit a wide range of capabilities, including mathematical reasoning, code generation, and linguistic behaviors. We show that Transformer architectures contain small subsets of attention heads that are necessary for certain capabilities. Zeroing out as few as five task-specific heads can degrade performance by up to

60\%

on standard benchmarks measuring the capability of interest, while largely preserving performance on unrelated tasks. We introduce a compressed sensing-based method that exploits the sparsity of these heads to identify them via strategic knockouts and a small number of model evaluations. We validate these findings across Llama and Qwen models ranging from 1B to 14B parameters and a diverse set of capabilities including mathematical abilities and code generation, revealing a modular organization in which specialized capabilities are dependent on sparse, functionally distinct components. Overall, our results suggest that capability localization is a general organizational principle of Transformer language models, with implications for interpretability, model editing, and AI safety. Code is released at https://github.com/locuslab/llm-components.

Abstract

Causal Interventions in Language Models

Explore similar work

Compressed Sensing for Capability Localization in Large Language Models

The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason

Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations