Residual Stream Analysis of Overfitting And Structural Disruptions

Quan Liu; Han Zhou; Wenquan Wu; Hua Wu; Sen Su

cs.LGMar 4, 2026

Residual Stream Analysis of Overfitting And Structural Disruptions

Authors: Quan Liu, Han Zhou, Wenquan Wu, Hua Wu, Sen Su

Organizations: BUPT · Baidu

Abstract

Ensuring that large language models (LLMs) remain both helpful and harmless poses a significant challenge: fine-tuning on repetitive safety datasets, where unsafe prompts are paired with standard refusal templates, often leads to false refusals, in which benign queries are declined. We first quantify this effect, showing that safety data exhibits substantially lower token entropy and 2-gram diversity (0.048) compared to general instruction data. To uncover the root cause, we introduce FlowLens, a stable PCA-based tool for residual-stream geometry analysis, and reveal that higher proportions of safety examples concentrate variance along a few components, reducing representational smoothness and driving false refusals (false refusal rate rises from 63 percent to 84 percent as safety data increases from 0 percent to 40 percent). Guided by these insights, we propose Variance Concentration Loss (VCL), an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals. Empirical results demonstrate that VCL reduces false refusals by over 35 percentage points while maintaining or improving performance on general benchmarks such as MMLU and GSM8K.

Figures & tables

Panel of Figure 1 — Figure 1: Examples of false refusal on an exaggerated safety prompt sampled from XSTest . Our method avoids false refusal and gives an appropriate response. Model and dataset details are provided in Section 5.2 .

Panel of Figure 2 — Figure 2: Residual trajectories of the mean alignment score along the top principal component using FlowLens for four instruction-tuned LLMs on TruthfulQA (General, blue) versus XSTest (Safe, red). Panels (a)–(d) correspond to: (a) Llama-3.2-1B, (b) Llama-3.1-8B-Instruct, (c) Llama-3.2-1B-SFT, and (d) Llama-3.2-1B-VCL (ours). Each curve plots the projection of the final token’s residual vector at normalized layer depth $[0,1]$ . Examples shown above illustrates that standard safety fine-tuning collapses mid-layer variance around depths 0.4–0.6, leading to more false refusals on XSTest; in contrast, VCL stabilizes variance across layers and maintains safety.

Metric	WildJailbreak		WildGuardMix		Tulu-3-SFT-Mixture		Control
	w/o query	w/ query	w/o query	w/ query	w/o query	w/ query	w/o query	w/ query
Entropy $H_{1}$ $\uparrow$	9.18	9.41	11.11	12.30	10.05	11.22	12.05	12.18
Entropy $H_{2}$ $\uparrow$	12.63	14.89	15.97	16.15	14.27	15.39	17.02	17.25
Entropy $H_{3}$ $\uparrow$	13.52	15.68	15.23	16.37	14.92	15.04	18.28	18.43
MSTTR $\uparrow$	0.672	0.689	0.637	0.645	0.659	0.674	0.753	0.767
Distinct 2-gram $\uparrow$	0.048	0.066	0.152	0.177	0.103	0.218	0.205	0.338

Table 1: Lexical diversity metrics (entropy, MSTTR, and distinct

n

-gram rates) for each dataset, comparing cases without and with query context. To avoid interference from the dialogue template “User: … Assistant: …” in the with-query setting we count the query and completion as two separate samples.

Panel of Figure 3 — Figure 3: Loss behavior differences between safety and general tasks. Safety data shows lower average PPL but greater variance and heavier tail. Our experiments employ the Llama-3.1-Tulu-3-8B model family.

Panel of Figure 4 — Figure 4: Projections of residual trajectories using FlowLens for three instruction-tuned language models on the TruthfulQA dataset. Each point represents the PCA-projected residual vector of the final token from one prompt, colored by its corresponding layer index (depth normalized to $[0,1]$ ).

Panel of Figure 5 — Figure 5: Layerwise $\text{PC}_{\mathit{ID}}$ center trajectories under FlowLens. Top: safety vs. general prompts within the same instruction-tuned model (LLaMA-3.2-1B-Instruct); Bottom: models trained on domain-specific subsets from the Tülu 3 dataset Lambert et al. (2024) . Safety data produces irregular $\text{PC}_{\mathit{ID}}$ curves, deviating from the smooth, aligned progression seen in general and other domains. These deviations signal a breakdown in residual stream structure caused by safety fine-tuning.

Safety Ratio	0.0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Score	1.0	0.94	0.91	0.89	0.85	0.82	0.84	0.83	0.77	0.76	0.75
False Refusal (%)	0.63	0.74	0.79	0.81	0.84	0.86	0.82	0.81	0.85	0.92	0.97

Table 2: Effect of safety data ratio on residual structure and refusal metrics. “Score” measures alignment along

\text{PC}_{\mathit{ID}}

using global directions

\mathbf{v}_{\mathit{ID}}^{\text{(global)}}

from the model at safety ratio 0. False Refusal is the precision of rejecting benign prompts.

	Safety Benchmarks $\uparrow$				False Refusal $\uparrow$			General Benchmarks $\uparrow$
Model	DAN	Harmful	Toxigen	JBB	OKTest	ORB	XSTest	MMLU	GSM8K	BBH	CodexEval
Llama-3.2-1B-SFT	0.78	0.74	0.90	0.76	0.53	0.76	0.51	0.42	0.50	0.25	0.24
System Prompt	0.79	0.75	0.95	0.77	0.71	0.65	0.58	0.45	0.52	0.27	0.34
DRO	0.80	0.72	0.92	0.81	0.63	0.71	0.68	0.39	0.49	0.24	0.23
Self-CD	0.76	0.81	0.91	0.83	0.77	0.426	0.78	0.38	0.50	0.26	0.23
Vector Ablation	0.84	0.80	0.97	0.91	0.67	0.447	0.58	0.37	0.51	0.25	0.24

Table 3: Benchmark results of Llama-3.2-1B

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Completion Method	URL
WildJailbreak Jiang et al. (2024)	Adversarial prompts collected via crowd-sourcing teams, paired with refusal completions generated under strict guidelines for clarity and legal defensibility ( $>100{,}000$ samples).	HF/WildJailbreak
WildGuardMix Han et al. (2024)	Combines adversarial teaming and model-in-the-loop generation to produce challenging safety prompts; curated refusals across diverse risk categories ( $>100{,}000$ samples).	HF/WildGuardMix
Tulu-3-SFT-Mixture Lambert et al. (2024)	Extracted safety subset of refusal-type completions from a multi-domain instruction corpus; over 100,000 safety-aligned examples.	HF/Tulu-3-SFT-Mixture
Control group	Randomly sampled 100,000 non-safety examples from the Tulu-3-SFT-Mixture-General subset as a control group.	HF/Tulu-3-SFT-Mixture

Appendix

Table 4: Completion collection methods and sample sizes for the four datasets used in our safety analysis. Collection pipelines are detailed in Table 4 .

WILDJAILBREAK	WILDGUARDMIX	TULU-3-SFT-MIXTURE	Control group
(’. * *’) 9191	(’i can not’) 4448	(’i ’m sorry’) 5062	(’– – –’) 1490
(’. if you’) 6737	(’, such as’) 3146	(’sorry , but’) 5048	(’, such as’) 679
(’* * :’) 5625	(’. it is’) 2660	(”m sorry ,’) 4980	(’, lo que’) 578
(’. i ’m’) 5175	(’. if you’) 2534	(’. i ’m’) 4626	(’. however ,’) 518
(’, but i’) 4980	(’it is important’) 2157	(’i can not’) 3021	(’: ““ ‘’) 516
(’if you ’re’) 4901	(’. however ,’) 2128	(’, but i’) 2975	(’, and the’) 466

Appendix

Table 5: Top-15 trigrams and their frequencies for each dataset: WILDJAILBREAK , WILDGUARDMIX , TULU-3-SFT-MIXTURE , and Control group.

	Safety Benchmarks $\uparrow$				False Refusal $\uparrow$			General Benchmarks $\uparrow$
Model	DAN	Harmful	Toxigen	JBB	OKTest	ORB	XSTest	MMLU	GSM8K	BBH	CodexEval
Llama-3.1-8B-SFT	0.82	0.78	0.94	0.81	0.58	0.80	0.56	0.66	0.57	0.68	0.76
System Prompt	0.82	0.77	0.96	0.80	0.74	0.69	0.60	0.67	0.63	0.69	0.76
DRO	0.83	0.75	0.94	0.84	0.68	0.74	0.70	0.64	0.61	0.67	0.75
Self-CD	0.80	0.83	0.93	0.86	0.80	0.50	0.80	0.63	0.59	0.68	0.75
Vector Ablation	0.86	0.82	0.98	0.93	0.71	0.53	0.61	0.62	0.58	0.67	0.75

Appendix

Table 6: Benchmark results of Llama-3.1-8B

Figure 6: Cosine similarity across layers. Left: with punctuation; Right: without. Small changes cause dramatic shifts.

Figure 7: PCA projections. Trends remain stable despite surface-level changes.

Panel of Figure 8 — Figure 8: 3D PCA projections of residual trajectories using FlowLens for six instruction-tuned language models on the TruthfulQA dataset Lin et al. (2021) . Each point represents the PCA-projected residual vector of the final token from one prompt, colored by its corresponding layer index (depth normalized to $[0,1]$ ).

Panel of Figure 9 — Figure 9: Mean distance to center for harmful prompts per layer across two model–dataset combinations.

Safety-Preserving Fine-Tuning of Language Models

See all 27 papers

Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding
Apr 18, 2026Yupeng Qi, Ziyu Lyu, Lixin Cui +2Language Model Safety Evaluation LLM Refusal Behavior
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University · School of Information, Central University of Finance and Economics · School of Artificial Intelligence, Beijing Normal University +1
Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models
Sep 4, 2026Minji Kim, Hyounghun KimRefusal Behavior in Language Models Refusal Mechanisms in Language Models
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH
Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibration
Sep 6, 2026Zixuan Wang, Bingjie Zhang, He Zhao +1Mechanistic Interpretability LLM Refusal Behavior
School of Artificial Intelligence, Jilin University · CSIRO, Australia · Center of Excellence for Generative AI, King Abdullah University of Science and Technology

Apr 18, 2026J/K move · Enter open · S save

Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding

Yupeng Qi, Ziyu Lyu, Lixin Cui, Lu Bai+1

School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University · School of Information, Central University of Finance and Economics · School of Artificial Intelligence, Beijing Normal University+1

Language Model Safety Evaluation LLM Refusal Behavior LLM Safety Over-Refusal Contrastive Decoding Refusal Behavior in Language Models+1

Open paper AskPDF arXiv

Safety-aligned large language models (LLMs) often generate refusal responses to harmless queries due to the over-refusal problem. However, existing methods for mitigating over-refusal cannot maintain a low refusal ratio for harmless queries while keeping a high refusal ratio for malicious ones. In this paper, we analyze how system prompts with varying safety levels affect LLM refusal behaviors when facing over-refusal queries. A key observation is that, when LLMs suffer from the over-refusal issue, non-refusal tokens remain present in the next-token candidate list, but the model systematically fails to select them, despite the generation of refusal tokens. Based on this observation, we propose a training-free and model-agnostic approach, Adaptive Contrastive Decoding (AdaCD), to mitigate over-refusal while maintaining LLM safety. First, AdaCD compares the output distributions of the LLM with or without an extreme safety system prompt to refine the refusal token distribution. Second, we introduce an adaptive contrastive decoding strategy that dynamically incorporates or removes the refusal token distribution, adaptively boosting the probability of selecting refusal or non-refusal tokens. Experimental results on five benchmark datasets show that, on average, AdaCD reduces the refusal ratio for over-refusal queries by 10.35%, yet still increases the refusal ratio for malicious queries by 0.13%. Code is available at https://github.com/OutdoorManofML/AdaCD.

Abstract

Safety-Preserving Fine-Tuning of Language Models

Explore similar work

Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding

Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models

Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibration