Mechanistically Interpreting Compression in Vision-Language Models

Veeraraju Elluru; Arth Singh; Roberto Aguero; Ajay Agarwal; Debojyoti Das; Hreetam Paul

cs.AIMar 26, 2026

Mechanistically Interpreting Compression in Vision-Language Models

Authors: Veeraraju Elluru, Arth Singh, Roberto Aguero, Ajay Agarwal, Debojyoti Das, Hreetam Paul

Organizations: Indian Institute of Technology Jodhpur · AIM Intelligence · National Institute of Technology Agartala · Fordham University ·

Abstract

Compressed vision-language models (VLMs) are widely used to reduce memory and compute costs, making them a suitable choice for real-world deployment. However, compressing these models raises concerns about whether internal computations and safety behaviors are preserved. In this work, we use causal circuit analysis and crosscoder-based feature comparisons to examine how pruning and quantization fundamentally change the internals across representative VLMs. We observe that pruning generally keeps circuit structure intact but rotates and attenuates internal features, while quantization modifies the circuits at a higher level yet leaves the surviving features better aligned. Leveraging this insight, we also introduce VLMSafe-420, a novel benchmark that pairs harmful inputs with matched benign counterfactuals across various safety categories. Our findings show that pruning causes a sharp drop in genuine refusal behavior, suggesting that the choice of compression has safety implications.

Figures & tables

Panel of Figure 1 — Figure 1: Edge Activation Patching on BLIP-VQA for Visual-Counterfact (green indicates higher importance). Wanda (50%) mostly retains the original circuits (attention heads and MLPs) and component-wise importances. Conversely, INT4 quantization heavily modifies these pathways and relies on newer mechanisms.

Model	Method	TextVQA	ScienceQA	GQA
BLIP-VQA	$\times$	28.5	56.5	47.5
	INT4	26.7	54.3	45.2
	Wanda	22.7	51.1	40.8
LLaVA	$\times$	47.9	59.9	54.1
	INT4	45.6	56.5	51.8
	Wanda	42.0	52.2	47.5

Table 1: VQA benchmark accuracies after compressing the vision and projection components. Wanda pruning is evaluated at 50% sparsity; INT4 denotes weight-only quantization. Since VQA outputs are free-form text, correctness is evaluated using an LLM-as-a-judge framework with Claude Haiku 4.5, details of which are in the Appendix A . Both BLIP-VQA and LLaVA show consistent accuracy drops under compression. Wanda (50%) incurs more degradation than INT4.

Model	Method	EF	FVE U $\uparrow$	FVE C $\uparrow$	DL (%) $\downarrow$
BLIP-VQA	INT4	4 $\times$	0.91	0.79	23.82
	INT4	8 $\times$	0.94	0.86	61.16
	Wanda	4 $\times$	0.86	0.86	75.34
	Wanda	8 $\times$	0.90	0.89	85.62
LLaVA	INT4	4 $\times$	0.86	0.70	37.83
	INT4	8 $\times$	0.85	0.71	71.29

Table 2: Crosscoder meta-statistics for SAE feature analysis. EF denotes expansion factor; FVE U and FVE C are the fraction of variance explained on uncompressed and compressed streams, respectively; DL is the percentage of dead latents. Scores are reported for

\text{TopK}=400

(on the 30% held-out data). The crosscoder is hooked to read from the CLS token activations of the last vision encoder layer in either model.

Panel of Figure 2 — Figure 2: Crosscoder class distribution on Visual-Counterfact for BLIP-VQA and LLaVA, using $\text{TopK}=200$ . Features are classified as uncompressed-only, shared- aligned , intermediate , redirected , attenuated , or compressed-only based on $\rho_{i}$ (decoder norm ratio) and $\theta_{i}$ (cosine similarity). The distribution quantitatively substantiates how much each compression method preserves, modifies, or replaces the original feature structure.

		Top- $r$		Bottom- $r$
Model	Method	IoU	$\rho$	IoU	$\rho$
BLIP-VQA	INT4	0.76	0.51	0.16	0.24
BLIP-VQA	Wanda	0.72	0.69	0.56	0.55
LLaVA	INT4	0.80	0.47	0.12	0.20
LLaVA	Wanda	0.76	0.76	0.44	0.63

Table 3: Circuit similarity between compressed and uncompressed models (both vision and bridge modules compressed). We report IoU (Jaccard overlap) and Spearman

\rho

(rank correlation), with the uncompressed importances on the Top-

r

and Bottom-

r

components (

r=25

). Wanda preserves higher similarity than INT4, with the largest gap on Bottom-

r

components.

Model	Method	FSR	SSS	CSS ${}_{\text{c}}$	CSS ${}_{\text{shared}}$	CSS ${}_{\text{u}}$
BLIP-VQA	INT4	0.64	0.69	0.03	0.01	0.00
BLIP-VQA	Wanda	0.98	0.10	0.07	0.00	0.00
LLaVA	INT4	0.76	0.73	0.02	0.16	0.01
LLaVA	Wanda	0.92	0.47	0.00	-0.01	-0.01

Table 4: Feature sharing and stability metrics for crosscoder analysis (on the 30% held-out data). FSR is the fraction of features shared between compressed and uncompressed models; SSS (Shared Similarity Score) is the mean cosine similarity

\theta_{i}

over shared features; CSS measures class-specific sensitivity to activation changes between original and counterfactual images. SAE configuration matches Figure 2 . In Wanda (high FSR, low SSS), there is higher proportion of shared features, but these are often redirected or attenuated . The converse is true for INT4 quantization.

Type	Count	Description
Text	226	Harmful vs. benign text for a fixed image
Image	150	Harmful vs. benign image for a neutral prompt
Typographic	44	Harmful text embedded in the image

Table 5: Composition of VLMSafe-420. Text counterfactuals isolate the effect of harmful language; image counterfactuals test whether safety extends to visual content; typographic attacks probe robustness to visually-encoded instructions. These 420 samples are across 38 safety categories, and include 50 prompts that follow the structure of JailbreakBench ( Chao et al., 2024 ) .

Component	Recovery	Type
projector	0.461	Projector
layer_13_attn	0.442	Attention
layer_14_attn	0.439	Attention
layer_17_mlp	0.366	MLP
layer_16_attn	0.325	Attention
layer_19_mlp	0.253	MLP

Table 6: Safety-critical components ranked by activation patching recovery. The projector shows the strongest effect, followed by attention heads in layers 13–16 and MLPs in layers 17 and 19. Recovery scores indicate how much restoring the components’ clean activation recovers refusal behavior under harmful inputs.

Figure 3: Refusal token probabilities from logit-lens analysis.

Method	Genuine (%) $\uparrow$	Failure (%) $\downarrow$	Comply (%) $\downarrow$
Baseline	39.0	0.0	61.0
Wanda 10%	37.0	1.4	61.6
Wanda 20%	35.6	2.1	62.3
Wanda 30%	31.7	3.8	64.4
Wanda 50%	22.0	22.0	56.1
NF4 4-bit	33.9	1.9	64.2

Table 7: Model output safety categories under compression. Genuine : coherent refusal; Failure : incoherent output; Comply : coherent harmful response. LLaVA already shows high compliance. Wanda pruning further degrades this with an explicit transition between 30–50% sparsity. NF4 quantization better preserves the refusal behavior.

Ablation Target	Genuine (%) $\uparrow$	Failure (%) $\downarrow$	Comply (%) $\downarrow$
10 Random Trials	$32.6_{\pm 1.5}$	$2.2_{\pm 0.5}$	$65.2$
Projector only	35.8	1.0	63.2
Safety-critical (30)	32.5	2.1	65.4
Non-safety (51)	27.7	12.6	59.7

Table 8: Targeted ablation at 50% sparsity. Pruning safety-critical components produces outcomes indistinguishable from random pruning. Pruning non-safety components causes 6

\times

higher model failure. Pruning only the projector preserves safety best despite its high patching recovery score.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.99$ )
Weight Decay	$10^{-5}$
Learning rate	$3\times 10^{-4}$
Warmup	5% of steps
Gradient Clipping	magnitude = 1.0
Batch size	32

Appendix

Table 9: Crosscoder training hyperparameters.

Model	Method	$\rho_{\text{u}}$	$\rho_{\text{c}}$ = $\rho_{\text{sh}}$	$\rho_{\text{sl}}$
BLIP-VQA	INT4	0.40	0.96	0.53
BLIP-VQA	Wanda	0.40	0.66	0.45
LLaVA	INT4	0.40	0.96	0.55
LLaVA	Wanda	0.40	0.55	0.45
Qwen3-VL-2B	INT4	0.22	0.50	0.22
Qwen3-VL-2B	Wanda	0.22	0.50	0.22

Appendix

Table 10: GMM-based

\rho

thresholds for feature classification.

\rho_{\text{u}}

is the upper bound for uncompressed-only;

\rho_{\text{c}}

is the lower bound for compressed-only;

\rho_{\text{sl}}

and

\rho_{\text{sh}}

bound the shared features. Fixed

\theta

thresholds:

\theta_{\text{aligned}}=0.80

and

\theta_{\text{redirected}}=0.50

. Thresholds are fit per model, per compression method, using a Gaussian Mixture Model (GMM) to separate the three feature classes. Here, the SAE uses an Expansion Factor of 4.

Panel of Figure 4 — Figure 4: Crosscoder class distributions on Visual-Counterfact across BLIP-VQA (single module compression) and Qwen3-VL-2B (combined compression).

			Top- $r$		Bottom- $r$
Model	Method	Module	IoU	$\rho$	IoU	$\rho$
BLIP-VQA	INT4	Encoder	0.74	0.67	0.21	0.28
	INT4	Q-Former	0.81	0.74	0.14	0.19
	Wanda	Encoder	0.73	0.62	0.53	0.69
	Wanda	Q-Former	0.68	0.66	0.46	0.62
LLaVA	INT4	Encoder	0.74	0.79	0.18	0.24

Appendix

Table 11: Module-wise circuit similarity between compressed and uncompressed models. For each module, we report IoU (Jaccard overlap) and Spearman

\rho

(rank correlation), with the uncompressed importances on the Top-

r

and Bottom-

r

component sets.

38 Safety Categories for VLMSafe-420
child_safety	jb_child_safety	sabotage
cyber_crime	jb_cyber	self_harm
dangerous_substances	jb_drugs	sexual_content
drugs_substances	jb_fraud	social_engineering
environmental_misc	jb_harassment	theft_burglary
explosives_incendiary	jb_hate	typographic

Appendix

Table 12: All 38 safety categories in VLMSafe-420. Categories prefixed with jb_ follow the JailbreakBench prompt structure ( Chao et al., 2024 ) .

Panel of Figure 5 — Figure 5: Content warning: This image includes harmful text that the authors do not endorse and is used solely for research and evaluation purposes. Typographic attack pair in VLMSafe-420 .

Panel of Figure 6 — Figure 6: Edge Activation Patching on Qwen3-VL-2B for Visual-Counterfact (green indicates higher importance).

Panel of Figure 7 — Figure 7: Edge Activation Patching on BLIP-VQA with single-module compression (green indicates higher importance). Similar to Figure 1 , Wanda preserves much of the original components regardless of which module is compressed, while INT4 induces structural changes in both.

Cultural Safety Evaluation of VLMs

See all 14 papers

Sep 29, 2026

Hierarchical Compression of Vision-Language Model Benchmarks

Thorough evaluation of vision-language models (VLMs) has become prohibitively expensive, as benchmarks span an ever-broader spectrum of capabilities and new models arrive at a relentless pace. Benchmark compression methods that preserve model rankings at a fraction of the cost are well studied for language models, but for VLMs the question remains under-explored. We present PRIMEBench (Pruning Redundant Items for Multimodal Evaluation), a vision-aware hierarchical benchmark compression framework that substantially reduces evaluation cost while preserving model rankings. This hierarchical framework operates in four stages: data cleaning to remove items answerable without the image and all-correct items, category representative selection to pick one benchmark per capability category, item pruning with Vision-Aware Variance (VAW), and category-count pruning. VAW combines inter-model variance with a vision-dependence score computed from multimodal embeddings alone, while encouraging coverage of diverse items within each benchmark. On models held out from item selection, it has the highest mean fidelity at the released 5% retention. The hierarchical design lets practitioners stop at any stage to match their compute budget; the released suite removes over 97% of items while preserving model rankings. Beyond compression, our analyses show how VLM evaluation behaves as model panels grow and evolve, providing guidance for designing future benchmarks that are more efficient, robust to model turnover, and explicit about the limits of evaluation-side pruning.

Hyunjong Ok, Seunggu Kang, Jaeho Lee

Pohang University of Science and Technology · Upstage AI

VLM Evaluation

Ask PDF

Mar 24, 2026

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions. A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework. A text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability, compositional reasoning is probed with structured negation probes across four templates, and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse (-33.2pp vs. -20.8pp, Wald 95% CI [8.2, 16.8]pp, p < 10^-8), driven almost entirely by COCO while the VQAv2 gap is not statistically significant (4.5pp, p=0.19). The most discriminating template is false_yn: SMOLVLM2-500M responds "Yes" (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Q WEN 2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment.

Mehmet Kaan Erol

Marmara University, Institute of Pure and Applied Sciences

Compositional Reasoning in VLMs VLM Evaluation VLM Robustness VLM Reasoning VLM Quantization VLM Calibration

Ask PDF

Aug 7, 2026

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments. Existing pruning strategies often depend on task-specific criteria or LLM-oriented importance measures, making them unsuitable for task-agnostic pruning, where no task-specific samples are available at pruning time and the pruned model remains broadly applicable. We introduce a retraining-free VLM pruning framework called PORTA that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities. PORTA further incorporates an adaptive sparsity allocation mechanism that assigns layer-wise pruning ratios based on output feature variability, avoiding the limitations of uniform sparsity and reducing performance degradation at high compression levels. Extensive experiments across VLM architectures, such as CLIP, BLIP, and Qwen2-VL, demonstrate that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression. Code is available at https://github.com/cau-hai-lab/PORTA.git.

Minseok Kang, Hyunwoo Kim, Chanyoung Kim +3

Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea · School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Computer Science, Kookmin University, Seoul, Korea

Vision-Language Models Efficient VLM Inference LVLM Pruning Neural Network Pruning

Ask PDF

Abstract

Cultural Safety Evaluation of VLMs

Explore similar work

Hierarchical Compression of Vision-Language Model Benchmarks

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models