VTC-AUDIT: An Interventional Reliability Audit for Visual-Token Compression under Paired Image Shifts

Isaac; Li

cs.CVOct 25, 2026

VTC-AUDIT: An Interventional Reliability Audit for Visual-Token Compression under Paired Image Shifts

Authors: Isaac, Li

Organizations: Rucheng · University of Pittsburgh

Abstract

Visual-token compression is usually evaluated by clean accuracy and efficiency, but these averages do not reveal failures that compression introduces under an image shift where the uncompressed model stays correct. We introduce VTC-AUDIT, which estimates this incremental risk only on pairs where the full model is correct on both images and the compressed model is correct on the clean image. The four arms then cross two binary choices: which image selects the retained token positions, and which image supplies the visual features. We audit three open 8-9B vision-language models on four tasks and eight pre-specified shifts. We evaluate two training-free compressors---hard-pruning DivPrune and prune-and-merge ApET---at a matched 25% retained-position budget, producing 708,000 inference records. Reselecting positions on the shifted image, rather than reusing the clean-image positions, raises the failure rate in all six model-compressor settings. The increase is 3.04-9.60 percentage points for DivPrune and 2.05-4.20 percentage points for ApET, with every 95% interval above zero. Repeating the audit on a separately resampled second manifest reproduces all six increases. In a pooled monitor audit across the four benchmarks, support Jaccard instability fails the practical AUROC criterion of 0.65 in all six settings, whereas clean compressed-model uncertainty clears it. Thus, support turnover is a weak ranking signal under the audited benchmark mixture, while clean compressed-model uncertainty is substantially more informative. These results suggest that future visual-token compression evaluations should report conditional reliability alongside clean accuracy and efficiency.

Figures & tables

		Full model				DivPrune, 25%			ApET, 25%
Model	Task	Clean score	Clean $\sigma{=}1$	Shifted score	Shifted $\sigma{=}1$	Clean score	Clean $\sigma{=}1$	Eligibility	Clean score	Clean $\sigma{=}1$	Eligibility
Qwen3.5-9B	All	82.89	80.85	82.24	80.18	72.72	70.75	68.39	70.02	68.50	65.89
	ChartQA	89.40	89.40	88.50	88.50	62.00	62.00	60.52	64.60	64.60	62.45
	TextVQA	83.36	75.20	82.75	74.52	77.26	69.40	66.42	57.46	51.40	49.83
	GQA	70.80	70.80	69.73	69.73	65.40	65.40	62.28	69.80	69.80	65.72
	POPE	88.00	88.00	87.98	87.98	86.20	86.20	84.33	88.20	88.20	85.58

Table 1: Full-model performance, clean compressed performance, and eligibility (seed 42). “Score” is the mean benchmark score

\sigma

, and “

\sigma=1

” is the percentage of outputs receiving full credit. Shifted full-model values pool the eight shifts. Eligibility is the percentage of configurations satisfying the conditions in Sec. 3.3 . All entries are percentages.

Model	Compressor	Dynamic SCFR	Frozen SCFR	Counterfactual SCFR	$\Delta_{u,m}$ [95% CI]	Persistence
Qwen3.5-9B	DivPrune	7.12	4.08	8.10	3.04 [2.21, 3.85]	77.07
	ApET	5.77	3.72	5.14	2.05 [1.30, 2.82]	55.58
LLaVA-OneVision-2- 8B-Instruct	DivPrune	5.65	2.13	5.50	3.52 [2.81, 4.25]	74.82
	ApET	4.99	2.52	4.38	2.46 [1.87, 3.08]	58.76
InternVL3.5-8B	DivPrune	14.24	4.63	14.20	9.60 [8.18, 11.13]	81.60
	ApET	9.88	5.68	9.18	4.20 [3.15, 5.28]	64.05

Table 2: Primary intervention results (seed 42). SCFR values and persistence are percentages.

\Delta_{u,m}

is the dynamic–frozen difference in percentage points. Persistence is the fraction of dynamic failures that also fail in the counterfactual arm. Contrasts are computed from unrounded rates.

		All tasks			TextVQA only
Model	Compressor	Dynamic	Frozen	$\Delta_{\rm score}$ [95% CI]	Dynamic	Frozen	$\Delta_{\rm score}$ [95% CI]
Qwen3.5-9B	DivPrune	7.23	4.20	3.04 [2.21, 3.88]	4.88	4.23	0.65 [ $-$ 0.82, 2.10]
	ApET	5.89	3.79	2.09 [1.33, 2.82]	10.40	5.62	4.78 [2.60, 7.02]
LLaVA-OneVision-2- 8B-Instruct	DivPrune	5.82	2.24	3.58 [2.87, 4.32]	5.34	1.75	3.59 [2.32, 4.93]
	ApET	5.08	2.59	2.49 [1.88, 3.13]	5.82	2.10	3.73 [2.28, 5.38]
InternVL3.5-8B	DivPrune	14.39	4.66	9.73 [8.24, 11.23]	13.44	4.82	8.63 [6.17, 11.28]

Table 3: Partial-credit sensitivity of the reselection effect (seed 42). Score loss is

1-\sigma

and is reported as a percentage, macro-averaged over task–shift-family cells on the same eligible population as SCFR.

\Delta_{\rm score}

is dynamic minus frozen score loss in percentage points, with 95% benchmark-stratified image-cluster bootstrap intervals. Contrasts are computed from unrounded rates. The TextVQA panel isolates the graded benchmark.

		DivPrune				ApET
Model	Task	JPEG	Noise	Blur	Bright.	JPEG	Noise	Blur	Bright.
Qwen3.5-9B	ChartQA	9.12	8.21	8.79	10.23	3.37	2.55	2.89	2.72
	TextVQA	0.15	0.75	0.30	1.50	7.06	6.40	3.62	1.40
	GQA	2.25	0.48	1.95	2.07	0.31	1.66	$-$ 0.15	0.45
	POPE	0.48	1.07	0.95	0.36	$-$ 0.12	0.00	0.82	$-$ 0.12
LLaVA-OneVision-2- 8B-Instruct	ChartQA	9.76	8.51	8.43	10.33	6.16	3.04	7.64	4.17

Table 4: Task–shift heterogeneity of the reselection contrast (seed 42). Each entry is

\Delta_{u,m}=\mathrm{SCFR}_{\mathrm{dyn}}-\mathrm{SCFR}_{\mathrm{frz}}

in percentage points, computed over both severities within each shift family. The table reports all 96 model–compressor–task–shift-family cells. Eligible cell sizes range from 256 to 857 configurations (129–430 unique image clusters), with medians of 617 configurations and 312 clusters. Cell-level contrasts characterize heterogeneity; aggregate contrasts and their intervals are reported in Table 2 .

Model	Compressor	Dynamic SCFR, mild $\rightarrow$ strong	Frozen SCFR, mild $\rightarrow$ strong	$\Delta_{u,m}$ , mild $\rightarrow$ strong
Qwen3.5-9B	DivPrune	6.67 $\rightarrow$ 7.58	3.47 $\rightarrow$ 4.69	3.20 $\rightarrow$ 2.88
	ApET	5.30 $\rightarrow$ 6.25	3.40 $\rightarrow$ 4.04	1.90 $\rightarrow$ 2.21
LLaVA-OneVision-2- 8B-Instruct	DivPrune	5.34 $\rightarrow$ 5.95	1.84 $\rightarrow$ 2.42	3.51 $\rightarrow$ 3.53
	ApET	4.40 $\rightarrow$ 5.58	2.21 $\rightarrow$ 2.84	2.19 $\rightarrow$ 2.74
InternVL3.5-8B	DivPrune	14.24 $\rightarrow$ 14.24	3.69 $\rightarrow$ 5.61	10.55 $\rightarrow$ 8.64
	ApET	9.20 $\rightarrow$ 10.56	4.76 $\rightarrow$ 6.63	4.44 $\rightarrow$ 3.94

Table 5: Shift-severity analysis (seed 42). For each model–compressor setting, the table reports dynamic SCFR, frozen SCFR, and their difference

\Delta_{u,m}

at mild and strong severity. SCFR values are percentages;

\Delta_{u,m}

is in percentage points. Differences are computed from unrounded rates.

Model	Compressor	Jaccard AUROC [95% CI]	$U_{\pm}$	Uncertainty AUROC [95% CI]	$L_{95}$
Qwen3.5-9B	DivPrune	0.492 [0.465, 0.519]	0.535	0.714 [0.678, 0.747]	0.678
	ApET	0.564 [0.537, 0.592]	0.592	0.705 [0.663, 0.745]	0.663
LLaVA-OneVision-2- 8B-Instruct	DivPrune	0.423 [0.395, 0.453]	0.605	0.783 [0.744, 0.815]	0.744
	ApET	0.494 [0.465, 0.524]	0.535	0.806 [0.778, 0.833]	0.778
InternVL3.5-8B	DivPrune	0.426 [0.395, 0.457]	0.605	0.785 [0.754, 0.812]	0.754
	ApET	0.488 [0.459, 0.517]	0.541	0.844 [0.819, 0.867]	0.819

Table 6: Monitor audit (seed 42). AUROCs are reported with 95% benchmark-stratified image-cluster bootstrap intervals.

U_{\pm}

is the orientation-free upper 95% bound for Jaccard instability, and

L_{95}

is the lower 95% bound for clean uncertainty. The practical AUROC target is 0.65.

Visual Token Compression

Visual token compression reduces image and video representations to lower inference, memory, storage, or communication costs while retaining task-relevant information.

See all 270 papers

Sep 20, 2026

Query Independent Variable Rate Visual Token Coding

Visual-token compression for vision--language models is posed almost entirely as a selection problem: decide which tokens to keep and discard the rest. The criteria that work best rank tokens by the attention the language model pays them, which makes the ranking a function of the question being asked. That is invisible in a single-turn benchmark and decisive whenever a compressed representation is written once and read many times, as when it is cached across the turns of a conversation or transmitted between a device and a server. We take the other half of the classical transform-coding toolkit instead: keep every token and vary its rate. A transform code exposes each token's measured distortion--rate curve, and a fixed bit budget is distributed across tokens by exact integer rate--distortion optimisation on those curves. No text enters the pipeline, so one compressed representation serves any query. At equal bit budgets, on two datasets and two capacities, it preserves the model's output distribution and its answers better than uniform-rate coding, the closed-form water-fill and distortion-ranked pruning. It matches attention-ranked pruning on the question pruning was tuned for, and overtakes it once the compressed image must answer a different question about the same image.

Hongbo Zhang, Zihao Yang, Liuyang Song +4

Peking University · Uppsala University · University of Southern California

Visual Token Compression Token Compression Vision-Language Models Rate-Distortion Optimization

Ask PDF

Sep 30, 2026

Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses. We study adversarial images that preserve full-token correctness yet induce errors after compression, even when both inference paths succeed on the clean image. Creating such failures is challenging because perturbing token importance can also damage the visual content needed for full-token inference. We propose Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens. In the primary LLaVA-1.5-7B setting, FATA uses only vision-encoder gradients, without access to the deployed compressor, token budget, or downstream task. Across four visually dependent task subsets and four compressors under a controlled reconstruction protocol, FATA achieves SR = 96.3% full-token accuracy retention and CBR = 22.1% conditional blinding, compared with 89.8% and 15.7% for CAA. Ablations support the role of both objectives in balancing compressed-path failure against full-token preservation. FATA also has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate. These findings motivate assessing adversarial robustness jointly across full-token and compressed inference.

Shilinlu Yan, Bowen Chen, Yuechen Zhang +3

Beijing University of Posts and Telecommunications · Jiangnan University · Nanyang Technological University +1

VLM Robustness Adversarial Attacks on VLMs Visual Token Compression

Ask PDF

May 6, 2026

Visual Text Compression as Measure Transport

Visual text compression (VTC) promises efficient long-context processing by rendering text into an image and re-encoding it with a vision-language model, often producing

3

20\times

fewer decoder tokens than subword tokenization. Yet token savings do not translate predictably into downstream utility: on some tasks the visual path matches or exceeds the text path, on others it collapses, and the compression ratio itself does not predict which regime will occur. The missing quantity is therefore not another summary of efficiency, but a principled measure of task-relevant information loss induced by visual encoding. We address this problem by formulating VTC in the language of measure transport. Treating text and visual tokens as empirical probability measures, we show that the ViT patch encoder induces a push-forward map whose transport cost decomposes into a precision cost from within-patch aggregation and a coverage cost from cross-patch fragmentation. Both terms are estimable from downstream-label-free probes. This formulation yields two operational consequences: a downstream-label-free routing criterion that selects whether to use the visual path for a given input or benchmark instance, and a transport-informed foveation mechanism that re-encodes high-cost regions at higher resolution. Across

24

NLP datasets at Qwen3-4B, our label-free rule matches the per-dataset oracle on

17/24

datasets (

70.8\%

), and improves the average task score by

+3.3\%

with

-10.3\%

average tokens relative to a pure-LLM.

Lv Tang, Tianyi Zheng, Yang Liu +2

University of Alberta · vivo Mobile Communication Co., Ltd · Tsinghua University

Efficient VLM Inference Visual Tokenization Visual Token Compression

Ask PDF

Abstract

Visual Token Compression

Explore similar work

Query Independent Variable Rate Visual Token Coding

Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

Visual Text Compression as Measure Transport