VTC-AUDIT: An Interventional Reliability Audit for Visual-Token Compression under Paired Image Shifts

Isaac; Li

cs.CVOct 25, 2026

VTC-AUDIT: An Interventional Reliability Audit for Visual-Token Compression under Paired Image Shifts

Authors: Isaac, Li

Organizations: Rucheng · University of Pittsburgh

Abstract

Visual-token compression is usually evaluated by clean accuracy and efficiency, but these averages do not reveal failures that compression introduces under an image shift where the uncompressed model stays correct. We introduce VTC-AUDIT, which estimates this incremental risk only on pairs where the full model is correct on both images and the compressed model is correct on the clean image. The four arms then cross two binary choices: which image selects the retained token positions, and which image supplies the visual features. We audit three open 8-9B vision-language models on four tasks and eight pre-specified shifts. We evaluate two training-free compressors---hard-pruning DivPrune and prune-and-merge ApET---at a matched 25% retained-position budget, producing 708,000 inference records. Reselecting positions on the shifted image, rather than reusing the clean-image positions, raises the failure rate in all six model-compressor settings. The increase is 3.04-9.60 percentage points for DivPrune and 2.05-4.20 percentage points for ApET, with every 95% interval above zero. Repeating the audit on a separately resampled second manifest reproduces all six increases. In a pooled monitor audit across the four benchmarks, support Jaccard instability fails the practical AUROC criterion of 0.65 in all six settings, whereas clean compressed-model uncertainty clears it. Thus, support turnover is a weak ranking signal under the audited benchmark mixture, while clean compressed-model uncertainty is substantially more informative. These results suggest that future visual-token compression evaluations should report conditional reliability alongside clean accuracy and efficiency.

Figures & tables

		Full model				DivPrune, 25%			ApET, 25%
Model	Task	Clean score	Clean $\sigma{=}1$	Shifted score	Shifted $\sigma{=}1$	Clean score	Clean $\sigma{=}1$	Eligibility	Clean score	Clean $\sigma{=}1$	Eligibility
Qwen3.5-9B	All	82.89	80.85	82.24	80.18	72.72	70.75	68.39	70.02	68.50	65.89
	ChartQA	89.40	89.40	88.50	88.50	62.00	62.00	60.52	64.60	64.60	62.45
	TextVQA	83.36	75.20	82.75	74.52	77.26	69.40	66.42	57.46	51.40	49.83
	GQA	70.80	70.80	69.73	69.73	65.40	65.40	62.28	69.80	69.80	65.72
	POPE	88.00	88.00	87.98	87.98	86.20	86.20	84.33	88.20	88.20	85.58

Table 1: Full-model performance, clean compressed performance, and eligibility (seed 42). “Score” is the mean benchmark score

\sigma

, and “

\sigma=1

” is the percentage of outputs receiving full credit. Shifted full-model values pool the eight shifts. Eligibility is the percentage of configurations satisfying the conditions in Sec. 3.3 . All entries are percentages.

Model	Compressor	Dynamic SCFR	Frozen SCFR	Counterfactual SCFR	$\Delta_{u,m}$ [95% CI]	Persistence
Qwen3.5-9B	DivPrune	7.12	4.08	8.10	3.04 [2.21, 3.85]	77.07
	ApET	5.77	3.72	5.14	2.05 [1.30, 2.82]	55.58
LLaVA-OneVision-2- 8B-Instruct	DivPrune	5.65	2.13	5.50	3.52 [2.81, 4.25]	74.82
	ApET	4.99	2.52	4.38	2.46 [1.87, 3.08]	58.76
InternVL3.5-8B	DivPrune	14.24	4.63	14.20	9.60 [8.18, 11.13]	81.60
	ApET	9.88	5.68	9.18	4.20 [3.15, 5.28]	64.05

Table 2: Primary intervention results (seed 42). SCFR values and persistence are percentages.

\Delta_{u,m}

is the dynamic–frozen difference in percentage points. Persistence is the fraction of dynamic failures that also fail in the counterfactual arm. Contrasts are computed from unrounded rates.

		All tasks			TextVQA only
Model	Compressor	Dynamic	Frozen	$\Delta_{\rm score}$ [95% CI]	Dynamic	Frozen	$\Delta_{\rm score}$ [95% CI]
Qwen3.5-9B	DivPrune	7.23	4.20	3.04 [2.21, 3.88]	4.88	4.23	0.65 [ $-$ 0.82, 2.10]
	ApET	5.89	3.79	2.09 [1.33, 2.82]	10.40	5.62	4.78 [2.60, 7.02]
LLaVA-OneVision-2- 8B-Instruct	DivPrune	5.82	2.24	3.58 [2.87, 4.32]	5.34	1.75	3.59 [2.32, 4.93]
	ApET	5.08	2.59	2.49 [1.88, 3.13]	5.82	2.10	3.73 [2.28, 5.38]
InternVL3.5-8B	DivPrune	14.39	4.66	9.73 [8.24, 11.23]	13.44	4.82	8.63 [6.17, 11.28]

Table 3: Partial-credit sensitivity of the reselection effect (seed 42). Score loss is

1-\sigma

and is reported as a percentage, macro-averaged over task–shift-family cells on the same eligible population as SCFR.

\Delta_{\rm score}

is dynamic minus frozen score loss in percentage points, with 95% benchmark-stratified image-cluster bootstrap intervals. Contrasts are computed from unrounded rates. The TextVQA panel isolates the graded benchmark.

		DivPrune				ApET
Model	Task	JPEG	Noise	Blur	Bright.	JPEG	Noise	Blur	Bright.
Qwen3.5-9B	ChartQA	9.12	8.21	8.79	10.23	3.37	2.55	2.89	2.72
	TextVQA	0.15	0.75	0.30	1.50	7.06	6.40	3.62	1.40
	GQA	2.25	0.48	1.95	2.07	0.31	1.66	$-$ 0.15	0.45
	POPE	0.48	1.07	0.95	0.36	$-$ 0.12	0.00	0.82	$-$ 0.12
LLaVA-OneVision-2- 8B-Instruct	ChartQA	9.76	8.51	8.43	10.33	6.16	3.04	7.64	4.17

Table 4: Task–shift heterogeneity of the reselection contrast (seed 42). Each entry is

\Delta_{u,m}=\mathrm{SCFR}_{\mathrm{dyn}}-\mathrm{SCFR}_{\mathrm{frz}}

in percentage points, computed over both severities within each shift family. The table reports all 96 model–compressor–task–shift-family cells. Eligible cell sizes range from 256 to 857 configurations (129–430 unique image clusters), with medians of 617 configurations and 312 clusters. Cell-level contrasts characterize heterogeneity; aggregate contrasts and their intervals are reported in Table 2 .

Model	Compressor	Dynamic SCFR, mild $\rightarrow$ strong	Frozen SCFR, mild $\rightarrow$ strong	$\Delta_{u,m}$ , mild $\rightarrow$ strong
Qwen3.5-9B	DivPrune	6.67 $\rightarrow$ 7.58	3.47 $\rightarrow$ 4.69	3.20 $\rightarrow$ 2.88
	ApET	5.30 $\rightarrow$ 6.25	3.40 $\rightarrow$ 4.04	1.90 $\rightarrow$ 2.21
LLaVA-OneVision-2- 8B-Instruct	DivPrune	5.34 $\rightarrow$ 5.95	1.84 $\rightarrow$ 2.42	3.51 $\rightarrow$ 3.53
	ApET	4.40 $\rightarrow$ 5.58	2.21 $\rightarrow$ 2.84	2.19 $\rightarrow$ 2.74
InternVL3.5-8B	DivPrune	14.24 $\rightarrow$ 14.24	3.69 $\rightarrow$ 5.61	10.55 $\rightarrow$ 8.64
	ApET	9.20 $\rightarrow$ 10.56	4.76 $\rightarrow$ 6.63	4.44 $\rightarrow$ 3.94

Table 5: Shift-severity analysis (seed 42). For each model–compressor setting, the table reports dynamic SCFR, frozen SCFR, and their difference

\Delta_{u,m}

at mild and strong severity. SCFR values are percentages;

\Delta_{u,m}

is in percentage points. Differences are computed from unrounded rates.

Model	Compressor	Jaccard AUROC [95% CI]	$U_{\pm}$	Uncertainty AUROC [95% CI]	$L_{95}$
Qwen3.5-9B	DivPrune	0.492 [0.465, 0.519]	0.535	0.714 [0.678, 0.747]	0.678
	ApET	0.564 [0.537, 0.592]	0.592	0.705 [0.663, 0.745]	0.663
LLaVA-OneVision-2- 8B-Instruct	DivPrune	0.423 [0.395, 0.453]	0.605	0.783 [0.744, 0.815]	0.744
	ApET	0.494 [0.465, 0.524]	0.535	0.806 [0.778, 0.833]	0.778
InternVL3.5-8B	DivPrune	0.426 [0.395, 0.457]	0.605	0.785 [0.754, 0.812]	0.754
	ApET	0.488 [0.459, 0.517]	0.541	0.844 [0.819, 0.867]	0.819

Table 6: Monitor audit (seed 42). AUROCs are reported with 95% benchmark-stratified image-cluster bootstrap intervals.

U_{\pm}

is the orientation-free upper 95% bound for Jaccard instability, and

L_{95}

is the lower 95% bound for clean uncertainty. The practical AUROC target is 0.65.

Visual Token Compression

Visual token compression reduces image and video representations to lower inference, memory, storage, or communication costs while retaining task-relevant information.

See all 270 papers

Query Independent Variable Rate Visual Token Coding
Sep 20, 2026Hongbo Zhang, Zihao Yang, Liuyang Song +4Visual Token Compression Token Compression
Peking University · Uppsala University · University of Southern California
Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
Sep 30, 2026Shilinlu Yan, Bowen Chen, Yuechen Zhang +3VLM Robustness Adversarial Attacks on VLMs
Beijing University of Posts and Telecommunications · Jiangnan University · Nanyang Technological University +1
Visual Text Compression as Measure Transport
May 6, 2026Lv Tang, Tianyi Zheng, Yang Liu +2Efficient VLM Inference Visual Tokenization
University of Alberta · vivo Mobile Communication Co., Ltd · Tsinghua University

Sep 20, 2026J/K move · Enter open · S save

Query Independent Variable Rate Visual Token Coding

Hongbo Zhang, Zihao Yang, Liuyang Song, Daqian Yang+3

Peking University · Uppsala University · University of Southern California

Visual Token Compression Token Compression Vision-Language Models Rate-Distortion Optimization

Open paper AskPDF arXiv

Visual-token compression for vision--language models is posed almost entirely as a selection problem: decide which tokens to keep and discard the rest. The criteria that work best rank tokens by the attention the language model pays them, which makes the ranking a function of the question being asked. That is invisible in a single-turn benchmark and decisive whenever a compressed representation is written once and read many times, as when it is cached across the turns of a conversation or transmitted between a device and a server. We take the other half of the classical transform-coding toolkit instead: keep every token and vary its rate. A transform code exposes each token's measured distortion--rate curve, and a fixed bit budget is distributed across tokens by exact integer rate--distortion optimisation on those curves. No text enters the pipeline, so one compressed representation serves any query. At equal bit budgets, on two datasets and two capacities, it preserves the model's output distribution and its answers better than uniform-rate coding, the closed-form water-fill and distortion-ranked pruning. It matches attention-ranked pruning on the question pruning was tuned for, and overtakes it once the compressed image must answer a different question about the same image.

Abstract

Visual Token Compression

Explore similar work

Query Independent Variable Rate Visual Token Coding

Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

Visual Text Compression as Measure Transport