CoViST: Visual Token Compression via Composable States

Qi Zhang; Xiandong Meng; Ronggang Wang; Siwei Ma

cs.AISep 27, 2026

CoViST: Visual Token Compression via Composable States

Authors: Qi Zhang, Xiandong Meng, Ronggang Wang, Siwei Ma

Organizations: Peng Cheng Laboratory Shenzhen, China · Peking University Shenzhen Graduate School Peking University Beijing, China

Abstract

Visual token compression lowers the inference cost of vision--language models by representing images with fewer tokens. However, most existing methods compress visual tokens to a reduced set, leaving the amount of visual evidence represented by each token and its original spatial context implicit. Therefore, the compressed representation does not explicitly encode how much visual information each representative carries or where it lies in the original image. This limitation arises even after a single reduction and becomes more pronounced when compression is repeated across decoder layers. To address this issue, we propose CoViST, a training-free framework that represents a compressed image as a composable visual state. Specifically, the state combines representative features with original positions, effective contribution weights, and reusable selection metadata. CoViST constructs this state through coverage-guided selection and conservation-based contribution composition, and explicitly incorporates its contribution and positional information into decoder attention. Each component of the state retains its interpretation under successive reductions, enabling the same formulation to support both fixed compression before prefill and progressive compression within the decoder. Experimental results on seven LLaVA-1.5-7B benchmarks show that CoViST-Fixed retains 99.9%, 99.5%, and 98.1% of uncompressed performance at 192, 128, and 64 tokens, respectively, and CoViST-Pro retains 99.8%, 99.9%, and 99.1% at the corresponding layer-average budgets, outperforming state-of-the-art methods under their respective budget settings. Code will be released publicly.

Figures & tables

Method	Budget	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{v2} $}}$	$R$ (%)
Vanilla, $K=576$	–	62.0	64.0	1867	85.8	69.5	58.2	76.7	100.0
DivPrune (CVPR 2025)	fixed	58.9	63.1	1723	86.5	69.0	55.7	76.1	96.9
PyramidDrop (CVPR 2025)	avg.	57.3	63.3	1797	84.8	69.2	56.5	76.4	97.1
VisionZip (CVPR 2025)	fixed	59.3	63.0	1783	85.3	68.9	57.3	76.8	97.7
MMTok (ICLR 2026)	fixed	60.1	63.4	1774	86.4	68.8	57.7	77.1	98.2
ZOO-Prune (CVPR 2026)	fixed	60.0	62.9	1782	87.2	69.2	57.3	77.3	98.3

Table 1: Performance comparison on LLaVA-1.5-7B. Bold and underline mark the best and second-best values in each benchmark.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{v2} $}}$	$R$ (%)
Vanilla	63.3	68.7	1824	86.0	72.8	61.3	78.3	100.0
$K{=}192$ ( $\downarrow 66.7\%$ )
VisionZip	59.1	66.9	1754	85.1	73.5	59.5	78.1	97.6
DivPrune	59.4	66.6	1782	86.8	72.9	58.5	78.0	97.8
CDPruner	60.4	67.2	1776	86.6	72.4	58.7	78.4	97.8
ZOO-Prune	60.0	66.7	1762	86.7	73.1	59.1	78.7	98.1

Table 2: Comparison on LLaVA-1.5-13B.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{v2} $}}$	$R$ (%)
Vanilla	63.3	68.7	1824	86.0	72.8	61.3	78.3	100.0
$K{=}192$ ( $\downarrow 66.7\%$ )
VisionZip	59.1	66.9	1754	85.1	73.5	59.5	78.1	97.6
DivPrune	59.4	66.6	1782	86.8	72.9	58.5	78.0	97.8
CDPruner	60.4	67.2	1776	86.6	72.4	58.7	78.4	97.8
ZOO-Prune	60.0	66.7	1762	86.7	73.1	59.1	78.7	98.1

Table 2: Comparison on LLaVA-1.5-13B.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{v2} $}}$	MMB ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{CN} $}}$	$R$ (%)
Vanilla	59.8	83.4	2320	86.6	88.1	76.7	81.6	80.1	100.0
$K{=}512$ ( $\downarrow 60.5\%$ )
VisionZip	56.6	78.9	2317	85.8	80.5	–	80.7	–	95.8
PruneSID	59.8	80.9	2218	85.9	87.6	–	80.4	–	97.6
CoViST-Pro	58.0	82.1	2315	86.4	86.2	73.1	80.1	79.6	98.2
CoViST-Fixed	59.2	83.0	2308	85.5	87.6	76.7	80.9	80.2	99.4

Table 3: Comparison on Qwen2.5-VL-7B.

Method	Budget	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{v2} $}}$	$R$ (%)
Vanilla	–	64.2	64.3	1817	86.9	68.0	61.1	79.9	100.0
DivPrune (CVPR 2025)	fixed	61.6	65.4	1773	85.5	67.8	55.4	78.9	96.1
PyramidDrop (CVPR 2025)	avg.	62.9	66.5	1733	86.4	69.4	58.3	–	97.3
ApET (CVPR 2026)	avg. †	63.0	65.3	1815	87.2	–	57.9	79.2	97.5
ZOO-Prune (CVPR 2026)	fixed	62.2	65.2	1816	86.8	68.0	58.0	79.6	97.5
VisionZip (CVPR 2025)	fixed	61.3	66.3	1787	87.7	68.1	60.2	–	97.8

Table 4: Performance comparison on LLaVA-NeXT-7B.

K

counts content tokens, and CoViST additionally keeps the 48 newline tokens. † Layer average cannot be derived.

Method	$K$	TTFT (ms)	Generation (ms)		FLOPs (T)	KV (MiB)
Vanilla	–	263.94	289.64		43.408	1494.82
CoViST-Fixed	640	178.96	207.66	( $1.39\times$ )	10.004	374.81
CoViST-Fixed	320	125.85	153.32	( $1.89\times$ )	5.661	214.81
CoViST-Fixed	160	102.72	130.25	( $2.22\times$ )	3.530	134.81
CoViST-Pro	640	225.41	253.89	( $1.14\times$ )	10.058	374.81
CoViST-Pro	320	153.56	183.57	( $1.58\times$ )	5.675	214.81

Table 5: Efficiency of LLaVA-NeXT-7B on POPE.

K

counts content tokens (a layer average for CoViST-Pro), and CoViST also keeps 48 newline tokens. Parentheses give the generation speedup over Vanilla; the generation time includes visual encoding and compression. Table 27 gives the full measurements.

Method	$K$	TTFT (ms)	Generation (ms)		FLOPs (T)	KV (MiB)
Vanilla	–	263.94	289.64		43.408	1494.82
CoViST-Fixed	640	178.96	207.66	( $1.39\times$ )	10.004	374.81
CoViST-Fixed	320	125.85	153.32	( $1.89\times$ )	5.661	214.81
CoViST-Fixed	160	102.72	130.25	( $2.22\times$ )	3.530	134.81
CoViST-Pro	640	225.41	253.89	( $1.14\times$ )	10.058	374.81
CoViST-Pro	320	153.56	183.57	( $1.58\times$ )	5.675	214.81

Table 5: Efficiency of LLaVA-NeXT-7B on POPE.

K

	Configuration	$K{=}192$	$K{=}128$	$K{=}64$
(a)	CSI representatives at persistent positions, $m\equiv\mathbf{1}$	99.28	98.81	96.35
(b)	+ $m$ composed under the conservation constraint	99.74	99.18	97.10
(c)	+ coverage-gap reserve	99.84	99.27	97.05
(d)	+ closed-form head gains	99.91	99.50	97.49
(e)	+ $H$ composed with confidence gating	99.95	99.47	97.59
(f)	+ $P$ on the full original span (main)	99.96	99.49	98.10

Table 6: Composing the state (retention, %). Rows (a)–(f) build up CoViST-Fixed, with a span of 288 at

K=64

in (a)–(e); (g) and (h) run CoViST-Pro without and with state inheritance.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Value	Role
$\beta$	1.5	evidence exponent, Equation 4
$u$	0.40 / 0.30 / 0.25 at $K=64$ / 128 / 192, concentration-adaptive, at most 0.60	uniform component
$\lambda_{\max}$ , $\sigma_{0}$	0.30, 0.10	instruction residual bound and selectivity scale
$\lambda_{\max}$ (first-layer route)	0.15, without instruction anchors	used when no CLIP text encoder is available
$N_{\mathrm{r}}$	16 regions with a 10% uniform regional weight	region balancing
core fraction	$0.25+0.45\sqrt{c}$ of the budget	protected attention peaks

Appendix

Table 7: Constants of the main configuration, shared by both execution strategies and by all four models.

Method	Target $K$	Layers 0–13	Layers 14–23	Layers 24–31	Mean
CoViST-Pro	192	288	173	48	192.0625
CoViST-Pro	128	192	115	32	127.9375
CoViST-Pro	64	96	58	16	64.1250
CoViST-Fixed	$K$	$K$	$K$	$K$	$K$

Appendix

Table 8: Token schedules for the 32 decoder layers. The last column gives the exact layer average.

Method	$K$	GQA	MMB	MME-PC	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R_{\mathrm{PC}}$ (%)
Vanilla	576	62.0	64.0	1867	85.8	69.5	58.2	100.0
CoViST-Fixed	192	61.3	64.3	1901	86.2	69.1	57.6	100.0
CoViST-Fixed	128	61.3	64.1	1867	86.3	69.0	57.1	99.5
CoViST-Fixed	64	60.2	63.1	1819	85.8	68.5	56.4	98.1
CoViST-Pro	192	61.6	64.5	1855	86.2	69.3	57.9	99.8
CoViST-Pro	128	61.3	64.7	1890	86.1	69.1	57.7	100.0

Appendix

Table 9: Per-task performance of CoViST on LLaVA-1.5-7B. Retention

R_{\mathrm{PC}}

(%) averages six task-specific ratios to the uncompressed reference; MME-PC sums perception and cognition, POPE uses F1, and TextVQA uses the OCR-augmented prompt.

K

is exact at every layer for CoViST-Fixed and denotes the target layer average for CoViST-Pro (actual means

192.06

127.94

, and

64.13

). Aggregates are computed before rounding.

Setting	$N$	GQA	MMB	MME-PC	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R_{\mathrm{PC}}$ (%)
LLaVA-1.5-13B
Vanilla	576	63.3	68.7	1824	86.0	72.8	61.3	100.0
$K=192$ ( $\downarrow 66.7\%$ )	192	62.9	68.5	1821	87.0	73.1	60.6	99.9
$K=128$ ( $\downarrow 77.8\%$ )	128	62.7	68.0	1811	87.1	73.7	60.1	99.7
$K=64$ ( $\downarrow 88.9\%$ )	64	61.4	67.1	1775	86.3	73.0	59.5	98.3
LLaVA-NeXT-7B

Appendix

Table 10: Cross-model performance of CoViST-Fixed. Each model has its own uncompressed reference.

R_{\mathrm{PC}}

uses the same six tasks as Table 9 , including OCR-augmented TextVQA. For LLaVA-NeXT,

K

counts content tokens and

N

includes 48 retained newline tokens. For the other models,

K=N

. Aggregates are computed before rounding.

LLaVA-NeXT-7B
Setting	MME-P	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$ (no OCR)	SEED-I	MMMU
Vanilla	1496	64.6	69.9	36.3
$K=640$ ( $\downarrow 77.8\%$ )	1531	63.1	69.0	35.2
$K=320$ ( $\downarrow 88.9\%$ )	1482	59.4	67.7	35.3
$K=160$ ( $\downarrow 94.4\%$ )	1449	53.0	65.2	34.8
Qwen2.5-VL-7B

Appendix

Table 11: Additional task scores for the cross-model evaluation. TextVQA here omits OCR tokens from the prompt. SEED-I denotes the image subset. These scores are reported separately from the six-task aggregate in Table 10 .

$K$	Content	Attention	Perception	Cognition	Total
192	Yes	Yes	1537	364	1901
	No	Yes	1534	364	1898
	Yes	No	1485	349	1834
	No	No	1483	351	1834
288	Yes	Yes	1497	351	1848
	No	Yes	1496	351	1847

Appendix

Table 12: Controlled MME interventions at fixed representative sets. Content enables Equation 8 ; attention enables the complete visual attention correction. Each budget has its own paired reference run. Components and totals are rounded independently from the saved scores.

Span	$K=192$	$K=128$	$K=64$
288	98.49	98.06	97.59
432	99.11	99.24	98.17
576 (main)	99.96	99.49	98.10

Appendix

Table 13: Position span around the main configuration (retention, %). Representatives are placed on a proportionally compressed copy of the original grid spanning 288, 432, or 576 positions. The full span of 576 preserves the original grid geometry and is the main configuration.

Configuration	$K=192$	$K=128$	$K=64$
Reference	100.0	99.5	98.1
Head gains $[0.5,1.5]$	99.9	99.5	97.8
Head gains $[0.7,1.3]$	99.8	99.6	98.3
Coverage reserve $3\%$	99.9	99.5	98.3
Coverage reserve $8\%$	99.9	99.5	97.9
Evidence exponent $1.0$	99.6	99.6	98.1

Appendix

Table 14: CoViST-Fixed retention (%) under one-at-a-time changes. Each row uses one configuration across all three budgets. The reference is shared with Table 9 .

First-stage gain	$K=192$	$K=128$	$K=64$
$0.75$	99.9	99.9	99.2
$0.80$	99.8	100.0	99.2
$0.85$	99.7	99.9	99.3

Appendix

Table 15: CoViST-Pro retention (%) for nearby first-stage attention gains. Gains from layer 14 onward remain one. The main configuration uses

0.80

at all budgets.

	Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R$ (%)
(a) Fixed counts: MME-PC and $R_{\mathrm{PC}}$
	Vanilla, $K=576$	62.0	64.0	1867	85.8	69.5	58.2	100.0
	HoloV (NeurIPS 2025)	58.6	62.6	1779	85.0	67.3	55.8	96.5
	VisionZip R (CVPR 2025)	59.2	62.5	1749	85.2	68.7	55.8	96.7
	DivPrune (CVPR 2025)	58.9	63.1	1723	86.5	69.0	55.7	96.8
	ToMe (ICLR 2023)	59.5	62.6	1727	86.9	69.0	55.8	97.0

Appendix

Table 16: Pre-decoder compression on LLaVA-1.5-7B. Budget labels give the fixed visual count

K

entering the decoder. MME-PC includes perception and cognition; MME-P uses perception only.

R

averages the six ratios to each source’s full-model reference. Within each budget and metric block, methods are ordered by increasing unrounded

R

. Bold and underline indicate the two highest distinct displayed values in each performance column; ties share a mark and full-model rows are excluded. VisionZip R and VisionZip V use RESTORE and VisionTrim reports. Appendix E specifies sources and evaluation protocols.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R$ (%)
(a) Layer schedules: MME-PC and $R_{\mathrm{PC}}$
Vanilla, $K=576$	62.0	64.0	1867	85.8	69.5	58.2	100.0
$K=192$ ( $\downarrow 66.7\%$ )
SparseVLM (ICML 2025)	57.6	62.5	1721	83.6	69.1	56.1	95.9
PyramidDrop (CVPR 2025)	57.3	63.3	1797	84.8	69.2	56.5	97.0
ApET ‡ (CVPR 2026)	60.2	63.4	1808	86.3	68.5	54.4	97.5

Appendix

Table 17: Compression during decoder prefill on LLaVA-1.5-7B. Panels (a,b,d) use MME-PC and

R_{\mathrm{PC}}

; (c) uses MME-P and

R_{\mathrm{P}}

. In (a,c), budget headings give reported layer averages

K

; ‡ marks a budget as labeled by its source. Panel (b) uses post-pruning counts except SIEVE ‡ ; (d) uses sample-average final counts. T denotes training. Sorting and bold/underline marks follow Table 16 ; the single-method settings in (d) are unranked.

Method	Budget	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R$ (%)
PruneSID-Dyn S (ICLR 2026) PS	192	60.2	63.8	1797	87.1	69.1	56.9	98.5
PruneSID-Dyn S (ICLR 2026) PS	128	58.9	62.6	1760	86.9	68.8	55.1	96.9
PruneSID-Dyn S (ICLR 2026) PS	64	57.2	59.7	1734	84.1	68.1	54.2	94.5
OccamToken S (arXiv 2026) OC	128	60.9	63.9	1825	86.3	69.1	58.0	99.1
OccamToken S (arXiv 2026) OC	64	59.3	63.2	1801	86.2	69.0	57.4	98.1
OccamToken S (arXiv 2026) OC	32	59.1	62.9	1780	86.2	69.1	56.3	97.5

Appendix

Table 18: Additional reported settings with MME perception plus cognition. The budget column reproduces each source’s setting. T denotes training. Source codes are defined in Table 20 .

Method	Budget	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R$ (%)
TOPS ‡ (arXiv 2026) TP	128	60.5	62.5	1483	86.8	68.2	57.0	98.3
TOPS ‡ (arXiv 2026) TP	64	58.7	60.9	1443	86.5	68.6	56.2	96.8
TOPS ‡ (arXiv 2026) TP	32	56.7	59.5	1385	83.5	68.8	54.9	94.3
STAR-Pro (arXiv 2026) SR	128	60.8	62.4	1456	87.3	68.9	57.2	98.4
STAR-Pro (arXiv 2026) SR	64	59.4	61.0	1421	87.5	68.6	56.3	97.0
STAR-Pro (arXiv 2026) SR	32	57.2	60.1	1358	86.6	68.7	54.6	94.8

Appendix

Table 19: Additional reported settings with MME perception only. The budget column reproduces each source’s setting, and ‡ marks a budget as labeled by its source. T denotes training. Source codes are defined in Table 20 .

Code	Reporting paper	Table	Base
R	RESTORE ( Cho et al., 2026 )	1	B1
D	DeSAP ( Ma et al., 2026b )	1	B2
C	CRISP ( Li et al., 2026d )	1	B2
V	VisionTrim ( Yu et al., 2026 )	1	B2
T	CDPruner ( Zhang et al., 2025b )	1	B3
S	SCoRe ( Xu et al., 2026 )	1	B4

Appendix

Table 20: Sources for external task scores. Codes identify a specific report, including reproduced baseline rows. B1–B12 identify the complete six-task denominators in Table 21 .

Base	MME type	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$
B1	PC	61.9	64.6	1862	85.9	69.5	58.2
B2	PC	61.9	64.7	1862	85.9	69.5	58.2
B3	P	61.9	64.7	1507	85.9	69.5	58.2
B4	P	62.0	64.3	1511	85.9	66.8	58.2
B5	P	61.9	64.7	1513	85.9	69.6	58.2
B6	P	61.9	64.0	1509	85.8	69.6	58.3

Appendix

Table 21: Source-specific full-model scores. PC denotes perception plus cognition and P denotes perception only. Displayed denominators are rounded; retention uses the original reported precision.

LLaVA-NeXT-7B: six tasks
Source	GQA	MMB	MME	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$
ZOO-Prune (MME-PC)	64.2	67.9	1842	86.4	70.2	61.3
AnchorPrune (MME-P)	64.2	67.2	1529	86.4	70.2	61.3

Appendix

Table 22: External full-model references for cross-model retention. Each row lists scores in the order stated by its panel header. The CoViST references appear in Tables 10 and 11 . Source-specific ratios use the full available precision before these references are rounded for display.

(a) MME-PC and six-task retention
	Method	GQA	MMB	MME-PC	POPE	SQA	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R$ (%)
	Vanilla	64.2	64.3	1817	86.9	68.0	61.1	100.0
	Full (source)	64.2	67.9	1842	86.4	70.2	61.3	100.0
	DivPrune (CVPR 2025)	61.6	65.4	1773	85.5	67.8	55.4	95.7
	ZOO-Prune (CVPR 2026)	62.2	65.2	1816	86.8	68.0	58.0	97.2
	VisionZip (CVPR 2025)	61.3	66.3	1787	86.3	68.1	60.2	97.5

Appendix

Table 23: Pre-decoder comparisons on LLaVA-NeXT-7B. Panel (a) uses the MME-PC results reported by Kim et al. (2026) . Panel (b) uses the MME-P results reported by Oh & Kim (2026) , including AnchorPrune itself. Each panel averages its six displayed tasks using the full-model references of the corresponding source. CoViST uses its own references and OCR-augmented TextVQA. Published rows retain their source protocols. CoViST keeps

K+48

visual tokens including newline markers. Ranking and highlighting follow Table 16 and are independent within each panel and budget.

Qwen2.5-VL-7B: MMBench, MME-PC, and TextVQA
	Method	MMB	MME-PC	VQA ${}^{\scalebox{0.8}{$ \scriptstyle\mathrm{T} $}}$	$R_{3}$ (%)
	Vanilla	83.4	2320	82.4	100.0
	HiPrune E (ACL Findings 2026)	80.3	2177	75.8	93.3
	EADP E (ECCV 2026)	81.6	2213	78.6	95.5
	DivPrune C (CVPR 2025)	81.6	2279	81.8	98.0
	CoViST-Fixed	83.0	2308	81.5	99.3

Appendix

Table 24: Fixed-budget comparisons on Qwen2.5-VL-7B at

512/256/128

tokens. All three task scores are shown, and

R_{3}

averages their source-normalized ratios. Superscripts C and E identify Tables 4 and 3 of Zhang et al. (2025b) and Wang et al. (2026c) , respectively. CoViST TextVQA omits OCR tokens from the prompt. This three-task aggregate is distinct from the six-task result in Table 10 . The corresponding source references are reported in Appendix E.4 . Ranking and highlighting follow Table 16 .

(a) General visual reasoning
Method	GQA	MMB	POPE	SQA	$R_{4}$ (%)
Vanilla	59.8	83.4	86.6	88.1	100.0
Full (source)	60.8	83.8	86.3	88.2	100.0
$K=256$ ( $\downarrow 80.2\%$ )
VisionZip (CVPR 2025)	57.0	78.6	83.2	84.5	94.9
CoIn (CVPR 2026)	58.7	79.4	85.1	84.6	96.5

Appendix

Table 25: Additional fixed-budget comparisons on Qwen2.5-VL-7B over two independently defined four-task subsets. Panel (a) uses Table 3 of Du et al. (2026) . VisionZip is their reimplementation. Panel (b) uses Table 3 of Oh & Kim (2026) and OCR-augmented TextVQA for CoViST. Every aggregate uses the four displayed task scores and source-specific full-model references. The two

R_{4}

columns have different task sets and are ranked separately.

Method	Weight	Bound	Head	Position	Inherit	Content
ToMe	patch count	none	shared	n.a. (ViT)	within ViT	average
RESTORE	merge count	none	shared	original, distance factor	n.r.	base method
ERA	saliency-weighted count	none	shared	n.r.	no	recycled into anchors
CaRe	none	–	–	n.r.	no	confidence-gated
HiDrop	none	–	–	persistent IDs	no	none (trained)
CoViST	evidence-conserved transfer	$m_{\max}$	closed-form gains	original, distance factor	yes	confidence-gated, norm-preserving

Appendix

Table 26: Design comparison of attention-level corrections. Weight: the quantity attached to a representative in attention. Bound: the upper bound on that weight. Head: whether the correction differs across heads. Position: treatment of representative positions. Inherit: whether weights are carried from one reduction into the next. Content: treatment of representative features. Entries summarize the cited papers, and “n.r.” marks aspects those papers do not report.

(a) Measured latency and output rate
Method	$K$	Mean visual tokens	TTFT (ms)	LLM prefill (ms)	Generation (ms)	Generation speedup	Output (token/s)
Vanilla	All	2928	263.94	245.66	289.64	$1.00\times$	6.91
CoViST-Fixed	640	688	178.96	150.34	207.66	$1.39\times$	9.63
CoViST-Fixed	320	368	125.85	97.71	153.32	$1.89\times$	13.04
CoViST-Fixed	160	208	102.72	74.41	130.25	$2.22\times$	15.36
CoViST-Pro	640	688	225.41	197.16	253.89	$1.14\times$	7.88

Appendix

Table 27: Inference efficiency on LLaVA-NeXT-7B with POPE and natural EOS stopping, measured on an A800 GPU with FP16 and SDPA.

K

counts content tokens, exactly for CoViST-Fixed and as a layer average for CoViST-Pro. Mean visual tokens additionally include all 48 newline tokens. Latencies are means over 100 prompts with three repetitions, and peak allocated memory is the maximum over requests. Generation speedup is relative to Vanilla. FLOPs estimate decoder prefill matrix multiplications only.

Explore similar work

Jul 14, 2026cs.CV

VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.

Yupeng Zheng, Kai Zou, Bin Liu +1

Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China Hefei, China

Token Compression Data Compression Methods Autoencoder Architectures Intrinsic Encoders

Ask PDF

Aug 3, 2026cs.CV

Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

Modern vision language models (VLMs) turn high-resolution images into long sequences of visual tokens. Every token traverses the language decoder and persists in its prompt KV cache, inflating inference cost and motivating aggressive visual compression. Existing score-based methods assign each token an independent importance score and retain the Top-K. However, text queries consume collective, signed attention messages from the visual population, not isolated patches. Consequently, equally sized Top-K sets can repeatedly cover one salient region, omit sparse but complementary evidence and discard information carried by the removed population. We therefore formulate faithful visual compression as constructing a compact coreset for decoder messages, and introduce our training-free Grounded Message Coreset Pruning (GMC) which jointly allocates support across query-grounded, appearance, and coordinate-aware evidence, then transports discarded states into selected representatives at their original multimodal positions before physical compaction and native attention resume. This decomposes faithful compression into two coupled components, including selecting carriers that cover the required message modes and realizing the signed population message on those carriers. We further derive bounds connecting their errors to signed-message distortion, visual innovation, and candidate-margin stability. Experiments across multiple VLM families and diverse benchmarks demonstrate strong performance, with GMC-H2 retaining 97.78% Full-relative mean capability on Qwen2.5-VL-7B using 80.2% fewer visual tokens, while GMC-L16 reaches 100.36%. Controlled interventions verify that collective support and population realization jointly drive these gains.

Long Qian, Jiaqi Wei, Bingke Zhu +2

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Engineering, Cardiff University, Cardiff, United Kingdom · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China +1

Large Language Model Compression Learned Image Compression Data Compression Methods Importance Distortion

Ask PDF

Sep 30, 2026cs.CV

CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding

Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding expensive for the vision encoder and the language model. Existing methods often compress visual tokens after dense encoding, creating a mismatch between the representation used during training and the compact interface required at deployment. We present CoVisco, a codec-native vision encoder with native token compression for unified image-video understanding. By combining codec-native input support with segmented attention, CoVisco can encode long visual inputs in a single forward pass without forming dense patch-to-patch interactions across all frames. Each temporal segment is equipped with learnable abstract tokens that learn a compact segment-level representation, while fine-grained patch tokens remain available throughout the encoder. Alternating intra-segment and abstract-communication layers preserve video-level context through the abstract-token channel. A lightweight selector further exposes either abstract tokens alone or abstract tokens augmented with a runtime-selected subset of patch tokens, yielding a compact visual interface that reduces the visual context and prefill burden of downstream MLLMs while retaining fine-grained evidence when needed. Pretrained with contrastive objectives on 565M image--text pairs and 6.4M videos, CoVisco shows competitive performance on video-oriented embedding and multimodal understanding benchmarks. In the evaluated four-segment, 64-frame setting, abstract-only inference uses only 400 visual tokens while achieving video-understanding performance close to, and on some benchmarks exceeding, OneVision-Encoder. Selected patch tokens further improve fine-grained video reasoning. Project URL: https://github.com/ernie-research/CoVisco.git

Yulong Liu, Xiaotian Han, Junyuan Shang +6

ERNIE Team, Baidu Inc. · The Hong Kong University of Science and Technology · Institute of Automation, Chinese Academy of Sciences (CASIA)

Vision Encoders Token Compression Video Understanding Taco

Ask PDF