PACE: Pareto-Adaptive Compression for Efficient Native MLLMs

Lei Tian; Chunzheng Zhu; Kai Wu; Yifan Zhang; Haihua Yang

cs.CVOct 15, 2026

PACE: Pareto-Adaptive Compression for Efficient Native MLLMs

Authors: Lei Tian, Chunzheng Zhu, Kai Wu, Yifan Zhang, Haihua Yang

Organizations: ByteDance · Hunan University · Institute of Automation, Chinese Academy of Sciences

Abstract

Native multimodal large language models (native MLLMs) jointly encode visual and linguistic sequences within a unified autoregressive Transformer, emerging as a promising paradigm for multimodal understanding. Unlike conventional ViT-based MLLMs relying on external vision encoders, ViT-free native MLLMs tokenize pixel patches directly, potentially suffering severe visual token redundancy. A pilot study confirms this: unlike ViT-based models that degrade under compression, ViT-free MLLMs tolerate aggressive compression with near-lossless or even improved performance. Yet existing compression methods target ViT-based settings, leaving native MLLMs, which stand to gain the most, unexplored. To close this gap, we introduce an analysis pipeline centered on token discriminability and cross-dimensional heterogeneous redundancy, revealing that (i) redundancy is distributed independently and heterogeneously along height and width axes, and (ii) early-layer visual tokens are indiscriminable and uniformly redundant. Guided by these findings, we propose PACE (Pareto-Adaptive Compression for Efficient Native MLLMs), the first visual token compression framework tailored for native MLLMs with Multimodal-RoPE. Leveraging dimensional decoupling of positional encoding, PACE decomposes attention into two orthogonal spatial saliency estimates along height and width, gated by a positional attention signal, yielding direction-aware importance with negligible overhead. It further casts token budget allocation as Pareto optimization over performance preservation, inference efficiency, and spatial coverage, jointly identifying optimal compression depths and retention ratios. Experiments show PACE maintains near-lossless performance at only 25%-50% visual token retention, even improving several benchmarks, while delivering over 2x FLOPs reduction.

Figures & tables

Figure 1: Probing analysis and progressive compression in native MLLMs. (a) ViT-based MLLMs degrade consistently under compression, while native MLLMs could preserve or even improve performance. (b) Layer-wise discriminability reveals a depth-progressive ramp after the IFB, guiding compression stage placement. (c) Spatially decoupled saliency along height and width dimensions. (d) Progressive token retention across multiple compression stages.

Figure 2: Overview of PACE. (A) Model architecture with depth-progressive compression ( $r_{l_{1}}>r_{l_{2}}>\dots>r_{l_{L}}$ ) applied after the IFB. (B) Native-RoPE QK projection: queries / keys are split into temporal ( $\boldsymbol{q}_{t}$ ) and spatial ( $\boldsymbol{q}_{h},\boldsymbol{q}_{w}$ ) components with independent rotary encodings. (C) SDIE scoring: directional saliency scores are fused via spatial addition and temporal multiplicative gating for TopK selection. (D) PACS: discriminability probing identifies compression layers; a Pareto front over performance, efficiency, and coverage yields the optimal retention schedule.

Figure 3: Pareto front visualization for compression schedule optimization. (a) 3D Pareto front over performance preservation, FLOPs saving, and coverage variance; the gold star marks the selected operating point. (b) 2D projection of FLOPs saving vs. performance retention; color encodes coverage variance (green=low, red=high). (c) Per-layer optimal retention rate: three compression stages (orange/dark orange/red-orange) are aligned with the discriminability curve.

Figure 4: Training data distribution across four stages. Each donut chart shows the dataset composition and proportional contribution per stage.

Retention=25%
		MMB	AI2D	MME ⋆	HallBench ‡	RWQA	MMStar	SEED-IMG	MathVista
Ours-2B †	w/o compress	69.7	80.8	2084	42.7	61.8	52.9	71.8	55.7
Ours-2B †	w/ compress	+1.4% (70.7)	+2.8% (83.1)	+0.4% (2092)	-2.5% (41.6)	-1.0% (61.2)	-4.7% (50.4)	-0.6% (71.4)	-9.7% (50.3)
SAIL-7B †	w/o compress	70.1	76.7	1719	33.1	63.9	53.1	72.9	57.0
SAIL-7B †	w/ compress	-2.6% (68.3)	-2.2% (75.0)	-2.5% (1676)	-1.0% (32.8)	-1.1% (63.2)	-13.6% (45.9)	-3.0% (70.7)	-11.2% (50.6)
ONECAT-3B †	w/o compress	83.1	77.2	2064	39.4	64.8	55.6	72.8	60.7

Table 1: Compression robustness across visual token retention ratios. Each cell reports the relative change from the uncompressed baseline (absolute score in parentheses); green / red denote improvement / degradation . † ViT-free MLLMs. ⋆ MME: sum of perception and reasoning. ‡ HallusionBench: average of aAcc, qAcc, and fAcc as commonly done in prior works.

Figure 5: Progressive visual token retention maps at three compression stages. Retained tokens increasingly focus on semantically critical regions (e.g., text and objects) while preserving spatial coverage, validating the Pareto-adaptive schedule’s balance between performance and coverage.

Figure 6: Effect of the number of compression stages (

N

) on Ours-2B. The three-stage design (

N\!=\!3

) consistently preserves or improves baseline performance across all retention ratios, while

N\!=\!1

incurs notable degradation under aggressive compression.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Language	Size	Sampling Strategy	Notes
BLIP3o [ Chen et al., 2025 ]	EN	29.4M	full sampling	High-quality synthetic captions
COYO [ Byeon et al., 2022 ]	EN	314M	partly sampling	Web-crawled, noisy
Recap-DataComp [ Li et al., 2024b ]	EN	610M	partly sampling	High-quality recaptions
Relaion-COCO [ Schuhmann et al., 2022b ; Schuhmann et al., 2022a ]	EN	333M	partly sampling	High quality recaptions
ImageNet-21K [ Deng et al., 2009 ]	EN	10.2M	partly sampling	Classification labels as short captions
Total		$\sim$ 518M

Appendix

Table 6: Pre-training data summary. Total:

\sim

518M image-text pairs.

Dataset	Language	Size	Sampling Strategy	Notes
ImageNet-21K [ Deng et al., 2009 ]	EN, CH	1.6M	partly sampling	Classification labels as short captions
laionCN [ An et al., 2025 ]	EN, CH	6.4M	partly sampling	English & Chinese web-crawled image–text pairs
COYO [ Byeon et al., 2022 ]	EN, CH	15.1M	partly sampling	Web-crawled, noisy
Recap-DataComp [ Li et al., 2024b ]	EN, CH	6.2M	partly sampling	High-quality recaptions, rewritten for dense semantics
SAM-1B [ Kirillov et al., 2023 ]	EN, ZH	1.1M	partly sampling	Segmentation-grounded dense captions
MINT [ Awadalla et al., 2024 ]	EN, CH	4.1M	partly sampling	Multimodal interleaved documents

Appendix

Table 7: Mid-training data summary. Total:

\sim

89M samples.

Task Category	Datasets
Captioning	TextCaps (en) [ Sidorov et al., 2020 ] , ShareGPT4V (en&zh) [ Chen et al., 2024c ] , COCO [ Lin et al., 2014 ]
General QA	VQAv2 [ Goyal et al., 2017 ] , GQA [ Hudson and Manning, 2019 ] , OKVQA [ Marino et al., 2019 ] , A-OKVQA [ Schwenk et al., 2022 ] , VSR [ Shaharabany et al., 2022 ] , VisualDialog [ Das et al., 2017 ] , ViQuAE [ Lerner et al., 2022 ]
Science	AI2D [ Kembhavi et al., 2016 ] , ScienceQA [ Lu et al., 2022a ] , TQA [ Kembhavi et al., 2017 ]
Chart	ChartQA [ Masry et al., 2022 ] , DVQA [ Kafle et al., 2018 ] , PlotQA [ Methani et al., 2020 ] , IconQA [ Lu et al., 2021b ] , VisText [ Tang et al., 2023 ] , Chart-to-Text [ Kantharaj et al., 2022 ] , FigureQA [ Kahou et al., 2017 ]
Mathematics	Geo170K [ Gao et al., 2023 ] , GeoQA+ [ Chen et al., 2021 ] , TabMWP [ Lu et al., 2022b ] , CLEVR-Math [ Lindström and Abraham, 2022 ] , Geometry3K [ Lu et al., 2021a ]
OCR	OCRVQA [ Mishra et al., 2019 ] , TextVQA [ Singh et al., 2019 ] , InfoVQA [ Mathew et al., 2022 ] , DocVQA [ Mathew et al., 2021 ] , ST-VQA [ Biten et al., 2019 ] , SynthText [ Gupta et al., 2016 ]

Appendix

Table 8: SFT data summary. Multi-source instruction data spanning 8 task categories. Total: 12M samples.

Configuration	Pre-training	Mid-training	SFT	RL
Resolution	$112^{2}$ ~ $1024^{2}$	$256^{2}$ ~ $1024^{2}$	$256^{2}$ ~ $2048^{2}$	$256^{2}$ ~ $2048^{2}$
Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$ )
Weight decay	0.01
LR schedule	cosine	cosine	cosine	cosine
Peak LR	$5\times 10^{-4}$	$5\times 10^{-5}$	$5\times 10^{-5}$	$1\times 10^{-5}$
Training steps	250K	46K	19K	2K

Appendix

Table 10: Hyperparameter configurations for each training stage.

Configuration	Ours-2B
Backbone LLM	Qwen3-1.7B
Pre-Buffer layers ( $L_{\text{buf}}$ )	4
Post-LLM layers ( $L_{\text{post}}$ )	36
Total layers	40
Hidden dim. $d$	2048
Attention heads	16 (QK) / 8 (V)

Appendix

Table 11: Architecture configuration of our trained model.

Stage	Retention	MMB	AI2D	MME	HalBench	RWQA	MMStar	SEED	MathVista
Stage 2: Mid-training ( $\sim$ 89M samples)
Mid-train	100%	58.4	69.3	1782	35.6	52.1	43.8	62.4	42.1
	50%	-2.1% (57.2)	+0.4% (69.6)	-1.3% (1759)	-2.8% (34.6)	-1.5% (51.3)	-3.4% (42.3)	-1.0% (61.8)	-4.5% (40.2)
	25%	-5.8% (55.0)	-2.7% (67.4)	-4.2% (1707)	-7.3% (33.0)	-4.6% (49.7)	-8.2% (40.2)	-4.8% (59.4)	-11.6% (37.2)
Stage 3: Supervised Fine-Tuning ( $\sim$ 12.8M samples)
SFT	100%	67.8	79.2	2041	41.3	60.2	51.6	70.4	53.8

Appendix

Table 12: Per-stage benchmark performance: original vs. compressed. We report results after each training stage (starting from mid-training) on eight benchmarks. Pre-training results are omitted because the model has not yet acquired instruction-following ability at that stage. Green : improvement over the uncompressed baseline of that stage; red : degradation.

Configuration	Ret.	MMB	AI2D	HallBench
Full tokens	100%	69.7	80.8	42.7
PACE	25%	+1.4% (70.7 ± 0.06 )	+2.9% (83.1 ± 0.06 )	-2.5% (41.6 ± 0.1 )
	50%	+2.2% (71.2 ± 0.06 ))	+4.0% (84.0 ± 0.1 ))	+0.5% (42.9 ± 0.1 ))
	75%	+3.2% (71.9 ± 0.15 ))	+5.1% (84.9 ± 0.06 ))	+3.9% (44.4 ± 0.06 ))

Appendix

Table 13: Statistical Significance Analysis on 3 independently sampled calibration sets on MMB, AI2D and HallBench.

Figure 7: Qualitative comparison of token retention (Part 1). Left (object localization): our method retains tokens covering the camouflaged target and its spatial context, while PyramidDrop scatters tokens uniformly and SparseVLM over concentrates on a single cluster. Right (counting): our method distributes tokens across all fruit instances, whereas baselines exhibit concentration bias that drops peripheral ones.

Figure 8: Qualitative comparison of token retention (Part 2). Left (text recognition): our method preserves sign text via $s^{H}$ , while baselines introduce spatial gaps in lettering at high pruning. Right (fine grained detail): our method retains clock display tokens through $s^{P}$ , whereas baselines progressively lose critical pixels.

Retention	$N=1$	$N=2$	$N=3$ (Ours)	$N=4$
75%	65.8	66.4	67.3	67.2
50%	63.2	65.1	66.6	66.5
25%	55.7	62.8	65.6	65.4

Appendix

Table 16: Effect of number of compression stages. Average over MMB, AI2D, HalBench, RWQA, and SEED at various retention levels.

Retention	MMB	AI2D	MME	HalBench	RWQA	MMStar	SEED	MathVista
100% (Full)	69.7	80.8	2084	42.7	61.8	52.9	71.8	55.7
90%	70.5	81.3	2118	42.8	61.7	52.9	72.1	55.8
80%	72.0	83.1	2124	43.5	62.0	52.8	72.7	56.0
75%	71.9	84.9	2120	44.4	62.4	52.6	73.1	56.4
70%	71.6	84.5	2112	43.8	62.5	52.3	72.9	56.6
60%	71.4	84.2	2103	43.1	62.6	52.1	72.7	55.0

Appendix

Table 19: Fine-grained retention sweep on PACE-2B. Performance across eight benchmarks at retention levels from 10% to 90%.

Multimodal Token Compression

Multimodal token compression reduces image, video, audio, or 3D tokens to cut model compute and memory while retaining task-relevant evidence.

See all 96 papers

Apr 18, 2026

EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling

Recent Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language understanding tasks, yet their inference efficiency is often hampered by the large number of visual tokens, particularly in high-resolution or multi-image scenarios. To address this issue, we propose EvoComp, a visual token compression framework that significantly reduces token count while preserving task accuracy. EvoComp introduces a lightweight encoder-only transformer-based compressor that selects the most informative and non-redundant visual tokens by jointly considering visual and textual contexts. A core challenge lies in providing effective supervision for training the compressor. To this end, we design an evolutionary labeling strategy that searches for token subsets minimizing the MLLM's output loss, while enforcing semantic diversity through vocabulary-based token grouping. We further train the compressor using a tailored loss function combining the GHM loss to mitigate class and difficulty imbalance, and a cosine similarity regularization to encourage semantic separation between retained and discarded tokens. Extensive experiments across multiple vision-language benchmarks show that EvoComp outperforms existing methods based on attention or similarity heuristics. Notably, it retains 99.3% of the original accuracy under 3x token compression and delivers up to 1.6x speedup on mobile devices.

Jiafei Song, Fengwei Zhou, Jin Qu +7

OPPO CTG

Efficient Multimodal Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Compression

Ask PDF

Jul 14, 2026

VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.

Yupeng Zheng, Kai Zou, Bin Liu +1

Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China Hefei, China

Efficient VLM Inference Large Vision-Language Models Multimodal Token Compression Visual Token Compression

Ask PDF

Jun 30, 2026

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Yuhao Wang, Mu Qiao, Haiwen Diao +5

School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1

Efficient VLM Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF

Abstract

Multimodal Token Compression

Explore similar work

EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling

VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs