Bottleneck Tokens for Unified Multimodal Retrieval

Siyu Sun; Jing Ren; Zhaohe Liao; Dongxiao Mao; Xiangyuan Ren; Yiyi Zhang; Haohua Zhao; Weixiong Lin

cs.LGApr 13, 2026

Bottleneck Tokens for Unified Multimodal Retrieval

Authors: Siyu Sun, Jing Ren, Zhaohe Liao, Dongxiao Mao, Xiangyuan Ren, Yiyi Zhang, Haohua Zhao, Weixiong Lin,

Organizations: Shanghai Jiao Tong University · ByteDance

Abstract

Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).

Figures & tables

Figure 1: Bottleneck tokens for unified multimodal retrieval. The model uses a set of learnable bottleneck tokens (red) inserted between the query tokens and the target tokens in the LLM decoder. The query is embedded through these bottleneck tokens, while the model autoregressively predicts the target tokens. This structure enforces focused interaction between query and target, improving retrieval performance without altering the inference-time complexity.

Model	Size	Overall	Image (acc@1)					Video (acc@1)					VisDoc (ndcg@5)
			Avg	CLS	QA	RET	GD	Avg	CLS	QA	RET	MRET	Avg	VDRv1	VDRv2	VR	OOD
			36	10	10	12	4	18	5	5	5	3	24	10	4	6	4
Reference: Larger Scale / Different Data Conditions
LamRA [ 15 ]	8.3B	47.4	52.4	51.7	34.1	66.9	56.7	33.6	32.9	42.6	23.2	37.2	50.2	56.3	33.3	58.2	40.1
Gme [ 27 ]	8.3B	57.8	56.0	57.7	34.7	71.2	59.3	38.4	37.4	50.4	28.4	37.0	75.2	89.4	55.6	85.0	44.4
UniME-V2 [ 4 ]	8.0B	59.6	71.8	65.6	68.7	73.1	90.9	39.0	37.2	50.6	28.9	39.6	56.7	61.8	42.0	70.5	37.9

Table 1: Main results on MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks). The upper block lists reference methods that use larger backbones (

\geq

7 B), proprietary training data † , or chain-of-thought traces from external models ‡ . The lower block compares 2B-scale models trained on publicly available data without external augmentation. VLM2Vec-V2 is reproduced under identical training settings as our direct baseline. Bold and underline denote the best and second-best results in the 2B-scale block, respectively.

Variant	Overall	Image	Video	VisDoc
BToks (Full)	59.0	66.0	39.9	62.7
EOS pooling	56.1	63.8	37.1	58.9
w/o Condensation Mask	57.9	64.5	39.8	61.7
w/o Generative Objective	57.6	65.6	39.8	59.1

Table 2: Ablation study on key components evaluated on MMEB-V2 [ 16 ] . Starting from the full BToks framework (first row), we remove one component at a time: (i) replacing bottleneck tokens with last-token (EOS) pooling, (ii) removing the Condensation Mask while retaining bottleneck tokens and the generative objective, (iii) removing the generative objective (NTP loss) while keeping all other components. Bold and underline denote the best and second-best, respectively.

$K$	1	2	4	8	16	32
Overall	57.15	57.22	58.96	58.34	58.32	57.99
Image	65.40	65.62	65.97	65.82	66.03	65.86
Video	37.22	37.86	39.94	39.31	38.77	38.00
VisDoc	59.71	59.14	62.72	61.39	61.41	61.19

Table 3: Effect of the number of bottleneck tokens

K

on MMEB-V2 [ 16 ] . We report macro-averaged scores across all 78 datasets (Overall) and per-modality groups (Image, Video, VisDoc). Bold and underline denote the best and second-best, respectively. The highlighted column (

K{=}4

) is our default setting.

Modality	Image	Video	VisDoc	Overall
Datasets	36	18	24	78
Image (w/o BToks)	64.61	33.18	31.15	47.06
Image (BToks)	65.39 (+0.78)	39.42 (+6.24)	47.03 (+15.88)	53.75 (+6.69)
Video (w/o BToks)	27.00	28.00	47.43	33.79
Video (BToks)	30.92 (+3.92)	31.45 (+3.45)	48.19 (+0.76)	36.35 (+2.56)
VisDoc (w/o BToks)	44.96	26.61	53.54	43.37

Table 4: Performance comparison between BToks and the w/o-BToks baseline when models are trained on single-modality data (Image / Video / VisDoc). Numbers in parentheses denote absolute improvements over the corresponding w/o-BToks baseline. The results show that BToks more effectively leverages the same training data and yields robustness and cross-modal generalization gains under data-scarce conditions.

Method	Train (GPU $\cdot$ h)	p50 (ms)	p90 (ms)	Mean (ms)	Thrpt (/s)
VLM2Vec-V2 [ 16 ]	504	74.90	87.10	79.09	50.57
BToks (Ours)	695	76.48	84.73	80.06	49.96
Overhead of bottleneck tokens: mean $+$ 0.97 ms ( $+$ 1.2%), throughput $-$ 0.61/s ( $-$ 1.2%).

Table 5: Training and inference efficiency. BToks achieves superior performance under comparable efficiency. We report training cost (GPU

\cdot

hours on A100-80G), inference latency (p50/p90/mean in ms), and throughput (queries/s) on MMEB-V2.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Dataset	Source	Weight	# Samples
Image	ImageNet-1K	MMEB [ 9 ]	1.0	100,000
	N24News		1.0	48,988
	HatefulMemes		0.5	8,500
	VOC2007		0.5	7,844
	SUN397		0.5	19,850
	OK-VQA		0.5	9,009

Appendix

Table 6: Summary of the training data mixture. All data sources follow the VLM2Vec-V2 configuration. Weights are relative sampling weights within each training batch.

LoRA rank	Overall	Image	Video	VisDoc
$r=8$	53.9	62.2	34.5	56.1
$r=16$ (default)	59.0	66.0	39.9	62.7
$r=32$	58.5	65.6	39.3	62.4

Appendix

Table 7: Sensitivity of BToks to the LoRA rank on MMEB-V2. The default

r{=}16

achieves the best overall and per-modality scores.

$\lambda(t)$ schedule	Overall	Image	Video	VisDoc
$\lambda(t)\equiv 0$ (contrastive-only)	57.6	65.6	39.8	59.1
$0.1,\ t\!<\!1\mathrm{k}$ ; $0$ otherwise	58.9	66.6	39.2	62.1
$0.05,\ t\!<\!2\mathrm{k}$ ; $0$ otherwise	58.2	66.1	38.5	61.1
$0.1,\ t\!<\!2\mathrm{k}$ ; $0$ otherwise (default)	59.0	66.0	39.9	62.7
$0.2,\ t\!<\!2\mathrm{k}$ ; $0$ otherwise	58.3	66.3	38.9	60.7
$0.1,\ t\!<\!3\mathrm{k}$ ; $0$ otherwise	57.5	65.2	38.1	60.6

Appendix

Table 8: Sensitivity of BToks to the NTP loss weight schedule

\lambda(t)

on MMEB-V2. A short early NTP phase (

\lambda{=}0.1

, first

2\mathrm{k}

steps) works best; longer or constant schedules degrade performance.

Meta-task	Dataset	VLM2Vec-V2	BToks	$\Delta$
Classification	Average	64.5	64.3	$-$ 0.2
	VOC2007	87.0	85.7	$-$ 1.3
	N24News	79.9	74.0	$-$ 5.9
	SUN397	72.7	74.9	+2.2
	ObjectNet	60.0	68.5	+8.5
	Country211	25.0	25.2	+0.2

Appendix

Table 9: Per-dataset results on Image tasks from MMEB-V2. Meta-task averages are shown in bold .

Meta-task	Dataset	VLM2Vec-V2	BToks	$\Delta$
Classification	Average	39.1	43.7	+4.6
	K700	33.8	43.1	+9.3
	UCF101	59.8	69.3	+9.5
	HMDB51	38.5	47.1	+8.6
	SmthSmthV2	39.2	41.0	+1.8
	Breakfast	24.3	18.0	$-$ 6.3

Appendix

Table 10: Per-dataset results on Video tasks from MMEB-V2. Meta-task averages are shown in bold .

Meta-task	Dataset	VLM2Vec-V2	BToks	$\Delta$
ViDoRe-V1	Average	66.0	71.1	+5.1
	ArxivQA	69.5	76.5	+7.0
	DocVQA	30.9	37.2	+6.3
	InfoVQA	78.4	80.9	+2.5
	TabFQuAD	74.8	80.4	+5.6
	TATDQA	35.6	42.0	+6.4

Appendix

Table 11: Per-dataset results on Visual Document tasks from MMEB-V2. Meta-task averages are shown in bold .

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 123 papers

Jun 14, 2026

Stepwise Token Selection for Efficient Multimodal Large Language Models

In multimodal large language models (MLLMs), inference cost is largely dominated by the visual token prefix rather than the language backbone, making token reduction a key factor for improving efficiency. Existing approaches typically assign independent importance scores to visual tokens and retain a fixed number of top-ranked tokens, implicitly assuming token independence and a uniform compression ratio across inputs. In this work, we reformulate visual token pruning as a sequential decision-making process. Specifically, we introduce a pointer-style selection mechanism that iteratively chooses informative tokens, conditioning each decision on previously selected ones, and dynamically determines when to stop via a learned termination action. This enables joint optimization of both the selected subset and its size. To enable end-to-end training under standard language modeling objectives, we design a differentiable relaxation based on a variance-preserving noise interpolation scheme, allowing gradients to propagate through the discrete selection process. Extensive experiments on LLaVA-v1.5-7B and Qwen2.5-VL-7B demonstrate that our approach consistently outperforms fixed-ratio baselines across different compression levels. Under aggressive pruning that removes 88.9% of visual tokens, our method preserves 94.6% of the original accuracy while achieving a 1.88x speed-up in prefill latency.

Landi He, Shawn Young, Lijian Xu

Shenzhen University of Advanced Technology

Efficient Multimodal Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF

Jul 28, 2026

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

Recent multimodal large language models (MLLMs), such as Qwen2.5-VL and InternVL3, generate large numbers of vision tokens for high-resolution inputs, leading to substantial computational cost. Existing vision token pruning methods either depend on cross-modal attention and cannot prune before the prefill stage, or rely on diversity estimation with high computational overhead. We observe that attention scores from both vision and text tokens peak at modality separator tokens, suggesting that these separators bridge the two modalities. Based on this observation, we propose SepPrune, an efficient, training-free, plug-and-play pruning method that uses the separator token as a unified query to rank and select informative vision tokens. SepPrune reuses the LLM's built-in projection parameters and requires no architectural changes. Experiments on Qwen2.5-VL-7B show that SepPrune achieves state-of-the-art performance, retaining 96.3% of the original accuracy while removing 80.2% of vision tokens.

Yuchen Wang, Qihui Zhu, Yang Liu +2

MoE Key Lab of BIPC, NEL-BITA, University of Science and Technology of China, Hefei, China · ChangXin Memory Technologies, Hefei, China · APKL of BIIP, IAI, Hefei Comprehensive National Science Center, Hefei, China

Efficient Multimodal Inference Multimodal Large Language Models Visual Token Pruning LVLM Pruning

Ask PDF

Aug 28, 2026

Visual Token Coding for Video Multimodal Large Language Models

In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach

VTC_{Dy}

. To validate VTC, we apply it to three MLLMs and conduct experiments on multiple video understanding benchmarks. The experimental results show that VTC

_{\mathrm{Dy}}

achieves an average performance retention of 100.1% with a 50% token budget for Qwen3-VL, while still retaining 97.8% of the average performance when the token budget is reduced to 25%. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is available at https://github.com/Msr233/VTC.

Chenxin Fang, Tao Chen, JunChao You +3

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China

Multimodal Token Compression Video Token Compression Visual Token Compression Multimodal Large Language Models

Ask PDF

Abstract

Multimodal IR

Explore similar work

Stepwise Token Selection for Efficient Multimodal Large Language Models

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

Visual Token Coding for Video Multimodal Large Language Models