Bottleneck Tokens for Unified Multimodal Retrieval

Siyu Sun; Jing Ren; Zhaohe Liao; Dongxiao Mao; Xiangyuan Ren; Yiyi Zhang; Haohua Zhao; Weixiong Lin

cs.LGApr 13, 2026

Bottleneck Tokens for Unified Multimodal Retrieval

Authors: Siyu Sun, Jing Ren, Zhaohe Liao, Dongxiao Mao, Xiangyuan Ren, Yiyi Zhang, Haohua Zhao, Weixiong Lin,

Organizations: Shanghai Jiao Tong University · ByteDance

Abstract

Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).

Figures & tables

Figure 1: Bottleneck tokens for unified multimodal retrieval. The model uses a set of learnable bottleneck tokens (red) inserted between the query tokens and the target tokens in the LLM decoder. The query is embedded through these bottleneck tokens, while the model autoregressively predicts the target tokens. This structure enforces focused interaction between query and target, improving retrieval performance without altering the inference-time complexity.

Model	Size	Overall	Image (acc@1)					Video (acc@1)					VisDoc (ndcg@5)
			Avg	CLS	QA	RET	GD	Avg	CLS	QA	RET	MRET	Avg	VDRv1	VDRv2	VR	OOD
			36	10	10	12	4	18	5	5	5	3	24	10	4	6	4
Reference: Larger Scale / Different Data Conditions
LamRA [ 15 ]	8.3B	47.4	52.4	51.7	34.1	66.9	56.7	33.6	32.9	42.6	23.2	37.2	50.2	56.3	33.3	58.2	40.1
Gme [ 27 ]	8.3B	57.8	56.0	57.7	34.7	71.2	59.3	38.4	37.4	50.4	28.4	37.0	75.2	89.4	55.6	85.0	44.4
UniME-V2 [ 4 ]	8.0B	59.6	71.8	65.6	68.7	73.1	90.9	39.0	37.2	50.6	28.9	39.6	56.7	61.8	42.0	70.5	37.9

Table 1: Main results on MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks). The upper block lists reference methods that use larger backbones (

\geq

7 B), proprietary training data † , or chain-of-thought traces from external models ‡ . The lower block compares 2B-scale models trained on publicly available data without external augmentation. VLM2Vec-V2 is reproduced under identical training settings as our direct baseline. Bold and underline denote the best and second-best results in the 2B-scale block, respectively.

Variant	Overall	Image	Video	VisDoc
BToks (Full)	59.0	66.0	39.9	62.7
EOS pooling	56.1	63.8	37.1	58.9
w/o Condensation Mask	57.9	64.5	39.8	61.7
w/o Generative Objective	57.6	65.6	39.8	59.1

Table 2: Ablation study on key components evaluated on MMEB-V2 [ 16 ] . Starting from the full BToks framework (first row), we remove one component at a time: (i) replacing bottleneck tokens with last-token (EOS) pooling, (ii) removing the Condensation Mask while retaining bottleneck tokens and the generative objective, (iii) removing the generative objective (NTP loss) while keeping all other components. Bold and underline denote the best and second-best, respectively.

$K$	1	2	4	8	16	32
Overall	57.15	57.22	58.96	58.34	58.32	57.99
Image	65.40	65.62	65.97	65.82	66.03	65.86
Video	37.22	37.86	39.94	39.31	38.77	38.00
VisDoc	59.71	59.14	62.72	61.39	61.41	61.19

Table 3: Effect of the number of bottleneck tokens

K

on MMEB-V2 [ 16 ] . We report macro-averaged scores across all 78 datasets (Overall) and per-modality groups (Image, Video, VisDoc). Bold and underline denote the best and second-best, respectively. The highlighted column (

K{=}4

) is our default setting.

Modality	Image	Video	VisDoc	Overall
Datasets	36	18	24	78
Image (w/o BToks)	64.61	33.18	31.15	47.06
Image (BToks)	65.39 (+0.78)	39.42 (+6.24)	47.03 (+15.88)	53.75 (+6.69)
Video (w/o BToks)	27.00	28.00	47.43	33.79
Video (BToks)	30.92 (+3.92)	31.45 (+3.45)	48.19 (+0.76)	36.35 (+2.56)
VisDoc (w/o BToks)	44.96	26.61	53.54	43.37

Table 4: Performance comparison between BToks and the w/o-BToks baseline when models are trained on single-modality data (Image / Video / VisDoc). Numbers in parentheses denote absolute improvements over the corresponding w/o-BToks baseline. The results show that BToks more effectively leverages the same training data and yields robustness and cross-modal generalization gains under data-scarce conditions.

Method	Train (GPU $\cdot$ h)	p50 (ms)	p90 (ms)	Mean (ms)	Thrpt (/s)
VLM2Vec-V2 [ 16 ]	504	74.90	87.10	79.09	50.57
BToks (Ours)	695	76.48	84.73	80.06	49.96
Overhead of bottleneck tokens: mean $+$ 0.97 ms ( $+$ 1.2%), throughput $-$ 0.61/s ( $-$ 1.2%).

Table 5: Training and inference efficiency. BToks achieves superior performance under comparable efficiency. We report training cost (GPU

\cdot

hours on A100-80G), inference latency (p50/p90/mean in ms), and throughput (queries/s) on MMEB-V2.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Dataset	Source	Weight	# Samples
Image	ImageNet-1K	MMEB [ 9 ]	1.0	100,000
	N24News		1.0	48,988
	HatefulMemes		0.5	8,500
	VOC2007		0.5	7,844
	SUN397		0.5	19,850
	OK-VQA		0.5	9,009

Appendix

Table 6: Summary of the training data mixture. All data sources follow the VLM2Vec-V2 configuration. Weights are relative sampling weights within each training batch.

LoRA rank	Overall	Image	Video	VisDoc
$r=8$	53.9	62.2	34.5	56.1
$r=16$ (default)	59.0	66.0	39.9	62.7
$r=32$	58.5	65.6	39.3	62.4

Appendix

Table 7: Sensitivity of BToks to the LoRA rank on MMEB-V2. The default

r{=}16

achieves the best overall and per-modality scores.

$\lambda(t)$ schedule	Overall	Image	Video	VisDoc
$\lambda(t)\equiv 0$ (contrastive-only)	57.6	65.6	39.8	59.1
$0.1,\ t\!<\!1\mathrm{k}$ ; $0$ otherwise	58.9	66.6	39.2	62.1
$0.05,\ t\!<\!2\mathrm{k}$ ; $0$ otherwise	58.2	66.1	38.5	61.1
$0.1,\ t\!<\!2\mathrm{k}$ ; $0$ otherwise (default)	59.0	66.0	39.9	62.7
$0.2,\ t\!<\!2\mathrm{k}$ ; $0$ otherwise	58.3	66.3	38.9	60.7
$0.1,\ t\!<\!3\mathrm{k}$ ; $0$ otherwise	57.5	65.2	38.1	60.6

Appendix

Table 8: Sensitivity of BToks to the NTP loss weight schedule

\lambda(t)

on MMEB-V2. A short early NTP phase (

\lambda{=}0.1

, first

2\mathrm{k}

steps) works best; longer or constant schedules degrade performance.

Meta-task	Dataset	VLM2Vec-V2	BToks	$\Delta$
Classification	Average	64.5	64.3	$-$ 0.2
	VOC2007	87.0	85.7	$-$ 1.3
	N24News	79.9	74.0	$-$ 5.9
	SUN397	72.7	74.9	+2.2
	ObjectNet	60.0	68.5	+8.5
	Country211	25.0	25.2	+0.2

Appendix

Table 9: Per-dataset results on Image tasks from MMEB-V2. Meta-task averages are shown in bold .

Meta-task	Dataset	VLM2Vec-V2	BToks	$\Delta$
Classification	Average	39.1	43.7	+4.6
	K700	33.8	43.1	+9.3
	UCF101	59.8	69.3	+9.5
	HMDB51	38.5	47.1	+8.6
	SmthSmthV2	39.2	41.0	+1.8
	Breakfast	24.3	18.0	$-$ 6.3

Appendix

Table 10: Per-dataset results on Video tasks from MMEB-V2. Meta-task averages are shown in bold .

Meta-task	Dataset	VLM2Vec-V2	BToks	$\Delta$
ViDoRe-V1	Average	66.0	71.1	+5.1
	ArxivQA	69.5	76.5	+7.0
	DocVQA	30.9	37.2	+6.3
	InfoVQA	78.4	80.9	+2.5
	TabFQuAD	74.8	80.4	+5.6
	TATDQA	35.6	42.0	+6.4

Appendix

Table 11: Per-dataset results on Visual Document tasks from MMEB-V2. Meta-task averages are shown in bold .

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 124 papers

Stepwise Token Selection for Efficient Multimodal Large Language Models
Jun 14, 2026Landi He, Shawn Young, Lijian XuEfficient Multimodal Inference Multimodal Large Language Models
Shenzhen University of Advanced Technology
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Jul 28, 2026Yuchen Wang, Qihui Zhu, Yang Liu +2Efficient Multimodal Inference Multimodal Large Language Models
MoE Key Lab of BIPC, NEL-BITA, University of Science and Technology of China, Hefei, China · ChangXin Memory Technologies, Hefei, China · APKL of BIIP, IAI, Hefei Comprehensive National Science Center, Hefei, China
Visual Token Coding for Video Multimodal Large Language Models
Aug 28, 2026Chenxin Fang, Tao Chen, JunChao You +3Multimodal Token Compression Video Token Compression
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China

Jun 14, 2026J/K move · Enter open · S save

Stepwise Token Selection for Efficient Multimodal Large Language Models

Landi He, Shawn Young, Lijian Xu

Shenzhen University of Advanced Technology

Efficient Multimodal Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Open paper AskPDF arXiv

In multimodal large language models (MLLMs), inference cost is largely dominated by the visual token prefix rather than the language backbone, making token reduction a key factor for improving efficiency. Existing approaches typically assign independent importance scores to visual tokens and retain a fixed number of top-ranked tokens, implicitly assuming token independence and a uniform compression ratio across inputs. In this work, we reformulate visual token pruning as a sequential decision-making process. Specifically, we introduce a pointer-style selection mechanism that iteratively chooses informative tokens, conditioning each decision on previously selected ones, and dynamically determines when to stop via a learned termination action. This enables joint optimization of both the selected subset and its size. To enable end-to-end training under standard language modeling objectives, we design a differentiable relaxation based on a variance-preserving noise interpolation scheme, allowing gradients to propagate through the discrete selection process. Extensive experiments on LLaVA-v1.5-7B and Qwen2.5-VL-7B demonstrate that our approach consistently outperforms fixed-ratio baselines across different compression levels. Under aggressive pruning that removes 88.9% of visual tokens, our method preserves 94.6% of the original accuracy while achieving a 1.88x speed-up in prefill latency.

Abstract

Multimodal IR

Explore similar work

Stepwise Token Selection for Efficient Multimodal Large Language Models

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

Visual Token Coding for Video Multimodal Large Language Models