Rethinking Model Efficiency: Multi-Agent Inference with Large Models

Sixun Dong; Juhua Hu; Steven Li; Wei Wen; Qi Qian

cs.CVApr 6, 2026

Rethinking Model Efficiency: Multi-Agent Inference with Large Models

Authors: Sixun Dong, Juhua Hu, Steven Li, Wei Wen, Qi Qian

Organizations: Independent Researcher · University of Washington · Meta Reality Labs

Abstract

Most vision-language models (VLMs) apply a large language model (LLM) as the decoder, where the response tokens are generated sequentially through autoregression. Therefore, the number of output tokens can be the bottleneck of the end-to-end latency. However, different models may require vastly different numbers of output tokens to achieve comparable performance. In this work, we conduct a comprehensive analysis of the latency across different components of VLMs on simulated data. The experiment shows that a large model with fewer output tokens can be more efficient than a small model with a long output sequence. The empirical study on diverse real-world benchmarks confirms the observation that a large model can achieve better or comparable performance as a small model with significantly fewer output tokens. To leverage the efficiency of large models, we propose a multi-agent inference framework that keeps large models with short responses but transfers the key reasoning tokens from the small model when necessary. The comparison on benchmark tasks demonstrates that by reusing the reasoning tokens from small models, it can help approach the performance of a large model with its own reasoning, which confirms the effectiveness of our proposal.

Figures & tables

Panel of Figure 1 — Figure 1: Efficiency emerges with scale. (a) Latency grows almost linearly on the number of output tokens, and larger models have the higher per-token cost. (b)-(d) However, smaller models (2B/4B) require way more tokens to achieve a comparable performance as larger models (8B).

Qwen3-VL	256 2	512 2	1024 2	2048 2	4096 2
2B	0.84	1.49	7.21	64.83	335.19
4B	0.91	1.69	6.99	76.63	343.78
8B	0.92	1.58	6.46	75.59	341.61

Table 1: Qwen3-VL vision preprocessing latency (ms) with different image resolutions.

Qwen3-VL	256 2	512 2	1024 2	2048 2	4096 2
2B	12.69	12.84	24.41	172.28	1995.25
4B	12.36	12.56	24.91	180.10	2104.70
8B	13.79	14.26	34.40	272.90	3343.02

Table 2: Qwen3-VL vision encoder latency (ms) with different image resolutions.

Image Res.	$256^{2}$ (64 Tokens)			$512^{2}$ (256 Tokens)			$1024^{2}$ (1024 Tokens)
Query Tokens	64	256	512	64	256	512	64	256	512
Total Tokens	128	320	576	320	512	768	1088	1280	1536
2B	27.52	27.79	27.92	28.22	28.00	28.89	45.96	46.97	49.80
4B	32.21	32.15	34.66	32.79	32.86	39.02	65.87	69.33	77.86
8B	33.76	36.25	47.93	36.74	43.91	53.45	94.27	102.65	115.31

Table 3: Qwen3-VL Time-to-First-Token latency (ms) with different image resolutions and query token lengths.(Profiled to 8192 tokens, 1536 shown for brevity.)

Image Res.	$256^{2}$ (64 Image Tokens)								$512^{2}$ (256 Image Tokens)								Avg Tok/s
Input Tok.	128				320				320				512
Output Tok.	8	128	256	512	8	128	256	512	8	128	256	512	8	128	256	512
Qwen3-VL-2B	138.73	2030.41	4051.51	8072.48	139.68	2032.67	4052.28	8070.89	139.62	2038.54	4056.94	8080.48	138.45	2021.54	4035.18	8067.07	61.80
Qwen3-VL-4B	176.63	2636.46	5275.21	10515.65	177.41	2642.43	5266.63	10486.47	177.36	2644.82	5267.35	10487.88	176.38	2634.05	5261.63	10503.87	47.77
Qwen3-VL-8B	178.50	2651.51	5282.48	10517.45	180.89	2656.08	5300.42	10540.37	181.69	2660.82	5294.05	10563.77	186.50	2648.22	5271.76	10690.40	47.26

Table 4: Qwen3-VL decoding time with different image resolutions and query token lengths.

Figure 2: Illustration of the proposed multi-agent inference framework. (a) shows our empirical observation that a large model with a short response can achieve a similar performance as the small model with additional reasoning tokens. (b) demonstrates the proposed reasoning transfer strategy that can reuse the reasoning tokens output by the small model for the large model to improve its performance. (c) Our final proposal adopts mutual verification to further reduce the number of expensive model calls for efficient inference.

Qwen3-VL	POPE			MMMU			MMBench			ChartQA			$\text{InfoVQA}_{val}$			RealWorldQA
Qwen3-VL	S	E	R	S	E	R	S	E	R	S	E	R	S	E	R	S	E	R
2B	1.0	105.4	266.1	101.7	1448.9	2019.1	3.7	449.6	569.3	3.5	95.8	256.3	3.4	74.7	192.5	1.4	155.6	244.4
4B	1.0	97.0	168.6	47.8	1122.9	1822.2	2.9	336.1	475.2	3.6	276.1	352.2	3.6	240.7	288.0	1.1	209.0	346.2
8B	1.0	34.4	149.3	45.8	944.6	1548.7	2.2	106.0	426.5	3.5	249.8	294.3	3.5	177.9	228.2	1.0	221.5	297.0

Table 5: Comparison of the averaged number of generated tokens with Qwen3-VL models.

Qwen3-VL	POPE
Qwen3-VL	S	E	E(S2)	R	R(S2)
2B	88.6	88.8	89.2	88.4	89.9
4B	88.7	88.0	89.6	87.8	88.9
8B	87.1	87.3	87.8	88.2	89.5

Table 6: Comparison of performance with Qwen3-VL models on POPE. (S2) denotes the 2-stage strategy as in Eqn. 7 .

Qwen3-VL	MMBench			MMMU			ChartQA			InfoVQA			RealWorldQA
Qwen3-VL	S	E	R	S	E †	R †	S	E	R	S	E	R	S	E	R
2B	76.8	75.4	75.0	44.2	46.9	47.9	77.80	82.72	83.72	68.64	71.50	74.57	60.65	63.79	64.58
4B	83.7	85.0	84.5	48.3	60.0	61.3	83.12	83.04	83.16	79.96	82.87	83.08	71.24	70.59	69.93
8B	85.2	86.0	85.9	55.6	62.0	64.3	84.72	87.68	86.20	82.80	86.37	86.66	67.58	72.68	70.33

Table 7: Comparison of performance with Qwen3-VL models. † denotes original results without 2-stage strategy.

Qwen3-VL	POPE	MMMU	MMBench	ChartQA	InfoVQA	RealWorldQA
2B-R	88.4	47.9	72.8	72.2	71.4	60.5
4B-S	88.7	48.3	83.7	83.1	80.0	71.2
4B-R	88.9	61.3	84.5	83.2	83.1	69.9
4B + 2B-R	91.0 +2.6%	55.9 +15.7%	84.1 +0.5%	82.4 -0.8%	79.7 -0.4%	71.6 +0.6%
8B-S	87.1	55.6	85.2	84.7	82.8	67.6
8B-R	89.5	64.3	85.9	86.2	86.7	70.3

Table 8: Comparison of reasoning transfer that uses reasoning tokens from the 2B/4B model to replace the reasoning tokens for 4B/8B models with the 2-stage strategy as in Eqn. 7 . Percentage changes show improvement ( green ) or degradation ( red ) compared to baseline with the “Simple” prompt, i.e . , “-S”.

Config	8B-S	Sparse Reasoning Transfer								Full
Config	8B-S	L4	L8	L10	L12	L14	L16	L22	L24	Full
Avg. #T	-	145.9	153.6	85.9	90.5	89.0	121.1	112.1	209.5	266.1
F1	87.1	89.3	89.0	88.7	89.2	88.9	89.5	89.6	89.6	89.2

Table 9: Comparison of attention weights from different layers (L#) for sparse reasoning token transfer (8B+2B-R) performance on POPE. Tokens are sampled according to the attention weights and the total weight of sampled tokens is 80% of the original set. #T denotes the number of selected tokens.

Dataset	#Test	Multi-Agent Inference (MAI)				Baseline			Full RT
Dataset	#Test	MV 2B(S)/8B(S)	MV 2B(R)/8B(S)	RT	Final Perf.	2B(S)	2B(R)	8B(S)	8B+2B-R
POPE	9000	8507 (94.52%)	46 (0.51%)	447 (4.97%)	89.4	88.6	88.4	87.1	89.2
MMMU	900	498 (55.33%)	142 (15.78%)	260 (28.89%)	62.0	44.2	47.9	55.6	62.9
MMBench	4329	3736 (86.34%)	262 (6.05%)	330 (7.62%)	85.5	76.8	72.8	85.2	85.1
ChartQA	2500	1800 (72.00%)	195 (7.80%)	505 (20.20%)	86.8	77.8	72.2	84.7	85.3
InfoVQA	2801	1728 (61.69%)	271 (9.68%)	802 (28.63%)	82.9	68.6	71.4	82.8	82.8

Table 10: Multi-Agent Inference based on Qwen3-VL-2B/8B. “MV” denotes the mutual verification and “RT” is for reasoning transfer. The best result is in bold.

Method	POPE	MMBench	MMMU	ChartQA	InfoVQA	RealWorldQA
Full RT (s)	39,865	40,718	43,924	10,805	9,201	561
MAI (s)	4,954	8,676	25,025	4,158	5,094	268
Speedup	8.05 $\times$	4.69 $\times$	1.76 $\times$	2.60 $\times$	1.81 $\times$	2.09 $\times$

Table 11: Comparison of total inference time (s) on benchmarks.

Method	Backend	BS	Accept Rate	Total † (s) $\downarrow$	Latency † (s/sample) $\downarrow$	Speedup	ChartQA Acc. (%)
Method	Backend	BS	Accept Rate	Total † (s) $\downarrow$	Latency † (s/sample) $\downarrow$	Speedup	w/o S2	w/ S2
8B-R	HF	1	N/A	27,195	10.88	1.00 $\times$	77.08	85.52
SD (8B+2B)	HF	1	60.39%	31,540	12.62	0.86 $\times$	77.24	85.40
8B-R	vLLM	32	N/A	1,122	0.449	1.00 $\times$	77.68	86.32
MAI (8B+2B)	vLLM	32	N/A	468	0.187	2.40 $\times$	86.52	-

Table 12: Comparison of decoding strategies on ChartQA. † HF latency excludes model loading, whereas vLLM is end-to-end. All reported latencies exclude S2 overhead, which is required by baselines for best accuracy but avoided by MAI.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

InternVL3.5	POPE			MMMU			MMBench			ChartQA			InfoVQA			RealWorldQA
InternVL3.5	S	E	R	S	E	R	S	E	R	S	E	R	S	E	R	S	E	R
1B	1.1	26.0	39.7	2.5	302.2	425.8	3.9	31.1	82.3	3.5	37.9	66.2	3.5	57.8	42.5	1.4	36.4	69.5
2B	1.0	12.2	45.2	262.2	389.9	508.8	14.0	66.6	163.1	3.5	121.4	143.3	4.4	78.2	137.6	1.8	67.6	122.5
4B	1.0	23.3	57.1	5.9	358.9	491.7	2.3	55.1	154.7	3.6	173.4	200.1	3.9	152.6	181.6	1.3	70.0	103.9
8B	1.1	30.5	46.1	10.6	262.0	385.5	2.2	49.5	141.7	3.9	120.3	158.2	3.5	108.6	142.0	1.6	54.3	75.7
14B	1.4	19.3	47.0	11.2	419.0	484.1	2.5	46.7	161.0	3.6	126.7	147.7	3.4	52.5	117.6	3.8	60.8	71.9

Appendix

Table 13: Comparison of the averaged number of generated tokens with InternVL3.5 models.

InternVL	POPE			MMMU			MMBench			ChartQA			InfoVQA			RealWorldQA
InternVL	S	E	R	S	E $\dagger$	R $\dagger$	S	E	R	S	E	R	S	E	R	S	E	R
1B	83.8	85.6	83.0	39.3	43.0	41.6	68.2	67.8	68.1	70.7	65.5	67.2	56.1	51.9	51.7	51.9	58.3	56.1
2B	88.9	89.3	90.7	51.2	52.0	54.3	75.7	78.4	77.1	80.9	82.7	85.5	41.4	65.5	65.8	50.2	55.4	48.8
4B	88.8	93.1	91.7	55.8	60.4	59.4	76.5	81.7	82.5	83.7	88.6	87.8	68.9	73.6	73.9	59.5	60.7	42.6
8B	88.6	90.7	88.5	56.8	60.9	60.2	74.7	83.0	83.5	77.2	83.9	83.8	57.4	74.7	75.4	59.9	63.8	58.8
14B	85.8	86.5	88.3	58.2	62.2	61.6	82.8	84.0	83.7	87.3	88.8	88.4	69.6	74.1	77.1	52.8	67.7	67.8

Appendix

Table 14: Comparison of performance with InternVL3.5 models.

\dagger

denotes original results without 2-stage strategy.

Dataset	#Test	2B-R			2B-R(S2)			$\Delta$ Final
Dataset	#Test	MV $\|N$	RT $\|N$	Final	MV $\|N$	RT $\|N$	Final	$\Delta$ Final
POPE	9000	46 (0.51%)	447 (4.97%)	88.70	63 (0.70%)	430 (4.78%)	88.79	+0.09
MMMU	900	141 (15.67%)	284 (31.56%)	61.22	63 (7.00%)	362 (40.22%)	61.00	$-0.22$
MMBench	4329	262 (6.05%)	330 (7.62%)	85.48	263 (6.08%)	329 (7.60%)	85.48	0.00
ChartQA	2500	195 (7.80%)	505 (20.20%)	86.80	216 (8.64%)	484 (19.36%)	86.80	0.00
InfoVQA	2801	271 (9.68%)	802 (28.63%)	82.94	278 (9.93%)	795 (28.38%)	83.00	+0.06

Appendix

Table 15: Original 2B-R vs. 2B-R(S2) in the cascade. MV

|N

matches the main table

\mathrm{MV}_{2\mathrm{B(R)}/8\mathrm{B(S)}}

(count; % of all tests). RT

|N

matches the main RT column (count; % of all tests).Final: POPE/F1, InfoVQA/mean ANLS

\times 100

, others/accuracy.

Panel of Figure 3 — Figure 3: Illustration of total attention weights of all reasoning tokens and sparsity within reasoning tokens averaged over 32 heads. Sparsity is measured by the ratio of reasoning tokens that contribute to 80% total attention weights of reasoning tokens.

Layer	Model	N-Tok (80%)	1st	2nd	3rd	4th	5th	6th	7th	8th	9th	10th	11th	12th	13th	14th	15th
L1	8B+2BR	45.0% (54)	.	a	,	.	,	,	,	the	in	the	,	the	and	.	)
	8B+2BR	45.0% (54)	11.1%	3.1%	3.0%	2.9%	2.9%	2.9%	2.6%	2.6%	2.4%	2.3%	2.1%	2.0%	2.0%	1.8%	1.7%
	8B+8BR	40.6% (104)	.	.	the	,	of	.	).	:	,	,	the	.	,	in	and
	8B+8BR	40.6% (104)	7.5%	3.5%	3.2%	2.8%	2.2%	2.2%	2.1%	2.0%	2.0%	1.8%	1.7%	1.5%	1.4%	1.4%	1.3%
L6	8B+2BR	21.7% (26)	the	.	the	the	.	.	.	Therefore	.	,	the	The	picture	,	the
	8B+2BR	21.7% (26)	19.1%	17.6%	16.2%	3.7%	3.5%	3.3%	1.6%	1.5%	1.2%	1.0%	0.9%	0.9%	0.9%	0.9%	0.8%

Appendix

Table 16: Comparison of top tokens selected by 8B + 2B-R and 8B + 8B-R across different layers. Evidently, deeper layers focus on semantic reasoning tokens, where similar tokens can be selected from 2B-R and 8B-R tokens.

Figure 4: Head-wise attention weights of total reasoning tokens across different layers.

Figure 5: Head-wise sparsity within reasoning tokens across different layers.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
Apr 7, 2026Jun Zhang, Yicheng Ji, Feiyang Ren +7Efficient VLM Inference Large Vision-Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Empirical Recipes for Efficient and Compact Vision-Language Models
Mar 17, 2026Jiabo Huang, Zhizhong Li, Sina Sajadmanesh +2Vision-Language Models Efficient VLM Inference
Sony AI.
Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding
Mar 25, 2026Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella +5Efficient VLM Inference Large Vision-Language Models
Georgia Institute of Technology · Cisco Research

Apr 7, 2026J/K move · Enter open · S save

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li+6

The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security

Efficient VLM Inference Large Vision-Language Models Long-Context VLMs

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance. This overhead is driven by a multi-regime interplay between high-resolution feature extraction, quadratic attention scaling, and memory bandwidth constraints. We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding. Unlike prior reviews focused on isolated optimizations, we analyze the end-to-end pipeline to reveal how upstream decisions dictate downstream bottlenecks, covering compute-bound visual encoding, the intensive prefilling of massive contexts, and the ''visual memory wall'' in bandwidth-bound decoding. By decoupling the efficiency landscape into the axes of shaping information density, managing long-context attention, and overcoming memory limits, this work provides a structured analysis of how isolated optimizations compose to navigate the trade-off between visual fidelity and system efficiency. The survey concludes by outlining four future frontiers supported by pilot empirical insights, including hybrid compression based on functional unit sensitivity, modality-aware decoding with relaxed verification, progressive state management for streaming continuity, and stage-disaggregated serving through hardware-algorithm co-design. Our literature repository is at https://github.com/SuDIS-ZJU/Efficient-LVLMs-Inference.

Abstract

Efficient VLM Inference

Explore similar work

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

Empirical Recipes for Efficient and Compact Vision-Language Models

Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding