Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

Wenjie Huang; Jun Zhang; Hongyaoxing Gu; Haoyang Huang; Tianqi Xu; Xinyue Hao; Xinyi Hu; Xinle Du

cs.CLOct 29, 2026

Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

Authors: Wenjie Huang, Jun Zhang, Hongyaoxing Gu, Haoyang Huang, Tianqi Xu, Xinyue Hao, Xinyi Hu, Xinle Du,

Organizations: Zhejiang University · Alibaba Group · University of the Chinese Academy of Sciences · Carnegie Mellon University ·

Abstract

Multi-turn conversations with multimodal large language models (MLLMs) reuse input context, but long multimodal inputs increase time to first token (TTFT). Query-aware hard pruning reduces prefill tokens but permanently excludes information that later questions may require. We propose Raincheck, a framework for deferred multimodal prefill that decouples initial token selection from later information availability. An existing pruning method selects tokens for initial prefill; Raincheck preserves and progressively prefills unselected tokens in source order using unused compute capacity during decode. Unfinished prefill continues after answer return during available scheduling gaps. Attention visibility control and affine state mapping enable context updates after generation begins in full-attention and hybrid recurrent models. Prefill-decode fusion reduces repeated weight reads. We evaluate Qwen3-VL-8B and Qwen3.5-9B with 30% initial multimodal-token retention. Across both models and four pruning methods, Raincheck recovers on average 77.7% and 60.4% of the quality lost to matched hard pruning on MultiVerse and ConvBench, respectively. On MultiVerse, the SGLang implementation reduces mean first-turn TTFT by up to 24.1% relative to full prefill, with less than 0.5% increase in mean time per output token (TPOT).

Figures & tables

Figure 1: Schematic overview of Raincheck . Full prefill ( Full ) processes all input tokens before generation. Hard pruning ( Hard ) discards unselected tokens to shorten initial prefill. Raincheck ( Ours ) prefills important tokens first, processes deferred tokens during decode, and completes unfinished prefill after answer return. The next turn reuses the completed context. Blue , purple , black , and green denote important multimodal, deferred multimodal, fixed text/control, and generated tokens. Right: ConvBench mean score loss (%) relative to Full on Qwen3.5-9B with MMTok, 30% initial retention, and 31-token deferred chunks.

Figure 2: Motivation for deferred multimodal prefill: (a) later questions may need tokens omitted earlier; (b) unused compute capacity during decode allows deferred tokens to be processed; (c) time between turns can be used to finish deferred prefill.

Panel of Figure 3 — Figure 3: A later question targets a position omitted when pruning for the first question. Purple marks the jersey number; faded cells mark omitted tokens.

Panel of Figure 4 — Figure 4: Token-selection changes at 30% retention. Curves show the share of dialogues where first-question selection omits more than the specified percentage of (a) second-question selections or (b) question-and-answer selections.

Panel of Figure 5 — Figure 5: TPOT by deferred chunk size versus position-matched decode-only intervals.

Panel of Figure 6 — Figure 6: Multimodal follow-up message intervals in ShareChat and tool-call durations in TraceLab.

Symbol	Meaning
$K,D$	Important / deferred tokens
$\mathcal{T}_{\mathrm{tail}}$	Intervening input tokens
$Y$	Processed output tokens
$S_{\mathrm{prefix}}$	Prefix state
$S_{\mathrm{live}}$	Live state
$P_{\mathrm{tail}}$	Tail transition matrix

Table 1: Core method notation.

Figure 7: Deferred-prefill attention.

Figure 8: GDN state correction.

Model	Pruning method	Policy	MultiVerse				ConvBench
Model	Pruning method	Policy	T1	T2	T3	Avg	T1	T2	T3	Mean
Qwen3-VL-8B	N/A	Full	47.68	53.81	58.11	55.43	50.95	54.94	25.56	43.82
	MMTok †	Hard	32.78 (-31.3%)	46.92 (-12.8%)	55.20 (-5.0%)	48.77 (-12.0%)	40.55 (-20.4%)	48.09 (-12.5%)	25.65 (+0.4%)	38.10 (-13.1%)
	MMTok †	Ours	45.97 (-3.6%)	53.39 (-0.8%)	58.96 (+1.5%)	55.22 (-0.4%)	47.05 (-7.7%)	53.64 (-2.4%)	24.78 (-3.1%)	41.83 (-4.5%)
	HiPrune †	Hard	42.37 (-11.1%)	52.13 (-3.1%)	58.52 (+0.7%)	53.67 (-3.2%)	45.93 (-9.9%)	51.99 (-5.4%)	24.09 (-5.8%)	40.67 (-7.2%)
	HiPrune †	Ours	46.92 (-1.6%)	51.85 (-3.6%)	58.39 (+0.5%)	55.13 (-0.5%)	49.65 (-2.6%)	53.81 (-2.1%)	24.18 (-5.4%)	42.55 (-2.9%)

Table 2: Multi-turn quality at 30% initial retention. Parentheses show relative changes from Full ; bold marks the higher score in each Hard / Ours pair. Avg and Mean follow the aggregation in Section 3.1 .

Model	Policy	T1		T2		T3		Overall
Qwen3-VL-8B	Full	25.74		59.15		48.78		53.15
	Hard	25.08	(-2.6%)	57.44	(-2.9%)	46.26	(-5.2%)	50.68	(-4.6%)
	Ours	24.92	(-3.2%)	56.60	(-4.3%)	46.88	(-3.9%)	51.73	(-2.7%)
Qwen3.5-9B	Full	25.66		47.17		39.96		44.28
	Hard	26.07	(+1.6%)	45.14	(-4.3%)	41.19	(+3.1%)	42.87	(-3.2%)
	Ours	26.48	(+3.2%)	47.29	(+0.3%)	41.44	(+3.7%)	44.13	(-0.3%)

Table 3: MT-Video-Bench quality with MMTok at 30% initial retention. Overall includes all annotated turns; parentheses and bold follow Table 2 .

Policy	TTFT (ms)		TPOT (ms)		E2E (s)
MultiVerse: Qwen3-VL-8B
Full	169.94		6.1919		3.3340
Hard	133.17	(-21.64%)	6.1354	(-0.91%)	3.2684	(-1.97%)
Ours	128.96	(-24.11%)	6.1996	(+0.13%)	3.2970	(-1.11%)
MultiVerse: Qwen3.5-9B
Full	171.20		6.4934		3.4893

Table 4: Mean first-turn inference latency with matched outputs. Parentheses show relative changes from Full .

Model	Dataset	TPOT during restoration (ms)	Unfinished at answer completion		Post-response time (ms)
Model	Dataset	TPOT during restoration (ms)	Unfinished at answer completion		Mean	P95
Qwen3-VL-8B	MultiVerse	7.201	0/288	(0.00%)	0.049	0.060
Qwen3-VL-8B	MT-Video-Bench	8.685	120/288	(41.67%)	74.024	357.703
Qwen3.5-9B	MultiVerse	7.882	0/288	(0.00%)	0.089	0.100
Qwen3.5-9B	MT-Video-Bench	8.064	198/288	(68.75%)	162.802	371.782

Table 5: Deferred-prefill completion on 512-token MultiVerse traces and naturally terminated Full video traces replayed across policies. Restoration TPOT includes only deferred-prefill intervals; post-response statistics include all first-turn requests, including those already fully restored.

Panel of Figure 9 — Figure 9: Quality across initial retention ratios and deferred-prefill chunk sizes with MMTok. MV denotes MultiVerse; Video denotes MT-Video-Bench. Panels use separate vertical scales; MultiVerse shading shows 95% marginal confidence intervals.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Type / shape	Definition
Input and execution
$X,K,D$	Token sequences	Targeted multimodal tokens; important and deferred partitions in source order.
$\mathcal{T}_{\mathrm{tail}}$	Token sequence	Processed input between the insertion boundary and generated tokens, including text, control, or special tokens.
$\mathcal{T},I_{0}$	Token sequences	All fixed text/control/special tokens; initial input combining $\mathcal{T}$ and $K$ in source order.
$N_{m},b_{m},\rho$	Counts; scalar ratio	Span size, initial retained count, deferred fraction; $b_{m}=N_{m}-\lfloor\rho N_{m}\rfloor$ .
$I_{t}$	Token sequence	Materialized input before step $t$ : $I_{t}=I_{0}\\|D_{<t}$ .

Appendix

Table 6: Complete notation with object types and tensor shapes.

Component	Compared with Full	Compared with Hard
KV after full recovery	Same token count	$\|D\|b_{\mathrm{KV}}$ additional bytes
Deferred input features	Longer residency of cached features; no duplicate allocation required	Same residency extension if features would otherwise be evicted
GDN state mapping	Additional prefix checkpoints, transition matrices, join buffers, and convolution history	Same additional state structures
Scheduling metadata	Deferred indices, progress counters, and cache mappings	Same additional structures
Execution buffers	Fused-step and completion buffers; peak depends on buffer reuse and graph capture	Additional prefill buffers beyond decode-only execution

Appendix

Table 7: Component-level allocation requirements for Raincheck relative to the baselines at matched input and output lengths.

Suffix $\|Y\|$ (tokens)	Map (MiB)	Direct step ( $\mu$ s/token)	Online step ( $\mu$ s/token)	Increment ( $\mu$ s/token)	Join ( $\mu$ s)	Relative error (%)
64	2	5.242	7.323	2.081	10.601	0.0672
256	2	5.359	7.483	2.124	10.747	0.0749
1024	2	5.439	7.460	2.021	10.715	0.0851
4096	2	5.411	7.532	2.121	10.812	0.0775

Appendix

Table 8: Online GDN state-map scaling and numerical validation under identical recorded factors.

Dataset and split	Population	Reported metric	Task coverage
GQA testdev balanced	12,578 questions	Accuracy	Objects, attributes, spatial relations, and compositional reasoning
TextVQA validation	5,000 questions	VQA accuracy	Reading scene text and combining text with visual evidence
ScienceQA-IMG test	2,017 questions	Accuracy	Science questions with images and knowledge-based reasoning
MMBench EN dev	4,329 questions	CircularEval accuracy	Multiple-choice visual perception and reasoning
MathVerse testmini	3,940 samples	Proxy correctness	Five visual variants of mathematical reasoning problems
MultiVerse	647 dialogues, 2,536 scored turns	Checklist-weighted score	Image-grounded understanding across successive questions

Appendix

Table 9: Benchmark tasks, evaluated populations, and scoring metrics.

Parameter	Value
Generation mode	Non-thinking categorical sampling, one beam
Sampling switch	do_sample=true ; num_beams=1
Temperature	0.7
Top- $p$ / top- $k$	0.8 / 20
Min- $p$	0.0, which disables additional minimum-probability filtering
Presence / frequency / repetition penalty	1.5 / 0.0 / 1.0

Appendix

Table 10: Quality-generation settings and fixed task budgets for Qwen3-VL-8B and Qwen3.5-9B.

Model	Pruning method	Policy	GQA		TextVQA		SQA-IMG		MMBench		MathVerse
Qwen3-VL-8B	N/A	Full	60.56		79.10		90.83		88.96		45.61
	MMTok †	Hard	56.92	(-6.0%)	68.32	(-13.6%)	85.32	(-6.1%)	87.16	(-2.0%)	40.18	(-11.9%)
	MMTok †	Ours	56.92	(-6.0%)	68.32	(-13.6%)	85.32	(-6.1%)	87.13	(-2.1%)	42.61	(-6.6%)
	HiPrune †	Hard	58.04	(-4.2%)	75.73	(-4.3%)	88.94	(-2.1%)	88.50	(-0.5%)	43.86	(-3.8%)
	HiPrune †	Ours	58.06	(-4.1%)	75.73	(-4.3%)	88.94	(-2.1%)	88.50	(-0.5%)	44.64	(-2.1%)
	DivPrune †	Hard	59.08	(-2.4%)	74.50	(-5.8%)	86.07	(-5.2%)	87.80	(-1.3%)	41.62	(-8.7%)

Appendix

Table 11: Single-turn quality at 30% initial multimodal-token retention. Parentheses and bold follow Table 2 . Appendix C.2 defines the benchmark-specific evaluators.

Model	Pruning method	Hard (%)	Ours (%)
Qwen3-VL-8B	MMTok	30.07	38.84
	HiPrune	30.07	38.76
	DivPrune	30.07	38.60
	VisionZip-Select	30.07	38.77
Qwen3.5-9B	MMTok	30.07	43.24
	HiPrune	30.07	43.27

Appendix

Table 12: Decode-active multimodal-token ratios, averaged equally across the five single-turn datasets. Initial retention averages 30.07%.

Dataset	Pruning method	Full [95% CI]	Hard [95% CI]	Ours [95% CI]	$\Delta$
MMBench	MMTok	88.96 [88.03, 89.88]	87.16 [86.16, 88.15]	87.13 [86.12, 88.13]	$-0.03$
	HiPrune		88.50 [87.55, 89.44]	88.50 [87.53, 89.44]	$0.00$
	DivPrune		87.80 [86.86, 88.77]	87.83 [86.83, 88.80]	$+0.03$
	VisionZip-Select		88.33 [87.39, 89.30]	88.33 [87.36, 89.28]	$0.00$
MathVerse	MMTok	45.61 [44.06, 47.16]	40.18 [38.65, 41.73]	42.61 [41.07, 44.19]	$+2.43$
	HiPrune		43.86 [42.28, 45.41]	44.64 [43.10, 46.19]	$+0.78$

Appendix

Table 13: MMBench and MathVerse scores for Qwen3-VL-8B with cell-level 95% intervals. Delta is Ours minus Hard in score points.

Dataset	Pruning method	Full [95% CI]	Hard [95% CI]	Ours [95% CI]	$\Delta$
MMBench	MMTok	90.58 [89.70, 91.45]	87.92 [86.95, 88.87]	88.06 [87.06, 89.03]	$+0.14$
	HiPrune		89.74 [88.84, 90.62]	89.93 [89.00, 90.81]	$+0.19$
	DivPrune		88.33 [87.34, 89.28]	88.38 [87.41, 89.33]	$+0.05$
	VisionZip-Select		89.67 [88.77, 90.58]	89.74 [88.84, 90.64]	$+0.07$
MathVerse	MMTok	64.34 [62.84, 65.84]	47.84 [46.27, 49.39]	64.67 [63.20, 66.14]	$+16.83$
	HiPrune		61.22 [59.70, 62.69]	64.62 [63.12, 66.09]	$+3.40$

Appendix

Table 14: MMBench and MathVerse scores for Qwen3.5-9B with cell-level 95% intervals. Delta is Ours minus Hard in score points.

Model	Answer refusal	Memory recall	Object reference	Proactive interaction	Summary	Topic shifting
Qwen3-VL-8B	$-0.73$	$+3.18$	$+3.38$	$+0.40$	$+1.01$	$-0.90$
Qwen3.5-9B	$+3.17$	$-0.13$	$+1.55$	$+0.82$	$+2.66$	$-0.55$

Appendix

Table 15: MT-Video-Bench Ours minus Hard differences in score points. Columns show the six benchmark categories.

Policy	Answer excerpt	Score
Turn 4: How is marble formed?
Full	“Marble is a metamorphic rock that forms from the recrystallization of limestone or dolomite under high temperatures and pressures.”	60
Ours	“The distinct patterns and colors often seen in marble result from impurities present in the original limestone or dolomite.”	80
Turn 5: What happens inside the tank when the setup is turned on?
Full	“a blue light illuminates the tank and bubbles are actively rising from the internal structure, indicating that an aeration or filtration system has been activated.”	40
Ours	“water begins to flow through the PVC pipe maze inside the tank.”	20

Appendix

Table 16: Qwen3.5 answer excerpts and full-answer checklist scores on MT-Video-Bench with 64-token restoration. Questions are shortened; excerpts are verbatim.

Model	Retention (%)	Full	Hard	Ours	Gain [95% CI]
Qwen3-VL-8B	30	55.36	48.92	55.22	$+6.30$ –
	50	55.36	50.32	55.39	$+5.07$ –
	70	55.36	53.35	55.26	$+1.91$ –
Qwen3.5-9B	30	44.80	37.24	43.12	$+5.88$ –
	50	44.80	39.41	43.70	$+4.29$ –
	70	44.80	41.77	43.75	$+1.98$ –

Appendix

Table 17: MMTok initial-retention quality sweep on MultiVerse: 647 dialogues and 2,536 scored turns per cell. Ours uses a 16-token guard and

c=31

. The final column gives score gains; dashes mark unavailable paired intervals.

Model	Retention (%)	Full	Hard	Ours	Gain [95% CI]
Qwen3-VL-8B	30	43.82	38.10	41.83	$+3.73$ –
	50	43.82	38.45	40.73	$+2.28$ –
	70	43.82	40.01	41.19	$+1.18$ –
Qwen3.5-9B	30	44.57	35.53	43.79	$+8.26$ –
	50	44.57	39.46	43.13	$+3.67$ –
	70	44.57	40.87	43.44	$+2.57$ –

Appendix

Table 18: MMTok initial-retention quality sweep on ConvBench. Each cell contains all 577 dialogues. Scores use the caption-conditioned, independent-turn, dual-order protocol in Appendix C.2 . The final column gives score gains; dashes mark unavailable paired intervals.

Model	Retention (%)	Full	Hard	Ours	Gain [95% CI]
Qwen3-VL-8B	30	53.15	50.68	51.73	$+1.05$ [ $0.05$ , $2.03$ ]
	50	53.15	51.12	52.53	$+1.41$ [ $0.60$ , $2.23$ ]
	70	53.15	51.85	52.22	$+0.37$ [ $-0.53$ , $1.30$ ]
Qwen3.5-9B	30	44.28	42.87	44.13	$+1.26$ [ $0.34$ , $2.20$ ]
	50	44.28	43.62	44.86	$+1.24$ [ $0.08$ , $2.43$ ]
	70	44.28	43.91	44.57	$+0.66$ [ $-0.40$ , $1.75$ ]

Appendix

Table 19: MMTok initial-retention quality sweep on MT-Video-Bench. Each cell contains the same 119 videos, 902 dialogues, and 5,311 turns. The final column gives score gains with 95% paired video-cluster intervals.

Model	$c$	Full	Hard	Ours	Gain [95% CI]
Qwen3-VL-8B	7	52.12	47.16	52.96	$+5.80$ [ $3.52$ , $8.14$ ]
	15	52.40	47.12	53.35	$+6.23$ [ $3.65$ , $8.78$ ]
	31	52.20	47.61	51.54	$+3.93$ [ $1.50$ , $6.32$ ]
	63	52.34	47.39	53.14	$+5.75$ [ $3.30$ , $8.18$ ]
Qwen3.5-9B	7	41.73	36.17	38.90	$+2.73$ [ $-0.29$ , $5.78$ ]
	15	41.79	36.31	40.61	$+4.30$ [ $1.34$ , $7.41$ ]

Appendix

Table 20: MultiVerse chunk-quality sensitivity on 128 matched dialogues using MMTok. Scores use the DeepSeek-V4.1-Flash protocol in Appendix C.2 . The final column reports paired Ours gains over Hard with 95% dialogue-cluster intervals.

Model	$c$	Full	Hard	Ours	Gain [95% CI]
Qwen3-VL-8B	7	43.49	35.29	40.89	$+5.60$ [ $2.34$ , $8.85$ ]
	15	42.06	35.16	40.10	$+4.94$ [ $1.69$ , $8.33$ ]
	31	43.49	36.46	41.93	$+5.47$ [ $1.95$ , $8.98$ ]
	63	43.10	36.85	41.67	$+4.82$ [ $1.43$ , $8.20$ ]
Qwen3.5-9B	7	42.45	33.07	42.45	$+9.38$ [ $5.47$ , $13.28$ ]
	15	42.19	33.98	41.15	$+7.17$ [ $4.03$ , $10.42$ ]

Appendix

Table 21: ConvBench chunk-quality sensitivity on 128 matched dialogues using MMTok. Scores follow the ConvBench protocol in Appendix C.2 . The final column reports paired Ours gains over Hard with 95% dialogue-cluster intervals.

Model	$c$	Full	Hard	Ours	$\Delta$ from $c=64$ [95% CI]
Qwen3-VL-8B	7	53.15	50.68	51.93	$-0.01$ [ $-0.87$ , $0.81$ ]
	15	53.15	50.68	51.81	$-0.13$ [ $-0.98$ , $0.67$ ]
	31	53.15	50.68	51.73	$-0.21$ [ $-1.03$ , $0.57$ ]
	63	53.15	50.68	52.27	$+0.33$ [ $-0.39$ , $1.05$ ]
	64	53.15	50.68	51.94	Reference
Qwen3.5-9B	7	44.28	42.87	44.59	$-0.85$ [ $-1.67$ , $-0.03$ ]

Appendix

Table 22: MT-Video-Bench restoration-chunk sensitivity at 30% initial retention. Each cell contains the same 119 videos, 902 dialogues, and 5,311 turns. The final column compares each chunk setting with the 64-token configuration using 95% paired video-cluster intervals.

Model	$c$	Context adherence [95% CI]	Dialogue $S_{0}$ [95% CI]
Qwen3-VL-8B	7	$+1.86$ [ $-1.07$ , $4.79$ ]	$+3.71$ [ $0.00$ , $7.62$ ]
	15	$+3.81$ [ $1.27$ , $6.35$ ]	$+9.18$ [ $5.27$ , $13.28$ ]
	31	$+3.81$ [ $0.88$ , $6.93$ ]	$+6.25$ [ $1.95$ , $10.55$ ]
	63	$+2.44$ [ $-0.49$ , $5.47$ ]	$+2.54$ [ $-1.76$ , $7.03$ ]
Qwen3.5-9B	7	$+3.03$ [ $0.00$ , $5.96$ ]	$+8.20$ [ $3.32$ , $13.09$ ]
	15	$+5.66$ [ $2.25$ , $8.98$ ]	$+4.69$ [ $0.00$ , $9.38$ ]

Appendix

Table 23: ConvBench supporting endpoints on the 128-dialogue chunk cohort. Entries report the Ours preference score against Hard minus 50, with 95% dialogue-cluster intervals.

Turn	Full	Hard	Post-answer	Ours	Difference [95% CI]
Qwen3-VL-8B
First	47.88	32.46	32.46	45.97	$+13.51\ [11.54,15.54]$
Second	53.96	46.55	49.62	53.04	$+3.42\ [1.80,5.00]$
Third	57.87	55.27	56.99	58.40	$+1.41\ [-0.01,2.81]$
Fourth	62.60	61.62	62.39	63.47	$+1.08\ [-0.26,2.39]$
Follow-up	58.02	54.28	56.16	58.16	$+2.00\ [1.14,2.83]$

Appendix

Table 24: Restoration timing on MultiVerse. Scores use a 0 to 100 scale. The final column reports the paired score difference between Ours and Post-answer , with a 95% confidence interval.

Model	Turn	Metric	Hard vs. Full	Ours vs. Full	Ours vs. Hard
Qwen3-VL-8B	T1	TTFT	$-21.64\%$ [ $-24.97$ , $-18.65$ ]	$-24.11\%$ [ $-27.81$ , $-20.15$ ]	$-3.16\%$ [ $-7.09$ , $+1.44$ ]
	T1	TPOT	$-0.91\%$ [ $-1.14$ , $-0.71$ ]	$+0.13\%$ [ $-0.01$ , $+0.28$ ]	$+1.05\%$ [ $+0.72$ , $+1.42$ ]
	T2	TTFT	$-0.09\%$ [ $-0.84$ , $+0.65$ ]	$-0.45\%$ [ $-1.29$ , $+0.36$ ]	$-0.36\%$ [ $-1.04$ , $+0.32$ ]
	T2	TPOT	$-0.75\%$ [ $-0.89$ , $-0.63$ ]	$+0.03\%$ [ $-0.04$ , $+0.10$ ]	$+0.79\%$ [ $+0.66$ , $+0.92$ ]
	T3	TTFT	$+0.46\%$ [ $+0.07$ , $+0.88$ ]	$+0.80\%$ [ $+0.27$ , $+1.43$ ]	$+0.34\%$ [ $-0.29$ , $+1.03$ ]
	T3	TPOT	$-0.50\%$ [ $-0.72$ , $-0.31$ ]	$-0.03\%$ [ $-0.10$ , $+0.05$ ]	$+0.47\%$ [ $+0.28$ , $+0.70$ ]

Appendix

Table 25: Paired SGLang timing comparisons. Entries report candidate-minus-baseline relative changes with 95% dialogue-cluster confidence intervals; negative values indicate lower latency.

Model	Population	Metric	Full	Hard	Ours	Ours vs. Full	Ours vs. Hard
Qwen3-VL-8B	All turns	TTFT	56.245	46.965	45.919	$-18.358\%$	$-2.226\%$
	All turns	TPOT	6.278	6.231	6.279	$+0.010\%$	$+0.771\%$
	Continuations	TTFT	17.813	17.825	17.849	$+0.201\%$	$+0.134\%$
	Continuations	TPOT	6.308	6.263	6.306	$-0.028\%$	$+0.680\%$
Qwen3.5-9B	All turns	TTFT	69.788	62.523	62.090	$-11.030\%$	$-0.692\%$
	All turns	TPOT	6.526	6.508	6.536	$+0.157\%$	$+0.438\%$

Appendix

Table 26: All-turn and continuation aggregates for the paired SGLang experiment. Each model-policy pair contains 1,140 all-turn observations and 852 continuations, including T4 when present. Policy columns report mean latency in milliseconds. The final columns report relative mean changes.

Model	Blocking response		Cooperative response		Reduction		Response-to-prefix
Model	Mean	P95	Mean	P95	Estimate	[95% CI]	Median	P95
Qwen3-VL-8B	557.3	989.7	490.6	805.3	12.0%	[10.3%, 13.6%]	43.6	222.5
Qwen3.5-9B	694.9	1213.4	626.5	1032.9	9.8%	[8.8%, 10.9%]	81.5	216.9

Appendix

Table 27: Independent 16-token Transformers completion stress test. Latencies are in milliseconds. The reduction compares cooperative with blocking response-ready latency.

Model	Policy	T1 TTFT (s)	T1 TPOT (ms)	T2 TTFT (ms)	T3+ TTFT (ms)	Later TPOT (ms)
Qwen3-VL-8B	Full	3.569	6.680	18.15	18.01	6.759
	Hard	3.102	6.312	18.39	18.30	6.418
	Ours	3.107	8.014	91.76	18.00	6.762
Qwen3.5-9B	Full	3.480	6.631	34.80	34.35	6.647
	Hard	3.062	6.471	35.08	34.62	6.507
	Ours	3.063	7.622	196.92	34.30	6.653

Appendix

Table 28: Video timing means on the fixed 48-dialogue cohort. First-turn TTFT is in seconds; all other entries are in milliseconds. T2 is submitted immediately after T1, with no imposed think time, and includes parent-prefix waiting. T3+ reports later turns separately. Each TPOT entry averages per-request mean inter-token intervals.

Model	$c$	Mean TPOT (ms)			TTFT $\Delta$ (%)	TPOT $\Delta$ (%) [95% CI]
Model	$c$	Full	Hard	Ours	TTFT $\Delta$ (%)	TPOT $\Delta$ (%) [95% CI]
Qwen3-VL-8B	7	6.237	6.184	6.261	$-24.46$	$+0.386$ [ $+0.187$ , $+0.595$ ]
	15	6.216	6.154	6.219	$-25.00$	$+0.052$ [ $-0.106$ , $+0.224$ ]
	31	6.212	6.154	6.222	$-24.30$	$+0.157$ [ $-0.001$ , $+0.320$ ]
	63	6.211	6.155	6.237	$-24.09$	$+0.432$ [ $+0.197$ , $+0.689$ ]
Qwen3.5-9B	7	6.491	6.465	6.664	$-18.39$	$+2.669$ [ $+1.985$ , $+3.466$ ]

Appendix

Table 29: First-turn chunk-size timing sensitivity. Changes are Ours relative to the same run’s Full baseline; brackets give 95% paired intervals. Each policy has 96 observations from 48 dialogues.

Model	Retention (%)	Mean TPOT (ms)			TTFT $\Delta$ (%)	TPOT $\Delta$ (%) [95% CI]
Model	Retention (%)	Full	Hard	Ours	TTFT $\Delta$ (%)	TPOT $\Delta$ (%) [95% CI]
Qwen3-VL-8B	30.06	6.212	6.154	6.222	$-24.30$	$+0.157$ [ $-0.001$ , $+0.320$ ]
	50.01	6.228	6.183	6.231	$-17.32$	$+0.052$ [ $-0.091$ , $+0.197$ ]
	70.08	6.228	6.188	6.222	$-10.77$	$-0.103$ [ $-0.237$ , $+0.032$ ]
Qwen3.5-9B	30.06	6.499	6.469	6.524	$-18.22$	$+0.381$ [ $+0.206$ , $+0.567$ ]
	50.01	6.496	6.464	6.507	$-13.69$	$+0.160$ [ $+0.040$ , $+0.288$ ]

Appendix

Table 30: First-turn retention timing sensitivity at

c=31

and a 16-token guard. Retention is the fraction of visual tokens retained, pooled over the cohort. Intervals follow Table 29 .

Model	Policy	Mean TTFT (ms)	Mean TPOT (ms)
Qwen3-VL-8B	Full	163.135	6.228
	Hard	125.963	6.176
	Ours , fused	125.648	6.238
	Ours , serial	121.181	6.630
Qwen3.5-9B	Full	164.611	6.479
	Hard	136.965	6.452

Appendix

Table 31: Four-policy first-turn fusion control, measured in a separate batch with 48 dialogues and two repetitions per policy. Both Raincheck schedules restore all deferred tokens.

Model	$n$	Mean [95% CI]	$n$	Mean [95% CI]
	All answers		High-quality answers
Qwen3-VL-8B	48	47.14 [43.55, 50.74]	12	41.95 [35.59, 48.28]
Qwen3.5-9B	48	42.48 [38.35, 46.61]	11	39.39 [32.26, 47.61]

Appendix

Table 32: Answer-conditioned selection differences at 30% retention. Values report the percentage of answer-conditioned selections absent from the question-only selection. High-quality answers satisfy every reference checklist item and receive a quality score of at least 7/10.

(a) ShareChat multimodal follow-up intervals
Cleaning unit	Count	P5 (s)	Median (s)	P75 (s)	$\geq$ 60 s
Conversation	102	18.1	101.4	387.6	72.5%

Appendix

Table 33: ShareChat multimodal follow-up message intervals and TraceLab tool-call durations. (a) ShareChat quantiles. (b) Duration bins for the 102 cleaned ShareChat observations. (c) Published TraceLab positive tool-call durations.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 231 papers

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models
Sep 12, 2026Hansen Zhang, Landi He, Mingde Yao +1Efficient VLM Inference Visual Token Pruning
Shenzhen University of Advanced Technology, Shenzhen, China · School of Computer Science, Nanjing University, Nanjing, China · CUHK MMLab, CPII under InnoHK, Hong Kong, China
Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM Evaluation Confidence Estimation for VLMs
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Jul 28, 2026Yuchen Wang, Qihui Zhu, Yang Liu +2Efficient Multimodal Inference Multimodal Large Language Models
MoE Key Lab of BIPC, NEL-BITA, University of Science and Technology of China, Hefei, China · ChangXin Memory Technologies, Hefei, China · APKL of BIIP, IAI, Hefei Comprehensive National Science Center, Hefei, China

Sep 12, 2026J/K move · Enter open · S save

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models

Hansen Zhang, Landi He, Mingde Yao, Lijian Xu

Shenzhen University of Advanced Technology, Shenzhen, China · School of Computer Science, Nanjing University, Nanjing, China · CUHK MMLab, CPII under InnoHK, Hong Kong, China

Efficient VLM Inference Visual Token Pruning LVLM Pruning Multimodal Large Language Models

Open paper AskPDF arXiv

Visual prefixes account for a major portion of the per-layer computation in multimodal large language models (MLLMs), making visual-token pruning a direct approach to accelerating inference. Existing top-K methods typically evaluate tokens independently and apply a uniform budget to all inputs, overlooking both selection-dependent interactions and variations in visual complexity across samples. In contrast, we propose StepPrune, which formulates visual-token pruning as an adaptive sequential decision process. Conditioned on previously selected tokens and textual context, StepPrune progressively constructs the retained subset and automatically determines its size through a learned STOP action. During training, a variance-preserving noise gate provides a differentiable surrogate for the discrete selection process, whereas during inference, unselected tokens are physically removed before language-model prefill. A grouped selection mechanism further extends StepPrune to high-resolution inputs. Experiments across LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, and InternVL3 show that StepPrune achieves the best average normalized performance retention across all evaluated pruning rates on LLaVA-1.5, Qwen2.5-VL, and InternVL3, while remaining competitive on the substantially longer AnyRes prefixes of LLaVA-NeXT. On LLaVA-1.5, StepPrune retains 94.6% of the full-prefix normalized performance while pruning 88.9% of the visual tokens. At a mean retained count of 64, StepPrune reduces prefill latency from 59.95 ms to 40.05 ms, corresponding to a 1.50x prefill speed-up.

Abstract

Efficient Multimodal Inference

Explore similar work

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models

Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models