VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

Yuan Feng; Qize Yang; Ruizhe Chen; Sibo Song; Haolin He; Muzhi Zhu; Zihan Liu; Yunfei Chu

cs.CVOct 6, 2026

VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

Authors: Yuan Feng, Qize Yang, Ruizhe Chen, Sibo Song, Haolin He, Muzhi Zhu, Zihan Liu, Yunfei Chu, +4 more

Organizations: University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · The Chinese University of Hong Kong · Zhejiang University

Abstract

Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.

Figures & tables

Figure 1: Weaving elastic visual representations. More examples appear in Appendix C .

Figure 2: VisionWeave achieves content-adaptive token savings with near-native quality, whereas fix-budget baselines degrade sharply on challenging tasks. Visual case studies on ScreenSpotV2 and DocVQA illustrate this difference (Appendix B ). Our VisionWeave still retains higher quality even when baselines are granted its adaptive saving ratios (Appendix E.3 ).

Figure 3: Overall architecture of VisionWeave. The gated spatial pooler complements native fine-grained tokens $V_{\mathrm{fine}}$ with coarse-grained representations $V_{\mathrm{coarse}}$ within the same MRoPE coordinate, while the granularity router adaptively weaves them into a mixed-granularity sequence for the LLM.

	Stage 1	Stage 2	Stage 3
Trainable components	Gated Spatial Pooler	Granularity Router	Pooler, LLM
Teacher	Native model	Native model	Native model
Routing	Bypassed (all coarse)	Soft-mixing routing	Hard routing
Loss terms	$\mathcal{L}_{\mathrm{distill}}$	$\mathcal{L}_{\mathrm{distill}}+0.02\mathcal{L}_{\mathrm{bal}}$	$\mathcal{L}_{\mathrm{distill}}$
Student visual tokens	$25\%$	$100\%$	Content-adaptive

Table 1: Three-stage self-distillation for elastic visual processing.

Source	Domain	Approx. training samples
LLaVA-OneVision ( An et al., 2025 )	Image understanding	246K
LLaVA-Video-178K ( Zhang et al., 2025b )	Video understanding	281K
UGround ( Gou et al., 2025 )	GUI grounding	250K
Total		777K

Table 2: Training-data composition for Qwen3.8-27B.

Setting	Stage 1	Stage 2	Stage 3
Peak learning rate	$1\times 10^{-4}$	$1\times 10^{-5}$	$3{\times}10^{-6}$
Minimum learning rate	$5{\times}10^{-6}$	$0$	$0$
Warmup fraction	$5\%$	$5\%$	$2\%$
Global batch size	$32$	$64$	$512$
Training steps	$5{,}257$	$2{,}500$	$1{,}000$
Gradient clipping	$1.0$	$1.0$	$0.5$

Table 3: Training hyperparameters for Qwen3.8-27B.

Panel of Figure 4 — Figure 4: Training dynamics of three-stage self-distillation.

	Native Model	Fixed Savings (s=50%)			Content-adaptive Savings
Qwen3.5-4B	512 token/Image	Downsampling	FastV †	VisionZip	VisionWeave	Savings (%)
RealWorldQA	74.64	72.16 ( $\downarrow$ 3.32%)	70.98 ( $\downarrow$ 4.90%)	70.72 ( $\downarrow$ 5.25%)	73.46 ( $\downarrow$ 1.58%)	43.2
DocVQA	91.92	82.71 ( $\downarrow$ 10.02%)	78.69 ( $\downarrow$ 14.39%)	76.57 ( $\downarrow$ 16.70%)	91.03 ( $\downarrow$ 0.97%)	16.0
InfoVQA	67.44	52.87 ( $\downarrow$ 21.60%)	51.75 ( $\downarrow$ 23.27%)	50.89 ( $\downarrow$ 24.54%)	66.11 ( $\downarrow$ 1.97%)	10.6
ScreenSpotV2	89.86	80.42 ( $\downarrow$ 10.51%)	72.56 ( $\downarrow$ 19.25%)	70.44 ( $\downarrow$ 21.61%)	89.47 ( $\downarrow$ 0.43%)	24.1
HallusionBench	69.88	69.09 ( $\downarrow$ 1.13%)	69.26 ( $\downarrow$ 0.89%)	68.64 ( $\downarrow$ 1.77%)	69.26 ( $\downarrow$ 0.89%)	44.0

Table 4: Content-adaptive token savings with near-native quality.

Figure 5: VisionWeave achieves consistently favorable trade-offs across tasks and input resolutions.

Model	Throughput (req/min) $\uparrow$	Mean TTFT (s) $\downarrow$	P95 TTFT (s) $\downarrow$	Mean TPOT (ms) $\downarrow$	P95 TPOT (ms) $\downarrow$
Native Model	0.90	169.90	281.41	355.64	715.58
VisionWeave	2.07 ( $2.30\times$ )	77.54 ( $\downarrow$ 54.4%)	120.59 ( $\downarrow$ 57.1%)	140.29 ( $\downarrow$ 60.6%)	289.19 ( $\downarrow$ 59.6%)

Table 5: VisionWeave accelerates Qwen3.8-27B serving on SGLang.

Figure 8: VisionWeave enables flexible inference-time control of token savings while closely matching native-model performance when savings are disabled.

Benchmark	VisionWeave	Shuffled	$\Delta$	VisionWeave	Shuffled	$\Delta$
	Qwen3.5-4B			Qwen3.8-27B
RealWorldQA	73.46	73.46	$0.00$	77.91	76.60	$-1.31$
DocVQA	91.03	88.70	$-2.32$	91.33	88.35	$-2.98$
InfoVQA	66.11	64.83	$-1.28$	71.61	69.04	$-2.58$
ScreenSpotV2	89.47	80.03	$-9.43$	91.43	59.28	$-32.15$
HallusionBench	69.26	67.32	$-1.95$	73.60	71.66	$-1.95$

Table 6: Learned granularity allocation improves performance over shuffled routing.

Figure 9: VisionWeave maintains spatial coverage through mixed granularity, whereas FastV † drops tokens covering task-relevant content.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: VisionWeave retains full spatial coverage through mixed-granularity representations, whereas FastV † drops visual tokens.

Figure 11: VisionWeave adapts visual granularity to document content rather than imposing fixed token savings.

Figure 12: Elastic visual representations on Qwen3.5-4B at a 2048-token input budget.

Figure 13: Elastic visual representations on Qwen3.8-27B at a 1024-token input budget.

Figure 14: Additional Qwen3.5-4B visualizations at a 2048-token input budget.

Figure 15: Qwen3.8-27B visualizations corresponding to Figure 14 , with the same 2048-token input budget.

Figure 16: VisionWeave allocates visual granularity at a finer spatial scale than tile-level routing. ViCO panels are cropped from Cui et al. (2025)

Benchmark	Without Stage 3	With Stage 3	$\Delta$ (points)
RealWorldQA	79.08	77.91	$-1.18$
DocVQA	90.76	91.33	$+0.56$
InfoVQA	70.66	71.61	$+0.96$
ScreenSpotV2	66.59	91.43	$+24.84$
HallusionBench	72.10	73.60	$+1.51$
VideoOCR	69.85	70.51	$+0.67$

Appendix

Table 7: Stage 3 improves Qwen3.8-27B performance on seven of eight benchmarks.

Figure 17: On- and off-policy self-distillation yield comparable performance on Qwen3.5-4B.

	Upper bound	Matched savings
Qwen3.5-4B	512 tokens	FastV †	VisionZip	VisionWeave
RealWorldQA	74.64	71.76	72.42	73.46
DocVQA	91.92	91.16	91.09	91.03
InfoVQA	67.44	66.66	65.68	66.11
ScreenSpotV2	89.86	87.34	86.08	89.47
HallusionBench	69.88	69.09	69.44	69.26

Appendix

Table 8: Performance at matched dataset-level token savings.

Explore similar work

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
Sep 28, 2026Jingdi lei, Junxian Li, Di Zhang +3Long Visual-Token Sequences Multimodal Large Language Models
Nanyang Technological University · Shanghai Jiao Tong University · Fudan University +2
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Jul 28, 2026Yuchen Wang, Qihui Zhu, Yang Liu +2Visual Token Pruning Long Visual-Token Sequences
MoE Key Lab of BIPC, NEL-BITA, University of Science and Technology of China, Hefei, China · ChangXin Memory Technologies, Hefei, China · APKL of BIIP, IAI, Hefei Comprehensive National Science Center, Hefei, China
S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangVisual Token Pruning Multimodal Large Language Models
College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China

Sep 28, 2026, cs.CVJ/K move · Enter open · S save

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

Jingdi lei, Junxian Li, Di Zhang, Zhanqiu Zhang+2

Nanyang Technological University · Shanghai Jiao Tong University · Fudan University+2

Long Visual-Token Sequences Multimodal Large Language Models Vision State Space Models Visual Memory Multilayer Perceptrons

Open paper AskPDF arXiv

Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models~(MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose

δ

-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks,

δ

-Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens.

Abstract

Explore similar work

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

S2^22Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models