Recent advances in large vision models (LVMs) have shifted from modality-specific designs toward unified architectures that jointly process images, videos, and 3D data. However, existing unified LVMs primarily pursue functional integration, while overlooking the deeper goal of cross-vision synergy: the ability to reason over complementary priors across visual modalities. To address this, we present PolyV, a unified LVM that achieves cross-vision synergy at both the architectural and training levels. Architecturally, PolyV adopts a sparse Mixture-of-Experts LVM coordinated by a dynamic modality router, allowing each expert to specialize in modality-specific priors while enabling bidirectional interaction and mutual refinement across modalities. Training-wise, a synergy-aware paradigm combines modality-specific pretraining with coarse-to-fine synergy tuning via knowledge distillation and object-/relation-level alignment. Extensive experiments on 10 benchmarks spanning image, video, and 3D understanding, including synergy-focused datasets requiring spatial or temporal priors, demonstrate that PolyV consistently outperforms existing models, achieving over 10% average improvement over its backbone. Overall, PolyV establishes a unified framework for synesthetic visual reasoning, advancing toward truly synergistic LVMs. Project page: https://sqwu.top/PolyV.
Figures & tables
Figure 1: (a) Human perception integrates visual, spatial, and temporal cues synergistically, enabling reasoning across modalities. (b) Examples illustrate such synergy, inferring motion from static images and transferring 3D priors to improve video understanding.
Figure 2: An illustration of PolyV , where an MoE architecture is designed to enable synergistic learning across image, video, and 3D modalities. Fire denotes the trainable parameters.
Figure 3: Illustration of detailed training stages. Stage-1(-1/2) focuses on enabling model understanding of each vision modality. Stage-2(-1): introduces coarse-grained synergistic learning, where a video and 3D foundation model distill temporal and geometric priors into the MoE-LLM. During this process, the model generates latent synergy tokens wrapped in <synergy> , which are optimized via MSE loss to align with the knowledge extracted from foundation models, thereby fostering cross-modality reasoning.
Figure 4: Illustration of cross-vision synergy question-answer pairs. Inspired by [ 61 ] , we leverage the universal scene graph constructed from image-video and image-3D (multi-view) to construct the object-/relation-level cross-synergy question-answer pairs, which are then utilized to enable the model to achieve fine-grained cross-vision synergy.
Model
Image Understanding
Video Understanding
MMStar [ 11 ]
3DSRBench real [ 45 ]
MMSI-Bench [ 68 ]
VSI-Bench [ 66 ]
CVBench [ 83 ]
VideoMME w/o sub. [ 21 ]
GPT-4o [ 30 ]
64.7
44.2
30.3
34.0
69.1
71.9
Gemini-2.0-Flash [ 56 ]
-
49.8
69.4
45.4
-
-
LLaVA-OV-7B [ 33 ]
61.7
54.4
24.5
32.4
52.6
58.2
LLaVA-OV-1.5-8B [ 2 ]
67.7
57.8
29.6
46.3
21.7
56.3
InternVL2.5-8B [ 15 ]
62.8
50.9
28.7
41.6
59.4
64.2
Table 1: Comparison of PolyV with existing MLLMs on image and video understanding benchmarks. All models are evaluated under the official benchmark metrics, and average scores are reported. Improvements over the backbone Qwen2.5-VL-7B [ 6 ] are marked in red .
Model
ScanQA val [ 5 ]
SQA3D test [ 46 ]
Open-EQA HM3D [ 47 ]
BLEU-1
BLEU-4
METEOR
ROUGE-L
CIDEr
EM-1
EM-R1
Qwen2.5-VL-7B [ 6 ]
27.8
3.0
11.4
29.3
53.9
46.5
49.8
56.6
LLaVA-Video-7B [ 75 ]
39.7
3.1
17.7
44.6
88.7
48.5
-
Spatial-MLLM [ 60 ]
44.4
14.8
18.4
45.0
91.8
55.9
58.7
-
Scene-LLM [ 22 ]
-
11.7
15.8
35.9
80.0
53.6
-
-
LL3DA [ 13 ]
-
-
15.9
37.3
76.8
-
-
-
Table 2: Evaluation of 3D question-answering. General 2D VLMs [ 6 , 75 ] are evaluated in a zero-shot setting.
Setting
MMStar [ 11 ]
Open-EQA [ 47 ]
VSI-Bench [ 66 ]
Video-MME [ 21 ]
w/o expert
68.9
60.3
45.8
66.4
w/ expert
71.4
63.4
52.7
69.6
∙ Expert Numbers
2
69.5
61.2
48.3
67.5
3
70.1
62.7
50.6
68.4
4
71.4
63.4
52.7
69.6
Table 3: Ablation Studies on the MoE design in PolyV, examining its necessity, expert number, and architectural placement.
Setting
Open-EQA
3DSRBench real
VSI-Bench
Video-MME
PolyV(Full)
63.4
63.4
52.7
69.6
∙ Training Strategy
only Coarse-grained
61.9
61.0
51.6
68.2
only Fine-grained
62.5
59.8
50.2
68.9
∙ Foundation Model
w/ VideoFM
62.8
61.4
52.1
69.3
Table 4: Comparison studies on the training strategies and foundation model utilization in PolyV.
Figure 5: Token distribution across experts on different benchmarks, illustrating load balance and routing diversity that reflect PolyV’s adaptive expert specialization.
Figure 6: Token distribution across experts by modality. “Percentage” denotes the token fraction per expert across all MoE layers.
Figure 7: Qualitative comparisons between PolyV and existing models. Case 1 is from MMSI-Bench [ 68 ] , Case 2 from DSI-Bench [ 77 ] , Case 3 from VSI-Bench [ 66 ] , and Case 4 from Open-EQA (HM3D) [ 47 ] .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Illustration of CSQA examples, which is constructed based on a paired Image-Video scene graph and Image-3D scene graph.
Name
Experts
Top-k
MoE Layers
Embedding
Width
Layers
FFN
FFN Factor
Heads
Activated Param.
Total Param.
Base
-
-
-
152064
3584
28
18944
3
28
7.0B
7.0B
First Half
4
2
7
152064
3584
28
18944
3
28
8.4B
11.3B
Last Half
4
2
7
152064
3584
28
18944
3
28
8.4B
11.3B
Interval(4)
4
2
7
152064
3584
28
18944
3
28
8.4B
11.3B
Full
4
2
14
152064
3584
28
18944
3
28
9.9B
15.6B
Appendix
Table 5: Detailed Architecture of PolyV and Qwen2.5-VL(Base). “Width” represents the dimension of the hidden states. “FFN” denotes the dimension of the feed-forward network’s intermediate layer. “FFN Factor” represents the quantity of linear layers in the FFN. “Activated” or “Total Param” refers to the activated or total number of parameters. We highlight the architecture of PolyV in blue .
Parameters
Stage-1-1
Stage-1-2
Stage-2-1
Stage-2-2
Optimizer
AdamW
AdamW
AdamW
AdamW
Scheduler
Cosine
Cosine
Cosine
Constant
Learning rate
2e-5
2e-6
1e-4
1e-5
Weight Decay
0.05
0.05
0.05
0.1
Warmup Ratio
0.03
0.03
0.05
0.05
Updated Module
Projector
FFN
Router FFN (Experts) Alignment
Router FFN (Experts)
Appendix
Table 6: The hyperparameter settings in each stage.
Model
Coarse Perception
Fine-grained Perception
Instance Reasoning
Logical Reasoning
Math
Science Technology
Average
LLaVA-NeXT-Video-7B [ 74 ]
52.4
25.6
41.6
28.0
29.2
25.2
33.7
LLaVA-OneVision-7B [ 33 ]
65.2
51.2
64.8
58.0
53.2
44.4
56.1
LLaVA-OneVision1.5-8B [ 2 ]
68.4
53.6
70.0
70.8
77.6
59.2
66.6
InternVL3-8B [ 15 ]
74.0
59.2
72.0
65.6
64.4
54.4
64.9
Qwen3-VL-7B [ 6 ]
75.2
58.0
69.2
51.2
34.8
41.6
55.0
SpaceR [ 49 ]
66.0
48.4
68.0
56.0
44.8
38.4
53.6
Appendix
Table 7: Detailed evaluation results on MMStar [ 11 ] . Best results are marked in bold .
Model
Overall
Height
Loc.
Orient.
Multi.
LLaVA-v1.5-7B [ 41 ]
38.1
39.1
46.9
28.7
34.7
LLaVA-NeXT-Video-7B [ 74 ]
49.4
52.6
55.3
42.6
48.5
LLaVA-OneVision-7B [ 33 ]
54.4
56.8
61.3
46.1
50.3
LLaVA-OneVision1.5-8B [ 2 ]
57.8
57.4
66.5
49.9
53.4
InternVL2.5-8B [ 15 ]
50.9
45.9
68.1
38.7
43.3
InternVL3-8B [ 82 ]
58.1
61.3
67.6
48.3
53.8
Appendix
Table 8: Detailed evaluation results on 3DSRBench real [ 45 ] . ‘Loc.’ denotes location, ‘Orient.’ means orientation, and ‘Multi.’ is multi-object.
Model
Positional Relationship
Attribute
Motion
MSR
Average
Cam.–Cam.
Obj.–Obj.
Reg.–Reg.
Cam.–Obj.
Obj.–Reg.
Cam.–Reg.
Meas.
Appr.
Cam.
Obj.
LLaVA-v1.5-7B [ 42 ]
33.3
31.9
22.2
29.1
22.3
20.5
25.0
22.7
14.9
21.0
22.2
24.2
LLaVA-NeXT-Video-7B [ 74 ]
21.5
22.3
24.7
26.7
23.5
22.9
32.8
27.3
16.2
26.3
27.8
24.9
Qwen2.5-VL-7B [ 6 ]
24.7
24.5
24.7
25.6
29.4
26.5
25.0
18.2
20.3
39.5
25.8
25.9
Qwen3-VL-8B [ 6 ]
31.2
27.7
28.4
24.4
28.2
34.9
35.9
15.1
27.0
38.2
28.3
29.0
InternVL2.5-8B [ 15 ]
32.3
27.7
29.6
32.6
24.7
32.5
26.6
27.3
16.2
31.6
30.3
28.7
Appendix
Table 9: Detailed evaluation results on MMSI-Bench [ 68 ] . ‘Cam.’ denotes the camera, ‘Obj.’ means the object, ‘Reg.’ is the region, ‘Meas.’ is the measurement, and ‘Appr.’ is the appearance.
Table 11: Detailed evaluation results on CVBench [ 83 ] . The tasks include: multi-view scene understanding (M. SU), multi-video temporal reasoning (M. TR), joint-video spatial navigation (J. SN), video difference captioning (VDC), cross-video counterfactual reasoning (C. CR), joint-video summarization (J.S), and cross-video procedural transfer (C. PT).
Model
Obj. Rec.
Attr. Rec.
Spatial Und.
Obj. State Rec.
Func. Rea.
World Know.
Obj. Loc.
Average
LLaVA-NeXT-Video-7B [ 74 ]
31.5
34.1
35.0
31.2
51.0
31.7
30.2
34.9
LLaVA-OneVision-7B [ 33 ]
35.1
42.5
33.8
35.0
53.7
35.9
34.6
38.7
LLaVA-OneVision1.5-8B [ 2 ]
37.5
45.6
36.8
37.2
58.9
38.0
36.4
41.2
InternVL2.5-8B [ 15 ]
42.3
51.0
40.0
41.3
68.0
41.5
40.5
46.7
InternVL3-8B [ 15 ]
44.4
51.1
42.5
43.5
68.7
43.6
42.5
48.0
Qwen2.5-VL-7B [ 6 ]
50.4
62.1
49.3
66.4
61.4
60.3
47.6
56.6
Appendix
Table 12: Detailed evaluation results on OpenEQA HM3D [ 47 ] . ‘Obj. Rec.’ denotes object recognition, ‘Attr. Rec.’ is attribute recognition, ‘Spatial Und.’ is the spatial understanding, ‘Obj. State Rec.’ is the object state recognition, ‘Func. Rea.’ is the function reasoning, ‘World Know.’ is the world knowledge, ‘ Obj. Loc.’ is the object localization.
Table 15: Detailed evaluation results on DSI-Bench [ 77 ] across different video sources.
Model
Static Understanding
Dynamic Understanding
Average
Dim. Meas
Spatial Relation
3D Video Grounding
Disp. & P.L.
Speed & Acc.
Ego Orient.
Traj. Desc.
Pose Est.
LLaVA-NeXT-Video-7B [ 74 ]
22.1
42.5
19.9
21.5
21.1
27.0
42.3
18.9
23.6
LLaVA-OneVision-7B [ 33 ]
25.6
30.8
24.6
25.1
30.8
41.6
51.3
55.8
34.2
LLaVA-OneVision1.5-8B [ 2 ]
34.3
29.4
33.1
24.9
35.0
63.8
44.9
55.6
39.0
InternVL2.5-8B [ 15 ]
26.3
50.7
31.5
20.9
29.3
66.5
46.1
56.7
38.0
InternVL3-8B [ 15 ]
26.3
52.0
27.1
21.5
33.2
0.2
25.6
40.6
30.4
Appendix
Table 16: Detailed evaluation results on STI-Bench [ 34 ] . ‘Dim. Meas’ is the Dimensional Measurement, ‘Disp. & P.L.’ means Displacement & Path Length, ‘Speed & Acc.’ denotes Speed & Acceleration, ‘Ego Orient.’ indicates Ego-Centric Orientation, ‘Traj. Desc.’ is Trajectory Description, and ‘Pose Est.’ is Pose Estimation.
Figure 9: Qualitative comparisons between PolyV and existing models. Case 1 is from 3DSRBench [ 45 ] , Case 2 from DSI-Bench [ 77 ] , Case 3 from CV-Bench [ 57 ] , and Case 4 from STI-Bench [ 34 ] .
Multimodal large language models are increasingly expected to perform thinking with images, yet existing visual latent reasoning methods still rely on explicit textual chain-of-thought interleaved with visual latent tokens. This interleaved design limits efficiency and keeps reasoning fragmented across separate text and vision channels. We propose UniVLR, a unified visual latent reasoning framework that treats textual reasoning and auxiliary visual evidence as a shared visual workspace. Instead of preserving text CoT as an independent inference-time path, UniVLR renders reasoning traces together with auxiliary images and learns to compress this unified representation into compact visual latent tokens. At inference time, the model reasons only through visual latents and directly decodes the final answer, avoiding both external tool calls and verbose text reasoning. Experiments on real-world perception and visual reasoning tasks show that UniVLR outperforms prior visual latent reasoning methods while using substantially fewer generated reasoning tokens, suggesting a more unified and efficient paradigm for visual thinking in MLLMs.
Houcheng Jiang, Jiajun Fu, Junfeng Fang +4
University of Science and Technology of China · Zhongguancun Academy · National University of Singapore +1
While Vision-Language Models (VLMs) demonstrate strong capabilities, they continue to suffer from a critical limitation: insufficient fine-grained visual perception, which fundamentally limits their multimodal understanding. We attribute this bottleneck to text-dominant optimization biases during pre-training, which encourage the model to overlook fine-grained visual details, thereby limiting the capability of multimodal understanding. We investigate that overcoming this bottleneck requires two key elements: (1) a unified token space paradigm that ensures stable training dynamics, and (2) a modality-aligned dense visual supervision signal enriched with both structural granularity and semantic information to capture critical visual representations. Based on these insights, we propose VIVAS, a framework built upon the unified token space paradigm, which introduces a dense-structural-semantic vision tokenizer, which expands the textual vocabulary into a unified vision-language vocabulary by incorporating a visual vocabulary. During pretraining, VIVAS performs vision-language unified autoregressive supervision over both visual details and linguistic content, thereby enhancing visual perception to improve multimodal understanding. Trained end-to-end on 12.4T tokens, VIVAS achieves state-of-the-art performance across 7 tasks and 39 multimodal benchmarks.
Zhehan Kan, Yubo Zhu, Xinghua Jiang +10
Tsinghua University · Tencent Youtu Lab · Nanjing University
Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective. In this work, we study the feasibility and limits of token-reduction-based acceleration for unified VLM training. Through a systematic analysis of layerwise attention allocation, we uncover a fundamental asymmetry: visual understanding exhibits substantial late-layer visual redundancy, whereas visual generation maintains persistent dependence on image tokens across depth. Guided by this observation, we design task-specific accelerators that selectively reduce image-token computation for each objective. While these methods achieve significant efficiency gains in isolated settings, we observe a consistent synergy loss under unified training -- task-specific token dropping necessitates divergent parameter pathways and eliminates the mutual performance gains typically observed in joint optimization. Our findings suggest that efficient unified modeling requires preserving shared cross-task structures, highlighting the need for synergy-aware acceleration strategies. Project page: https://chicychen.github.io/TokenReductionUnifiedVLM/.