Modeling Cross-vision Synergy for Unified Large Vision Model

Shengqiong Wu; Lanhu Wu; Mingyang Bao; Wenhao Xu; Hanwang Zhang; Shuicheng Yan; Hao Fei; Tat-Seng Chua

cs.CVMar 3, 2026

Modeling Cross-vision Synergy for Unified Large Vision Model

Authors: Shengqiong Wu, Lanhu Wu, Mingyang Bao, Wenhao Xu, Hanwang Zhang, Shuicheng Yan, Hao Fei, Tat-Seng Chua

Organizations: National University of Singapore · Nanyang Technological University

Abstract

Recent advances in large vision models (LVMs) have shifted from modality-specific designs toward unified architectures that jointly process images, videos, and 3D data. However, existing unified LVMs primarily pursue functional integration, while overlooking the deeper goal of cross-vision synergy: the ability to reason over complementary priors across visual modalities. To address this, we present PolyV, a unified LVM that achieves cross-vision synergy at both the architectural and training levels. Architecturally, PolyV adopts a sparse Mixture-of-Experts LVM coordinated by a dynamic modality router, allowing each expert to specialize in modality-specific priors while enabling bidirectional interaction and mutual refinement across modalities. Training-wise, a synergy-aware paradigm combines modality-specific pretraining with coarse-to-fine synergy tuning via knowledge distillation and object-/relation-level alignment. Extensive experiments on 10 benchmarks spanning image, video, and 3D understanding, including synergy-focused datasets requiring spatial or temporal priors, demonstrate that PolyV consistently outperforms existing models, achieving over 10% average improvement over its backbone. Overall, PolyV establishes a unified framework for synesthetic visual reasoning, advancing toward truly synergistic LVMs. Project page: https://sqwu.top/PolyV.

Figures & tables

Figure 1: (a) Human perception integrates visual, spatial, and temporal cues synergistically, enabling reasoning across modalities. (b) Examples illustrate such synergy, inferring motion from static images and transferring 3D priors to improve video understanding.

Figure 2: An illustration of PolyV , where an MoE architecture is designed to enable synergistic learning across image, video, and 3D modalities. Fire denotes the trainable parameters.

Figure 3: Illustration of detailed training stages. Stage-1(-1/2) focuses on enabling model understanding of each vision modality. Stage-2(-1): introduces coarse-grained synergistic learning, where a video and 3D foundation model distill temporal and geometric priors into the MoE-LLM. During this process, the model generates latent synergy tokens wrapped in <synergy> , which are optimized via MSE loss to align with the knowledge extracted from foundation models, thereby fostering cross-modality reasoning.

Figure 4: Illustration of cross-vision synergy question-answer pairs. Inspired by [ 61 ] , we leverage the universal scene graph constructed from image-video and image-3D (multi-view) to construct the object-/relation-level cross-synergy question-answer pairs, which are then utilized to enable the model to achieve fine-grained cross-vision synergy.

Model	Image Understanding			Video Understanding
Model	MMStar [ 11 ]	3DSRBench ${}_{\text{real}}$ [ 45 ]	MMSI-Bench [ 68 ]	VSI-Bench [ 66 ]	CVBench [ 83 ]	VideoMME ${}_{\text{w/o sub.}}$ [ 21 ]
GPT-4o [ 30 ]	64.7	44.2	30.3	34.0	69.1	71.9
Gemini-2.0-Flash [ 56 ]	-	49.8	69.4	45.4	-	-
LLaVA-OV-7B [ 33 ]	61.7	54.4	24.5	32.4	52.6	58.2
LLaVA-OV-1.5-8B [ 2 ]	67.7	57.8	29.6	46.3	21.7	56.3
InternVL2.5-8B [ 15 ]	62.8	50.9	28.7	41.6	59.4	64.2

Table 1: Comparison of PolyV with existing MLLMs on image and video understanding benchmarks. All models are evaluated under the official benchmark metrics, and average scores are reported. Improvements over the backbone Qwen2.5-VL-7B [ 6 ] are marked in red .

Model	ScanQA ${}_{\text{val}}$ [ 5 ]					SQA3D ${}_{\text{test}}$ [ 46 ]		Open-EQA ${}_{\text{HM3D}}$ [ 47 ]
Model	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	EM-1	EM-R1	Open-EQA ${}_{\text{HM3D}}$ [ 47 ]
Qwen2.5-VL-7B [ 6 ]	27.8	3.0	11.4	29.3	53.9	46.5	49.8	56.6
LLaVA-Video-7B [ 75 ]	39.7	3.1	17.7	44.6	88.7	48.5	-
Spatial-MLLM [ 60 ]	44.4	14.8	18.4	45.0	91.8	55.9	58.7	-
Scene-LLM [ 22 ]	-	11.7	15.8	35.9	80.0	53.6	-	-
LL3DA [ 13 ]	-	-	15.9	37.3	76.8	-	-	-

Table 2: Evaluation of 3D question-answering. General 2D VLMs [ 6 , 75 ] are evaluated in a zero-shot setting.

Setting	MMStar [ 11 ]	Open-EQA [ 47 ]	VSI-Bench [ 66 ]	Video-MME [ 21 ]
w/o expert	68.9	60.3	45.8	66.4
w/ expert	71.4	63.4	52.7	69.6
$\bullet$ Expert Numbers
2	69.5	61.2	48.3	67.5
3	70.1	62.7	50.6	68.4
4	71.4	63.4	52.7	69.6

Table 3: Ablation Studies on the MoE design in PolyV, examining its necessity, expert number, and architectural placement.

Setting	Open-EQA	3DSRBench ${}_{\text{real}}$	VSI-Bench	Video-MME
PolyV(Full)	63.4	63.4	52.7	69.6
$\bullet$ Training Strategy
only Coarse-grained	61.9	61.0	51.6	68.2
only Fine-grained	62.5	59.8	50.2	68.9
$\bullet$ Foundation Model
w/ VideoFM	62.8	61.4	52.1	69.3

Table 4: Comparison studies on the training strategies and foundation model utilization in PolyV.

Figure 5: Token distribution across experts on different benchmarks, illustrating load balance and routing diversity that reflect PolyV’s adaptive expert specialization.

Figure 6: Token distribution across experts by modality. “Percentage” denotes the token fraction per expert across all MoE layers.

Figure 7: Qualitative comparisons between PolyV and existing models. Case 1 is from MMSI-Bench [ 68 ] , Case 2 from DSI-Bench [ 77 ] , Case 3 from VSI-Bench [ 66 ] , and Case 4 from Open-EQA (HM3D) [ 47 ] .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Illustration of CSQA examples, which is constructed based on a paired Image-Video scene graph and Image-3D scene graph.

Name	Experts	Top-k	MoE Layers	Embedding	Width	Layers	FFN	FFN Factor	Heads	Activated Param.	Total Param.
Base	-	-	-	152064	3584	28	18944	3	28	7.0B	7.0B
First Half	4	2	7	152064	3584	28	18944	3	28	8.4B	11.3B
Last Half	4	2	7	152064	3584	28	18944	3	28	8.4B	11.3B
Interval(4)	4	2	7	152064	3584	28	18944	3	28	8.4B	11.3B
Full	4	2	14	152064	3584	28	18944	3	28	9.9B	15.6B

Appendix

Table 5: Detailed Architecture of PolyV and Qwen2.5-VL(Base). “Width” represents the dimension of the hidden states. “FFN” denotes the dimension of the feed-forward network’s intermediate layer. “FFN Factor” represents the quantity of linear layers in the FFN. “Activated” or “Total Param” refers to the activated or total number of parameters. We highlight the architecture of PolyV in blue .

Parameters	Stage-1-1	Stage-1-2	Stage-2-1	Stage-2-2
Optimizer	AdamW	AdamW	AdamW	AdamW
Scheduler	Cosine	Cosine	Cosine	Constant
Learning rate	2e-5	2e-6	1e-4	1e-5
Weight Decay	0.05	0.05	0.05	0.1
Warmup Ratio	0.03	0.03	0.05	0.05
Updated Module	Projector	FFN	Router FFN (Experts) Alignment	Router FFN (Experts)

Appendix

Table 6: The hyperparameter settings in each stage.

Model	Coarse Perception	Fine-grained Perception	Instance Reasoning	Logical Reasoning	Math	Science Technology	Average
LLaVA-NeXT-Video-7B [ 74 ]	52.4	25.6	41.6	28.0	29.2	25.2	33.7
LLaVA-OneVision-7B [ 33 ]	65.2	51.2	64.8	58.0	53.2	44.4	56.1
LLaVA-OneVision1.5-8B [ 2 ]	68.4	53.6	70.0	70.8	77.6	59.2	66.6
InternVL3-8B [ 15 ]	74.0	59.2	72.0	65.6	64.4	54.4	64.9
Qwen3-VL-7B [ 6 ]	75.2	58.0	69.2	51.2	34.8	41.6	55.0
SpaceR [ 49 ]	66.0	48.4	68.0	56.0	44.8	38.4	53.6

Appendix

Table 7: Detailed evaluation results on MMStar [ 11 ] . Best results are marked in bold .

Model	Overall	Height	Loc.	Orient.	Multi.
LLaVA-v1.5-7B [ 41 ]	38.1	39.1	46.9	28.7	34.7
LLaVA-NeXT-Video-7B [ 74 ]	49.4	52.6	55.3	42.6	48.5
LLaVA-OneVision-7B [ 33 ]	54.4	56.8	61.3	46.1	50.3
LLaVA-OneVision1.5-8B [ 2 ]	57.8	57.4	66.5	49.9	53.4
InternVL2.5-8B [ 15 ]	50.9	45.9	68.1	38.7	43.3
InternVL3-8B [ 82 ]	58.1	61.3	67.6	48.3	53.8

Appendix

Table 8: Detailed evaluation results on 3DSRBench

{}_{\text{real}}

[ 45 ] . ‘Loc.’ denotes location, ‘Orient.’ means orientation, and ‘Multi.’ is multi-object.

Model	Positional Relationship						Attribute		Motion		MSR	Average
Model	Cam.–Cam.	Obj.–Obj.	Reg.–Reg.	Cam.–Obj.	Obj.–Reg.	Cam.–Reg.	Meas.	Appr.	Cam.	Obj.	MSR	Average
LLaVA-v1.5-7B [ 42 ]	33.3	31.9	22.2	29.1	22.3	20.5	25.0	22.7	14.9	21.0	22.2	24.2
LLaVA-NeXT-Video-7B [ 74 ]	21.5	22.3	24.7	26.7	23.5	22.9	32.8	27.3	16.2	26.3	27.8	24.9
Qwen2.5-VL-7B [ 6 ]	24.7	24.5	24.7	25.6	29.4	26.5	25.0	18.2	20.3	39.5	25.8	25.9
Qwen3-VL-8B [ 6 ]	31.2	27.7	28.4	24.4	28.2	34.9	35.9	15.1	27.0	38.2	28.3	29.0
InternVL2.5-8B [ 15 ]	32.3	27.7	29.6	32.6	24.7	32.5	26.6	27.3	16.2	31.6	30.3	28.7

Appendix

Table 9: Detailed evaluation results on MMSI-Bench [ 68 ] . ‘Cam.’ denotes the camera, ‘Obj.’ means the object, ‘Reg.’ is the region, ‘Meas.’ is the measurement, and ‘Appr.’ is the appearance.

Model	Average	Obj. Count	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Appr. Order
LLaVA-Video-7B [ 75 ]	35.6	48.5	14.0	47.8	24.2	43.5	42.4	34.0	30.6
LLaVA-NeXT-Video-7B [ 74 ]	29.9	53.6	30.6	21.6	38.8	27.7	33.9	31.4	16.1
LLaVA-OneVision-7B [ 33 ]	32.4	47.7	20.2	47.4	12.3	42.5	35.2	29.4	24.4
LLaVA-OneVision1.5-8B [ 2 ]	46.3	71.2	35.1	69.3	60.6	38.3	39.1	29.9	25.2
InternVL3-8B [ 82 ]	50.7	75.8	52.2	65.2	70.4	36.8	36.4	37.3	37.2
Video3d-LLM [ 79 ]	15.8	17.5	0.0	0.0	0.0	29.6	33.4	26.3	24.8

Appendix

Table 10: Detailed evaluation results on VSI-Bench [ 66 ] . object count (Obj. Count), absolute distance (Abs. Dist.), object size (Obj. Size), room size, relative distance (Rel. Dist.), relative direction (Rel. Dir.), route plan, appearance order (Appr. Order)

Model	Average	M. SU	M. TR	J. SN	VDC	C.CR	J.S	C.PT
LLaVA-NeXT-Video-7B [ 74 ]	30.9	5.3	14.6	14.6	20.0	17.3	5.8	7.8
Qwen2.5-VL-7B [ 6 ]	51.3	80.0	22.7	26.2	50.9	55.8	69.2	60.8
InternVL2.5-8B [ 15 ]	59.4	83.6	26.7	50.0	60.0	69.2	67.3	68.6
Qwen2.5-VL-8B [ 6 ]	51.1	80.0	31.1	47.6	71.1	66.7	71.1	68.0
LLaVA-OneVision-7B [ 33 ]	52.6	83.6	40.0	38.1	45.5	42.3	61.5	52.9
LLaVA-OneVision1.5-8B [ 2 ]	43.3	7.5	23.4	23.4	28.8	27.7	8.7	12.5

Appendix

Table 11: Detailed evaluation results on CVBench [ 83 ] . The tasks include: multi-view scene understanding (M. SU), multi-video temporal reasoning (M. TR), joint-video spatial navigation (J. SN), video difference captioning (VDC), cross-video counterfactual reasoning (C. CR), joint-video summarization (J.S), and cross-video procedural transfer (C. PT).

Model	Obj. Rec.	Attr. Rec.	Spatial Und.	Obj. State Rec.	Func. Rea.	World Know.	Obj. Loc.	Average
LLaVA-NeXT-Video-7B [ 74 ]	31.5	34.1	35.0	31.2	51.0	31.7	30.2	34.9
LLaVA-OneVision-7B [ 33 ]	35.1	42.5	33.8	35.0	53.7	35.9	34.6	38.7
LLaVA-OneVision1.5-8B [ 2 ]	37.5	45.6	36.8	37.2	58.9	38.0	36.4	41.2
InternVL2.5-8B [ 15 ]	42.3	51.0	40.0	41.3	68.0	41.5	40.5	46.7
InternVL3-8B [ 15 ]	44.4	51.1	42.5	43.5	68.7	43.6	42.5	48.0
Qwen2.5-VL-7B [ 6 ]	50.4	62.1	49.3	66.4	61.4	60.3	47.6	56.6

Appendix

Table 12: Detailed evaluation results on OpenEQA

{}_{\text{HM3D}}

[ 47 ] . ‘Obj. Rec.’ denotes object recognition, ‘Attr. Rec.’ is attribute recognition, ‘Spatial Und.’ is the spatial understanding, ‘Obj. State Rec.’ is the object state recognition, ‘Func. Rea.’ is the function reasoning, ‘World Know.’ is the world knowledge, ‘ Obj. Loc.’ is the object localization.

Model	Short	Mid	Long	Average
LLaVA-NeXT-Video-34B [ 74 ]	61.7	50.1	44.3	52.0
Qwen3-VL-8B [ 6 ]	58.1	49.2	49.1	52.1
LLaVA-3D [ 81 ]	27.1	27.0	25.4	26.5
SpaceR [ 49 ]	67.0	55.4	48.2	56.9
Spatial-MLLM [ 60 ]	54.4	42.0	35.8	44.1
Video3d-LLM [ 79 ]	47.7	42.0	37.1	41.9

Appendix

Table 13: Detailed evaluation results on VideoMME

{}_{\text{w/o sub}}

[ 21 ] .

Model	2D		3D		Average
Model	Relation	Count	Depth	Distance	Average
Qwen2.5-VL-7B [ 6 ]	77.4	63.2	74.2	44.0	64.8
Qwen3-VL-8B [ 6 ]	92.5	71.7	95.0	58.7	79.1
SpaceR [ 49 ]	83.7	65.1	78.5	66.0	72.9
Spatial-MLLM [ 60 ]	62.3	53.7	60.5	65.7	60.1
Video3d-LLM [ 79 ]	53.2	18.4	56.2	60.5	45.1

Appendix

Table 14: Detailed evaluation results on CV-Bench [ 57 ] .

Model	CamerBench [ 40 ]	SynFMC [ 53 ]	internet	k700 [ 55 ]	llava178k [ 75 ]	Average
LLaVA-NeXT-Video-7B [ 74 ]	36.0	28.4	36.9	33.6	35.8	35.6
LLaVA-OneVision-7B [ 33 ]	46.0	33.0	48.1	47.5	45.3	46.4
LLaVA-OneVision1.5-8B [ 2 ]	50.4	48.6	50.3	57.0	49.6	51.1
InternVL2.5-8B [ 15 ]	47.6	48.6	50.4	57.4	46.8	50.4
InternVL3-8B [ 15 ]	52.8	43.1	45.9	47.5	47.6	47.2
Qwen2.5-VL-7B [ 6 ]	40.0	38.5	39.4	41.2	48.1	40.5

Appendix

Table 15: Detailed evaluation results on DSI-Bench [ 77 ] across different video sources.

Model	Static Understanding			Dynamic Understanding					Average
Model	Dim. Meas	Spatial Relation	3D Video Grounding	Disp. & P.L.	Speed & Acc.	Ego Orient.	Traj. Desc.	Pose Est.	Average
LLaVA-NeXT-Video-7B [ 74 ]	22.1	42.5	19.9	21.5	21.1	27.0	42.3	18.9	23.6
LLaVA-OneVision-7B [ 33 ]	25.6	30.8	24.6	25.1	30.8	41.6	51.3	55.8	34.2
LLaVA-OneVision1.5-8B [ 2 ]	34.3	29.4	33.1	24.9	35.0	63.8	44.9	55.6	39.0
InternVL2.5-8B [ 15 ]	26.3	50.7	31.5	20.9	29.3	66.5	46.1	56.7	38.0
InternVL3-8B [ 15 ]	26.3	52.0	27.1	21.5	33.2	0.2	25.6	40.6	30.4

Appendix

Table 16: Detailed evaluation results on STI-Bench [ 34 ] . ‘Dim. Meas’ is the Dimensional Measurement, ‘Disp. & P.L.’ means Displacement & Path Length, ‘Speed & Acc.’ denotes Speed & Acceleration, ‘Ego Orient.’ indicates Ego-Centric Orientation, ‘Traj. Desc.’ is Trajectory Description, and ‘Pose Est.’ is Pose Estimation.

Figure 9: Qualitative comparisons between PolyV and existing models. Case 1 is from 3DSRBench [ 45 ] , Case 2 from DSI-Bench [ 77 ] , Case 3 from CV-Bench [ 57 ] , and Case 4 from STI-Bench [ 34 ] .

Vision Foundation Models

Vision foundation models are large pretrained visual systems adapted to recognition, segmentation, geometry, and video tasks.

See all 139 papers

UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs
May 12, 2026Houcheng Jiang, Jiajun Fu, Junfeng Fang +4Efficient Multimodal Inference Multimodal Large Language Models
University of Science and Technology of China · Zhongguancun Academy · National University of Singapore +1
VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision
Aug 25, 2026Zhehan Kan, Yubo Zhu, Xinghua Jiang +10Vision-Language Pretraining Fine-Grained Visual Perception
Tsinghua University · Tencent Youtu Lab · Nanjing University
On the Limits of Token Reduction for Efficient Unified Vision Language Training
May 31, 2026Siyi Chen, Weiming Zhuang, Jingtao Li +1Vision-Language Models Unified Multimodal Models
University of Michigan · Sony AI

May 12, 2026J/K move · Enter open · S save

UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs

Houcheng Jiang, Jiajun Fu, Junfeng Fang, Chen Gao+3

University of Science and Technology of China · Zhongguancun Academy · National University of Singapore+1

Efficient Multimodal Inference Multimodal Large Language Models Latent Visual Reasoning VLM Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning+1

Open paper AskPDF arXiv

Multimodal large language models are increasingly expected to perform thinking with images, yet existing visual latent reasoning methods still rely on explicit textual chain-of-thought interleaved with visual latent tokens. This interleaved design limits efficiency and keeps reasoning fragmented across separate text and vision channels. We propose UniVLR, a unified visual latent reasoning framework that treats textual reasoning and auxiliary visual evidence as a shared visual workspace. Instead of preserving text CoT as an independent inference-time path, UniVLR renders reasoning traces together with auxiliary images and learns to compress this unified representation into compact visual latent tokens. At inference time, the model reasons only through visual latents and directly decodes the final answer, avoiding both external tool calls and verbose text reasoning. Experiments on real-world perception and visual reasoning tasks show that UniVLR outperforms prior visual latent reasoning methods while using substantially fewer generated reasoning tokens, suggesting a more unified and efficient paradigm for visual thinking in MLLMs.

Abstract

Vision Foundation Models

Explore similar work

UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs

VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision

On the Limits of Token Reduction for Efficient Unified Vision Language Training