DiFuse: Fusing Native Diffusion Representations for Spatial Reasoning in Unified Multimodal Models

Ruixin Fan; Weining Wang; Jiahui Sun; Tailai Chen; Jing Liu

cs.CVOct 29, 2026

DiFuse: Fusing Native Diffusion Representations for Spatial Reasoning in Unified Multimodal Models

Authors: Ruixin Fan, Weining Wang, Jiahui Sun, Tailai Chen, Jing Liu

Organizations: Institute of Automation, Chinese Academy of Sciences

Abstract

Unified multimodal models (UMMs) jointly support visual understanding and image generation, yet information flow between these two capabilities remains largely asymmetric: understanding representations routinely condition generation, whereas generative representations are rarely reused for multimodal reasoning. In this work, we investigate whether intermediate representations from a UMM's native diffusion branch can enhance spatial reasoning without compromising general multimodal understanding. We propose DiFuse, which selectively incorporates intermediate diffusion representations into the understanding pathway of UMMs. Through systematic layer-wise analysis, we identify which diffusion representations are most informative and where they can be integrated with minimal interference to semantic processing. Guided by these findings, DiFuse spatially aligns diffusion features with visual tokens and injects them into selected multimodal decoder layers via residual fusion. Extensive experiments on UniWorld-V1 show that, DiFuse consistently improves spatial reasoning over a matched SFT baseline while maintaining broadly comparable performance on general multimodal benchmarks. Further analyses show that these gains depend critically on the choice of diffusion representations, injection positions, and the preservation of spatial correspondence between diffusion features and visual tokens. Results on OmniGen2 further demonstrate the transferability of DiFuse across unified multimodal architectures. Overall, our findings show that native diffusion representations can provide complementary spatial cues for multimodal reasoning when selectively integrated in a structure-preserving manner.

Figures & tables

Figure 1: Overview of DiFuse and its impact on unified multimodal models. (a) DiFuse establishes a generation-to-understanding pathway by reusing intermediate representations from the native Diffusion Transformer (DiT) branch and integrating them into the visual understanding pathway of a unified multimodal model. (b) General–spatial capability landscape. Models are positioned by their average general-understanding (Avg-G) and spatial-reasoning (Avg-S) scores. Marker shapes distinguish general-purpose MLLMs, UMMs, spatially enhanced models, and DiFuse. (c) Gains relative to Qwen2.5-VL-7B. We compare four Qwen2.5-VL-7B-based models against the same baseline. Blue and green bars show changes in Avg-G and Avg-S, respectively. UniWorld-V1 + DiFuse improves Avg-S by 2.60 points while leaving Avg-G nearly unchanged (+0.02 points).

Figure 2: Paradigms for generation-assisted and structure-enhanced multimodal understanding. Existing methods use diffusion objectives, explicit visual states, external diffusion features, or dedicated geometry branches, whereas DiFuse directly reuses spatially aligned representations from the UMM’s native DiT branch.

Figure 3: DiFuse architecture. (a) Overall pipeline. Given an input image and a question, the UMM’s understanding branch processes the visual and text tokens, while its native generation branch receives the same image and a fixed generation prompt “Generate the depth map of this image.” DiFuse extracts an intermediate spatial representation from the frozen DiT at a selected layer and denoising timestep, without decoding a final image. (b) DiFuse module. The extracted feature is spatially pooled and compressed with PCA, then normalized and projected through a convolutional adapter. Bilinear resizing aligns the resulting feature map with the visual-token grid. DiFuse adds the aligned features to visual-token states at selected decoder layers through residual connections; text-token states receive no direct residual update.

		Summary		General Multimodal Understanding					Spatial Reasoning
Backbone	Variant	Avg-G	Avg-S	MME norm	MMB	MMStar	GQA	RWQA	VSR	EmbSpatial	CVB-3D	3DSR-C
UniWorld-V1 [-1pt] (Qwen2.5-VL-7B)	Base	71.92	71.08	82.35	83.93	62.74	60.92	69.67	87.32	69.97	81.08	45.94
	+ SFT	71.51 (-0.41)	71.88 (+0.80)	78.23	84.88	63.76	60.90	69.80	87.81	70.99	81.92	46.79
	+ DiFuse	71.94 (+0.02)	73.68 (+2.60)	80.82	84.54	62.97	61.70	69.67	89.36	72.39	84.58	48.38
OmniGen2 [-1pt] (Qwen2.5-VL-3B)	Base	66.04	65.01	76.55	77.49	56.72	60.22	59.22	80.77	61.46	74.08	43.73
	+ SFT	67.69 (+1.65)	65.37 (+0.36)	76.71	78.26	58.00	61.17	64.31	82.41	62.20	74.75	42.12

Table 1: Controlled comparison of SFT and DiFuse on unified multimodal models. For each backbone, SFT and DiFuse use identical instruction-tuning data and optimization settings. Avg-G averages MME norm , MMB, MMStar, GQA, and RWQA; Avg-S averages VSR, EmbSpatial, CVB-3D, and 3DSR-C. Numbers in parentheses denote absolute changes over the corresponding base UMM. Bold numbers indicate the best result within each backbone family.

		Summary		General Multimodal Understanding					Spatial Reasoning
Method	Params.	Avg-G	Avg-S	MME norm	MMB	MMStar	GQA	RWQA	VSR	EmbSpatial	CVB-3D	3DSR-C
General-purpose VLMs
InternVL3	8B	74.24	72.41	85.18	85.74	68.65	62.73	68.89	87.07	74.45	84.00	44.11
LLaVA-1.5	7B	56.51	47.16	66.89	64.00	33.76	61.93	55.95	51.47	45.77	57.00	34.38
Llama-3.2-Vision-Instruct	11B	52.37	52.77	62.46	72.77	48.44	21.70	56.47	72.59	51.76	64.58	22.14
Idefics2	8B	59.46	62.50	63.99	75.26	47.34	51.22	59.48	78.56	53.49	71.00	46.93

Table 2: Contextual comparison with representative general-purpose VLMs, unified multimodal models (UMMs), and spatial-reasoning-enhanced models. All models are evaluated using the same benchmark splits and scoring implementation under deterministic decoding. Avg-G and Avg-S follow the definitions in Table 1 . Bold numbers in the summary columns indicate the best result within each model category. The shaded row denotes our DiFuse-enhanced UMM.

	Summary		General Multimodal Understanding					Spatial Reasoning
Source layer	Avg-G	Avg-S	MME norm	MMB	MMStar	GQA	RWQA	VSR	EmbSpatial	CVB-3D	3DSR-C
$g17$	71.44	72.14	79.00	83.76	62.61	61.11	70.72	88.30	71.18	81.58	47.51
$\mathbf{g18}$	71.89	73.08	81.31	83.51	64.03	60.92	69.67	89.28	72.12	82.83	48.09
$g28$	71.38	73.18	79.95	83.93	62.39	61.23	69.41	88.87	72.17	83.67	48.03
$g53$	71.74	73.06	79.95	84.36	62.46	60.92	70.98	89.12	71.95	83.00	48.19

Table 3: Diffusion source-layer screening. All candidates are evaluated under the same reduced screening configuration with all other settings fixed. These values are not directly comparable to the final DiFuse results. Bold numbers indicate the best result in each column, and the shaded row denotes the selected source layer.

		Summary		General Multimodal Understanding					Spatial Reasoning
Factor	Variant	Avg-G	Avg-S	MME norm	MMB	MMStar	GQA	RWQA	VSR	EmbSpatial	CVB-3D	3DSR-C
Diffusion timestep
$t$	50	71.52	72.98	79.84	83.76	62.87	61.48	69.67	88.95	71.76	83.58	47.64
$t$	100 (ours)	71.94	73.68	80.82	84.54	62.97	61.70	69.67	89.36	72.39	84.58	48.38
$t$	200	71.59	73.05	80.33	83.93	62.53	61.32	69.80	89.77	72.39	81.50	48.54
$t$	300	71.50	73.52	80.29	84.19	62.83	61.43	68.76	89.12	72.31	83.75	48.90

Table 4: Ablation of core DiFuse design choices on UniWorld-V1. Each block varies one factor while fixing the others; shaded entries denote the final configuration.

		Summary		General Multimodal Understanding					Spatial Reasoning
Condition	Variant	Avg-G	Avg-S	MME norm	MMB	MMStar	GQA	RWQA	VSR	EmbSpatial	CVB-3D	3DSR-C
Correspondence	Aligned features (ours)	71.94	73.68	80.82	84.54	62.97	61.70	69.67	89.36	72.39	84.58	48.38
Correspondence	Image-shuffled features	71.33	72.60	80.25	84.19	62.52	61.46	68.24	88.87	71.65	82.83	47.04
Correspondence	Token-shuffled features	71.80	71.84	81.59	83.93	62.41	61.14	69.93	87.81	70.80	81.42	47.33
Correspondence	Random features	71.91	72.80	80.36	84.45	64.73	60.74	69.28	88.95	72.01	82.00	48.23

Table 5: Effect of spatial correspondence in DiFuse. Image and token shuffling disrupt the corresponding image-level and token-level spatial alignment, while random features replace the diffusion representation. The shaded row denotes the final configuration.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	UniWorld-V1	OmniGen2
Understanding backbone	Qwen2.5-VL-7B	Qwen2.5-VL-3B
Decoder hidden dimension	3,584	2,048
Decoder layers	28	36
DiT source layer	18	6
Target timestep	100	900
Cached feature dimension	1,024	1,024

Appendix

Table 6: Backbone-specific DiFuse configurations. DiT layer denotes the source layer in the native generation branch; injection layers denote layers in the multimodal decoder.

Setting	Value
Training examples	296,401
Training epochs	3
Optimization steps	6,948
Hardware	$8\times$ NVIDIA A100-80GB
Per-device batch size	2
Gradient accumulation	8

Appendix

Table 7: Training configuration for the UniWorld-V1 experiments. The effective batch size includes gradient accumulation over all GPUs.

Backbone	LoRA Params.	Fusion Params.	LoRA Size	Fusion Size
UniWorld-V1	323.0M	207.7M	646.0 MB	415.3 MB
OmniGen2	239.5M	157.3M	957.9 MB	314.6 MB

Appendix

Table 8: Trainable parameter and checkpoint storage costs for the final DiFuse configurations. The reported values correspond to the main UniWorld-V1 experiment and the OmniGen2 generalization experiment. Parameter counts may vary across ablation settings that change the number of injection layers or the feature dimension.

		EmbSpatial				CV-Bench-3D
Backbone	Variant	AI2-THOR	MP3D	ScanNet	Overall	Depth	Distance	Overall
	Base	68.93	71.61	69.42	69.97	83.33	78.83	81.08
UniWorld-V1	+ SFT	70.86	72.11	70.00	70.99	83.17	80.67	81.92
	+ DiFuse	72.64	73.36	71.17	72.39	85.33	83.83	84.58
	Base	55.45	65.11	64.00	61.46	76.50	71.67	74.08
OmniGen2	+ SFT	56.74	64.61	65.42	62.20	79.50	70.00	74.75

Appendix

Table 9: Category-wise spatial reasoning results on EmbSpatial and CV-Bench-3D (%). EmbSpatial is divided into AI2-THOR, Matterport3D, and ScanNet. CV-Bench-3D reports Depth Order and Relative Distance subtasks. The Overall column denotes the official aggregate accuracy.

Source layers	Inject.	$\bm{\lambda}$	Avg-G	Avg-S	VSR	EmbSpatial	CVB-3D	3DSR-C
Single-layer $g18$	$0$ – $3$	0.5	71.89	73.08	89.28	72.12	82.83	48.09
$g18+g53$	$0$ – $3$	0.5	71.90	73.04	88.87	71.40	83.75	48.15
$g18+g28$	$0$ – $3$	0.5	71.63	73.23	89.12	71.59	84.08	48.13
Single-layer $g18$	$0$ – $7$	0.5	72.00	72.08	88.30	69.81	81.17	49.06
$g18+g28$	$0$ – $7$	0.5	71.51	73.40	89.12	72.34	83.75	48.38
$g18+g53$	$0$ – $7$	0.5	72.02	73.00	88.63	72.25	83.17	47.97

Appendix

Table 10: Exploratory anchor-residual multi-layer fusion. All multi-layer variants use

g18

as the anchor and add either

g28

g53

through a learnable bounded residual. Rows are grouped by decoder injection range and main fusion coefficient

\lambda

. Shaded rows denote the matched single-layer references, with the darker row indicating the final DiFuse configuration. Bold numbers indicate the best Avg-G and Avg-S within each group.

Figure 4: Diffusion responses across timesteps at fixed layer $g18$ . Response maps are shown at $t\in\{50,100,200,300\}$ for three representative images. Spatial organization is retained across timesteps, while the response granularity varies.

Figure 5: Diffusion responses across candidate layers at fixed timestep $t=100$ . Response maps are shown for the vital-layer candidates identified following Stable Flow ( Avrahami et al., 2025 ) . The spatial responses vary substantially across diffusion depth despite using the same image and timestep.

Figure 6: Qualitative comparison between SFT and DiFuse. Examples cover spatial relations, depth and 3D reasoning, and general and fine-grained visual understanding. Compared with the matched SFT baseline, DiFuse corrects several errors involving spatial relations and depth reasoning while largely preserving predictions on general visual understanding examples. Red and green responses denote incorrect and correct predictions, respectively.

Multimodal Spatial Reasoning

See all 56 papers

Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning
Sep 17, 2026Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang +3Unified Multimodal Models Multimodal Reasoning
University of California, San Diego · Meta · University of Pennsylvania
Representation Forcing for Bottleneck-Free Unified Multimodal Models
May 29, 2026Yuqing Wang, Zhijie Lin, Ceyuan Yang +10Unified Multimodal Models Conditional Image Generation
University of Hong Kong · ByteDance Seed · The Chinese University of Hong Kong +2
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
Jun 15, 2026Shuai Wang, Liang Li, Yang Chen +3Diffusion Transformer Unified Multimodal Models
Nanjing University · ByteDance Seed · University of Hong Kong

Sep 17, 2026J/K move · Enter open · S save

Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning

Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Jiatao Gu+2

University of California, San Diego · Meta · University of Pennsylvania

Unified Multimodal Models Multimodal Reasoning VLM Reasoning Multimodal Diffusion Models MLLM Reasoning

Open paper AskPDF arXiv

Multimodal models increasingly think with different modalities such as images, 3D point clouds, and robot states, not just text. Yet each modality is still encoded into its own representation space, creating a modality-switching gap whenever reasoning moves from one modality to another. In this paper, we introduce Uni-LaDiR (Unified Latent Diffusion Reasoner), a framework that unifies different modalities into a shared latent space for multimodal reasoning. A unified encoder maps teacher reasoning steps from different modalities into latent thought tokens in a shared space, trained to extract the information needed for later reasoning steps and the final output. A diffusion reasoner, trained jointly with the encoder, generates these tokens at inference without teacher reasoning steps. Across eleven vision-language model (VLM) benchmarks and two vision-language-action (VLA) suites, Uni-LaDiR achieves relative gains over the strongest baselines of 7.3% on four mathematical and logical VLM benchmarks and 6.1% on RLBench manipulation tasks. Controlled comparisons show increasing gains as more teacher modalities are unified. These results suggest that unification improves multimodal reasoning by weaving it into a single thread, where the model predicts successive thoughts in a common representation space.

Abstract

Multimodal Spatial Reasoning

Explore similar work

Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning

Representation Forcing for Bottleneck-Free Unified Multimodal Models

UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer