Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation

Mining Tan; Yinuo Wang; Ziqi Zhou; Weize Quan; Sifei Li; Jingdong Chen; DanDan Zheng; Libin Wang

cs.CVAug 24, 2026

Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation

Authors: Mining Tan, Yinuo Wang, Ziqi Zhou, Weize Quan, Sifei Li, Jingdong Chen, DanDan Zheng, Libin Wang,

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · School of Vehicle and Mobility, Tsinghua University · Ant Group, China

Abstract

Unified models for visual understanding and generation have made rapid progress, yet they still lack the ability to understand and manipulate the spatial states of object instances. Existing models can describe objects in natural language, but they struggle to precisely represent continuous object poses and generate geometrically consistent images under target viewpoints. To mitigate this, we propose Object-Uni, a unified model for object-centric spatial understanding and controllable generation. Specifically, we formulate object-centric spatial intelligence as a unified problem connecting pose perception, spatial reasoning, pose-conditioned generation, and object-centric novel view synthesis. We treat object pose as an explicit geometric variable shared by understanding and generation, rather than merely a prediction label or control signal. To make pose usable by multimodal large language models, we propose a viewpoint-based orientation abstraction that maps orientation into structured viewpoint descriptions while preserving continuous geometric supervision. We further construct an object-centric spatial benchmark (UniSpatial-80K) and train a unified model with an object-token-grounded pose anchor to associate each instance with its pose state. Experiments show that our model improves object-level pose understanding and pose-controllable generation, moving unified models from describing objects toward manipulating spatial states.

Figures & tables

Figure 1: Versatile capabilities of Object-Uni model. It unifies (a) object-centric generation and (c) understanding, also supporting orientation estimation and (b) object-centric novel view synthesis.

Figure 2: Overview of Object-Uni. Our model unifies object-centric spatial understanding and orientation-controllable generation. The dotted modules denote object-centric novel view synthesis.

		Approach	Azimuth [in ∘ ]				Polar [in ∘ ]				Rotation [in ∘ ]
		Approach	Error $\downarrow$	AUC@5/10/30 $\uparrow$			Error $\downarrow$	AUC@5/10/30 $\uparrow$			Error $\downarrow$	AUC@5/10/30 $\uparrow$
KITTI-Cityscapes	Single	Random	86.18	2.81	4.16	10.47	43.57	4.25	8.04	19.27	94.49	1.36	2.82	8.36
		GPT-4o	73.38	7.52	13.48	32.05	2.93	45.60	72.41	90.80	0.78	99.30	99.30	99.47
		Gemini-2.5-pro	69.32	10.14	16.21	31.78	2.92	47.87	72.64	90.88	0.80	99.26	99.63	99.88
		OriAny.V1	27.33	14.99	28.03	56.72	7.61	23.42	49.55	79.24	1.53	98.29	98.73	99.03
		OriAny.V2	29.94	15.48	27.07	55.40	2.34	59.72	79.44	93.09	3.65	31.46	65.61	88.54

Table 1: Evaluation results on object orientation estimation. The results are grouped by dataset and split. Best results in each dataset-split block are shown in bold, and second-best results are underlined. The Object-Uni w/o represents our model without object-token-grounded pose anchor.

Figure 3: Evaluation of pose-controllable generation in single- and multi-object scenarios. The reference image in the first row serves as the source for the input text and 9D pose annotations. Object-Uni shows better text alignment and pose consistency under diverse pose conditions.

			Localization		Azimuth [in ∘ ]			Polar [in ∘ ]			Rotation [in ∘ ]			Gen
		Approach	mIoU $\uparrow$	$Acc_{ls}$ $\uparrow$	Err $\downarrow$	AUC@10/30 $\uparrow$		Err $\downarrow$	AUC@10/30 $\uparrow$		Err $\downarrow$	AUC@10/30 $\uparrow$		CLIP $\uparrow$
ImageNet3D	Sgl.	SceneD.	66.45	68.81	62.10	13.30	24.59	15.77	35.25	58.92	12.12	41.95	70.78	0.333
	Sgl.	Object-Uni	74.31	83.40	34.40	30.14	50.74	10.61	47.32	71.77	9.68	55.57	78.12	0.341
	Mul.	SceneD.	39.03	20.89	67.83	13.99	25.67	20.24	32.48	54.65	16.69	46.01	69.54	0.334
	Mul.	Object-Uni	54.43	44.86	50.66	22.11	39.43	12.64	42.38	67.36	11.49	58.37	77.56	0.344
KITTI	Sgl.	SceneD.	38.40	26.45	68.17	10.40	24.60	4.03	70.79	87.28	7.86	53.15	76.30	0.319

Table 2: Evaluation results on pose-controllable generation. We compare SceneDesigner and our Object-Uni model on ImageNet3D, KITTI, and Objectron. The best results are marked in bold.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Angle type	Range	Language description
Azimuth $\phi_{i}$	$[337.5^{\circ},360^{\circ})\cup[0^{\circ},22.5^{\circ})$	front
	$[22.5^{\circ},67.5^{\circ})$	front-right quarter
	$[67.5^{\circ},112.5^{\circ})$	right side
	$[112.5^{\circ},157.5^{\circ})$	back-right quarter
	$[157.5^{\circ},202.5^{\circ})$	back
	$[202.5^{\circ},247.5^{\circ})$	back-left quarter

Appendix

Table 3: Mapping from continuous orientation angles to viewpoint descriptions.

Source	Train / Val	Test-Single	Test-Multi	File-level Total
KITTI-Cityscapes	1,266	363	303	1,932
ImageNet3D	39,765	1,229	146	41,140
Objectron	42,221	2,500	475	45,196
Total	83,252	4,092	924	88,268

Appendix

Table 4: Source and split composition of UniSpatial-80K.

Category	Obj Count	Obj %	Img Count	Img Coverage
shoes	11,962	13.09	6,981	8.39
chair	6,058	6.63	5,579	6.70
bottle	5,744	6.29	5,726	6.88
laptop	5,585	6.11	5,479	6.58
cup	5,582	6.11	5,427	6.52
camera	5,411	5.92	5,325	6.40

Appendix

Table 5: Top categories in UniSpatial-80K ranked by object count.

Figure 4: Additional results of pose-controllable generation in single- and multi-object scenarios. The reference image in the first row serves as the source for the input text and 9D pose annotations.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 295 papers

Steering Visual Generation in Unified Multimodal Models with Understanding Supervision
May 7, 2026Zeyu Liu, Zanlin Ni, Yang Yue +5Unified Multimodal Models Image Generation
1Tsinghua University · 2Kolors Team, Kuaishou Technology
PoseAdapter: Dual-Stream 2.5D Controllable Image Generation for Complex Multi-Object Scenes
Aug 16, 2026Yufeng Chi, Huimin Ma, Fan Gao +3Image Generation Controllable Image Generation
Tsinghua University Beijing, China · National University of Defense Technology Hunan, China
Think, Plan, Paint: Layout-Aware Reasoning for Controllable Image Generation in Unified Models
Jul 17, 2026Junhao Liu, Jian-Wei Zhang, Tao Huang +3Unified Multimodal Models Image Generation
Hunyuan, Tencent · School of Computer Science, Peking University

May 7, 2026J/K move · Enter open · S save

Steering Visual Generation in Unified Multimodal Models with Understanding Supervision

Zeyu Liu, Zanlin Ni, Yang Yue, Cheng Da+4

1Tsinghua University · 2Kolors Team, Kuaishou Technology

Unified Multimodal Models Image Generation Multimodal Generation Multimodal Foundation Models Multimodal Understanding

Open paper AskPDF arXiv

Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.

Abstract

Unified Multimodal Models

Explore similar work

Steering Visual Generation in Unified Multimodal Models with Understanding Supervision

PoseAdapter: Dual-Stream 2.5D Controllable Image Generation for Complex Multi-Object Scenes

Think, Plan, Paint: Layout-Aware Reasoning for Controllable Image Generation in Unified Models