Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · School of Vehicle and Mobility, Tsinghua University · Ant Group, China
Unified models for visual understanding and generation have made rapid progress, yet they still lack the ability to understand and manipulate the spatial states of object instances. Existing models can describe objects in natural language, but they struggle to precisely represent continuous object poses and generate geometrically consistent images under target viewpoints. To mitigate this, we propose Object-Uni, a unified model for object-centric spatial understanding and controllable generation. Specifically, we formulate object-centric spatial intelligence as a unified problem connecting pose perception, spatial reasoning, pose-conditioned generation, and object-centric novel view synthesis. We treat object pose as an explicit geometric variable shared by understanding and generation, rather than merely a prediction label or control signal. To make pose usable by multimodal large language models, we propose a viewpoint-based orientation abstraction that maps orientation into structured viewpoint descriptions while preserving continuous geometric supervision. We further construct an object-centric spatial benchmark (UniSpatial-80K) and train a unified model with an object-token-grounded pose anchor to associate each instance with its pose state. Experiments show that our model improves object-level pose understanding and pose-controllable generation, moving unified models from describing objects toward manipulating spatial states.
Figures & tables
Figure 1: Versatile capabilities of Object-Uni model. It unifies (a) object-centric generation and (c) understanding, also supporting orientation estimation and (b) object-centric novel view synthesis.
Figure 2: Overview of Object-Uni. Our model unifies object-centric spatial understanding and orientation-controllable generation. The dotted modules denote object-centric novel view synthesis.
Approach
Azimuth [in ∘ ]
Polar [in ∘ ]
Rotation [in ∘ ]
Error ↓
AUC@5/10/30 ↑
Error ↓
AUC@5/10/30 ↑
Error ↓
AUC@5/10/30 ↑
KITTI-Cityscapes
Single
Random
86.18
2.81
4.16
10.47
43.57
4.25
8.04
19.27
94.49
1.36
2.82
8.36
GPT-4o
73.38
7.52
13.48
32.05
2.93
45.60
72.41
90.80
0.78
99.30
99.30
99.47
Gemini-2.5-pro
69.32
10.14
16.21
31.78
2.92
47.87
72.64
90.88
0.80
99.26
99.63
99.88
OriAny.V1
27.33
14.99
28.03
56.72
7.61
23.42
49.55
79.24
1.53
98.29
98.73
99.03
OriAny.V2
29.94
15.48
27.07
55.40
2.34
59.72
79.44
93.09
3.65
31.46
65.61
88.54
Table 1: Evaluation results on object orientation estimation. The results are grouped by dataset and split. Best results in each dataset-split block are shown in bold, and second-best results are underlined. The Object-Uni w/o represents our model without object-token-grounded pose anchor.
Figure 3: Evaluation of pose-controllable generation in single- and multi-object scenarios. The reference image in the first row serves as the source for the input text and 9D pose annotations. Object-Uni shows better text alignment and pose consistency under diverse pose conditions.
Localization
Azimuth [in ∘ ]
Polar [in ∘ ]
Rotation [in ∘ ]
Gen
Approach
mIoU ↑
Accls↑
Err ↓
AUC@10/30 ↑
Err ↓
AUC@10/30 ↑
Err ↓
AUC@10/30 ↑
CLIP ↑
ImageNet3D
Sgl.
SceneD.
66.45
68.81
62.10
13.30
24.59
15.77
35.25
58.92
12.12
41.95
70.78
0.333
Object-Uni
74.31
83.40
34.40
30.14
50.74
10.61
47.32
71.77
9.68
55.57
78.12
0.341
Mul.
SceneD.
39.03
20.89
67.83
13.99
25.67
20.24
32.48
54.65
16.69
46.01
69.54
0.334
Object-Uni
54.43
44.86
50.66
22.11
39.43
12.64
42.38
67.36
11.49
58.37
77.56
0.344
KITTI
Sgl.
SceneD.
38.40
26.45
68.17
10.40
24.60
4.03
70.79
87.28
7.86
53.15
76.30
0.319
Table 2: Evaluation results on pose-controllable generation. We compare SceneDesigner and our Object-Uni model on ImageNet3D, KITTI, and Objectron. The best results are marked in bold.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Angle type
Range
Language description
Azimuth ϕi
[337.5∘,360∘)∪[0∘,22.5∘)
front
[22.5∘,67.5∘)
front-right quarter
[67.5∘,112.5∘)
right side
[112.5∘,157.5∘)
back-right quarter
[157.5∘,202.5∘)
back
[202.5∘,247.5∘)
back-left quarter
Appendix
Table 3: Mapping from continuous orientation angles to viewpoint descriptions.
Source
Train / Val
Test-Single
Test-Multi
File-level Total
KITTI-Cityscapes
1,266
363
303
1,932
ImageNet3D
39,765
1,229
146
41,140
Objectron
42,221
2,500
475
45,196
Total
83,252
4,092
924
88,268
Appendix
Table 4: Source and split composition of UniSpatial-80K.
Category
Obj Count
Obj %
Img Count
Img Coverage
shoes
11,962
13.09
6,981
8.39
chair
6,058
6.63
5,579
6.70
bottle
5,744
6.29
5,726
6.88
laptop
5,585
6.11
5,479
6.58
cup
5,582
6.11
5,427
6.52
camera
5,411
5.92
5,325
6.40
Appendix
Table 5: Top categories in UniSpatial-80K ranked by object count.
Figure 4: Additional results of pose-controllable generation in single- and multi-object scenarios. The reference image in the first row serves as the source for the input text and 9D pose annotations.
Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.
Zeyu Liu, Zanlin Ni, Yang Yue +5
1Tsinghua University · 2Kolors Team, Kuaishou Technology
While Text-to-Image (T2I) diffusion models have achieved remarkable success, precise spatial and orientational control in multi-object scenes remains a persistent challenge. Existing methods either rely on computationally expensive dense 3D maps or suffer from severe attribute leakage and "cut-and-paste" artifacts. To address these limitations, we propose PoseAdapter, a lightweight framework for high-fidelity 2.5D controllable image generation. Instead of dense spatial maps, it establishes precise spatial-angular anchors using an efficient condition layout: individual object captions, 2D bounding boxes, and 3D angles. To resolve the generative trade-off between strict instance isolation and global coherence, we introduce a Context-Aware Dual-Stream Representation. By injecting local object tokens and relation-enriched scene tokens into the visual stream of modern MM-DiT architectures via parallel masked and unmasked pathways, PoseAdapter eliminates attribute leakage while preserving natural inter-object relationships and scene-level coherence. To support this paradigm, we construct OrientLayout, a high-quality dataset featuring standardized 2.5D annotations and instance-level decoupled semantics. Extensive experiments demonstrate that PoseAdapter outperforms state-of-the-art baselines in spatial accuracy, orientational precision, and multi-object visual fidelity. Code and dataset will be available at https://github.com/cyf23/PoseAdapter.
Yufeng Chi, Huimin Ma, Fan Gao +3
Tsinghua University Beijing, China · National University of Defense Technology Hunan, China
Unified Multimodal Large Language Models (MLLMs) offer a promising paradigm for unifying visual understanding and generation, yet they still struggle to follow complex spatial instructions and logical constraints in controllable image generation. To address this gap, we present ATLAS, a unified framework that equips MLLMs with a human-like "Think, Plan, and Paint" paradigm. We adopt layout as the shared representation that connects the three stages, enabling the model to reason about spatial requirements, plan explicit object arrangements, and render the final image. We further improve plan-to-image fidelity with reinforcement-learning-based layout alignment. We instantiate ATLAS at 7B and 80B scales, achieving state-of-the-art performance among MLLMs on image generation benchmarks and an average 65.31% improvement over existing layout-based unified MLLMs. On spatially related tasks, ATLAS obtains an average 23.06% gain over the base models. Through the same layout interface, ATLAS also supports instruction-guided editing and multimodal grounding. We further introduce ATLAS-Reasoning, a benchmark for evaluating generation under complex spatial instructions.
Junhao Liu, Jian-Wei Zhang, Tao Huang +3
Hunyuan, Tencent · School of Computer Science, Peking University