PoseAdapter: Dual-Stream 2.5D Controllable Image Generation for Complex Multi-Object Scenes

Yufeng Chi; Huimin Ma; Fan Gao; Zhice Niu; Keqin Li; Jianmin Li

cs.CVAug 16, 2026

PoseAdapter: Dual-Stream 2.5D Controllable Image Generation for Complex Multi-Object Scenes

Authors: Yufeng Chi, Huimin Ma, Fan Gao, Zhice Niu, Keqin Li, Jianmin Li

Organizations: Tsinghua University Beijing, China · National University of Defense Technology Hunan, China

Abstract

While Text-to-Image (T2I) diffusion models have achieved remarkable success, precise spatial and orientational control in multi-object scenes remains a persistent challenge. Existing methods either rely on computationally expensive dense 3D maps or suffer from severe attribute leakage and "cut-and-paste" artifacts. To address these limitations, we propose PoseAdapter, a lightweight framework for high-fidelity 2.5D controllable image generation. Instead of dense spatial maps, it establishes precise spatial-angular anchors using an efficient condition layout: individual object captions, 2D bounding boxes, and 3D angles. To resolve the generative trade-off between strict instance isolation and global coherence, we introduce a Context-Aware Dual-Stream Representation. By injecting local object tokens and relation-enriched scene tokens into the visual stream of modern MM-DiT architectures via parallel masked and unmasked pathways, PoseAdapter eliminates attribute leakage while preserving natural inter-object relationships and scene-level coherence. To support this paradigm, we construct OrientLayout, a high-quality dataset featuring standardized 2.5D annotations and instance-level decoupled semantics. Extensive experiments demonstrate that PoseAdapter outperforms state-of-the-art baselines in spatial accuracy, orientational precision, and multi-object visual fidelity. Code and dataset will be available at https://github.com/cyf23/PoseAdapter.

Figures & tables

Figure 1. High-fidelity 2.5D controllable generation by PoseAdapter . (a) Single-object Setting : Precise adjustment of 3D orientations (azimuth, polar, rotation) and spatial bounding boxes. (b) Multi-object Setting : Strict attribute decoupling in complex scenes, assigning specific identities and colors to distinct entities while eliminating attribute leakage. (a) Single-object generation controlled by 3D angles and 2D bounding boxes. (b) Multi-object scenes demonstrating precise attribute assignment without semantic blending.

Method	Arbitrary 3D Orient.	Lightweight Cond.	Real-World Domain	Arch. Agnostic
Zero-1-to-3 ( Liu et al., 2023a )	$\checkmark$	$\times$	$\times$	$\checkmark$
CompassControl ( Parihar et al., 2025 )	$\times$	$\checkmark$	$\times$	$\checkmark$
ORIGEN ( Min et al., 2025 )	$\checkmark$	$\times$	$\checkmark$	$\times$
LOOSECONTROL ( Bhat et al., 2024 )	$\times$	$\times$	$\checkmark$	$\checkmark$
SceneDesigner ( Qin et al., 2025 )	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
PoseAdapter (Ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1. Comparison of orientation-aware image generation paradigms. “Arch. Agnostic” denotes seamless integration with standard T2I architectures. “Lightweight Cond.” indicates that the method does not require dense spatial maps (e.g., depth or CNOCS maps) as input conditions.

Figure 2. Overall pipeline of the proposed PoseAdapter. Given a set of instance conditions (captions, 2D bounding boxes, and 3D angles), we first encode them into unified object tokens $\mathcal{T}^{o}$ . A scene transformer then models their interactions to produce relation-enriched scene tokens $\mathcal{T}^{s}$ . Finally, these dual-stream representations are exclusively injected into the pre-trained MM-DiT visual stream via a masked local stream (enforcing precise boundaries) and an unmasked global stream (preserving inter-object coherence). The overall architecture diagram of the proposed PoseAdapter pipeline. On the left, Condition Encoding processes object captions, 2D bounding boxes, and angles through respective encoders and a fusion MLP to generate object tokens. In the middle, a Shared Object Encoder (Transformer) converts these local tokens into relation-enriched scene tokens, establishing a Dual-Stream Representation. On the right, within an MM-DiT block, the visual hidden states interact with these tokens via two parallel cross-attention branches: a Local Stream (using layout masks) for the object tokens, and an unmasked Global Stream for the scene tokens. The outputs are aggregated and residually added to the visual stream.

Method	Overall Quality			OrientLayout-Single-Front				OrientLayout-Single-Back				OrientLayout-Multi
Method	FID $\downarrow$	CLIP $\uparrow$	O-CLIP $\uparrow$	$Acc_{ls}$ $\uparrow$	mIoU $\uparrow$	$Acc_{22.5^{\circ}}$ $\uparrow$	$Acc_{BOV}$ $\uparrow$	$Acc_{ls}$ $\uparrow$	mIoU $\uparrow$	$Acc_{22.5^{\circ}}$ $\uparrow$	$Acc_{BOV}$ $\uparrow$	$Acc_{ls}$ $\uparrow$	mIoU $\uparrow$	$Acc_{22.5^{\circ}}$ $\uparrow$	$Acc_{BOV}$ $\uparrow$
LOOSECONTROL	36.78	0.287	0.232	34.72	32.94	10.18	12.28	35.23	34.87	6.47	8.82	24.20	29.45	2.43	6.40
SceneDesigner	25.10	0.347	0.288	75.79	69.79	73.68	86.67	74.71	66.45	62.35	80.58	47.48	46.38	56.00	80.00
PoseAdapter (Ours)	24.60	0.344	0.310	97.84	87.96	87.02	95.44	99.40	88.89	71.76	88.82	91.67	81.72	72.70	93.22

Table 2. Quantitative comparison on OrientLayout.

Acc_{22.5^{\circ}}

: automated accuracy (OrientAnything);

Acc_{BOV}

: human verification.

Figure 3. Qualitative comparison of controllable image generation. The proposed PoseAdapter demonstrates precise layout and orientation control across diverse settings. The Condition column displays the respective inputs: 2.5D visualizations for the proposed approach, CNOCS maps for SceneDesigner, and depth maps for LOOSECONTROL. A large grid of images comparing 2.5D controllable image generation methods. The top rows display the input spatial conditions, showing our lightweight 2D bounding boxes with orientation arrows versus the dense CNOCS and depth maps used by baselines. The grid is divided into columns representing different full prompts, such as a dog, a lion, a locomotive, an airplane, and multi-object scenes like a room with cameras or chairs. The rows show the generated results from LOOSECONTROL, SceneDesigner, and our PoseAdapter. Visually, our results align strictly with the colored bounding boxes and full prompts, while baselines show obvious misalignments, missing objects, or color blending in multi-object columns.

Method	Backbone	Tr. Params	Injection	Prep. (s) $\downarrow$	Net. (s) $\downarrow$
LOOSECONTROL	SD 1.5	0.7M	CN + LoRA	$\sim$ 14.2	$\sim$ 2.6
SceneDesigner	SD 3.5	1,487.7M	ControlNet	$\sim$ 13.7	$\sim$ 2.4
PoseAdapter (Ours)	SD 3.5	228.0M	IP-Adapter	0.0	$\sim$ 3.0

Table 3. Efficiency and latency comparison (RTX 5090, 3 objects, 28 steps). “Tr. Params” denotes trainable parameters; “Prep.” indicates external preprocessing time. Our method avoids spatial map rendering entirely.

Method	Single-Object			Multi-Object
Method	Text	Orient.	Spatial	Text	Orient.	Spatial
LOOSECONTROL	45.2	15.5	38.0	16.4	3.5	7.5
SceneDesigner	92.5	82.0	77.5	60.2	45.5	32.0
PoseAdapter (Ours)	98.5	97.0	98.5	95.0	87.5	95.5

Table 4. User study results (approval rate %).

Figure 4. Qualitative ablation results of relying solely on the global stream. A qualitative comparison matrix with three columns: Condition, Global Stream (Scene Token), and Dual Stream (Ours). It displays generated images for three prompts: a pink flamingo and a white rooster, a yellow bulldozer and a silver helicopter, and a red oven and a silver coffee machine. The Global Stream shows attribute leakage, while the Dual Stream cleanly separates the attributes.

Method	Split	OrientLayout			Overall
Method	Split	$Acc_{ls}$ (%) $\uparrow$	mIoU (%) $\uparrow$	$Acc_{BOV}$ (%) $\uparrow$	CLIP $\uparrow$	O-CLIP $\uparrow$
w/o Box Mask Loss	Single	97.56	87.32	90.55	0.342	0.309
w/o Box Mask Loss	Multi	86.84	79.06	90.61	0.342	0.309
w/o $t$ -Sampling Shift	Single	96.40	86.21	89.45	0.341	0.307
w/o $t$ -Sampling Shift	Multi	87.86	78.44	89.91	0.341	0.307
w/o Both Strategies	Single	94.79	82.66	86.81	0.323	0.300

Table 5. Quantitative ablation study of PoseAdapter.

Figure 5. Qualitative ablation results of relying solely on the local stream. A qualitative comparison matrix with three columns: Condition, Local Stream (Object Token), and Dual Stream (Ours). It displays generated images for three prompts: a golden retriever on a wooden boat, a grey wolf and a bald eagle, and a bedroom scene with a sofa. The Local Stream shows disjointed composition and unnatural clipping (highlighted by a red box on the sofa), while the Dual Stream achieves photorealistic harmony.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation
Jun 5, 2026Dian Gu, Zhengyi YangDiffusion Transformer T2I Generation
Institute of Automation, Chinese Academy of Sciences Beijing, China
ASTRA: Enhancing Multi-Subject Generation with Retrieval-Augmented Pose Guidance and Disentangled Position Embedding
Apr 15, 2026Tianze Xia, Zijian Ning, Zonglin Zhao +1Multi-Subject Image Generation Image Generation
Huazhong University of Science and Technology
Camera Control for Text-to-Image Generation via Learning Viewpoint Tokens
Apr 21, 2026Xinxuan Lu, Charless Fowlkes, Alexander C. BergT2I Generation Controllable Image Generation
University of California, Irvine

Jun 5, 2026J/K move · Enter open · S save

TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation

Dian Gu, Zhengyi Yang

Institute of Automation, Chinese Academy of Sciences Beijing, China

Diffusion Transformer T2I Generation Image Generation T2I Diffusion Models Multimodal Diffusion Models Layout-Guided T2I Generation

Open paper AskPDF arXiv

Pose-guided text-to-image generation often suffers from limb distortions and feature crosstalk in complex multi-person scenarios. While existing UNet-based adapters struggle with long-range spatial dependencies, emerging Multimodal Diffusion Transformers (MM-DiTs) offer superior global modeling. However, naive signal concatenation in MM-DiTs severely disrupts pre-trained latent distributions. To address this, we propose TrioPose, a native pose-driven framework built upon the SD3.5M architecture. Specifically, we introduce a Triple-Stream Pose-Aware DiT (TSPA-DiT) that treats pose as an independent modality. It employs layer-wise activation and zero-initialized dual-residual injection to smoothly enforce geometric constraints while preserving pre-trained latent stability. To resolve severe multi-instance occlusions, we design a Learnable Relational Bias Mask that categorizes topological connectivity into fine-grained physical states, mapping them into continuous attention soft constraints to effectively decouple inter-instance interference. Furthermore, a Pose-Guided Spatial Loss Weighting strategy modulates the native diffusion objective using heatmap-derived error maps, focusing anatomical supervision strictly on distortion-prone regions. Extensive experiments demonstrate that TrioPose achieves state-of-the-art performance across challenging benchmarks, including Human-Art, CrowdPose, and OCHuman. Notably, it attains an AP of

64.33

on Human-Art, representing a

30\%

improvement over prior arts, while setting new standards for visual fidelity and text-image semantic alignment in complex multi-human generation.

Abstract

Image Generation

Explore similar work

TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation

ASTRA: Enhancing Multi-Subject Generation with Retrieval-Augmented Pose Guidance and Disentangled Position Embedding

Camera Control for Text-to-Image Generation via Learning Viewpoint Tokens