Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generation

Derui Li; Qian Qiao; Yuhao Sun; Wenhao Guo; Peng Lu

cs.CVAug 21, 2026

Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generation

Authors: Derui Li, Qian Qiao, Yuhao Sun, Wenhao Guo, Peng Lu

Organizations: Beijing University of Posts and Telecommunications Beijing, China

Abstract

Panoramic image generation is increasingly important for immersive applications such as virtual reality, augmented reality, and 3D content creation. Unlike perspective images, panoramic images represent a viewer-centered $360^\circ$ surrounding space, where directional expressions such as left, right, front, and behind play a central role in spatial understanding. However, existing text-to-panorama methods largely rely on implicit spatial reasoning and often fail to faithfully ground object-level directional descriptions in spherical panoramic scenes. A straightforward alternative is to introduce explicit layouts, but requiring manually specified spatial conditions reduces the flexibility of language-based interaction and does not directly resolve the misalignment between egocentric directional language and panoramic image space. To address this issue, we propose PanoCtrl, an object-centric framework for controllable text-to-panorama generation. Our method explicitly bridges natural language and spherical panoramic space by converting textual descriptions into structured object-level spherical conditions and integrating them into the diffusion process. Specifically, we introduce PanoParse, a text-conditioned parser that predicts object semantics and spherical bounding field-of-view (BFoV) parameters, and PanoControl, which injects object-level semantic and spatial guidance into the diffusion transformer through object-aware attention and spatial residual enhancement. To support this task, we construct PanoGround, a dataset with object-level spherical annotations and diverse directional descriptions for controllable panoramic generation. Extensive experiments demonstrate that PanoCtrl achieves state-of-the-art performance in both spatial alignment and image quality.

Figures & tables

Figure 1. Motivation examples for controllable text-to-panorama generation. (a) Illustration of the alignment gap between egocentric directional space and ERP space. (b) Qualitative comparison with representative methods, including LayerPano3D ( Yang et al., 2025a ) and DiT360 ( Feng et al., 2025 ) , under directional descriptions. Red boxes indicate incorrect or spatially inconsistent object placements, while green boxes indicate correctly aligned ones.

Figure 2. Overall framework of PanoCtrl . PanoParse converts the input prompt into object-level spherical conditions, including semantic categories and BFoV parameters, and PanoControl integrates them into the diffusion transformer through object-aware attention and spatial residual enhancement for controllable panoramic image generation.

	Spatial Alignment			Image Quality
Method	OPR $\uparrow$	RTA $\uparrow$	SLE $\downarrow$	FID $\downarrow$	FAED $\downarrow$	IS $\uparrow$	CS $\uparrow$
PanFusion	76.61	19.87	52.30	86.31	9.46	2.59	23.21
SMGD	80.08	22.63	62.27	77.48	5.10	2.97	25.24
PAR	73.94	23.02	58.88	70.40	6.86	3.19	27.86
WorldGen	77.07	23.01	48.96	60.71	4.78	3.31	29.42
Matrix-3D	78.01	24.19	53.76	51.78	3.05	3.16	30.53

Table 1. Quantitative comparison on PanoGround.

Figure 3. Qualitative comparison on challenging panoramic scenes with multi-object directional descriptions. Captions are derived from the ground-truth panoramas, and colored phrases correspond to the colored boxes in the images. Compared with prior methods, Ours achieves more accurate directional grounding and more coherent layouts.

Model	Parse	Attn	Spa	OPR $\uparrow$	RTA $\uparrow$	SLE $\downarrow$	FID $\downarrow$	CS $\uparrow$
Baseline				82.47	24.35	44.82	50.94	27.83
I	✓			88.94	27.12	42.63	49.85	29.41
II	✓		✓	91.32	29.76	38.18	50.82	28.58
III	✓	✓		93.87	33.41	30.12	47.96	31.67
Ours	✓	✓	✓	98.59	36.91	25.34	46.86	32.54

Table 2. Ablation study of the main components in PanoCtrl .

Setting	OPR $\uparrow$	RTA $\uparrow$	SLE $\downarrow$	FID $\downarrow$	CS $\uparrow$
w/o $\mathcal{L}_{\text{parse}}$	90.26	32.84	31.92	47.28	31.42
w/o $\mathcal{L}_{\text{obj}}$	96.85	35.96	27.43	47.02	32.06
Ours	98.59	36.91	25.34	46.86	32.54

Table 3. Ablation study of the training objectives.

$\alpha$	OPR $\uparrow$	RTA $\uparrow$	SLE $\downarrow$	FID $\downarrow$	CS $\uparrow$
0.3	95.12	34.04	33.82	47.91	31.66
0.5	96.67	34.78	29.96	47.08	31.84
0.7	98.59	36.91	25.34	46.86	32.54
0.9	96.88	35.12	27.41	47.52	31.97

Table 4. Ablation study of the spatial blending coefficient

\alpha

Queries	OPR $\uparrow$	RTA $\uparrow$	SLE $\downarrow$	FID $\downarrow$	CS $\uparrow$
2	94.12	33.21	37.84	47.86	31.02
4	96.43	34.98	31.62	47.22	31.74
8	98.59	36.91	25.34	46.86	32.54
16	98.48	36.02	27.91	46.65	32.81

Table 5. Ablation study of the number of object queries.

Controllable Image Generation

Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.

See all 216 papers

Jul 9, 2026

Enhancing In-context Panoramic Generation via Geometric-aware Pretraining

In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream task-specific fine-tuning. To address the lack of large-scale, high-quality training data tailored to in-context panoramic tasks, we propose Canvas360Dataset, a collection of 1M high-quality paired panoramic samples for style transfer, inpainting, outpainting, and editing, enabling effective supervision across diverse in-context generation scenarios. On the modeling side, Canvas360 enhances text-to-panorama generation through parallel depth generation, velocity circular padding, and similarity loss regularization, enabling the model to learn geometry-aware representations, capture object distortion details, and improve geometric consistency and global coherence. Furthermore, empowered by strong panoramic priors, Canvas360 enables a unified in-context panoramic generation framework that supports diverse downstream tasks via token-level concatenation, surpassing prior methods in both task coverage and modeling flexibility. Extensive experiments show that Canvas360 improves panoramic image fidelity, achieving particularly strong performance on the panorama-specific FAED metric and competitive or leading results across the reported quantitative evaluations. More information can be found on our project page: https://zry000.github.io/Canvas360/

Haoran Feng, Ruiyang Zhang, Longyi Zhang +2

Insta360 Research · Tsinghua University · Beihang University +1

Conditional Image Generation Image Generation Image Editing

Ask PDF

May 13, 2026

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World

Multimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360-degree panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.

Changpeng Wang, Xin Lin, Junhan Liu +5

1Zhejiang University · University of California, San Diego · University of California, Irvine +1

3D Spatial Reasoning 3D Scene Understanding 3D VLMs

Ask PDF

Mar 14, 2026

Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation

We present GuidedSceneGen, a text-to-3D generation framework that produces metrically accurate, globally consistent, and semantically interpretable indoor scenes. Unlike prior text-driven methods that often suffer from geometric drift or scale ambiguity, our approach maintains an absolute world coordinate frame throughout the entire generation process. Starting from a textual scene description, we predict a global 3D layout encoding both semantic and geometric structure, which serves as a guiding proxy for downstream stages. A semantics- and depth-conditioned panoramic diffusion model then synthesizes 360° imagery aligned with the global layout, substantially improving spatial coherence. To explore unobserved regions, we employ a video diffusion model guided by optimized camera trajectories that balances coverage and collision avoidance, achieving up to 10x faster sampling compared to exhaustive path exploration. The generated views are fused using 3D Gaussian Splatting, yielding a consistent and fully navigable 3D scene in absolute scale. GuidedSceneGen enables accurate transfer of object poses and semantic labels from layout to reconstruction, and supports progressive scene expansion without re-alignment. Quantitative results and a user study demonstrate greater 3D consistency and layout plausibility compared to recent panoramic text-to-3D baselines.

Stefan Ainetter, Thomas Deixelberger, Edoardo A. Dominici +3

Huawei Technologies, Austria · Graz University of Technology, Austria · Huawei Technologies, Switzerland

Diffusion Model Guidance 3D Scene Generation 3D Scene Layout Generation Text-to-3D Generation Indoor Scene Generation

Ask PDF

Abstract

Controllable Image Generation

Explore similar work

Enhancing In-context Panoramic Generation via Geometric-aware Pretraining

PanoWorld: Towards Spatial Supersensing in 360∘^\circ∘ Panorama World

Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World