StructDiff: A Structure-Preserving and Spatially Controllable Diffusion Model for Single-Image Generation
Authors: Yinxi He, Kang Liao, Chunyu Lin, Tianyi Wei, Yao Zhao
Organizations: Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China, and also with Visual Intelligence +X International Cooperation Joint Laboratory of MOE, Beijing 100044, China. · College of Computing and Data Science, Nanyang Technological University, Singapore.
This paper introduces StructDiff, a generative framework based on a single-scale diffusion model for single-image generation. Single-image generation aims to synthesize diverse samples with similar visual content to the source image by capturing its internal statistics, without relying on external data. However, existing methods often struggle to preserve the structural layout, especially for images with large rigid objects or strict spatial constraints. Moreover, most approaches lack spatial controllability, making it difficult to guide the structure or placement of generated content. To address these challenges, StructDiff introduces an adaptive receptive field module to maintain both global and local distributions. Building on this foundation, StructDiff incorporates 3D positional encoding (PE) as a spatial prior, allowing flexible control over positions, scale, and local details of generated objects. To our knowledge, this spatial control capability represents the first exploration of PE-based manipulation in single-image generation. Furthermore, we propose a novel evaluation criterion for single-image generation based on large language models (LLMs). This criterion specifically addresses the limitations of existing objective metrics and the high labor costs associated with user studies. StructDiff also demonstrates broad applicability across downstream tasks, such as text-guided image generation, image editing, outpainting, and paint-to-image synthesis. Extensive experiments demonstrate that StructDiff outperforms existing methods in structural consistency, visual quality, and spatial controllability. The project page is available at https://butter-crab.github.io/StructDiff/.
Figures & tables
Fig. 1: StructDiff enables diverse generation and controllable editing from a single training image. Top : Diverse generation at arbitrary scales. Middle : Controllable generation through PE-guided geometric editing and mask-guided fine-tuning. Bottom : Various applications without retraining. Results demonstrate effectiveness in structural preservation and spatial controllability.
Fig. 2: Architectural paradigm comparison in single-image generation. (a) Multi-scale methods train separate generators at different resolutions, enabling multi-scale perception but suffering from error accumulation during hierarchical generation, leading to structural distortions in generated results. (b) Single-scale methods train one generator at fixed resolution, avoiding error accumulation but struggling with fixed receptive fields that cannot simultaneously capture fine textures and global structures, resulting in incomplete structure preservation. (c) Our approach varies receptive field sizes at fixed resolution, achieving multi-scale perception without error accumulation while maintaining structural consistency.
Fig. 3: Overview of StructDiff framework. (a) StructDiff trains on randomly cropped patches from a single image to learn internal statistics and generate diverse outputs. (b) The fully convolutional architecture uses Adaptive Receptive Field (ARF) Blocks with residual connections, removing downsampling and attention modules to prevent overfitting. Time embeddings are injected into each block and Fourier-embedded positional encodings enable spatial control. (c) ARF Blocks contain multiple parallel branches with different kernel sizes to extract features at varying receptive fields. The attention mechanism dynamically assigns weights to different branches, with larger receptive fields receiving higher weights when processing images containing large foreground structures, enabling adaptive balance between fine-detail modeling and structure preservation. (d) StructDiff supports default mode for random sampling and controllable mode guided by customized positional encodings, enabling spatial manipulation while maintaining structural coherence.
Fig. 4: Qualitative comparison with other methods. StructDiff vs. other methods on large-object images (rows 1st-2nd) and natural images (rows 3rd-4th). StructDiff achieves better visual quality and structural consistency across various image types.
Dataset
Method
Metrics
Point of Scoring (↑)
SIFID (↓)
MUSIQ (↑)
DB-CNN (↑)
LPIPS (↑)
Generation
Absence of
Total (5)
Quality (2)
Distortion (3)
Places50
SinGAN [ 1 ]
0.09
46.80
0.57
0.266
-
-
-
SinDDM [ 2 ]
0.34
44.27
0.57
0.210
-
-
-
SinFusion [ 5 ]
0.64
48.40
0.57
0.368
-
-
-
SinDiffusion [ 3 ]
0.06
46.79
0.56
0.387
-
-
-
TABLE I: Quantitative comparison on three datasets
Fig. 5: Validation of LLM-based evaluation through user study comparison. Overall preference percentages show high correlation between LLM-based scores (SIQS-G, SIQS-A) and user study results (User-G, User-A). Each bar shows the proportion where StructDiff is preferred ( teal ), results are tied ( gray ), or competing methods are preferred ( orange ). The high correlation validates LLM-based evaluation as an efficient alternative to labor-intensive user studies.
Metric
SIQS-G
MUSIQ
DB-CNN
Agreement Rate ( ↑ )
0.925
0.706
0.714
Cohen’s Kappa ( ↑ )
0.783
-0.063
-0.057
TABLE II: Agreement Analysis Between Metrics and User Study
Fig. 6: Qualitative comparison of spatially controllable generation. StructDiff with positional encoding guidance vs. SinDDM with ROI guidance for controlling foreground objects ( yellow boxes). While SinDDM generates blurry edges and artifacts, StructDiff produces sharper edges and maintains background layout. Results validate the effectiveness of 3D positional encoding for spatial control.
Fig. 7: Qualitative comparison between DiT-based large models and StructDiff on diverse generation from a single image. While DiT-based models struggle to capture fine-grained internal statistics and exhibit limited diversity or distorted artifacts, StructDiff faithfully preserves local structural patterns and generates highly diverse and visually realistic variations.
Fig. 8: Fine-grained local control through mask modification. StructDiff enables precise facial feature adjustments by modifying only the mask component of positional encoding. While SinDDM produces severe distortions, StructDiff achieves intuitive local control while preserving facial structure.
Region
Metric
Method
StructDiff
SinDDM
Background
PSNR-BG ( ↑ )
21.90
16.62
SSIM-BG ( ↑ )
0.65
0.53
Foreground
PSNR-FG ( ↑ )
34.44
17.94
SSIM-FG ( ↑ )
0.96
0.89
TABLE III: Quantitative Evaluation of Spatial Control
Fig. 9: Ablation study of StructDiff components. Top: ARF and foreground-aware loss effects. Without both, structures fragment. Combined components achieve coherent structure. Bottom left: Positional encoding strategies. 2D PE causes artifacts when shifting objects. 3D PE with mask enables precise foreground-background separation. Bottom right: Fourier embedding decouples spatial priors, preventing background shifts during foreground translation.
Fig. 10: StructDiff applications across diverse tasks without retraining. Text-guided Generation: CLIP guidance enables style control, content adaptation, and layout adjustment. Paint-to-Image/Editing: Structure-preserving sampling maintains global structure while allowing local modifications. Style Transfer/Harmonization: Preserves structural coherence during transformation. Outpainting: Seamless content extension maintains visual consistency. Results demonstrate high quality and versatility across image synthesis tasks.
Fig. 11: Failure cases of StructDiff. The method struggles with foreground-background separation in complex backgrounds and cannot complete occluded parts as humans would expect due to limited semantic information.
Generative models trained on a single image often struggle to balance global structural integrity and local diversity. Existing single-image generation methods commonly rely on random noise to drive the generation process and lack explicit global structural constraints, making the generated results prone to spatial structural misalignment when structural variations occur. To address the issue, Feature Rearrangement in Pre-Shape Space for Single-Image Generation (FRPSS) is proposed in this paper. The core of FRPSS is the Manifold Structural Rearrangement with Feature Augmentation on Geodesic Surface (MSR-FAGS) module. MSR-FAGS replaces the randomly initialized features of the low-scale generator with rearranged Pre-Shape features and uses the features to guide image generation at subsequent scales, thereby reducing the risk of structural misalignment. To support downstream tasks such as stylization, a Scale-adaptive Sliding-window Patch Extraction (SSPE) strategy is further designed, and a directional Contrastive Language-Image Pre-training supervision module with SSPE (CLIP-SSPE) is constructed. Qualitative and quantitative experiments demonstrate that FRPSS achieves the best Single Image Fréchet Inception Distance (SIFID) scores on all three datasets while maintaining competitive Learned Perceptual Image Patch Similarity (LPIPS). Further qualitative experiments verify the effectiveness of FRPSS across multiple downstream tasks with the CLIP-SSPE module.
Yuexing Han, Haoxuan Zhang, Bing Wang
School of Computer Engineering and Science, Shanghai University, 99 Shangda Road, Shanghai 200444, People’s Republic of China · Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education
While Text-to-Image (T2I) diffusion models have achieved remarkable success, precise spatial and orientational control in multi-object scenes remains a persistent challenge. Existing methods either rely on computationally expensive dense 3D maps or suffer from severe attribute leakage and "cut-and-paste" artifacts. To address these limitations, we propose PoseAdapter, a lightweight framework for high-fidelity 2.5D controllable image generation. Instead of dense spatial maps, it establishes precise spatial-angular anchors using an efficient condition layout: individual object captions, 2D bounding boxes, and 3D angles. To resolve the generative trade-off between strict instance isolation and global coherence, we introduce a Context-Aware Dual-Stream Representation. By injecting local object tokens and relation-enriched scene tokens into the visual stream of modern MM-DiT architectures via parallel masked and unmasked pathways, PoseAdapter eliminates attribute leakage while preserving natural inter-object relationships and scene-level coherence. To support this paradigm, we construct OrientLayout, a high-quality dataset featuring standardized 2.5D annotations and instance-level decoupled semantics. Extensive experiments demonstrate that PoseAdapter outperforms state-of-the-art baselines in spatial accuracy, orientational precision, and multi-object visual fidelity. Code and dataset will be available at https://github.com/cyf23/PoseAdapter.
Yufeng Chi, Huimin Ma, Fan Gao +3
Tsinghua University Beijing, China · National University of Defense Technology Hunan, China
Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/
Xin Lin, Haodong Li, Zhifei Zhang +5
University of California, San Diego · Adobe Research · Nanyang Technological University