Organizations: East China Normal University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
Pixel-space diffusion models directly model image distributions but remain difficult to optimize. Recent methods alleviate this challenge through target reparameterization, while still relying on a fixed clean-image target throughout denoising. Through empirical analysis, we identify a scale-time mismatch: image structures become predictable from coarse to fine as noise decreases, whereas existing models are forced to predict the full image even under high noise, resulting in low-SNR gradients that hinder optimization. To resolve this mismatch, we propose Observation Operator Diffusion, a unified framework that aligns both the supervision trajectory and feature refinement with the intrinsic recovery order of image structures. Specifically, we replace fixed full-image supervision along the standard flow path with a time-indexed observation trajectory that evolves from coarse structures to the full image during denoising. This trajectory is instantiated with a family of Gaussian-Lanczos operators at varying observation scales, yielding a path-consistent training objective. We further introduce GL-CoDA, a decoder that injects scale-specific Gaussian-Lanczos observations across decoding stages for coarse-to-fine feature refinement. Extensive experiments show that the proposed approach converges substantially faster while consistently improving generation quality, achieving an FID of 1.52 on ImageNet-256.
Figures & tables
Figure 1: Fixed full-image supervision vs. time-indexed observation trajectory. Existing pixel-space diffusion models use the full clean image x0 as the target throughout the denoising path. We instead introduce a time-indexed observation trajectory yt=At(x0) , whose image content evolves from coarse structure to the full image as denoising proceeds.
Figure 2: Scale-dependent predictability during denoising. We decompose the single-step prediction residual into multiscale structural bands and measure the prediction error at each scale. The heatmap (denoising time × spatial scale) and representative trajectories (inset) show that coarse structures become recoverable earlier than fine details, exposing the mismatch with fixed full-image supervision.
Figure 3: Optimization benefits of the observation trajectory. Left: Observation Operator Diffusion converges substantially faster than conventional full-image supervision while achieving a lower final FID. Right: Gradient signal-to-noise ratio (SNR) over diffusion time. By avoiding supervision of unresolved fine details, the observation trajectory maintains higher-SNR gradients in the high-noise regime, leading to more efficient optimization.
Figure 4: Overview of the proposed Observation Operator Diffusion framework. Top: Gaussian–Lanczos observation trajectory, where a family of operators At maps the clean image x0 to time-indexed observations yt=At(x0) , progressively transitioning from coarse structures to the full image along denoising. Bottom: two-stage Patch–Pixel backbone, comprising a Patch-DiT encoder and a GL-CoDA decoder. The GL structure branch injects scale-specific structural responses across decoding stages. The right panel details the construction of each Gaussian–Lanczos operator At .
Figure 5: Spatial and frequency responses of the Gaussian–Lanczos kernel. (a) Spatial response. Gaussian smoothing attenuates the signed sidelobes of the Lanczos component, reducing potential ringing. (b) Frequency response. At a matched −3dB cutoff, the Gaussian–Lanczos kernel exhibits a narrower transition band than the Gaussian-only kernel, yielding sharper scale selectivity and stronger attenuation of high-frequency detail.
Space
Method
Epochs
#params
gFID ↓
sFID ↓
IS ↑
Precision ↑
Recall ↑
Latent
REPA ( Yu et al. 2024 )
800
675M
1.42
4.70
305.7
0.80
0.65
SVG-XL ( Shi et al. 2025 )
1400
675M
1.92
–
264.9
–
–
DDT-XL ( Wang et al. 2025 )
400
675M
1.26
–
310.6
0.79
0.65
RAE-XL ( Zheng et al. 2025 )
800
839M
1.13
–
262.6
0.78
0.67
Pixel
FractalMAR-H ( Li et al. 2025 )
600
844M
6.15
–
348.9
0.81
0.46
FARMER ( Zheng et al. 2026 )
320
1.9B
3.60
–
269.2
0.81
0.51
Table 1: Quantitative results on ImageNet 256×256 for class-conditioned generation.
Figure 6: Class-conditional samples on ImageNet. Top: samples at 512×512 . Bottom: samples at 256×256 .
Space
Method
gFID ↓
sFID ↓
IS ↑
Precision ↑
Recall ↑
Latent
DiT-XL ( Peebles and Xie 2023a )
3.04
5.02
240.8
0.84
0.54
SiT-XL ( Ma et al. 2024 )
2.62
4.18
252.2
0.84
0.57
MaskDiT ( Zheng et al. 2023 )
2.50
5.10
256.2
0.83
0.56
U-ViT-H ( Bao et al. 2023 )
4.05
–
263.8
0.84
0.48
REPA ( Yu et al. 2024 )
2.08
4.19
274.6
0.83
0.58
RAE-XL ( Zheng et al. 2025 )
1.13
–
259.6
0.80
0.63
Table 2: Quantitative comparison on ImageNet 512×512 .
Architecture
#params
Full-image path
G–L path
JiT
2B
4.11
3.79
PixelDiT
797M
2.36
2.14
Ours
798M
2.22
1.95
Table 3: Observation-Path Ablation across Architectures, measured by FID.
(a) Kernel
(b) Depth assignment
Kernel
FID ↓
Assignment
FID ↓
Identity
2.22
Coarse only
2.28
Gaussian
2.10
Fine only
2.12
Lanczos
2.08
Fine → Coarse
2.30
G–L
1.95
Coarse → Fine
1.95
Table 4: Kernel-Component and Depth-Assignment Ablations.
Initial kernel scales
Schedule
FID ↓
Lanczos f0
Gaussian σ0
4
1
Linear
2.00
2
2
Linear
2.06
2
1
Linear
1.95
2
1
Cosine
2.08
Table 5: Kernel-Scale and Temporal-Schedule Sensitivity.
Figure 7: Depth-wise feature visualization at t=0.25 . The first column shows x0 (top) and z0.25 (bottom); B1–B4 show block input features (top) and our injected Gaussian–Lanczos structural responses (bottom).
Pixel-space diffusion has recently emerged as a promising direction for high-fidelity image generation by modeling images directly in the original pixel domain. However, pixel-space diffusion is computationally demanding due to the extremely high dimensionality of natural images. For efficiency, existing pixel diffusion models either compromise fine details with large-patch tokenization or confine subsequent refinement within individual patches. Such intra-patch refinement inevitably restricts structural continuity across patch boundaries and long-range token interactions, limiting refinement quality. To address these issues, we propose PixSGR, a novel Pixel diffusion framework with Sparse Global Refinement tailored for modeling the distribution of natural images directly in pixel space. PixSGR starts from a supervised low-channel bottleneck to efficiently capture the low-dimensional manifold of natural images. It then progressively expands the channel dimensionality and spatial resolution to recover increasingly fine-grained structures. At the spatial refinement stage, coarse-scale attention maps preselect globally relevant interactions to pre-sparsify fine-scale attention, enabling non-local refinement beyond isolated patches without the quadratic cost of dense attention. Extensive experiments on ImageNet validate the effectiveness of PixSGR. It achieves an FID of 1.51 at 256×256 and maintains performance when scaled to 512×512, attaining an FID of 1.60.
Weiyi You, Jinhua Zhang, Xingyu Zhou +3
University of Electronic Science and Technology of China
Recent advances in pixel-space diffusion models have narrowed the image quality gap with latent-space diffusion, but still converge more slowly and lag behind in final image quality. We argue that a key reason is the lack of an explicit representation prior: unlike latent diffusion, which usually denoises in a compact and structured latent space, pixel diffusion needs to learn denoising-friendly representations and pixel generation simultaneously from raw RGB space. To address this problem, we propose PixelDiT2, an end-to-end pixel-space diffusion model designed to decouple representation learning from pixel generation without introducing an autoencoder or latent reconstruction bottleneck. We propose representation grounding that uses a frozen pretrained vision foundation model to provide explicit per-patch representation guidance throughout denoising, allowing the pixel diffusion transformer to focus more on pixel generation. On ImageNet-256x256, PixelDiT2 achieves an FID of 1.46 after 600 epochs; at 512x512 resolution, PixelDiT2 achieves an FID of 1.48 after 680 epochs. Project page: https://pixeldit.github.io/pixeldit2/
Pixel-space diffusion models are trained on full-bandwidth noisy images, yet the useful signal available to the denoiser is strongly frequency dependent. Under rectified-flow diffusion and natural-image power-law spectra, the per-band data-to-noise contour k∗(t)=(1−t)−2/α separates a signal-bearing low-frequency region from a noise-dominated high-frequency region at each time t. We show that this implicit coarse-to-fine structure is not merely descriptive: it induces a capacity-allocation problem. A standard pixel-space denoiser must discover the moving bandwidth boundary internally and can spend computation on frequency-time regions where the optimal prediction collapses to deterministic baselines rather than data-distribution modeling. To make this boundary explicit, we introduce Spectral Forcing, a parameter-free, time-conditional 2D-DCT low-pass operator applied to the noisy input before the patch embedder. Its cutoff expands monotonically with the diffusion time and becomes the identity at the data endpoint. Through controlled synthetic experiments, we identify the regime in which the operator is beneficial: coarse patch tokenization and data whose high-frequency content is predominantly noise rather than essential signal. On ImageNet-256 with JiT-700M/32, Spectral Forcing consistently improves both FID and Inception Score across different training epochs, demonstrating robust gains throughout training; at finer tokenization, the spectral forcing is still competitive. We further insert the unchanged operator into SenseNova-U1, a unified text-to-image model, where it improves DPG-Bench and GenEval, showing that the input-side spectral prior transfers beyond class-conditional generation. These results suggest a route to capacity-efficient pixel-space diffusion by showing the signal and hiding the noise.