Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning

Hanqiu Li Cai; Chema Garabito

cs.CVOct 7, 2026

Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning

Authors: Hanqiu Li Cai, Chema Garabito

Organizations: SperidLabs

Abstract

Pixel-space diffusion models avoid the lossy VAE of latent models, which suggests an advantage on downstream tasks where fine-grained detail matters. We test this claim along both routes to a pixel-space backbone. We pretrain Iris-3B, a 3B-parameter pixel-space text-to-image transformer, from scratch through a $256\to512\to1024$ curriculum, after first ablating the prediction target and representation alignment at $256^2$ to decide what to scale. We also convert a pretrained latent model, FLUX.2 Klein base 4B, to pixel space. We fine-tune both families for monocular depth estimation and for image restoration/super-resolution. We find no significant improvement from using a pixel-space generative prior. Fine-tuned for depth with one matched direct-regression recipe, Iris-3B is level with the latent FLUX.2 Klein and the converted pixel FLUX.2 Klein falls behind it, and on $4\times$ DIV2K restoration neither pixel model beats a latent FLUX.2 Klein fine-tune, the converted one trailing it slightly. We document the recipes, the failure modes and the remaining confounds behind this negative result. Nevertheless, Iris-3B shows that pixel-space pretraining with the pixel-transformer (PiT) head of PixelDiT scales to 3B parameters and to text-to-image quality competitive with latent models, matching Qwen-Image on OneIG under the official evaluators at $1024^2$ . We release its weights and training code in the hope that they help pave the way for further work on pixel-space generation.

Figures & tables

Fig. 2:

x

- versus

v

-prediction at

256^{2}

, absolute scores at matched steps (EMA, 25-step FlowDPM++, CFG 4.5). Following JiT [ 5 ] , the

x

-prediction arm also uses its logit-normal timestep distribution

(0.8,0.8)

instead of

(0.0,1.0)

, so the contrast does not isolate the prediction target.

Fig. 3: Representation-alignment ablations at

256^{2}

, absolute scores at matched steps (same protocol as Fig. 2 ). All arms share the same model and data and differ only in the alignment objective. Each y-axis is broken to fit Self-Flow on the same scale.

Stage	Steps	Batch	Samples
256	0–367K	1024	375.8M
512 multi-AR	367K–530K	1024	166.9M
1024 multi-AR	530K–625K	512	48.6M
SFT	625K–665K	512	20.5M

Tab. 1: Training stages. Samples = steps

\times

global batch.

Model	GenEval	DPG	LongText	OneIG
FLUX.1-dev 12B [ 33 ]	0.66	83.8	0.607	0.434
i1 3B ∗ [ 51 ]	0.84	86.7	0.922	–
Z-Image 6B ∗ [ 39 ]	0.84	88.1	0.935	0.546
Qwen-Image 20B [ 3 ]	0.87	88.3	0.943	0.539
Iris-3B	0.798	86.52	0.857	0.540

Tab. 2: Official benchmarks at

1024^{2}

. References are copied from the respective papers; FLUX.1-dev from Qwen-Image [ 3 ] (GenEval, DPG), X-Omni [ 49 ] (LongText) and OneIG-Bench [ 50 ] (OneIG, English overall). ∗ Uses prompt rewriting.

Fig. 5: FLUX.2 Klein conversion probes after only 1K steps at $1024^{2}$ , one column per input initialization and trunk LR, with the median loss over steps 900–1000 (PiT head, global batch 8; “Gain ridge” is the gain-matched ridge). These are early diagnostic samples used to compare initializations, not final results; see Fig. 6 for the converted model.

Fig. 6: Samples from PiT 30K, the converted FLUX.2 Klein used downstream, at $1024^{2}$ (CFG 4, 100 steps), generated directly in pixel space.

	Depth		Restoration
Parent	Pixel	Latent	Pixel	Latent
Iris-3B	✓	n/a	✓	n/a
FLUX.2 Klein	✓	✓	✓	✓

Tab. 3: Downstream coverage. ✓: scored; n/a: no latent twin exists.

Fig. 7: Depth on four Hypersim test frames from the three arms of Tab. 4 , each trained with the same direct-regression recipe for 10K steps. Predictions are fitted to the ground truth with a scale and shift in log depth and shown as inverse depth on a shared per-row colour scale; black marks invalid ground truth.

	NYUv2		KITTI		ETH3D		ScanNet		DIODE		Mean		Hypersim
Parent	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	SEE $\downarrow$
Klein latent	.050	.967	.086	.921	.058	.971	.060	.954	.107	.924	.072	.947	.528
Klein pixel	.064	.953	.087	.919	.065	.960	.075	.931	.117	.906	.081	.934	.526
Iris-3B	.057	.961	.083	.927	.064	.965	.069	.939	.082	.935	.071	.946	.496

Tab. 4: Depth after 10K steps of the same direct-regression recipe, zero-shot on five benchmarks. AbsRel

\downarrow

and

\delta_{1}

\uparrow

per benchmark, their mean over the five benchmarks, and SEE at kernel 3 on Hypersim

\downarrow

Parent	Hypersim	KITTI	ETH3D	DIODE
Klein latent	0.97	0.99	0.99	0.98
Klein pixel	1.31	1.44	1.36	1.34
Iris-3B	2.14	1.61	2.21	2.25

Tab. 5: Patch-grid ratio of the raw depth predictions of Tab. 4 : mean absolute second difference of the prediction on pixels adjacent to

16{\times}16

patch borders over its mean elsewhere, both image axes, 60 images per benchmark. 1 means no grid

\downarrow

. NYUv2 and ScanNet are omitted because the latent model, which has no patch grid, already scores well below one there.

Fig. 8: $4\times$ restoration of two DIV2K validation images with native Real-ESRGAN degradations (the Tab. 6 pairs). Left: the full Iris-3B output. Right: the marked regions for each method, cropped from full-resolution outputs. HYPIR-SD2 is the public release. All methods use an empty prompt and one forward pass. We chose the images and regions after viewing the outputs.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	DeQA $\uparrow$
Klein latent	20.99	0.557	0.275	3.26	67.11	4.06
Klein pixel	20.87	0.504	0.274	4.11	67.03	4.04
Iris-3B	20.68	0.481	0.292	3.94	68.01	3.87
R-ESRGAN [ 25 ]	21.76	0.579	0.386	3.79	60.37	3.85
StableSR [ 20 ]	21.91	0.557	0.407	4.10	53.91	3.59
DiffBIR v2 [ 21 ]	21.29	0.476	0.471	3.39	66.21	3.88

Tab. 6: DIV2K

4\times

restoration on validation images with native

4\times

Real-ESRGAN degradations; every method receives the same saved input pairs and is scored with the same PyIQA metric settings and DeQA-Score revision. PSNR and SSIM are computed on RGB. Public methods run at their released settings, except that all text conditioning (including tag and caption generators) is replaced by an empty prompt. Grouped as ours, GAN, multi-step diffusion, one-step diffusion.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 9: Depth on the five zero-shot benchmarks after 10K steps of the same direct-regression recipe.

Fig. 10: $4\times$ restoration of further DIV2K validation images with native Real-ESRGAN degradations (the Tab. 6 pairs).

Fig. 11: $4\times$ restoration of further DIV2K validation images (continued).

Fig. 12: $4\times$ restoration of real-world low-quality images from RealLQ250 [ 55 ] (no ground truth).

Fig. 13: $4\times$ restoration of four real-world example inputs from the public HYPIR repository [ 12 ] (no ground truth).

Explore similar work

Sep 21, 2026cs.CV

PixelDiT2: Representation-Grounded Pixel Diffusion Transformers

Recent advances in pixel-space diffusion models have narrowed the image quality gap with latent-space diffusion, but still converge more slowly and lag behind in final image quality. We argue that a key reason is the lack of an explicit representation prior: unlike latent diffusion, which usually denoises in a compact and structured latent space, pixel diffusion needs to learn denoising-friendly representations and pixel generation simultaneously from raw RGB space. To address this problem, we propose PixelDiT2, an end-to-end pixel-space diffusion model designed to decouple representation learning from pixel generation without introducing an autoencoder or latent reconstruction bottleneck. We propose representation grounding that uses a frozen pretrained vision foundation model to provide explicit per-patch representation guidance throughout denoising, allowing the pixel diffusion transformer to focus more on pixel generation. On ImageNet-256x256, PixelDiT2 achieves an FID of 1.46 after 600 epochs; at 512x512 resolution, PixelDiT2 achieves an FID of 1.48 after 680 epochs. Project page: https://pixeldit.github.io/pixeldit2/

Yongsheng Yu, Wei Xiong, Yichen Sheng +2

NVIDIA · University of Rochester

Contextual Grounding

Ask PDF

Jul 20, 2026cs.CV

Pixel-Space Diffusion Transformers

Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space. However, fixed visual tokenizers can discard fine textures and structural details, while separate representation and diffusion training creates a mismatch between reconstruction and generation objectives. These limitations have renewed interest in pixel-space diffusion, which models raw pixels directly, removes the VAE bottleneck, and supports end-to-end optimization. This formulation better matches the demands of high-fidelity generation but introduces challenges in high-dimensional modeling, including noise scheduling, loss weighting, token efficiency, and scalable architecture design. Pixel-space modeling also offers a promising basis for unified multimodal systems: raw pixels, text, and task conditions can be represented in a shared token space and jointly processed by a single Transformer, narrowing the gap between visual understanding and generation. This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling. We summarize representative methods, identify key technical challenges, and discuss future directions toward high-fidelity, end-to-end vision foundation models that integrate generation and understanding.

Renye Yan, Jikang Cheng, You Wu +8

Peking University, China · Nanjing University, China · Stanford University, USA +3

Diffusion Transformers Visual Tokenizers Transformer Architectures

Ask PDF

May 12, 2026cs.CV

L2P: Unlocking Latent Potential for Pixel Generation

Pixel diffusion models have recently regained attention for visual generation. However, training advanced pixel-space models from scratch demands prohibitive computational and data resources. To address this, we propose the Latent-to-Pixel (L2P) transfer paradigm, an efficient framework that directly harnesses the rich knowledge of pre-trained LDMs to build powerful pixel-space models. Specifically, L2P discards the VAE in favor of large-patch tokenization and freezes the source LDM's intermediate layers, exclusively training shallow layers to learn the latent-to-pixel transformation. By utilizing LDM-generated synthetic images as the sole training corpus, L2P fits an already smooth data manifold, enabling rapid convergence with zero real-data collection. This strategy allows L2P to seamlessly migrate massive latent priors to the pixel space using only 8 GPUs. Furthermore, eliminating the VAE memory bottleneck unlocks native 4K ultra-high resolution generation. Extensive experiments across mainstream LDM architectures show that L2P incurs negligible training overhead, yet performs on par with the source LDM on DPG-Bench and reaches 93% performance on GenEval.

Zhennan Chen, Junwei Zhu, Xu Chen +7

Nanjing University · Tencent Youtu Lab · Hainan-biuh +1

Conditional Variational Autoencoder

Ask PDF