We introduce Elastic Looped Transformers (ELT), a highly parameter-efficient class of visual generative models based on a recurrent transformer architecture. While conventional generative models rely on deep stacks of unique transformer layers, our approach employs iterative, weight-shared transformer blocks to drastically reduce parameter counts while maintaining high synthesis quality. To effectively train these models for image and video generation, we propose the idea of Intra-Loop Self Distillation (ILSD), where student configurations (intermediate loops) are distilled from the teacher configuration (maximum training loops) to ensure consistency across the model's depth in a single training step. Our framework yields a family of elastic models from a single training run, enabling Any-Time inference capability with dynamic trade-offs between computational cost and generation quality, with the same parameter count. ELT significantly shifts the efficiency frontier for visual synthesis. With 4× reduction in parameter count under iso-inference-compute settings, ELT achieves a competitive FID of 2.0 on class-conditional ImageNet 256×256 and FVD of 72.8 on class-conditional UCF-101.
Figures & tables
Figure 1: Class-conditional Image Generation on ImageNet 256×256 . Comparing our proposed Elastic Looped Transformers (left) against vanilla looped transformers (right), which simply repeat transformer layers during a forward pass of diffusion. As shown, standard looping only produces a coherent image when the inference loop count exactly matches its training setup ( L=8 ), with quality severely degrading at all other values. In contrast, our approach maintains high-fidelity generation across several evaluated compute budgets.
Figure 2: Latent Trajectories of Standard vs. Elastic Looped Transformers . Xkint & Xkmax represent output of intermediate ( Lint ) & final loops ( Lmax ) respectively for kth generation sampling step. Unlike standard recurrent models (left) where only the final iteration Xkmax reaches the solution space, our ILSD training (right) guides intermediate states Xkint also toward the target space. This transformation shifts the model from a fixed-depth architecture to an Any-Time inference framework, supporting flexible computational budgets through early exits within a sampling step.
Figure 3: Overview of ELT framework . Training (left) utilizes shared parameters Θ with recurrent loops and Intra-Loop Self Distillation (ILSD) to improve intermediate representations. This enables Any-Time inference (right), allowing the model to exit early and predict Xk+1 from any intermediate block via a shared MLM head. Note that Xk represents the input for kth sampling iteration and output images in ELT Any-Time Inference (right) assume k=K (last sampling step). Refer Section 3 for details of annotations.
Model
AR
FID ↓
IS ↑
# params
# steps
# Gflops
BigGAN-deep Δ [ 5 ]
7.0
171.4
160M
1
-
StyleGAN-XL Δg [ 60 ]
2.3
265.1
166M
1
-
Improved DDPM Δ [ 52 ]
12.3
-
280M
250
> 150k
ADM + Upsample Δg [ 13 ]
3.9
215.8
554M
250
371k
LDM-4 Δg∗ [ 56 ]
3.6
247.7
400M
250
51.5k
DiT-XL/2 Δg∗ [ 53 ]
2.3
278.2
675M
250
59.5k
Table 1: Class-conditional Image Generation on ImageNet 256×256 . “# steps” refers to the number of neural network runs. Δ denotes values taken from prior publications. ∗ indicates usage of extra training data. g denotes use of classifier-free guidance [ 28 ] . Note that L in ( N×L ) notation is inference loop count per sampling step.
Figure 4: Pareto front of FID vs. Inference GFLOPs . The black curve denotes the fit (FID = 1922.5⋅G−0.95+1.48 ) over pareto-optimal configurations, representing the best achievable FID for a given computational budget. Points are labeled as N layers×L loops . Results demonstrate that ELT scales as effectively as the baseline while remaining significantly more parameter-efficient. Scaling both the model dimension ( d ) and the number of loops ( L ) follows a predictable efficiency frontier, where larger models with fewer loops often compete with smaller models with more loops at specific target GFLOPs. Trends across faded points show scaling along number of loops L in inference, from a single run trained with a certain Lmax loops. Refer Table 6 for the exact baseline configuration, including the specific number of layers for each comparison point.
Figure 5: Scaling with Parameters . We show the best-achievable FID as a function of model parameters (log scale). Each series corresponds to iso-inference-compute configuration for a specific model width ( d ). Points are labeled as Nlayers×Lloops . Results show that increasing parameter count via model width yields superior FID. Proposed ELT mechanism shows that even parameter-constrained models can achieve similar performance through recursive depth. Each ELT point in the figure is best inference configuration ( N×L ) chosen from its corresponding training run ( N×Lmax ). Refer Table 6 for the exact baseline configuration, including the specific number of layers for each comparison point.
Table 7
Figure 6: Faster Training Convergence with ELT . While using 2× less parameters, ELT improves the training efficiency in diffusion framework.
Figure 9
Figure 8: Impact of ILSD across generation processes. ILSD improves performance for all L in inference especially when L=Lmax . For w/o ILSD, Lmax here refers to the number of loops the model is trained with. Results are on ImageNet 256×256 .
Figure 9: Class-conditional Image Generation on ImageNet 256×256 . Comparing our proposed Elastic Looped Transformers (left) against vanilla looped transformers (right), which simply repeat transformer layers during a forward pass of diffusion. As shown, standard looping only produces a coherent image when the inference loop count exactly matches its training setup ( L=8 ), with quality severely degrading at all other values. In contrast, our proposed ELT enhances vanilla looping with Self Distillation, maintaining a high-fidelity generation across several evaluated compute budgets.
Figure 10: Class-conditional Image Generation on ImageNet 256×256 . Comparing our proposed Elastic Looped Transformers (left) against vanilla looped transformers (right), which simply repeat transformer layers during a forward pass of diffusion. As shown, standard looping only produces a coherent image when the inference loop count exactly matches its training setup ( L=8 ), with quality severely degrading at all other values. In contrast, our proposed ELT enhances vanilla looping with Self Distillation, maintaining a high-fidelity generation across several evaluated compute budgets.
Figure 11: Class-conditional Image Generation on ImageNet 256×256 . Qualitative results for ELT in diffusion framework, 16N×2L inference configuration & trained with Lmax=2 .
Figure 12: Class-conditional Image Generation on ImageNet 256×256 . Qualitative results for ELT in masked generative framework. Model scale: ELT-G, 8N×3L inference config & trained with Lmax=4 (FID=1.9).
Masked Generative Models
Diffusion Models
Config
ImageNet
UCF-101
ImageNet
Model Input
Latent shape
(1, 16, 16)
(4, 16, 16)
(4, 32, 32)
Codebook Size
1024
1024
—
Embed / Latent dim
1024
1024
4096
Training / Optimization
Table 5: Implementation Details. Hyperparameters and Configurations for Masked Generative Models (on ImageNet and UCF-101) and Diffusion Models (on ImageNet).
Model Scale
d_model
MLP dim
Heads
Layers (in Baseline)
S
384
1536
6
12
B
768
3072
12
12
L
1024
4096
16
24
XL
1152
4608
16
28
H
1280
5120
16
32
g
1408
6144
16
40
Table 6: Transformer Model Scaling . Architecture details for various model sizes.
Model Scale
d_model
MLP dim
Heads
Layers (in Baseline)
Small
2048
8192
16
16
Large
2048
8192
16
32
Table 7: Architecture Details for ELT in Diffusion framework.
We introduce Loop Flow Transformers (LiFT), a family of looped generative models that scales computation by repeatedly applying a shared Diffusion Transformer (DiT) core, with only light changes to the standard architecture. Rather than asking every recurrent step for the final prediction, LiFT trains each step with a single regression target: a point on a straight path from the model's initial estimate to the flow-matching target. Because we index these targets by a continuous depth coordinate, a trained model can loop far beyond its training depth with no retraining, early exits, or other modifications. In our experiments, these longer rollouts improve generation, so inference computation can grow without adding parameters. On ImageNet at 256x256, LiFT-L/2 achieves an FID 3.34 points lower than our dense DiT-XL/2 baseline while using approximately 60% fewer parameters, 32% fewer training FLOPs, and 52% fewer inference FLOPs.
Mohammad Mahdi Derakhshani, Pedro M. P. Curvo, Gertjan J. Burghouts +2
VISLab · University of Amsterdam · TNO, Intelligent Imaging +1
Modern computer vision pipelines remain fragmented, with tasks such as text-to-image generation, editing, restoration, and classical perception handled by separate models. We study Unified Visual Generation (UVG), where a single model produces diverse image-valued outputs through a unified multimodal interface. While diffusion-based systems dominate UVG due to strong quality and controllability, their iterative sampling incurs substantial inference latency, limiting practical deployment. To address these limitations, we propose UniGen-AR, a framework that pairs a general-purpose multi-modal language model (MLLM) with an efficient next-scale visual auto-regressive (VAR) decoder. This design retains the flexibility of MLLM-based conditioning while leveraging the sampling efficiency and latent unification properties of VAR models. In our framework, the MLLM encodes free-form instructions and control signals into a unified sequence, which guides the VAR decoder to generate image-valued outputs for over 15 tasks spanning four families. Empirically, UniGen-AR achieves up to 19× lower inference latency than diffusion-based baselines while maintaining or improving output quality. Our ablations further reveal that VQ-VAE tokenizer design, particularly codebook size and hierarchy, is a critical factor for VAR scalability in UVG. These results establish visual auto-regressive modeling as a compelling and efficient backbone for unified visual generation. Our project page is at https://zpbao.github.io/projects/unigenar.
Zhipeng Bao, Zhen Zhu, Nupur Kumari +4
1Carnegie Mellon University · University of Illinois Urbana-Champaign · 3Toyota Research Institute
Diffusion transformers (DiTs) achieve high generative quality but lock FLOPs to image resolution, limiting principled latency-quality trade-offs, and allocate computation uniformly across input spatial tokens, wasting resource allocation to unimportant regions. We introduce Elastic Latent Interface Transformer (ELIT), a drop-in, DiT-compatible mechanism that decouples input image size from compute. Our approach inserts a latent interface, a learnable variable-length token sequence on which standard transformer blocks can operate. Lightweight Read and Write cross-attention layers move information between spatial tokens and latents and prioritize important input regions. By training with random dropping of tail latents, ELIT learns to produce importance-ordered representations with earlier latents capturing global structure while later ones contain information to refine details. At inference, the number of latents can be dynamically adjusted to match compute constraints. ELIT is deliberately minimal, adding two cross-attention layers while leaving the rectified flow objective and the DiT stack unchanged. Across datasets and architectures (DiT, U-ViT, HDiT, MM-DiT), ELIT delivers consistent gains. On ImageNet-1K 512px, ELIT delivers an average gain of 35.3% and 39.6% in FID and FDD scores. Project page: https://snap-research.github.io/elit/
Moayed Haji-Ali, Willi Menapace, Ivan Skorokhodov +6