Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware
Organizations: Department of Signal Theory and Communications, Universidad Carlos III de Madrid, Spain · Department of Applied Mathematics and Computer Science, Technical University of Denmark (DTU), Denmark · Math Institute, Universidad de Granada, Spain
Abstract
Frontier text-to-image (T2I) foundation models predominantly rely on multi-billion parameter Diffusion Transformers (DiTs) pretrained across massive compute clusters, creating an accessibility barrier for academic laboratories. When training DiTs from scratch on single consumer GPUs (e.g., an NVIDIA RTX 4090 with 24,GB VRAM), practitioners encounter three primary bottlenecks: quadratic attention sequence costs, Just-In-Time (JIT) compilation graph breaks induced by variable aspect-ratio bucketing, and an autoencoder throughput wall where frozen Variational Autoencoder (VAE) operations dominate step latency. In this work, we introduce Asuka-FM, an end-to-end system and architectural framework demonstrating that competitive dual-stream DiTs (206M parameters) can be pretrained from scratch on a single consumer GPU in 160 GPU hours for under $55. Asuka-FM integrates four core designs: (i) an AdaLN-free dual-stream DiT backbone reducing parameter count; (ii) structured sequence sparsification via Sparse-Dense Residual Fusion (SPRINT) to cut attention FLOPs by over 50%; (iii) continuous aspect-normalized Rotary Position Embeddings (RoPE) coupled with shifted square crop training, guaranteeing static tensor shapes while unlocking zero-shot virtual camera control; and (iv) a pixel adaptation phase utilizing a lightweight (10M parameter) Patch Detailer Head operating under velocity regression (-prediction). Evaluated on open-domain benchmarks without benchmark-specific fine-tuning, Asuka-FM achieves GenEval 0.588 / DPG 78.96 scores at 256px and GenEval 0.609 / DPG 78.51 scores at 512px, setting a new Pareto-optimal efficiency standard for democratic foundation modeling.
Figures & tables
| Architectural Hyperparameter | Specification / Value |
|---|---|
| Latent Channel Dimension ( ) | 32 (FLUX.2-dev VAE, ) |
| Latent Patch Dimension ( ) | (Effective ) |
| Hidden Dimension ( ) | 768 |
| Number of Attention Heads ( ) | 12 |
| Attention Head Dimension ( ) | 64 |
| Feedforward Network (FFN) | SwiGLU ( ) |
| Stage / Phase | Res. ( ) | Drop ( ) | Samples | Throughput (img/s) | MFU (%) Theo / Act † | Compute (EFLOPs) ∗ | Wall Time | Cost ($) |
|---|---|---|---|---|---|---|---|---|
| Stage 1: Base Semantic Pretraining ( , Latent Patch Grid Image Tokens) | ||||||||
| Offline VAE Encoding ‡ | – | 0.625M | 45.2 | – | – | 3.8 h | $1.27 | |
| 1A: SPRINT Sparse | 0.75 | 46.88M | 282.6 | 74.4% / 86.8% | 23.74 | 47.0 h | $15.51 | |
| 1B: Dense Refine | 0.00 | 12.50M | 172.5 | 62.9% / 73.4% | 8.73 | 21.0 h | $6.93 | |
| Stage 2: High-Resolution Adaptation ( , Latent Patch Grid Image Tokens) | ||||||||
| Offline VAE Encoding ‡ | – | 0.625M | 20.6 | – | – | 8.4 h | $2.78 | |
| Model | Params | Hardware Budget | Training Data | GenEval Total | DPG-Bench Total |
|---|---|---|---|---|---|
| Industrial Foundation Models (Multi-Billion Parameters, Datacenter Compute) | |||||
| SDXL Base [ 35 ] | 2.6B | Hundreds of A100s | Millions (Web) | 0.406 ∗ | 72.40 ∗ |
| SD3 Medium [ 16 ] | 2.0B | Massive Cluster | Millions (Web) | 0.620 ∗ | 84.08 ∗ |
| FLUX.1 [Dev] [ 3 ] | 12.0B | Massive Cluster | Web-Scale | 0.660 ∗ | 83.84 ∗ |
| Janus-Pro-7B [ 9 ] | 7.0B | Large Cluster | Multimodal Web | 0.800 ∗ | 84.19 ∗ |
| (Open Foundation) [ 49 ] | 3.0B | TPU v5p-128 Pod | Curated Open | 0.840 | 86.73 |
| Model | Params | Hardware | Wall Time | GPU-h | Cost ($) |
| FLUX.1 [Dev] | 12.0B | H100 SXM5 Pod | Weeks | >\175\text{k}$ | |
| SD3 Medium | 2.0B | H100 SXM5 Pod | Weeks | >\70\text{k}$ | |
| . | 3.0B | TPU v5p-128 | 29.5 Days | \approx\380\text{k}$ | |
| PixelGen-XXL | 1.1B | 8 H800 | 6.0 Days | 1,152 | \approx\2,880$ |
| MiniT2I-B/32 | 0.26B | 8 H100 | 3.0 Days | 576 | \approx\2,016$ |
| Looped-DiT | 0.26B | 8 H100 | 3.0 Days | 576 | \approx\2,016$ |
| Hardware Device | Peak VRAM Allocated | Step Latency Breakdown (ms) | Total Step Latency | Backbone Throughput | MFU (%) (Isolated / Sys) | ||||
|---|---|---|---|---|---|---|---|---|---|
| Data / IO | Online VAE | Text Enc. | Backbone (F+B) | Opt. & Clip | |||||
| Production Latent Pretraining: Kernel-Fused Tensor Saturation ( , In-RAM Latents) | |||||||||
| NVIDIA A40 | 13,254.5 MB | 1.44 ms | – | 99.98 ms | 469.88 ms | 36.33 ms | 607.65 ms | 44.43 TFLOPS | 26.90% / 31.71% |
| RTX 4090 | 15,249.5 MB | 0.27 ms | 0.28 ms | 38.94 ms | 247.91 ms | 20.87 ms | 308.27 ms | 84.22 TFLOPS | 50.98% / 63.25% |
| Online Autoencoder Wall ( , FLUX.2 VAE On-the-Fly Encoding, Dense Tokens) | |||||||||
| NVIDIA A40 | 35,201.0 MB | 0.33 ms | 2,263.52 ms | 111.34 ms | 1,396.95 ms | 36.28 ms | 3,808.42 ms | 51.39 TFLOPS | 31.11% / 15.27% |
| Stage / Phase | Res. ( ) | Drop ( ) | Samples | Throughput (img/s) | MFU (%) Theo / Act † | Compute (EFLOPs) ∗ | Wall Time | Cost ($) |
|---|---|---|---|---|---|---|---|---|
| Phase 0: Offline Synth Generation (A40) ‡ | – | 0.10M | 1.85 img/s | – | – | 15.0 h | $6.00 | |
| Stage 1: Decoupled PDH Warmup (Synth) | 0.75 | 0.50M | 61.8 img/s | 62.4% / 72.8% | 0.52 | 2.25 h | $0.74 | |
| Stage 2A: Joint SPRINT Sparse (Synth) | 0.75 | 2.50M | 57.7 img/s | 58.1% / 67.8% | 2.60 | 12.0 h | $3.98 | |
| Stage 2B: Full-Token Dense Refine (Synth) | 0.00 | 1.50M | 44.4 img/s | 59.8% / 69.8% | 2.71 | 9.4 h | $3.10 | |
| Stage 3: Real-Data SFT Refinement (Real) | 0.00 | 5.16M | 42.1 img/s | 56.7% / 66.2% | 9.32 | 34.0 h | $11.22 | |
| Cumulative Pixel DiT Adaptation Total (Stages 1–3) | 9.66M | 46.5 img/s | 57.8% / 67.5% | 15.15 | 57.65 h | $19.04 | ||
| Configuration / Stage | Patch Size ( ) | Target Objective | Training Schedule | GenEval Total | DPG-Bench Total | Sampling Latency / Img (20-step / 30-step) |
|---|---|---|---|---|---|---|
| Baseline Latent Foundation Model (Pretraining Anchor) | ||||||
| Asuka-512 (Base Latent) | -pred | 139 Epochs (81.0h) | 0.609 | 78.51 | 0.250 s / 0.374 s | |
| Latent-to-Pixel Adaptation Progression (DiP Architecture, Zero VAE Decode) | ||||||
| Stage 2A: Joint SPRINT Sparse ( ) | -pred | 25 Epochs (12.0h) | 0.465 | 69.63 | 0.329 s / 0.493 s | |
| Stage 2B: Full-Token Refine (Synth, -pred) | -pred | 15 Epochs (9.4h) | 0.505 | 72.35 | 0.329 s / 0.493 s | |
| Stage 2B: Full-Token Refine (Synth, -pred) | -pred | 15 Epochs (9.4h) | 0.477 | 71.49 | 0.329 s / 0.493 s | |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage / Phase | Res. ( ) | Batch Size | Drop ( ) | Epochs | Samples | Avg Tokens | Throughput (img/s) | Token Rate (tok/s) | MFU (%) Theo / Act † | Compute (EFLOPs) ∗ | Wall Time | Cost ($) |
| Stage 1: Base Semantic Pretraining ( , Latent Patch Grid Image Tokens) | ||||||||||||
| Offline VAE Encoding ‡ | 64 | – | – | 0.625M | – | 45.2 | – | – | – | 3.8 h | $1.27 | |
| 1A: SPRINT Sparse | 256 | 0.75 | 75 | 46.88M | 351.1 tok | 282.6 | 99.2k | 74.4% / 86.8% | 23.74 | 47.0 h | $15.51 | |
| 1B: Dense Refine | 256 | 0.00 | 25 | 12.50M | 484.0 tok | 172.5 | 83.5k | 62.9% / 73.4% | 8.73 | 21.0 h | $6.93 | |
| Stage 2: High-Resolution Adaptation ( , Latent Patch Grid Image Tokens) | ||||||||||||
| Offline VAE Encoding ‡ | 32 | – | – | 0.625M | – | 20.6 | – | – | – | 8.4 h | $2.78 | |
| Model | Params | Hardware Budget | Training Data | GenEval Task Breakdown | GenEval Overall | DPG-Bench Total Score | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Single | Two-Obj | Count | Colors | Position | Color- Attr | ||||||
| Industrial Foundation Models (Multi-Billion Parameters, Datacenter Compute) | |||||||||||
| SDXL Base [ 35 ] | 2.6B | Hundreds of A100s | Millions (Web) | – | – | – | – | – | – | 0.406 ∗ | 72.40 ∗ |
| SD3 Medium [ 16 ] | 2.0B | Massive Cluster | Millions (Web) | 0.98 | 0.84 | 0.66 | 0.74 | 0.40 | 0.43 | 0.620 ∗ | 84.08 ∗ |
| FLUX.1 [Dev] [ 3 ] | 12.0B | Massive Cluster | Web-Scale | 0.99 | 0.81 | 0.79 | 0.74 | 0.20 | 0.47 | 0.660 ∗ | 83.84 ∗ |
| Janus-Pro-7B [ 9 ] | 7.0B | Large Cluster | Multimodal Web | – | – | – | – | – | – | 0.800 ∗ | 84.19 ∗ |
| Hardware Device | Res. Canvas | Batch Size (Tokens/Step) | Peak VRAM Allocated | Step Latency Breakdown (ms) | Total Step Latency | Backbone Throughput | MFU (%) (Iso. / Sys) | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Data/ IO | Online VAE | Text Enc. | Backbone (F+B) | Opt. & Clip | |||||||
| Production Latent Pretraining: Kernel-Fused Tensor Saturation ( , In-RAM Latents) | |||||||||||
| NVIDIA A40 | 64 (16,384) | 13,254.5 MB | 1.44 ms | – | 99.98 ms | 469.88 ms | 36.33 ms | 607.65 ms | 44.43 TFLOPS | 26.90% / 31.71% | |
| RTX 4090 | 64 (16,384) | 15,249.5 MB | 0.27 ms | 0.28 ms | 38.94 ms | 247.91 ms | 20.87 ms | 308.27 ms | 84.22 TFLOPS | 50.98% / 63.25% | |
| Online Autoencoder Wall ( , FLUX.2 VAE On-the-Fly Encoding, Dense Tokens) | |||||||||||
| NVIDIA A40 | 64 (65,536) | 35,201.0 MB | 0.33 ms | 2,263.52 ms | 111.34 ms | 1,396.95 ms | 36.28 ms | 3,808.42 ms | 51.39 TFLOPS | 31.11% / 15.27% | |
| Resolution | Mode | Steps | Total Time | Latency / Image | Step Cost ( ) | Fixed Overhead ( ) | Peak VRAM | Rate |
| Latent + VAE | 30 | 46.43 s | 2.902 s / img | 6.75 GB | 0.34 img/s | |||
| Latent + VAE | 20 | 32.56 s | 2.035 s / img | 6.75 GB | 0.49 img/s | |||
| Pixel (DiP) | 30 | 53.28 s | 3.330 s / img | 13.22 GB | 0.30 img/s | |||
| Pixel (DiP) | 20 | 35.55 s | 2.222 s / img | 13.22 GB | 0.45 img/s | |||
| Latent + VAE | 30 | 5.99 s | 0.374 s / img | 2.78 GB | 2.67 img/s | |||
| Pixel (DiP) | 30 | 7.89 s | 0.493 s / img | 4.40 GB | 2.03 img/s |
| Backbone / Architecture Variant | Params | BS | Epochs | FID |
| Part I: Latent Architecture & Conditioning Ablations | ||||
| Single-Stream DiT ( ) | 218M | 256 | 600 | 3.89 |
| Efficient U-Net Baseline | 185M | 256 | 400 | 3.45 |
| Dual-Stream DiT + MLP Text Adapter | 200M | 256 | 600 | 3.42 |
| Dual-Stream DiT + LLM (Frozen LFM2.5) | 206M | 512 | 1000 | 3.12 |
| Dual-Stream Baseline (Constant LR) | 206M | 512 | 600 | 3.06 |
| Stage / Phase | Modules Trained | Patch Size ( ) | Drop ( ) | Micro-BS (Per GPU) | Peak LR ( ) | Warmup / Schedule | Epochs | Samples | Throughput (img/s) | MFU (%) Theo / Act | Wall Time | Cost ($) |
| Phase 0: Synth Gen (A40) | – (Sampling) | – | – | 32 | – | – | – | 0.10M | 1.85 | – | 15.0 h | $6.00 |
| Stage 1: PDH Warmup | Embed + PDH | 16 | 0.75 | 16 | 25% Const | 5 | 0.50M | 61.8 | 62.4% / 72.8% | 2.25 h | $0.74 | |
| Stage 2A: Joint SPRINT | Full Model | 16 | 0.75 | 16 | 10% Const | 25 | 2.50M | 57.7 | 58.1% / 67.8% | 12.0 h | $3.98 | |
| Stage 2B: Dense Refine | Full Model | 16 | 0.00 | 8 | 10% WSD | 15 | 1.50M | 44.4 | 59.8% / 69.8% | 9.4 h | $3.10 | |
| Stage 3: Real Data SFT | Full Model | 16 | 0.00 | 8 | 10% WSD | 13 | 5.16M | 42.1 | 56.7% / 66.2% | 34.0 h | $11.22 | |
| Cumulative Pixel DiT Adaptation Total (Stages 1–3) | 58 | 9.66M | 46.5 | 57.8% / 67.5% | 57.65 h | $19.04 | ||||||
| Configuration / Stage | Patch Size ( ) | Target Obj. | Steps / Epochs | GenEval Task Breakdown | GenEval Overall | DPG-Bench Total | Latency / Img (20-step / 30-step) | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Single | Two-Obj | Count | Colors | Position | Color- Attr | |||||||
| Baseline Latent Foundation Model (Pretraining Anchor) | ||||||||||||
| Asuka-512 (Base Latent) | -pred | 139 Ep | 0.950 | 0.811 | 0.347 | 0.817 | 0.278 | 0.450 | 0.609 | 78.51 | 0.250 s / 0.374 s | |
| Stage 2: Synthetic Data SPRINT Adaptation Progression (100k Synthetic Images) | ||||||||||||
| Stage 2A: SPRINT Sparse ( ) | -pred | 25 Ep (12.0h) | 0.928 | 0.543 | 0.259 | 0.662 | 0.183 | 0.215 | 0.465 | 69.63 | 0.329 s / 0.493 s | |
| Stage 2B: Dense Refine ( ) | -pred | 15 Ep (9.4h) | 0.928 | 0.586 | 0.291 | 0.628 | 0.195 | 0.233 | 0.477 | 71.49 | 0.329 s / 0.493 s | |
| Phase | Epochs | Drop | RoPE Mode | Samples | Patches | PFLOPs | Time |
|---|---|---|---|---|---|---|---|
| 1A: Base Sparse | 0–100 | 0.75 | Discrete | 33.70M | 8.63B | 7.2 | 28.0 h |
| 1B: RoPE Adaptation | 100–120 | 0.75 | Continuous | 6.74M | 1.73B | 1.4 | 5.6 h |
| 1C: Dense FT | 120–145 | 0.00 | Continuous | 8.43M | 2.16B | 4.2 | 10.0 h |
| 1D: Cosine Decay | 145–170 | 0.00 | Continuous | 8.43M | 2.16B | 4.2 | 10.0 h |
| Total Stage 1 (Swapped Run) | 57.30M | 14.68B | 17.0 | 53.6 h | |||
| Stage / Phase | Res. ( ) | Drop ( ) | Epochs | Samples | Throughput (img/s) | MFU (%) Theo / Act † | Compute (EFLOPs) ∗ | Wall Time | Cost ($) |
|---|---|---|---|---|---|---|---|---|---|
| Stage 1: Base Semantic Pretraining ( , Latent Patch Grid Image Tokens) | |||||||||
| 1A: SPRINT Sparse | 0.75 | 100 | 33.70M | 334.3 | 69.1% / 80.6% | 13.42 | 28.0 h | $9.24 | |
| 1B: Dense Refine | 0.00 | 50 | 16.85M | 234.0 | 71.7% / 83.6% | 9.94 | 20.0 h | $6.60 | |
| 1C: WSD Cooldown | 0.00 | 25 | 8.43M | 234.0 | 71.7% / 83.6% | 4.97 | 10.0 h | $3.30 | |
| Stage 2: High-Resolution Adaptation ( , Latent Patch Grid Image Tokens) | |||||||||
| 2A: High-Res Sparse | 0.75 | 25 | 8.43M | 70.9 | 34.3% / 40.0% | 7.84 | 33.0 h | $10.89 | |
| Pipeline Component | Micro-BS 16 (Accum 8) | Micro-BS 32 (Accum 4) | ||
|---|---|---|---|---|
| Latency (ms) | Share (%) | Latency (ms) | Share (%) | |
| Data Loading / IO (CPU) | 0.16 ms | 0.0% | 0.18 ms | 0.0% |
| Online Frozen VAE Encoding | 2,548.10 ms | 72.6% | 4,906.91 ms | 72.5% |
| Text Encoder Forward Pass | 31.84 ms | 0.9% | 63.23 ms | 0.9% |
| Backbone Forward Pass (Sparse) | 229.31 ms | 6.5% | 457.20 ms | 6.8% |
| Backbone Backward Pass (Sparse) | 659.65 ms | 18.8% | 1,299.48 ms | 19.2% |
T2I Diffusion Models
Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.
See all 444 papers