An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains. Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for pixels at every length. In the rate-distortion sense a caption is therefore not the low-rate code of its image, unless the code is built to make it one. We study that construction and what it costs. We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last, so that each stage is compared with its own optimum. Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and it costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length. We build RUNG, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it. With data and capacity held fixed, ordering by language instead of reconstruction raises prefix-caption information 2.9 times and GenEval from 0.76 to 0.83, at a measured cost of 2.8 dB PSNR at a quarter of the code and 0.1 dB at full length. RUNG-XL (3.4B) reaches 0.88 GenEval and 87.6 DPG-Bench in 32 network evaluations and 0.91 GenEval when errors are redrawn at the rung where they appear; its coarse and fine captions agree 94.1% of the time, and it edits images without inversion.
Figures & tables
Figure 1: Allocation of capacity in ordered visual codes. Comparison between words-first ( Rung , top) and standard pixel-first (nested dropout, bottom) ordering at identical code lengths and bitrates. In Rung , early rungs are supervised by natural language summaries whose boundaries T1,…,T5 are fixed by empirical rate matching, leaving subsequent rungs to encode visual variation unconstrained by text. Conversely, pixel-reconstruction objectives allocate early bits to high-variance global illumination and color, starving early rungs of caption information.
Figure 2: Empirical validation of rate–distortion properties. Evaluated on RUNG-L ( R^L≈2.5k nats). (a) Mutual information between code prefix and captions demonstrates that pixel-first codes discard semantic content at low rates. (b) Reconstruction fidelity shows words-first sacrifices 2.8 dB PSNR within the language ladder, catching up to within 0.1 dB at full code length. (c) Measured excess rate π^(D) adheres strictly to the theoretical bounds of Proposition 3 , tapering to 73 nats (2.9% of the code) at full fidelity.
Caption Retention & Use
Quarter Code ( k/L=1/4 )
Full Code ( k/L=1 )
Ordering Signal
I^25↑
GenEval ↑
NCC (%)
PSNR ↑
LPIPS ↓
rFID ↓
PSNR ↑
rFID ↓
Gen. FID ↓
Unordered (Vanilla)
0.31
0.71
22.1
–
–
–
26.2
0.70
6.8
Pixel-First (Nested Dropout)
0.68
0.76
41.5
19.3
0.36
2.3
26.1
0.71
6.3
DINOv3 Feature Alignment
0.84
0.78
48.3
18.8
0.38
2.2
26.0
0.72
6.3
Words-First (RUNG-L)
1.97
0.83
94.1
16.5
0.45
2.1
26.0
0.74
6.2
Table 1: Effect of visual code ordering signals. Evaluated on RUNG-L ( 2562 ) under identical capacity and data. I^25 denotes the prefix–caption mutual information at one-quarter code length ( k/L=1/4 ). Quarter-code and full-code reconstructions evaluate image fidelity (PSNR, LPIPS) and distribution realism (rFID), alongside zero-shot text-to-image generation FID on MJHQ-30K.
Paradigm
Model
#Par
Single
Two
Count
Colors
Position
Attribute
Overall
Generation
SDXL
2.6B
0.98
0.74
0.39
0.85
0.15
0.23
0.55
SD3-Medium
2.0B
0.99
0.94
0.72
0.89
0.33
0.60
0.74
Lumina-Image 2.0
2.6B
0.94
0.87
0.67
0.89
0.62
0.62
0.73
FLUX.1-dev
12B
0.98
0.81
0.74
0.79
0.22
0.45
0.66
Unified AR
Emu3-Gen
8.0B
0.98
0.71
0.34
0.81
0.17
0.21
0.54
Show-o2
7.0B
1.00
0.87
0.58
0.92
0.52
0.62
0.76
Table 2: Fine-grained visual compositionality on GenEval. Evaluated under the standard GenEval suite across object count, color binding, and spatial positioning without prompt expansion. #Par denotes generator parameters during sampling. Lumina-DiMOO incorporates template-level GRPO alignment during post-training.
Model
GenEval
DPG
T2I-C++
TIIF
WISE
FID ↓
Total NFE ↓
Latency (s) ↓
SDXL
0.55
74.7
0.408
51.2
0.43
8.8
50
3.1
FLUX.1-dev
0.66
83.8
0.502
72.6
0.50
6.1
50
4.8
Lumina-Image 2.0
0.73
87.2
0.487
69.1
0.43
6.8
40
3.4
Janus-Pro
0.80
84.2
0.471
66.3
0.35
8.0
576
9.1
BAGEL
0.82
85.1
0.524
71.4
0.52
6.4
50
7.2
Lumina-DiMOO
0.88
86.0
0.551
74.8
0.60
6.9
64
6.3
Table 3: Instruction following, compositional synthesis, and efficiency. Benchmarked across GenEval, DPG-Bench, T2I-CompBench++ (T2I-C++), TIIF-Bench, WISE, and MJHQ-30K FID. Total NFE aggregates all sampling network evaluations (generator, latent flow decoder, and verifier passes); CFG forward passes are batched. Latency is measured at 5122 resolution with batch size 1 on an NVIDIA A100 GPU.
Figure 3: Left: the order is a dial. Sweeping λ trades quarter-code PSNR for caption information (GenEval in parentheses); λ=0 is nested dropout, and DINOv3 alignment lies inside the trade-off curve. Right: test-time compute. Rung-level correction (redraw budgets B=1 to 4 ; Appendix C ) against best-of- N with the same verifier, in network evaluations relative to one uncorrected sample.
Model
CapA.
DCS
CHAIR s↓
NCC (%)
GranBench
Qwen2.5-VL-7B (Reference)
−12.3
42.8
11.2
61.4
44.7
InternVL3-8B (Reference)
−18.7
39.5
13.5
57.9
41.2
BAGEL
−46.5
31.2
18.4
48.2
33.6
Lumina-DiMOO
−53.2
29.8
21.0
45.1
31.9
Rung Descent
-24.9
38.6
12.8
94.1
71.3
Table 4: Multimodal captioning and structural nesting consistency (NCC). Evaluated on CapArena-Auto (CapA., [−100,100] ), DeCapBench DCScore (DCS, [0,100] ), CHAIR s ( [0,100] , lower is better), and GranBench . NCC measures the fraction of attribute tuples in a model’s coarse caption entailed by its fine caption on identical images.
Construction
δ^
GenEval
NCC (%)
Independent per length
0.83
0.74
61.8
Truncated long caption
0.04
0.76
90.4
Successive summary
0.12
0.79
92.7
Table 5: Validation of Proposition 1 on caption hierarchy construction. Evaluated on Rung -L ( 2562 , 12M data). δ^=H^(Tm∣Tm+1) quantifies inter-stage information leakage in nats per token. Violating the Markov summarization condition imposes severe rate penalties and degrades semantic nesting consistency.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Level
Avg. Words
Avg. Tokens
H^(Tm) (nats)
Nats / Token
Cut (% of R^L )
Cut Rung
Entailed (%)
T1
8.1
10
30
3.0
1.2%
1
97.9
T2
19.6
25
72
2.9
2.9%
1
97.2
T3
44.8
57
150
2.6
6.0%
2
96.6
T4
89.5
110
275
2.5
11.0%
4
95.8
T5
178.3
210
525
2.5
21.0%
7
Root ( 100.0 )
Appendix
Table 6: Hierarchical characteristics of LadderCap across five granularity tiers. Evaluated with the Qwen3 tokenizer. H^(Tm) indicates empirical entropy under fine-tuned Qwen3-0.6B. Cut positions denote the cumulative code capacity budget where R^k≥H^(Tm) . Entailment reflects human-verified semantic agreement on held-out images.
Module
Architecture
Parameters
Inference Role
Base VAE
SD3 VAE (Frozen)
84M
Latent Decoding Only
Ladder Encoder
ViT-B, FSQ [8,8,8,5,5,5]
86M
Offline Tokenization
Image Decoder
DiT (24 layers, width 1152)
0.58B
8 Euler Steps
Language Decoder
Qwen3-1.7B + LoRA + Attribute Head
1.7B
Active in Self-Correction
Ascent (RUNG-L)
MMDiT (24 layers, width 1536)
1.3B
24 Flow Steps
Ascent (RUNG-XL)
MMDiT (32 layers, width 2048)
3.4B
24 Flow Steps
Appendix
Table 7: Network parameter breakdown and sampling roles. Active parameters during sampling exclude offline encoders and frozen base components.
Cut Rung
Cumulative Rate (nats)
Preserved Tiers
Kept Tuples (%)
DINO Sim. ↑
LPIPS ↓
2
156
T1 – T3
41.0
0.52
0.63
4
312
T1 – T4
63.0
0.61
0.56
8
625
T1 – T5
96.0
0.74
0.47
20
1560
T1 – T5
98.0
0.91
0.24
Appendix
Table 8: Controlled variation via latent code truncation. Truncating at rung k and resampling subsequent blocks on 1,000 validation images. Preserved tuples quantify retention of level-5 semantic attributes; DINO and LPIPS evaluate structural and perceptual distance to the source image.
Component Variant
GenEval
NCC (%)
Latency (s) ↓
Ascent Model Formulation
Autoregressive over Rungs
0.76
89.4
2.6
Single-Step Full-Code Diffusion
0.77
87.2
7.9
Block-Wise Conditional Flow (Default)
0.79
92.7
0.9
Step Allocation across Blocks
Uniform Step Distribution
0.76
91.5
0.9
Appendix
Table 9: Ablation studies on architectural and algorithmic components. Evaluated on Rung -L ( 2562 , 12M data). Shaded rows indicate default system configurations.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.