Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels

Ruilin Gong; Hao Gong

cs.AIOct 12, 2026

Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels

Authors: Ruilin Gong, Hao Gong

Organizations: Henan University of Technology · University of Chinese Academy of Sciences

Abstract

An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains. Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for pixels at every length. In the rate-distortion sense a caption is therefore not the low-rate code of its image, unless the code is built to make it one. We study that construction and what it costs. We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last, so that each stage is compared with its own optimum. Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and it costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length. We build RUNG, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it. With data and capacity held fixed, ordering by language instead of reconstruction raises prefix-caption information 2.9 times and GenEval from 0.76 to 0.83, at a measured cost of 2.8 dB PSNR at a quarter of the code and 0.1 dB at full length. RUNG-XL (3.4B) reaches 0.88 GenEval and 87.6 DPG-Bench in 32 network evaluations and 0.91 GenEval when errors are redrawn at the rung where they appear; its coarse and fine captions agree 94.1% of the time, and it edits images without inversion.

Figures & tables

Panel of Figure 1 — Figure 1: Allocation of capacity in ordered visual codes. Comparison between words-first ( Rung , top) and standard pixel-first (nested dropout, bottom) ordering at identical code lengths and bitrates. In Rung , early rungs are supervised by natural language summaries whose boundaries $T_{1},\dots,T_{5}$ are fixed by empirical rate matching, leaving subsequent rungs to encode visual variation unconstrained by text. Conversely, pixel-reconstruction objectives allocate early bits to high-variance global illumination and color, starving early rungs of caption information.

Figure 2: Empirical validation of rate–distortion properties. Evaluated on RUNG-L (

\hat{R}_{L}\approx 2.5\text{k}

nats). (a) Mutual information between code prefix and captions demonstrates that pixel-first codes discard semantic content at low rates. (b) Reconstruction fidelity shows words-first sacrifices 2.8 dB PSNR within the language ladder, catching up to within 0.1 dB at full code length. (c) Measured excess rate

\hat{\pi}(D)

adheres strictly to the theoretical bounds of Proposition 3 , tapering to 73 nats (2.9% of the code) at full fidelity.

	Caption Retention & Use			Quarter Code ( $k/L=1/4$ )			Full Code ( $k/L=1$ )
Ordering Signal	$\hat{I}_{25}\uparrow$	GenEval $\uparrow$	NCC (%)	PSNR $\uparrow$	LPIPS $\downarrow$	rFID $\downarrow$	PSNR $\uparrow$	rFID $\downarrow$	Gen. FID $\downarrow$
Unordered (Vanilla)	0.31	0.71	22.1	–	–	–	26.2	0.70	6.8
Pixel-First (Nested Dropout)	0.68	0.76	41.5	19.3	0.36	2.3	26.1	0.71	6.3
DINOv3 Feature Alignment	0.84	0.78	48.3	18.8	0.38	2.2	26.0	0.72	6.3
Words-First (RUNG-L)	1.97	0.83	94.1	16.5	0.45	2.1	26.0	0.74	6.2

Table 1: Effect of visual code ordering signals. Evaluated on RUNG-L (

256^{2}

) under identical capacity and data.

\hat{I}_{25}

denotes the prefix–caption mutual information at one-quarter code length (

k/L=1/4

). Quarter-code and full-code reconstructions evaluate image fidelity (PSNR, LPIPS) and distribution realism (rFID), alongside zero-shot text-to-image generation FID on MJHQ-30K.

Paradigm	Model	#Par	Single	Two	Count	Colors	Position	Attribute	Overall
Generation	SDXL	2.6B	0.98	0.74	0.39	0.85	0.15	0.23	0.55
	SD3-Medium	2.0B	0.99	0.94	0.72	0.89	0.33	0.60	0.74
	Lumina-Image 2.0	2.6B	0.94	0.87	0.67	0.89	0.62	0.62	0.73
	FLUX.1-dev	12B	0.98	0.81	0.74	0.79	0.22	0.45	0.66
Unified AR	Emu3-Gen	8.0B	0.98	0.71	0.34	0.81	0.17	0.21	0.54
	Show-o2	7.0B	1.00	0.87	0.58	0.92	0.52	0.62	0.76

Table 2: Fine-grained visual compositionality on GenEval. Evaluated under the standard GenEval suite across object count, color binding, and spatial positioning without prompt expansion. #Par denotes generator parameters during sampling. Lumina-DiMOO incorporates template-level GRPO alignment during post-training.

Model	GenEval	DPG	T2I-C++	TIIF	WISE	FID $\downarrow$	Total NFE $\downarrow$	Latency (s) $\downarrow$
SDXL	0.55	74.7	0.408	51.2	0.43	8.8	50	3.1
FLUX.1-dev	0.66	83.8	0.502	72.6	0.50	6.1	50	4.8
Lumina-Image 2.0	0.73	87.2	0.487	69.1	0.43	6.8	40	3.4
Janus-Pro	0.80	84.2	0.471	66.3	0.35	8.0	576	9.1
BAGEL	0.82	85.1	0.524	71.4	0.52	6.4	50	7.2
Lumina-DiMOO	0.88	86.0	0.551	74.8	0.60	6.9	64	6.3

Table 3: Instruction following, compositional synthesis, and efficiency. Benchmarked across GenEval, DPG-Bench, T2I-CompBench++ (T2I-C++), TIIF-Bench, WISE, and MJHQ-30K FID. Total NFE aggregates all sampling network evaluations (generator, latent flow decoder, and verifier passes); CFG forward passes are batched. Latency is measured at

512^{2}

resolution with batch size 1 on an NVIDIA A100 GPU.

Figure 3: Left: the order is a dial. Sweeping

\lambda

trades quarter-code PSNR for caption information (GenEval in parentheses);

\lambda{=}0

is nested dropout, and DINOv3 alignment lies inside the trade-off curve. Right: test-time compute. Rung-level correction (redraw budgets

B=1

4

; Appendix C ) against best-of-

N

with the same verifier, in network evaluations relative to one uncorrected sample.

Model	CapA.	DCS	CHAIR ${}_{s}\!\downarrow$	NCC (%)	GranBench
Qwen2.5-VL-7B (Reference)	$-12.3$	42.8	11.2	61.4	44.7
InternVL3-8B (Reference)	$-18.7$	39.5	13.5	57.9	41.2
BAGEL	$-46.5$	31.2	18.4	48.2	33.6
Lumina-DiMOO	$-53.2$	29.8	21.0	45.1	31.9
Rung Descent	-24.9	38.6	12.8	94.1	71.3

Table 4: Multimodal captioning and structural nesting consistency (NCC). Evaluated on CapArena-Auto (CapA.,

[-100,100]

), DeCapBench DCScore (DCS,

[0,100]

), CHAIR s (

[0,100]

, lower is better), and GranBench . NCC measures the fraction of attribute tuples in a model’s coarse caption entailed by its fine caption on identical images.

Construction	$\hat{\delta}$	GenEval	NCC (%)
Independent per length	0.83	0.74	61.8
Truncated long caption	0.04	0.76	90.4
Successive summary	0.12	0.79	92.7

Table 5: Validation of Proposition 1 on caption hierarchy construction. Evaluated on Rung -L (

256^{2}

, 12M data).

\hat{\delta}=\hat{H}(T_{m}\mid T_{m+1})

quantifies inter-stage information leakage in nats per token. Violating the Markov summarization condition imposes severe rate penalties and degrades semantic nesting consistency.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Avg. Words	Avg. Tokens	$\hat{H}(T_{m})$ (nats)	Nats / Token	Cut (% of $\hat{R}_{L}$ )	Cut Rung	Entailed (%)
$T_{1}$	8.1	10	30	3.0	1.2%	1	97.9
$T_{2}$	19.6	25	72	2.9	2.9%	1	97.2
$T_{3}$	44.8	57	150	2.6	6.0%	2	96.6
$T_{4}$	89.5	110	275	2.5	11.0%	4	95.8
$T_{5}$	178.3	210	525	2.5	21.0%	7	Root ( $100.0$ )

Appendix

Table 6: Hierarchical characteristics of LadderCap across five granularity tiers. Evaluated with the Qwen3 tokenizer.

\hat{H}(T_{m})

indicates empirical entropy under fine-tuned Qwen3-0.6B. Cut positions denote the cumulative code capacity budget where

\hat{R}_{k}\geq\hat{H}(T_{m})

. Entailment reflects human-verified semantic agreement on held-out images.

Module	Architecture	Parameters	Inference Role
Base VAE	SD3 VAE (Frozen)	84M	Latent Decoding Only
Ladder Encoder	ViT-B, FSQ $[8,8,8,5,5,5]$	86M	Offline Tokenization
Image Decoder	DiT (24 layers, width 1152)	0.58B	8 Euler Steps
Language Decoder	Qwen3-1.7B + LoRA + Attribute Head	1.7B	Active in Self-Correction
Ascent (RUNG-L)	MMDiT (24 layers, width 1536)	1.3B	24 Flow Steps
Ascent (RUNG-XL)	MMDiT (32 layers, width 2048)	3.4B	24 Flow Steps

Appendix

Table 7: Network parameter breakdown and sampling roles. Active parameters during sampling exclude offline encoders and frozen base components.

Cut Rung	Cumulative Rate (nats)	Preserved Tiers	Kept Tuples (%)	DINO Sim. $\uparrow$	LPIPS $\downarrow$
2	156	$T_{1}$ – $T_{3}$	41.0	0.52	0.63
4	312	$T_{1}$ – $T_{4}$	63.0	0.61	0.56
8	625	$T_{1}$ – $T_{5}$	96.0	0.74	0.47
20	1560	$T_{1}$ – $T_{5}$	98.0	0.91	0.24

Appendix

Table 8: Controlled variation via latent code truncation. Truncating at rung

k

and resampling subsequent blocks on 1,000 validation images. Preserved tuples quantify retention of level-5 semantic attributes; DINO and LPIPS evaluate structural and perceptual distance to the source image.

Ascent Model Formulation
Component Variant	GenEval	NCC (%)	Latency (s) $\downarrow$
Autoregressive over Rungs	0.76	89.4	2.6
Single-Step Full-Code Diffusion	0.77	87.2	7.9
Block-Wise Conditional Flow (Default)	0.79	92.7	0.9
Step Allocation across Blocks
Uniform Step Distribution	0.76	91.5	0.9

Appendix

Table 9: Ablation studies on architectural and algorithmic components. Evaluated on Rung -L (

256^{2}

, 12M data). Shaded rows indicate default system configurations.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Aug 3, 2026

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

Zhipeng Liu, Haochen Wang, Zhaoxiang Zhang

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

Image Captioning Image Captioning Evaluation T2I Generation Multimodal Understanding

Ask PDF

Mar 10, 2026

RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning

Dense image captioning is critical for vision-language pretraining and for text-to-image generation, but scaling expert-quality annotations is prohibitively expensive. While synthesizing captions from strong vision-language models (VLMs) is a practical alternative, supervised distillation often yields limited output diversity and weak generalization. Reinforcement learning (RL) could overcome these, but its successes have so far been concentrated in verifiable domains that rely on deterministic checkers---a luxury not available in open-ended captioning. We address this verification bottleneck with RubiCap, an RL framework that derives fine-grained, sample-specific rewards from LLM-written rubrics. For each image, an LLM rubric writer compares captions from a diverse committee of VLMs to identify consensus strengths and diagnose the current policy's deficiencies. These findings are converted into explicit evaluation criteria, enabling an LLM judge to decompose quality assessment and replace coarse scalar rewards with structured, multi-faceted assessments. Across extensive benchmarks, RubiCap achieves the highest win rates on CapArena, outperforming supervised distillation, prior RL methods, human-expert annotations, and GPT-4V-augmented outputs. On CaptionQA, it shows superior word efficiency: our 7B model matches Qwen2.5-VL-32B-Instruct, and our 3B model surpasses its 7B counterpart. Remarkably, using a compact RubiCap-3B as a captioner produces stronger pretrained VLMs than those trained on captions from proprietary models.

Tzu-Heng Huang, Sirajul Salekin, Javier Movellan +2

Work done during an internship at Apple · University of Wisconsin—Madison · Apple

Image Captioning RL for VLMs Reward Modeling Vision-Language Pretraining

Ask PDF

Apr 14, 2026

ReflectCAP: Detailed Image Captioning with Reflective Memory

Detailed image captioning demands both factual grounding and fine-grained coverage, yet existing methods have struggled to achieve them simultaneously. We address this tension with Reflective Note-Guided Captioning (ReflectCAP), where a multi-agent pipeline analyzes what the target large vision-language model (LVLM) consistently hallucinates and what it systematically overlooks, distilling these patterns into reusable guidelines called Structured Reflection Notes. At inference time, these notes steer the captioning model along both axes -- what to avoid and what to attend to -- yielding detailed captions that jointly improve factuality and coverage. Applying this method to 8 LVLMs spanning the GPT-4.1 family, Qwen series, and InternVL variants, ReflectCAP reaches the Pareto frontier of the trade-off between factuality and coverage, and delivers substantial gains on CapArena-Auto, where generated captions are judged head-to-head against strong reference models. Moreover, ReflectCAP offers a more favorable trade-off between caption quality and compute cost than model scaling or existing multi-agent pipelines, which incur 21--36% greater overhead. This makes high-quality detailed captioning viable under real-world cost and latency constraints.

Kyungmin Min, Minbeom Kim, Kang-il Lee +2

IPAI, Seoul National University · Dept. of ECE, Seoul National University · Adobe Research

Image Captioning VLM Hallucination Mitigation Memory-Augmented VLMs

Ask PDF

Abstract

T2I Generation

Explore similar work

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning

ReflectCAP: Detailed Image Captioning with Reflective Memory