The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation

Ashritha Gonuguntla

cs.CVAug 22, 2026

The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation

Authors: Ashritha Gonuguntla

Organizations: Carnegie Mellon University

Abstract

Reasoning-augmented text-to-image models such as GoT-R1 emit an explicit textual plan - object names, attributes, and bounding boxes - before generating image tokens. When such a model fails a compositional prompt, is the plan wrong, or is the plan right and the decoder unfaithful? Because the plan is machine-readable it can be edited before decoding, which makes the two separable. We first validate the ruler. Swapping the two bounding boxes inside the model's own chain demonstrably flips the generated layout: detector-based accuracy falls 0.75 -> 0.48 (p<1e-3), while a widely used VQA-based spatial metric rises. A five-rater human study agrees with the detector on 81% of items and with the VQA judge on 57%. All spatial results therefore use geometric scoring. Under sound measurement the decoder is a faithful executor: 94% of generated layouts realize the planned relation, and object-box binding survives reordering of the plan's object segments. The planner is the bottleneck. It writes wrong relations for phrasing-dependent reasons - 98% accuracy on "left" against 54% on "right" for semantically identical layouts, a raster-order bias we isolate with a mention-order control - and cluttered geometry that the decoder faithfully reproduces. Editing the plan therefore fixes the image without retraining: symbolic verification with resampling gives +5.0 points (p<1e-3), minimal in-place repair +6.0 (p=.02), rewriting only box geometry +10.7 (p<1e-4), and replacing the plan outright +13.3 (p=1e-4). Gains are indifferent to the plan's prose style and to its likelihood under the planner, but not to its geometry. Modular planner-decoder designs are therefore viable, provided the plan is internally consistent: box-text contradictions induce object duplication and identity fusion. We release the plan-fidelity evaluation protocol, all plans, and 12k generated images.

Figures & tables

Condition	vs. control	Relation $\checkmark$ $\uparrow$	$\Delta$ [95% CI]	$p$
Unperturbed control (own chains)	—	0.750	—	—
Native-path baseline (fresh chains)	—	0.773	—	—
Verify-then-generate ( $K{=}5$ )	native baseline	0.823	$+5.0$ $[+2.3,+8.0]$	$.0007$
Minimal in-place repair	control	0.810	$+6.0$ $[+1.0,+11.0]$	$.021$
Geometric plan repair ( $m{=}0.2$ )	control	0.857	$+10.7$ $[+6.7,+14.7]$	$<.0001$
Oracle: minimal text, clean boxes	control	0.883	$+13.3$ $[+8.3,+18.3]$	$.0001$

Table 1: The intervention ladder (OWLv2 relation correctness, spatial_val ,

n{=}300

per condition, paired sign-flip permutation tests). Verification and repair give significant training-free gains; replacing the plan outright helps most—provided the boxes are clean and well-separated. Text style (minimal vs. verbose) is irrelevant within each box type; plans mimicking the planner’s own style and box statistics gain nothing.

Panel of Figure 1 — Figure 1: “A cat on the top of a sofa.” Verified chains preserve the baseline’s coherence. Internally contradictory plans (boxes disagreeing with prose, as in box-swap and our legacy oracle construction) produce stacked, duplicated objects—the failure signature of conflicting conditioning, not of plan novelty.

Condition (spatial)	Relation $\checkmark$ $\uparrow$	Plan IoU	Followed plan	Obj. detected
Unperturbed control	0.750	0.747	0.938	.957/.950
Box swap	0.477	0.629 †	0.755 †	.947/.917
Boxes removed	0.600	—	—	.903/.867
Object segments reversed	0.747	0.282	0.096	.957/.947
Chain ablated	0.453	—	—	.800/.723

Table 2: Spatial chain perturbations (OWLv2;

n{=}300

; † fidelity scored against the swapped plan). The decoder reads boxes (swap flips layouts,

-27.3

p{=}10^{-4}

; removal costs 15 points), reads them semantically (reversing segment order changes nothing about the image—“followed the literal reordered plan” collapses to 0.096 while prompt accuracy is untouched), and depends on the chain for object identity itself (ablation:

-29.7

relation,

-16

points object detection).

Condition (color subset)	Presence $\uparrow$	Color Y/N $\uparrow$	Color match $\uparrow$	Combined $\uparrow$
Unperturbed control	0.9219	0.9160	0.7884	0.8564
Attribute swap in chain	0.9238	0.7751	0.5605	0.7233
Boxes removed from chain	0.8320	0.8431	0.7370	0.7386
Chain ablated entirely	0.6934	0.6816	0.6146	0.5234

Table 3: Color chain perturbations (diagnostic BLIP-VQA, reliable for color;

n{=}256

of 300 prompts, the remainder not parsed by the benchmark’s question generator). Swapping the two colors inside the chain—prompt untouched—transfers the swap to the image: color match falls 23 points with object presence unchanged. The chain dominates the prompt for attribute binding, though it does not fully override it.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Relation	Plan accuracy (original phrasing)	Plan accuracy (inverted phrasing)
on the left of	97.8%	98.0%
on the right of	55.1%	53.9%
on the top of	89.5%	87.5%
on the bottom of	73.7%	77.9%

Appendix

Table 4: Planner accuracy by relation and mention order (chains only, 3 seeds per prompt). Accuracy follows the relation word regardless of which object is mentioned first—semantically identical layouts differ by up to 44 points depending on phrasing—consistent with a raster-order prior (first-mentioned entity placed at the top-left origin) rather than object-specific or mention-order effects alone.

Compositional T2I Generation

Compositional text-to-image generation creates images that faithfully combine multiple objects, attributes, relationships, and styles specified in a prompt.

See all 99 papers

May 12, 2026

EPIC: Efficient Predicate-Guided Inference-Time Control for Compositional Text-to-Image Generation

Recent text-to-image (T2I) generators can synthesize realistic images, but still struggle with compositional prompts involving multiple objects, counts, attributes, and relations. We introduce EPIC (Efficient Predicate-Guided Inference-Time Control), a training-free inference-time refinement framework for compositional T2I generation. EPIC casts refinement as predicate-guided search: it parses the original prompt once into a fixed visual program of object variables and typed predicates, covering checkable conditions such as object presence, counts, attributes, and relations. Each generated or edited image is verified against this program using visual evidence extracted from that image. An image is judged to satisfy the prompt only when all predicates are satisfied; otherwise, failed predicates decide the next step, routing local failures to targeted editing and global failures to resampling while the fixed visual program remains unchanged. On GenEval2, EPIC improves prompt-level accuracy from 34.16% for single-pass generation with the base generator to 71.46%. Under the same generator/editor setting and maximum image-model execution budget, EPIC outperforms the strongest prior refinement baseline by 19.23 points while reducing realized cost by 31% in image-model executions, 72% in MLLM calls, and 81% in MLLM tokens per prompt.

Sunung Mun, Sunghyun Cho, Jungseul Ok

Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science & Engineering, POSTECH

Compositional T2I Generation Controllable Image Generation Multi-Subject Image Generation

Ask PDF

Jun 3, 2026

Evaluating Reasoning Fidelity in Visual Text Generation

Recent text-to-image (T2I) models can render highly legible and well-structured text within images, enabling applications including document generation and slide generation. However, it remains unclear whether such systems faithfully preserve reasoning ability when complex solutions must be expressed directly through rendered text, or whether they merely imitate surface-level patterns. We investigate this question by evaluating reasoning fidelity in visual text generation, where models must express complete reasoning processes as images. Our evaluation includes long text rendering, factual knowledge probing, context understanding, and multi-step reasoning. Across these settings, we find that current T2I models frequently produce semantic errors, logical inconsistencies, and incorrect intermediate steps, even when the rendered text appears visually clear. These failures contrast with the strong reasoning performance of text-only models on the same tasks. Our findings reveal a substantial gap between visual text generation and procedural reasoning, motivating more reliable visual text reasoning.

Jiajun Hong, Jiawei Zhou

Stony Brook University

T2I Generation Evaluation Text Rendering in T2I Generation

Ask PDF

May 5, 2026

Large Language Models are Universal Reasoners for Visual Generation

Text-to-image generation has advanced rapidly with diffusion models, progressing from CLIP and T5 conditioning to unified systems where a single LLM backbone handles both visual understanding and generation. Despite the architectural unification, these systems frequently fail to faithfully align complex prompts during synthesis, even though they remain highly accurate at verifying whether an image satisfies those same prompts. We formalize this as the understanding-generation gap and propose UniReasoner, a framework that leverages the LLM as a universal reasoner to convert its understanding strength into direct generation guidance. Given a prompt, the LLM first produces a coarse visual draft composed of discrete vision tokens. It then performs a self-critique by evaluating the draft for prompt consistency, producing a grounded textual evaluation that pinpoints what needs to be corrected. Finally, a diffusion model is conditioned jointly on the prompt, the visual draft, and the evaluation, ensuring that generation is guided by explicit corrective signals. Each signal addresses a limitation of the other: the draft provides a concrete, scene-level anchor that reduces under-specification in text-only conditioning, while the evaluation turns verification into grounded, actionable constraints that correct omissions, hallucinations, and relational errors. Experiments show that UniReasoner improves compositional alignment and semantic faithfulness under the same diffusion backbone while maintaining image quality, demonstrating a practical way to exploit LLM reasoning to close the understanding-generation gap.

Sucheng Ren, Chen Chen, Zhenbang Wang +5

1Johns Hopkins University · 2Apple

Diffusion Model Guidance Image Generation Compositional T2I Generation VLM Reasoning

Ask PDF

Abstract

Compositional T2I Generation

Explore similar work

EPIC: Efficient Predicate-Guided Inference-Time Control for Compositional Text-to-Image Generation

Evaluating Reasoning Fidelity in Visual Text Generation

Large Language Models are Universal Reasoners for Visual Generation