Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

Yucheng Yang; Xiaoxiao Sun; Shusheng Yang; Jiakang Chen; Serena Yeung-Levy

cs.CVOct 9, 2026

Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

Authors: Yucheng Yang, Xiaoxiao Sun, Shusheng Yang, Jiakang Chen, Serena Yeung-Levy

Organizations: Stanford University · Tsinghua University · New York University

Abstract

Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint attention. These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the prompt, or preserve them? We find that image-to-text (I2T) feedback can preserve the objects and attributes already represented in the evolving image, including errors. Transplanting I2T messages can rescue failures or disrupt successes. At matched intervention strengths, suppressing the influence of I2T feedback improves generation, whereas opposite controls degrade it. These findings motivate Semantic Feedback Guidance (SFG), which suppresses the influence of I2T feedback by guiding away from an auxiliary branch that strengthens this feedback and weakens text-to-image conditioning. The in-place variant, iSFG, directly suppresses I2T feedback and strengthens text-to-image conditioning, requiring no extra model evaluations. Temporal interventions motivate running the SFG branch during the first 20% of image sampling steps, requiring about $1.2\times$ the base NFE without classifier-free guidance (CFG). Experiments across three image-generation backbones show gains in alignment and preference-model scores. On FLUX, adding SFG to CFG raises GenEval from $0.6661$ to $0.7479$ ; on SD3.5M, iSFG adds $+1.33$ HPSv2.1 over CFG at zero extra NFE. The control also transfers to text-to-video and image-to-video generation, improving VBench scores. Code is available at https://github.com/yyc1215lll-creator/SFG.

Figures & tables

Figure 1: Matched qualitative comparisons on FLUX. When dealing with more complex prompts, our method aligns better with the objects and attributes specified in the complex prompts.

Figure 2: Image-to-text feedback. The current image writes into internal text states through $B_{i\rightarrow t}$ , which later blocks read to update the image. If the image misses an object, does this feedback help recover it or preserve the incomplete scene? Text labels and AI-generated fruit images illustrate schematic states and possible outcomes; the input prompt stays fixed.

Figure 3: Message transplantation and local response directions. (a) Transplant donor image-to-text messages into recipient text states, keeping the recipient prompt and initial noise fixed. Success-to-failure and failure-to-success transplants test rescue and disruption. (b) $J_{t}$ and $-J_{t}$ are opposite local responses; $R_{t}^{S}$ and $R_{t}^{\rm rand}$ are velocity changes from successful-donor and random-message interventions. Text labels, AI-generated fruit images, and vector angles are schematic.

	SD3M			FLUX
Method	GenEval $\uparrow$	T2I-Comp Bench++ $\uparrow$	FID $\downarrow$	GenEval $\uparrow$	T2I-Comp Bench++ $\uparrow$
Base	0.3294	0.3230	41.76	0.3604	0.3254
S 2 -Guidance ( Chen et al., 2026a )	0.4866	0.3667	26.00	0.4990	0.3900
SEG ( Hong, 2024 )	0.5322	0.4331	28.19	0.5548	0.4505
PAG ( Ahn et al., 2024 )	0.5604	0.4278	28.53	0.4866	0.3843
SFG-S (ours)	0.4678	0.4249	40.19	0.4471	0.3961

Table 1: Text-to-image generation without CFG on SD3M and FLUX. All methods use

\mathrm{cfg}=1

. GenEval and T2I-CompBench++ are averaged over six and eight categories, respectively. FID uses 5k COCO images on SD3M; CFG ( Ho & Salimans, 2022 ) at

s=7.5

gives

32.49

for reference. Dashes denote unavailable results.

		GenEval $\uparrow$				HPS $\uparrow$	ImageReward $\uparrow$
Model	Method	Count.	Pos.	C.-Attr.	Overall	Avg.	Mean
SD3.5M	CFG ( Ho & Salimans, 2022 )	0.6458	0.2150	0.5767	0.6890	31.65	1.1102
	CFG+PAG ( Ahn et al., 2024 )	0.6458	0.2017	0.5233	0.6807	31.51	1.1038
	CFG+SEG ( Hong, 2024 )	0.6646	0.2017	0.6017	0.6890	32.11	1.1180
	CFG+S 2 ( Chen et al., 2026a )	0.6458	0.1967	0.5717	0.6811	31.71	1.1152
	CFG+TACA ( Lv et al., 2025 )	0.6771	0.1900	0.5950	0.6960	31.58	1.1298

Table 2: Text-to-image generation with CFG on SD3.5M and FLUX. Each additional guidance method is combined with CFG. Count., Pos., and C.-Attr. denote counting, position, and color-attribute GenEval scores, selected to show categories with larger SFG gains; Overall averages all six categories. HPSv2.1 averages the four style scores; ImageReward reports the mean over the same images, with identical prompts and seeds for every method.

		GenEval $\uparrow$				HPSv2.1 $\uparrow$	ImageReward $\uparrow$
Model	Method	Count.	Pos.	C.-Attr.	Overall	Avg.	Mean
SD3.5M	CFG ( Ho & Salimans, 2022 )	0.6458	0.2150	0.5767	0.6890	31.65	1.1102
SD3.5M	CFG+iSFG (ours)	0.6813	0.2367	0.5950	0.7047	32.98	1.1850
FLUX	CFG ( Ho & Salimans, 2022 )	0.6458	0.1850	0.5233	0.6661	30.96	0.9837
FLUX	CFG+iSFG (ours)	0.6563	0.2200	0.5417	0.6868	31.28	1.0369

Table 3: Improving CFG with iSFG at zero additional NFE on SD3.5M and FLUX. iSFG edits the conditional forward pass without increasing model evaluations relative to CFG. Count., Pos., and C.-Attr. are selected GenEval categories with larger gains: counting, position, and color attribute; Overall averages six categories. HPSv2.1 averages the four style scores; ImageReward reports the mean over the same images.

	Total	Subject	Background	Temporal	Motion	Dynamic	Aesthetic	Imaging
Method	Score	Cons.	Cons.	Flicker	Smooth.	Degree	Quality	Quality
Base (no guidance)	82.58	92.11	91.98	98.01	99.39	97.92	50.88	63.65
CFG (all steps) ( Ho & Salimans, 2022 )	83.52	92.79	92.65	97.20	98.78	96.53	58.97	64.94
CFG (first 14 steps) ( Ho & Salimans, 2022 )	83.58	93.08	92.67	97.38	99.00	96.53	57.81	64.90
S 2 -Guidance ( Chen et al., 2026a )	82.96	92.35	92.43	97.49	99.35	98.61	52.69	64.62
CFG-Zero ⋆ ( Fan et al., 2025 )	83.41	92.74	92.38	96.97	98.86	97.92	58.41	64.86

Table 4: Text-to-video generation with different guidance methods on HunyuanVideo. We report seven VBench quality dimensions and the aggregate score (Total). SFG uses

\mathrm{cfg}=1

without an unconditional branch; the two CFG rows differ in their guidance window. Cons., Flicker, and Smooth. denote consistency, temporal flickering, and motion smoothness.

	Total	I2V	I2V	Subject	Background	Motion	Dynamic	Aesthetic	Imaging
Method	Score	Subj.	Bg.	Cons.	Cons.	Smooth.	Degree	Quality	Quality
Base (no guidance)	88.50	96.51	97.92	89.83	91.73	98.99	85.37	57.62	67.88
CFG ( Ho & Salimans, 2022 )	89.43	97.82	98.59	92.88	94.43	99.10	66.67	60.44	70.24
S 2 -Guidance ( Chen et al., 2026a )	88.66	96.63	97.98	90.01	91.74	98.93	86.59	58.12	67.90
CFG-Zero ⋆ ( Fan et al., 2025 )	87.49	95.79	97.19	86.59	90.61	98.87	85.77	57.23	67.74
SFG (ours)	90.04	98.16	99.06	93.04	94.57	99.50	70.44	60.56	70.44

Table 5: Image-to-video generation with different guidance methods on HunyuanVideo. We report eight VBench dimensions and the aggregate score (Total), covering reference-image consistency (I2V Subj./Bg.) and video quality (e.g., Subject/Background Cons.).

Figure 6: Validation of the SFG-X guidance direction. GenEval on SD3.5M (

\mathrm{cfg}=1

), comparing SFG-X (

u<0

) with the opposite-sign branch (

u>0

) at matched

|u|

and

\omega

. Suppression/amplification labels denote the net guidance effect. Base marks the unguided score.

Figure 7: Guidance components and guidance windows. (a) GenEval at the Table 1 operating points. (b) Both FLUX curves use 28 steps; SD3.5M CFG+SFG and SD3M SEG use 40. FLUX CFG returns to scale 1 outside its guidance window. (c) HunyuanVideo T2V scores with 25 total sampling steps. SFG is active for the first 5, 7, 10, or 25 steps; CFG is active for the first 7, 14, or 25 steps. Guidance is disabled after the selected prefix.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Complementary directions. SFG-X and SFG-S corrections have low cosine similarity with CFG and with each other, both over prompts and through sampling.

Benchmark	Prompts	Seeds/prompt	Total outputs
GenEval	$553$	$6$	$3{,}318$
T2I-CompBench++	$300$	$8$	$2{,}400$
HPSv2.1	$800$	$4$	$3{,}200$
ImageReward	$800$	$4$	$3{,}200$
VBench (T2V)	$326$	$3$	$978$
VBench (I2V)	$355$	$3$	$1{,}065$

Appendix

Table 6: Evaluation sample counts per model and method configuration. Seeds/prompt gives the number of generation seeds for each distinct prompt, or prompt–reference-image pair for I2V. The 800 HPSv2.1 prompts are the total across four styles; four seeds per prompt yield 3,200 images. ImageReward reuses these images without additional generation.

$\omega$	GenEval $\uparrow$
5	0.5811
6	0.5787
7	0.5751
8	0.5730

Appendix

Table 7: SFG scale sweep on SD3M without CFG ( Ho & Salimans, 2022 ) . GenEval versus guidance scale

\omega

; bold marks the best tested score.

Figure 9: SD3M without CFG / GenEval guidance-scale sweep.

$\omega$	GenEval $\uparrow$
7	0.6065
9	0.6233
10	0.6475
11	0.6655
12	0.6739
13	0.6779

Appendix

Table 8: SFG scale sweep on FLUX without CFG ( Ho & Salimans, 2022 ) . GenEval versus

\omega

, with built-in guidance set to 1; bold marks the best tested score.

Figure 10: FLUX without CFG / GenEval guidance-scale sweep.

$\omega$	GenEval $\uparrow$
7	0.7421
7.5	0.7479
8	0.7432
8.5	0.7442

Appendix

Table 9: SFG scale sweep on FLUX with CFG ( Ho & Salimans, 2022 ) . GenEval versus

\omega

at CFG scale 3.5; bold marks the best tested score.

Figure 11: FLUX with CFG / GenEval guidance-scale sweep.

$\omega$	HPSv2.1 $\uparrow$
5	31.54
5.5	31.54
6	31.58
6.5	31.57
7.5	31.66

Appendix

Table 10: SFG scale sweep on FLUX with CFG ( Ho & Salimans, 2022 ) . HPSv2.1 versus

\omega

; bold marks the best tested score.

Figure 12: FLUX with CFG / HPSv2.1 guidance-scale sweep.

Scale ( $w$ )	GenEval $\uparrow$
0.5	0.4696
1.25	0.5521
2.5	0.5580
3	0.5604
4	0.5563

Appendix

Table 11: PAG on SD3M without CFG. ( Ahn et al., 2024 ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=3

Figure 13: PAG. Five-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
0.2	0.4264
0.5	0.4684
1	0.5322
1.25	0.5303
3	0.5212

Appendix

Table 12: SEG on SD3M without CFG. ( Hong, 2024 ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=1

Figure 14: SEG. Five-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
0.2	0.3814
0.5	0.4415
1	0.4646
2	0.4866
3	0.4754

Appendix

Table 13: S 2 -Guidance on SD3M without CFG. ( Chen et al., 2026a ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=2

Figure 15: S 2 . Five-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
0.05	0.6729
0.1	0.6807
0.2	0.6669
0.75	0.6650
1	0.6568
1.5	0.6499

Appendix

Table 14: PAG on SD3.5M with CFG. ( Ahn et al., 2024 ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=0.1

Figure 16: PAG. Six-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
0.05	0.6729
0.2	0.6890
0.5	0.6741

Appendix

Table 15: SEG on SD3.5M with CFG. ( Hong, 2024 ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=0.2

Figure 17: SEG. Three-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
0.25	0.6811
1.5	0.6274
2	0.5846
3	0.4977
4.5	0.4227

Appendix

Table 16: S 2 -Guidance on SD3.5M with CFG. ( Chen et al., 2026a ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=0.25

Figure 18: S 2 . Five-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
1	0.4626
1.5	0.4728
2	0.4866
2.5	0.4706
3	0.4757

Appendix

Table 17: PAG on FLUX without CFG. ( Ahn et al., 2024 ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=2

Figure 19: PAG. Five-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
2	0.5315
2.5	0.5295
3	0.5548
3.5	0.5544
4	0.5525

Appendix

Table 18: SEG on FLUX without CFG. ( Hong, 2024 ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=3

Figure 20: SEG. Five-point weight sweep.

Scale ( $w$ )	GenEval $\uparrow$
1	0.4876
2	0.4990
2.5	0.4709
3	0.4548
3.5	0.4347

Appendix

Table 19: S 2 -Guidance on FLUX without CFG. ( Chen et al., 2026a ) GenEval versus guidance weight

w

; bold marks the best tested setting (

w=2

Figure 21: S 2 . Five-point weight sweep.

Method	Extra NFE	Time per image (s)	Relative time
CFG	0	$10.800\pm 0.020$	$1.000\times$
CFG+iSFG (ours)	0	$11.108\pm 0.018$	$1.029\times$

Appendix

Table 20: iSFG runtime. Mean

\pm

standard deviation over three paired runs.

Message path	Donor $\to$ recipient	Steps	Outcome	Count (rate)
Image-to-text	$S\to F$ , $\lambda=1$	1–40	Rescue	270/400 (67.5%)
Image-to-text	$S\to F$ , $\lambda=0.5$	1–40	Rescue	207/400 (51.8%)
Image-to-text	Random, $\lambda=0.5$	1–40	Rescue	5/400 (1.3%)
Image-to-text	$F\to F$	1–40	Rescue	38/200 (19.0%)
Text-to-image	$S\to F$ , $\lambda=1$	1–40	Rescue	74/400 (18.5%)
Text-to-image	$S\to F$ , $\lambda=1$	1–8	Rescue	151/400 (37.8%)

Appendix

Table 21: Message transplantation and controls on paired generation trajectories.

S

and

F

denote successful and failed trajectories; arrows run from donor to recipient. Image-to-text and text-to-image transplantation use the same 400 success/failure pairs, while failed-to-failed transplantation uses 200 failure/failure pairs.

\lambda

is the message mixing weight. Steps specify the transplant window. Counts and percentages report rescued failures or disrupted successes.

Metric	Condition	Single	Two	All
Warning coverage (concepts)	Any step	166/233 (71.2%)	224/306 (73.2%)	390/539 (72.4%)
	By step 20	118/233 (50.6%)	185/306 (60.5%)	303/539 (56.2%)
Failure rescue (paths)	Guided	188/223 (84.3%)	161/262 (61.5%)	349/485 (72.0%)
Success disruption	Guided	13/228 (5.7%)	68/258 (26.4%)	81/486 (16.7%)
Final failure after warning	Base	158/166 (95.2%)	201/205 (98.0%)	359/371 (96.8%)
	Guided	31/166 (18.7%)	81/205 (39.5%)	112/371 (30.2%)

Appendix

Table 22: Omission warnings, generation outcomes, and probe states on 500 SD3.5M trajectory pairs. Single and Two denote single- and two-object prompts; All combines both. Guided uses SFG-X throughout 40 sampling steps at

\mathrm{cfg}=1

. Count-based rows report events over the eligible concepts or paths, with percentages in parentheses. Symptom persistence and terminal non-recovery are percentages averaged with equal prompt weights.

Failure rescue $\uparrow$
Window	S	T	All
Base	0.0	0.0	0.0
1–10	81.4	42.3	69.4
1–20	83.1	57.7	75.3
10–25	67.8	42.3	60.0
1–40	86.4	69.2	81.2

Appendix

Table 23: SFG-X window effects on final images in the 85-pair SD3.5M cohort. Windows specify inclusive sampling steps. S/T denote single-/two-object prompts; All combines both. Values are percentages. Arrows indicate the preferred direction; bold marks the best guided window per column and metric.

Figure 22: Top to bottom: failure rescue, success disruption, and warned-failure rescue.

Symptom persistence $\downarrow$
Window	S	T	All
Base	75.7	59.6	69.1
1–10	26.8	37.9	31.4
1–20	24.0	25.5	24.6
10–25	44.0	34.5	40.1
1–40	21.3	28.2	24.1

Appendix

Table 24: SFG-X window effects on omission-probe states in the 85-pair SD3.5M cohort. S/T denote single-/two-object prompts. Percentages use equal prompt weights on fixed baseline-warned concepts. Lower is better; bold marks the best guided window per column and metric.

Figure 23: Top to bottom: symptom persistence and no terminal recovery.

Class	Threshold	Detect	False alarm
S	q90	40.9	18.2
S	q95	27.3	9.1
T	q90	50.0	20.0
T	q95	20.0	10.0

Appendix

Table 25: Omission-probe threshold evaluation on a held-out SD3.5M seed. S/T denote single-/two-object prompts. q90/q95 are the 90th/95th percentiles of probe scores on successful calibration trajectories. Detect is the percentage of failed test trajectories flagged by the probe; False alarm is the percentage of successful test trajectories flagged.

Figure 24: Qualitative comparisons on SD3.5M. Each triplet compares CFG+PAG, CFG+SEG, and CFG+SFG (ours) with the same prompt and seed. Prompt excerpts highlight the requested subjects and visual attributes.

Figure 25: Compositional comparisons on FLUX. Each pair compares CFG and CFG+SFG (ours) with the same prompt and seed. Prompt excerpts appear above the pairs.

Figure 26: FLUX with CFG+SFG (ours) across styles and scenes. Selected outputs from our 800-prompt HPSv2.1 evaluation set; labels describe the images and are not the original prompts.

Explore similar work

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
Aug 14, 2026Nikolai Röhrich, Isabell Hans, Felix Krause +1Diffusion Model Guidance T2I Generation
LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany
PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latents
Sep 28, 2026Vighnesh Subramaniam, Boris Katz, Brian Cheung +3Diffusion Model Guidance T2I Generation
MIT CSAIL · Google Cloud AI Research · UCSF
TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers
Jan 5, 2026Binglei Li, Mengping Yang, Zhiyu Tan +2Diffusion Transformer Compositional T2I Generation
Fudan University, Shanghai 200433, China · Shanghai Academy of AI for Science, Shanghai 200030, China · Shanghai Innovation Institute, Shanghai 200030, China

Aug 14, 2026J/K move · Enter open · S save

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Nikolai Röhrich, Isabell Hans, Felix Krause, Björn Ommer

LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany

Diffusion Model Guidance T2I Generation Controllable Image Generation Diffusion Guidance Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance

Abstract

Explore similar work

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latents

TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers