Diffusion models generate samples through a multi-step denoising trajectory, yet condition every step on the same instruction, so the trajectory cannot be addressed at the level of individual operations. We introduce Dancing Diffusion, which assigns each span of timesteps a dedicated continuous condition block and thereby turns conditioning into a latent program. A compiler maps text features into this program, and a fixed pretrained generator executes its blocks over successive sampling intervals. The trajectory can then dance, keeping its executed history while its continuation is redirected. A shared prefix is executed once and reused, and a later operation is rewritten while content fixed by earlier operations is preserved. The compiler is trained through the images it produces, with visual feedback on both the requested change and the retained content. On SANA, reusing an executed prefix preserves markedly more of the scene than regenerating from noise at a similar strength of edit, learned conditions preserve more background than native prompt switching from the same saved state, and the learned program improves generation quality over native SANA. Later branches weaken the edit, which leaves semantic control over individual operations limited. By converting a single-instruction trajectory into a program of addressable operations, Dancing Diffusion lays a foundation for structured intermediate computation in diffusion models.
Figures & tables
Figure 1: Dancing: editing the generation path. (a) A background-only first block produces a common parent, shown as a predicted-clean readout. Both continuations reuse the same saved noisy state and introduce a sailboat or yacht through future conditions. A native same-noise generation is shown for orientation. Curves and intervals are schematic; the images illustrate the operation and are not a final-checkpoint comparison or evidence of native-method superiority. (b) A compiler produces conditions for a shared generator, with training-only feedback from a visual observer.
Figure 2: Training the conditions used after a saved prefix. The reference path supplies a detached sampler state and source image. The target compiler provides future conditions; the suffix output receives observer feedback for change and retention. Only compiler parameters are optimized. The source reference cannot move to make the edit loss easier. At deployment the observer and loss are absent.
Figure 3: Learning when to reconstruct foreground and background. (a) Exact-mask subject and background MSE on 1,024 validation images, normalized pointwise by the single-expert MSE, so that one denotes parity and lower is better. Every measured point is retained, and absolute curves are in Appendix F . (b) Predicted-clean x^0 readouts of separate pure-noise DDPM-200 trajectories with common initial and per-step noise, followed by the target condition image. Row colors match the curves in (a). The observer-guided model delays wall texture while keeping a recognizable sphere. These trajectories are not samples from the noised-reference evaluation in (a), and the example was selected post hoc for correct colors, so it does not estimate how often colors are correct.
Method
3/9/18
6/12/12
9/12/9
Full regeneration
0.383 / 13 / 27
0.334 / 11 / 29
0.307 / 14 / 29
Learned suffix
0.254 / 14 / 29
0.247 / 15 / 27
0.242 / 14 / 25
Native suffix
0.269 / 14 / 28
0.274 / 14 / 26
0.269 / 17 / 27
SDEdit–SANA
0.284 / 16 / 28
0.288 / 16 / 27
0.293 / 12 / 27
FlowEdit–SANA
0.143 / 13 / 27
0.138 / 12 / 25
0.142 / 12 / 24
Table 1: Matched category-cohort comparison: 12 families, three seeds, 36 attempts per schedule. Cells give BG LPIPS ↓ / DINO replacement count ↑ / OWLv2 replacement count ↑ . Mask coverage is 32/36, 33/36 and 33/36; counts use all attempts. Regeneration starts at 0; others at 9. Native uses the saved state; SANA-adapted image editors use the source PNG. Counts are proxies, not full-caption success; all starts appear in Appendix E .
Figure 4: Editing with visible scene retention. Genuine, uncropped outputs for the same source and request: final checkpoint, schedule 6/12/12, branch/start 9 (regeneration: 0). The learned suffix retains avenue and building structure; image-input SANA adaptations also retain substantial context. Their source access differs (saved state versus PNG). These selected illustrations do not estimate success rates; failure cases are described in Appendix P .
Method / schedule
Initialization
Update 600
Final 1423
Δ vs. native
Native SANA
7.349 (no adaptation)
—
3/9/18
7.248
7.203
7.315
-0.034
6/12/12
7.268
7.247
7.145
-0.204
9/12/9
7.411
7.466
7.375
+0.026
Table 2: Generation FID versus native SANA and across training stages. Negative final-minus-native differences indicate improvement. All entries use the same noise set; no training-seed uncertainty is inferred.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Source access
Question
Full regeneration
Same initial noise
Value of retaining source history
Native suffix
Same saved state
Value of learned future conditions
Initialized compiler
Same saved state
Effect of adaptation
SDEdit–SANA adaptation
Source PNG, re-noising
Image-input denoising control
FlowEdit–SANA adaptation
Source PNG, source/target velocities
Image-input flow editing control
Appendix
Table 3: Roles of the comparison methods.
Detector
Threshold
Audit precision
Audit recall
Grounding DINO tiny
0.40
0.886
0.819
OWLv2 base ensemble
0.31
0.900
0.861
Appendix
Table 4: Independent detector calibration on public images, not generated-editing accuracy.
Figure 5: Preservation and category change on a common evaluation cohort. Each point uses the same 12 category families and three seeds. Replacement proxies use all 36 attempts, and background LPIPS is defined for 32, 33 and 33 source masks across the schedules. Colors identify methods, marker shapes identify starts 9, 15 and 21, and the star denotes full regeneration. Curves connect descriptive means, not fitted frontiers, and the two detectors can order methods differently.
Method
Start
BG LPIPS ↓
Coverage
DINO T/R
OWLv2 T/R
Full regeneration
0
0.383
32/36
35/13
36/27
Learned suffix
9
0.254
32/36
36/14
36/29
Learned suffix
15
0.125
32/36
34/12
35/25
Learned suffix
21
0.037
32/36
30/0
17/8
Native suffix
9
0.269
32/36
36/14
36/28
Native suffix
15
0.122
32/36
34/13
35/23
Appendix
Table 5: Category proxies and preservation, schedule 3/9/18. T/R are counts out of 36 per detector; coverage is defined/36. Start is a saved-state branch for suffix methods and a re-noising or flow start for image-input methods.
Method
Start
BG LPIPS ↓
Coverage
DINO T/R
OWLv2 T/R
Full regeneration
0
0.334
33/36
34/11
35/29
Learned suffix
9
0.247
33/36
35/15
36/27
Learned suffix
15
0.129
33/36
34/13
33/26
Learned suffix
21
0.038
33/36
31/1
23/11
Native suffix
9
0.274
33/36
35/14
36/26
Native suffix
15
0.131
33/36
35/13
33/24
Appendix
Table 6: As Table 5 , schedule 6/12/12.
Method
Start
BG LPIPS ↓
Coverage
DINO T/R
OWLv2 T/R
Full regeneration
0
0.307
33/36
35/14
35/29
Learned suffix
9
0.242
33/36
35/14
34/25
Learned suffix
15
0.121
33/36
35/13
34/25
Learned suffix
21
0.043
33/36
30/2
21/6
Native suffix
9
0.269
33/36
36/17
36/27
Native suffix
15
0.126
33/36
36/11
34/23
Appendix
Table 7: As Table 5 , schedule 9/12/9.
Schedule
Method
Start 9
Start 15
Start 21
Coverage
3/9/18
SDEdit–SANA
0.265
0.288
0.269
63/72
3/9/18
FlowEdit–SANA
0.122
0.066
0.041
63/72
6/12/12
SDEdit–SANA
0.273
0.291
0.273
61/72
6/12/12
FlowEdit–SANA
0.122
0.067
0.042
61/72
9/12/9
SDEdit–SANA
0.282
0.296
0.282
60/72
9/12/9
FlowEdit–SANA
0.127
0.066
0.042
60/72
Appendix
Table 8: Background LPIPS of image-input SANA adaptations over all 72 source cases per schedule, using the historical source masks.
Figure 6: Absolute subject and background MSE on logarithmic scales, before normalization, with the fuller eight-step qualitative trajectory. The quantitative noised-reference evaluation and the selected pure-noise trajectory are separate experiments.
Schedule; branch
CLIP Δ
BG LPIPS ↓
BG PSNR ↑
BG SSIM ↑
3/9/18; 0
0.097 (72/72)
0.352 (63/72)
15.652 (63/72)
0.572 (63/72)
3/9/18; 9
0.092 (72/72)
0.225 (63/72)
19.073 (63/72)
0.701 (63/72)
3/9/18; 15
0.076 (72/72)
0.108 (63/72)
24.316 (63/72)
0.837 (63/72)
3/9/18; 21
0.028 (72/72)
0.030 (63/72)
31.710 (63/72)
0.946 (63/72)
6/12/12; 0
0.092 (72/72)
0.321 (61/72)
16.500 (61/72)
0.599 (61/72)
6/12/12; 9
0.088 (72/72)
0.227 (61/72)
19.096 (61/72)
0.697 (61/72)
Appendix
Table 9: Final edit-only 24-family diagnostics: means (defined/attempted cases). Three seeds per request; step 0 is full regeneration. PSNR with zero MSE is undefined numerically and excluded from its mean.
Schedule; branch
CLIP Δ
BG LPIPS ↓
BG PSNR ↑
BG SSIM ↑
3/9/18; 0
0.049 (72/72)
0.415 (10/72)
15.553 (10/72)
0.561 (10/72)
3/9/18; 9
0.042 (72/72)
0.198 (10/72)
21.826 (10/72)
0.780 (10/72)
6/12/12; 0
0.048 (72/72)
0.451 (11/72)
15.372 (11/72)
0.538 (11/72)
6/12/12; 9
0.041 (72/72)
0.244 (11/72)
20.267 (11/72)
0.729 (11/72)
9/12/9; 0
0.047 (72/72)
0.425 (12/72)
16.233 (12/72)
0.598 (12/72)
9/12/9; 9
0.042 (72/72)
0.252 (12/72)
20.327 (12/72)
0.745 (12/72)
Appendix
Table 10: As Table 9 , for the 24-case challenge pool.
Schedule; branch
Whole LPIPS
PNG MSE
Latent MSE
3/9/18; 9
0.050 (24/24)
0.003 (24/24)
0.020 (24/24)
3/9/18; 15
0.018 (24/24)
0.001 (24/24)
0.004 (24/24)
3/9/18; 21
0.005 (24/24)
0.000 (24/24)
0.000 (24/24)
6/12/12; 9
0.057 (24/24)
0.003 (24/24)
0.019 (24/24)
6/12/12; 15
0.031 (24/24)
0.001 (24/24)
0.006 (24/24)
6/12/12; 21
0.012 (24/24)
0.000 (24/24)
0.001 (24/24)
Appendix
Table 11: Same-prompt drift at the final checkpoint: eight evaluation families, three seeds. No target edit is requested; these distances measure change from the frozen-source continuation.
Branch
Control
IF
VC
VQ
9
Initial
11.1/6.9/22.2/59.7/0.0
9.7/12.5/34.7/43.1/0.0
9.7/22.2/41.7/26.4/0.0
9
Update 600
9.7/9.7/11.1/69.4/0.0
8.3/8.3/25.0/58.3/0.0
11.1/12.5/44.4/31.9/0.0
15
Initial
0.0/5.6/13.9/80.6/0.0
2.8/12.5/19.4/65.3/0.0
4.2/9.7/20.8/65.3/0.0
15
Update 600
0.0/5.6/9.7/84.7/0.0
0.0/5.6/18.1/76.4/0.0
2.8/8.3/18.1/70.8/0.0
21
Initial
0.0/0.0/0.0/100.0/0.0
0.0/0.0/0.0/100.0/0.0
0.0/0.0/0.0/100.0/0.0
21
Update 600
0.0/0.0/0.0/100.0/0.0
0.0/0.0/1.4/98.6/0.0
0.0/0.0/0.0/100.0/0.0
Appendix
Table 12: Final 3/9/18 automated comparison: percentages win/loss/order-disagreement/tie/missing for the final checkpoint versus each control. Each cell has 72 planned pairs, 24 families and both image orders. Missing judgments are retained in the denominator.
Figure 7: All pre-specified branch points under the final edit-only checkpoints. Each curve shows a preservation–semantic-proxy tradeoff, not a validated success frontier. Background means are conditional on valid source masks; semantic means use all attempts.
Branch
Native
Trained
Lower in
Δ [descriptive 95% interval]
9
0.2528
0.2296
41/60
−0.0216 [ −0.0461 , −0.0036 ]
15
0.1134
0.1134
28/60
+0.0000 [ −0.0069 , +0.0069 ]
21
0.0305
0.0398
8/60
+0.0090 [ +0.0062 , +0.0119 ]
Appendix
Table 16: Matched-prefix preservation at the final checkpoint. Background LPIPS uses historical source masks: 60/72 cases from 21 families per branch. Lower is better. The difference and interval give an equally weighted family estimate, whereas the two means average valid attempts.
Figure 8: Selected differences beyond prefix reuse. Both suffixes start from the same state. The trained output better retains the person’s scale and the wall in the first row, and the striped clothing and portrait scale in the second. The bicycle caption also requests napping, which neither edit realizes. Post-hoc examples from the final 9/12/9 compiler, uncropped and unretouched.
Schedule
pad
n
Regen.
FlowEdit
SDEdit
Native
3-9-18
0%
68/72
−0.122
+0.100
−0.040
−0.016
3-9-18
5%
65/72
−0.129
+0.097
−0.052
−0.017
3-9-18
10%
62/72
−0.127
+0.097
−0.056
−0.016
6-12-12
0%
66/72
−0.092
+0.105
−0.043
−0.024
6-12-12
5%
62/72
−0.094
+0.104
−0.055
−0.022
6-12-12
10%
60/72
−0.090
+0.100
−0.064
−0.023
Appendix
Table 17: Source-box background LPIPS on the full evaluation cohort. Entries are paired mean differences (ours branch 9 minus comparator; lower is better). n is defined/attempted, with missing cases retained.
Schedule
pad
n
Regen.
FlowEdit
SDEdit
Native
3-9-18
0%
33/36
−0.114
+0.114
−0.023
−0.011
3-9-18
5%
30/36
−0.124
+0.111
−0.043
−0.014
3-9-18
10%
30/36
−0.127
+0.107
−0.048
−0.012
6-12-12
0%
33/36
−0.083
+0.109
−0.035
−0.029
6-12-12
5%
30/36
−0.083
+0.106
−0.058
−0.031
6-12-12
10%
30/36
−0.083
+0.097
−0.070
−0.032
Appendix
Table 18: As Table 17 , on the eligible cohort.
Comparator
Schedule
pad
n
Δ [95% interval]
Regeneration
3-9-18
0%
68/72
−0.122[−0.155,−0.092]
Regeneration
3-9-18
5%
65/72
−0.129[−0.168,−0.095]
Regeneration
3-9-18
10%
62/72
−0.127[−0.170,−0.090]
Regeneration
6-12-12
0%
66/72
−0.092[−0.113,−0.073]
Regeneration
6-12-12
5%
62/72
−0.094[−0.116,−0.072]
Regeneration
6-12-12
10%
60/72
−0.090[−0.110,−0.070]
Appendix
Table 19: Selected 95% family-bootstrap intervals for the full evaluation cohort. All comparator intervals and RGB MSE are available in the analysis artifact.
Schedule
Branch
pad 0%
pad 5%
pad 10%
3-9-18
9
−0.016 (68/72)
−0.017 (65/72)
−0.016 (62/72)
3-9-18
15
+0.003 (68/72)
+0.003 (65/72)
+0.003 (62/72)
3-9-18
21
+0.001 (68/72)
+0.001 (65/72)
+0.001 (62/72)
6-12-12
9
−0.024 (66/72)
−0.022 (62/72)
−0.023 (60/72)
6-12-12
15
−0.002 (66/72)
−0.001 (62/72)
−0.000 (60/72)
6-12-12
21
+0.003 (66/72)
+0.003 (62/72)
+0.003 (60/72)
Appendix
Table 20: Native shared-path source-box LPIPS comparison, full evaluation cohort. Point differences are ours minus native; n is defined/attempted.
Schedule
Branch
pad 0%
pad 5%
pad 10%
3-9-18
9
−0.011 (33/36)
−0.014 (30/36)
−0.012 (30/36)
3-9-18
15
+0.003 (33/36)
+0.003 (30/36)
+0.003 (30/36)
3-9-18
21
+0.002 (33/36)
+0.002 (30/36)
+0.002 (30/36)
6-12-12
9
−0.029 (33/36)
−0.031 (30/36)
−0.032 (30/36)
6-12-12
15
−0.002 (33/36)
−0.002 (30/36)
−0.002 (30/36)
6-12-12
21
+0.003 (33/36)
+0.003 (30/36)
+0.003 (30/36)
Appendix
Table 21: As Table 20 , on the eligible cohort.
Schedule
Recomputed (s)
Cached (s)
Speedup
Time saved
3/9/18
9.28
7.19
1.29 ×
22.6%
6/12/12
9.20
7.18
1.28 ×
22.0%
9/12/9
9.39
7.19
1.31 ×
23.4%
Appendix
Table 22: Mean time over four scenes and three repetitions to produce four target continuations. Every cached endpoint matches its recomputed counterpart exactly.
Training schedule
3/9/18
6/12/12
9/12/9
3/9/18
0.074 (24) / 0.129 (24)
0.070 (24) / 0.133 (23)
0.075 (24) / 0.139 (22)
6/12/12
0.076 (24) / 0.138 (24)
0.073 (24) / 0.138 (23)
0.078 (24) / 0.144 (22)
9/12/9
0.077 (24) / 0.160 (24)
0.070 (24) / 0.144 (23)
0.074 (24) / 0.143 (22)
Initialization
0.072 (24) / 0.126 (24)
0.073 (24) / 0.134 (23)
0.076 (24) / 0.143 (22)
Appendix
Table 23: Complete crossing, shown at branch 15. Cells report CLIP paired margin change / background LPIPS, with defined denominators in parentheses (24 attempted). These proxies do not establish edit success.
Schedule; branch
Matched source
Own source
3/9/18; 0
0.101 (24) / 0.399 (24)
0.101 (24) / 0.390 (22)
3/9/18; 9
0.091 (24) / 0.266 (24)
0.092 (24) / 0.254 (22)
3/9/18; 15
0.074 (24) / 0.129 (24)
0.074 (24) / 0.126 (22)
3/9/18; 21
0.019 (24) / 0.036 (24)
0.022 (24) / 0.038 (22)
6/12/12; 0
0.095 (24) / 0.375 (23)
0.095 (24) / 0.369 (22)
6/12/12; 9
0.089 (24) / 0.272 (23)
0.088 (24) / 0.260 (22)
Appendix
Table 24: Diagonal source-path comparison. CLIP paired margin change / background LPIPS (defined denominators); different source modes use different source images.
Compiler
B0
B1
B2
all
nochange
Trained 3/9/18
0.001 (24) 0.179 (24)
0.015 (24) 0.306 (24)
0.090 (24) 0.237 (24)
0.101 (24) 0.399 (24)
-0.000 (24) 0.147 (24)
Trained 6/12/12
0.001 (24) 0.204 (23)
0.040 (24) 0.301 (23)
0.048 (24) 0.114 (23)
0.095 (24) 0.375 (23)
0.000 (24) 0.070 (23)
Trained 9/12/9
0.004 (24) 0.221 (22)
0.059 (24) 0.245 (22)
0.027 (24) 0.094 (22)
0.093 (24) 0.336 (22)
0.000 (24) 0.078 (22)
Init. 3/9/18
0.000 (24) 0.088 (24)
0.012 (24) 0.319 (24)
0.087 (24) 0.191 (24)
0.099 (24) 0.411 (24)
0.000 (24) 0.000 (24)
Init. 6/12/12
-0.000 (24) 0.190 (23)
0.039 (24) 0.312 (23)
0.047 (24) 0.078 (23)
0.093 (24) 0.382 (23)
0.000 (24) 0.000 (23)
Init. 9/12/9
0.006 (24) 0.208 (22)
0.060 (24) 0.256 (22)
0.026 (24) 0.039 (22)
0.094 (24) 0.343 (22)
0.000 (24) 0.000 (22)
Appendix
Table 25: Block interventions at branch 0, where all blocks are active. CLIP paired margin change / background LPIPS (defined denominators). Initial and trained methods share the frozen source; these are descriptive measurements.
Figure 9: Generation along the learned path. Final compiler, identical initial noise, with source, subject-changed and background-changed captions. Columns show x0=xt−σtvθ(xt,ck) from the last executed denoiser evaluation in each block (steps 6, 18, 30), not decodes of noisy states. B0 exposes coarse scene appearance, B1 makes the subject recognizable and B2 completes the image. Changing the scene also changes pose and appearance, so stage progression alone is not factor independence.
Figure 10: Edit strength against source closeness, all five starts. Each curve traces one method across starts 0, 6, 9, 15 and 21, for the whole cohort and the two non-empty source-size strata. Later starts stay closer to the source but edit less, so low LPIPS at a late start indicates a weak edit. Subject panels use both-detector strict replacement, and background panels the uncalibrated CLIP scene margin. The medium stratum holds only eight sources.
Start
Method
DINO
OWLv2
Both
BG LPIPS ( n )
0
B0-isolated learned
19
48
19
0.173 (48)
0
Native shared state
9
48
9
0.451 (48)
0
FlowEdit PNG, CFG4
9
48
9
0.297 (48)
0
FlowEdit PNG, CFG6
11
48
11
0.355 (48)
0
FlowEdit PNG, CFG8
13
48
13
0.397 (48)
0
FlowEdit latent, CFG4
8
48
8
0.293 (48)
Appendix
Table 26: B0-isolated source cohort: subject replacement. Strict category replacement requires source category present in the source, target present after editing and source category absent after editing. Counts use all 48 attempts; Both requires both calibrated detectors. BG LPIPS uses 5% source-box padding; parentheses give defined backgrounds. Lower LPIPS is useful only with successful editing.
Start
Method
DINO
OWLv2
CLIP
FG LPIPS ( n )
0
B0-isolated learned
48
48
0.103
0.657 (48)
0
Native shared state
48
48
0.101
0.664 (48)
0
FlowEdit PNG, CFG4
48
48
0.089
0.572 (48)
0
FlowEdit PNG, CFG6
48
48
0.092
0.605 (48)
0
FlowEdit PNG, CFG8
48
48
0.094
0.642 (48)
0
FlowEdit latent, CFG4
48
48
0.088
0.577 (48)
Appendix
Table 27: B0-isolated source cohort: background changes. Retained counts require the original subject category in both source and output, using all 48 attempts. CLIP denotes scene-margin change; FG LPIPS uses 5% padded source boxes, with defined-region counts in parentheses.
Direction
Start
Both/retained (pp)
95% interval
Region LPIPS
subject
0
+20.8
[+4.2, +45.8]
-0.277
subject
6
+6.2
[+0.0, +18.8]
-0.007
subject
9
+8.3
[+2.1, +14.6]
-0.001
subject
15
+8.3
[+2.1, +16.7]
-0.001
subject
21
+4.2
[+0.0, +10.4]
-0.001
background
0
+0.0
[+0.0, +0.0]
-0.007
Appendix
Table 28: B0-isolated learned minus native shared state, all 48 paired cases. Detector entries are percentage-point differences in strict replacement (subject) or source-subject retention (background); LPIPS entries are differences in the 5% padded region metric, where a negative value is closer to the source. Intervals are unadjusted descriptive family bootstrap percentiles.
Branch
Method
D/O/B
BG ↓
FG ↓
Scene Δ
9
Learned suffix
13/48/13
0.1195
0.2953
0.0844
9
Native shared-state
8/48/8
0.1210
0.3095
0.0826
9
FlowEdit PNG, CFG 4
10/48/10
0.0762
0.1890
0.0597
9
FlowEdit PNG, CFG 6
14/48/14
0.1059
0.2424
0.0816
9
FlowEdit PNG, CFG 8
12/48/12
0.1407
0.3065
0.0869
9
FlowEdit latent, CFG 4
10/48/10
0.0697
0.2002
0.0637
Appendix
Table 29: Complete bidirectional grid, 48 attempts per direction and row. D/O/B are strict replacement counts for DINO, OWLv2 and both. BG and FG are non-target LPIPS for subject and background edits (lower is better), and Scene Δ is the CLIP margin change.
Figure 11: Bidirectional stress-test tradeoffs for all sources and for large source boxes. Lines connect the three measured branch points. Upper panels use strict replacement, lower panels the scene-direction CLIP change.
Figure 12: Prefix reuse retains what regeneration discards. Regeneration adds a second rider, reframes the plants and replaces the room and the lettering. Prefix reuse keeps the single rider and herd, the potted plants and their layout, and the candlelit room, the bow tie and the “me to you” lettering, while changing the requested subject.
Recent diffusion editors perform diverse instruction-based edits while conditioning on the source image at every denoising step. Yet persistent source-image conditioning can limit how fully an edit is executed and how natural the result appears, especially when the target scene diverges substantially from the input. We introduce DuET (Dual Expert Trajectories), a training-free inference method that temporarily relaxes source-image conditioning by transitioning through a text-to-image phase before returning to edit mode, allowing the denoising trajectory to move toward the target distribution while retaining the structural benefits of image-conditioned editing. Without modifying model weights or increasing sampling cost, DuET consistently improves instruction relevance, semantic fidelity, and perceptual quality across diverse models and benchmarks. In some cases, these gains come with a modest reduction in source-image preservation, revealing a predictable trade-off between source preservation and edit fidelity.
Lidia Troeshestova, Alexander Ustyuzhanin, Sergey Kastryulin
Diffusion models can produce striking images and videos, but they still struggle with the compositional details that make a generation faithful to a prompt, such as object counts, attribute binding, spatial relations, and temporally grounded actions. A common way to improve prompt satisfaction is to spend more compute at test time through Best-of-N sampling, but final-sample selection is fixed. Best-of-N can only choose among completed outputs and cannot repair a promising trajectory before it fails. We introduce PreviewDiff, a training-free test-time search method that turns diffusion sampling from scalar search into a multimodal critic-guided search over intermediate latents. At selected denoising checkpoints, PreviewDiff decodes a partial preview, asks a multimodal judge to score and critique it, and uses the resulting natural-language feedback to branch over semantic prompt edits and locally re-noised latent continuations. These branches are then scored and selectively rolled forward, allowing verifier compute to guide generation while the sample is still editable. Across image and video generation benchmarks, PreviewDiff consistently improves over budget-matched Best-of-N selection and strong scalar-search baselines. Ablations show that earlier interventions and increased search width provide the largest gains, while deeper search and additional semantic variants offer complementary improvements. PreviewDiff demonstrates that multimodal feedback is most useful not only as a final verifier, but as an active controller inside the denoising process.
Real-world image generation often involves multi-turn editing, where users iteratively modify small regions while most image content remains unchanged. However, existing diffusion transformer (DiT)-based editing pipelines recompute the entire image at every turn, causing substantial redundant computation. Existing DiT acceleration methods further ignore semantic correspondence across prompts, leading to unnecessary recomputation or unsafe reuse that harms editing quality. To address this, we propose RegionCache, a semantic-aware reuse framework for multi-turn image editing that selectively reuses diffusion states from unchanged regions. RegionCache detects reusable regions through semantic overlap between consecutive prompts and cross-attention localization, and adopts an adaptive reuse schedule based on prompt similarity and contextual consistency. Experiments on PixArt-alpha demonstrate that RegionCache achieves 1.43x--2.55x end-to-end speedup while maintaining comparable image quality. Code is available at https://github.com/hebutBryant/RegionCache.
Peizheng Li, Xin Ai, Hanyuan Liu +2
School of Computer Science and Engineering, Northeastern University, China · School of Computer Science