ProgressNet: Sketching and Prompting with a Frozen Text-to-Image Model

Arkaprabha Basu; Chaitat Utintu; Yi-Zhe Song

cs.CVOct 2, 2026

ProgressNet: Sketching and Prompting with a Frozen Text-to-Image Model

Authors: Arkaprabha Basu, Chaitat Utintu, Yi-Zhe Song

Organizations: SketchX, CVSSP, University of Surrey · CoSTAR National Lab

Abstract

Humans draw progressively: a few strokes, a look at the result, a stroke erased, a prompt revised. Image generators do not work this way. They typically take a finished sketch and produce the image in a single pass, so every edit starts the picture again, and the models that do keep state across turns are driven by text, cannot take a stroke, and are too slow to draw with. We present ProgressNet, a training-free framework that lets a frozen text-to-image model follow a drawing session as it unfolds: strokes are added and erased, the prompt is revised, and the image keeps up at about a second per turn. It needs no new parameters because the frozen model already has what a progressive generator needs, a pathway through which the previous turn can be remembered, layers that can carry appearance forward without freezing structure, and an internal signal of how far to trust an unfinished sketch; three inference-time mechanisms (Previous-Concept Memory, Layer-Selective K/V Injection and Banded Adaptive Control) use each in turn. As a sketch fills in, every existing method degrades, the FID of the FLUX+ControlNet baseline doubling between 10% and 100% completion on FS-COCO, while ProgressNet's barely moves; it maintains strong fidelity and progressive coherence across three sketch domains and is preferred by users over five competitors, most widely on erasure.

Figures & tables

Figure 1: ProgressNet follows a drawing session turn by turn, training-free on frozen FLUX. Top : the cat’s body is erased and it moves onto a platform, stands once legs are drawn, loses its tail when it is erased, and walks into a river once the tail is redrawn. Bottom : a scene is built stroke by stroke, re-themed by the prompt from snowy to green, and finished with a house and birds.

Figure 2: Flux+ControlNet imprints raw stroke edges and visually drifts across progressive turns; ours adapts smoothly while maintaining temporal coherence.

Panel of Figure 3 — Figure 3: Pilot studies illustrating failure modes of existing methods: temporal drift, prompt-sketch conflict under partial strokes, and contradiction upon stroke removal. (a) Under progressive sketching, FLUX+ControlNet and StableFlow show temporal instability and artifacts on the partial strokes ( red borders), while ours remains stable across turns. (b) When strokes are removed and re-added (Seq. 1-3), FLUX+ControlNet fails to preserve established concepts, shifting identity across turns.

Figure 4: Overview of ProgressNet. Left : the previous output $I_{t-1}$ is prepared according to the current operation: addition keeps $I_{t-1}$ , removal white-fills ( $W$ ) the erased region $\mathbf{F}_{t}$ , and a prompt update gray-fills ( $N$ ) the uncommitted region $1-\mathbf{C}_{t}$ . Middle : the memory $M_{t}$ and current sketch $\mathbf{S}_{t}$ condition Union ControlNet $\mathbf{C}_{U}$ , while BAC scales the memory stream per layer band and solver step. Right : LS-KVI reuses previous-turn K/V features at six low-vitality FLUX layers.

Figure 5: Vital vs. non-vital layer. Vital-layer injection locks in early details, losing adaptability, while non-vital LS-KVI preserves appearance and sharpness as structure follows sketch.

Figure 6: Layer- and solver-step variation in control responses. Control-induced perturbations and their normalized magnitudes vary across the evaluated layer groups and solver steps during progression. Appendix Sections C.2 and C.3 report complementary feature-redistribution diagnostics on SD3.5 and across four sketch domains.

	10% Completion						50% Completion						100% Completion
Metric	S-a-S	SDXS	CB	Flux	SF	Ours	S-a-S	SDXS	CB	Flux	SF	Ours	S-a-S	SDXS	CB	Flux	SF	Ours
FS-COCO
FID-I $\downarrow$	192.3	40.3	195.9	32.9	36.8	36.5	200.1	41.4	209.9	44.7	40.7	36.0	219.2	49.6	208.3	67.9	52.8	37.3
FID-C $\downarrow$	0.432	0.161	0.400	0.149	0.185	0.152	0.434	0.170	0.396	0.188	0.201	0.141	0.475	0.188	0.407	0.264	0.217	0.143
DINOv2 $\uparrow$	0.286	0.331	0.328	0.375	0.397	0.400	0.305	0.337	0.340	0.386	0.384	0.415	0.276	0.328	0.317	0.361	0.359	0.413
CLIPScore $\uparrow$	0.267	0.299	0.293	0.298	0.301	0.306	0.268	0.297	0.291	0.285	0.296	0.305	0.256	0.291	0.289	0.265	0.289	0.303

Table 1: Comparison with SOTA methods across sketch completion levels and domains. Latency is measured per turn, independent of completion level. Bold : best; underline : second best.

		PFC-DINO $\uparrow$				PFC-LPIPS $\uparrow$				Cross-Comp. Robustness $\uparrow$
Dataset	Method	$10\mkern-4.0mu\blacktriangleright\mkern-4.0mu20$	$30\mkern-4.0mu\blacktriangleright\mkern-4.0mu50$	$80\mkern-4.0mu\blacktriangleright\mkern-4.0mu100$	Mean	$10\mkern-4.0mu\blacktriangleright\mkern-4.0mu20$	$30\mkern-4.0mu\blacktriangleright\mkern-4.0mu50$	$80\mkern-4.0mu\blacktriangleright\mkern-4.0mu100$	Mean	Dist.	Vis.	Text	Overall
FS-COCO	Sketch-a-Sketch	0.237	0.228	0.222	0.229	0.331	0.322	0.299	0.317	39.51	76.22	97.50	66.47
	SDXS	1.883	1.824	1.579	1.762	1.864	1.820	1.546	1.744	133.14	90.18	105.10	108.06
	Cond. Balance	0.261	0.236	0.229	0.242	0.318	0.298	0.299	0.305	42.74	87.68	104.06	73.06
	Flux Baseline	1.836	1.387	0.999	1.407	1.896	1.425	0.982	1.435	100.00	100.00	100.00	100.00
	StableFlow	2.198	2.120	1.682	2.000	2.174	2.151	1.726	2.017	119.70	100.23	105.41	108.14

Table 2: Progressive stability across sketch domains. PFC-DINO and PFC-LPIPS evaluate temporal coherence and generation quality at the three reported FID-I anchors. Cross-Comp. reports cross-completion robustness, normalized to the Flux baseline (

100

). Bold : best; underline : second best.

Panel of Figure 7 — Figure 7: Progressive sketching (left) and sketch removal (right). Competing methods frequently drift or introduce structural artifacts as strokes evolve, while ProgressNet preserves appearance and scene consistency while adapting to added or removed strokes.

	Study 1: Overall			Study 2: Operation-based
	Pref.	Coher.	Qual.	Erase	Edit	Prog.	F-Draw	F-Erase	Preserv.
S-a-S	2.9	4.9	3.4	1.4	1.9	3.0	5.1	4.2	3.8
SDXS	4.1	13.2	4.3	1.8	2.3	4.2	7.1	5.8	5.1
CB	5.1	10.1	7.2	2.1	4.1	12.1	7.6	7.2	7.4
Flux	5.9	6.8	10.2	2.8	5.3	7.6	8.1	7.8	8.2
SF	10.2	20.3	15.1	6.7	8.8	18.3	22.4	19.7	17.6

Table 3: User study preferences.

				50% Completion					100% Completion
Variant	PCM	KV	BAC	FID-I $\downarrow$	FID-C $\downarrow$	DINOv2 $\uparrow$	CLIP $\uparrow$	T5 $\uparrow$	FID-I $\downarrow$	FID-C $\downarrow$	DINOv2 $\uparrow$	CLIP $\uparrow$	T5 $\uparrow$	Latency $\downarrow$
Baseline	✗	✗	✗	44.7	0.188	0.386	0.285	0.642	67.9	0.264	0.361	0.265	0.641	0.761
+ PCM	✓	✗	✗	33.9	0.145	0.420	0.304	0.654	36.6	0.150	0.414	0.301	0.653	0.816 +0.055
+ LS-KVI	✓	✓	✗	35.7	0.161	0.415	0.303	0.654	37.9	0.162	0.415	0.300	0.653	0.816 +0.055
Ours	✓	✓	✓	36.0	0.141	0.415	0.305	0.656	37.3	0.143	0.413	0.303	0.655	0.829 +0.068

Table 4: Ablation on FS-COCO across sketch completion levels. Each row cumulatively enables PCM, LS-KVI, and BAC. Bold : best; underline : second best.

Figure 9: Without BAC, sketch-edge artifacts leak under prompt and sketch conflict or imperfect sketches from Pixelor ( Bhunia et al., 2020 ) and human drawn sketches. BAC reduces visible artifacts associated with these difficult sketch-prompt configurations.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Eval. $N$	Sketch domain	Sampling / pairing
FS-COCO	10,000	Freehand scenes	Full evaluation set; paired image/text
Photo-Sketching	500	Contour tracing	Evaluation subset of paired sketch/image data
Sketchy	1,250	Freehand objects	10/category across 125 categories

Appendix

Table 5: Datasets used for generation-quality and progressive evaluation. The same sample identities and progressive states are supplied to every method.

	20% Completion						30% Completion						70% Completion						80% Completion
Metric	S-a-S ( Sarukkai et al., 2024 )	SDXS ( Song et al., 2024 )	CB ( Cohen et al., 2025 )	Flux ( Black Forest Labs, 2024 )	SF ( Avrahami et al., 2025 )	Ours	S-a-S ( Sarukkai et al., 2024 )	SDXS ( Song et al., 2024 )	CB ( Cohen et al., 2025 )	Flux ( Black Forest Labs, 2024 )	SF ( Avrahami et al., 2025 )	Ours	S-a-S ( Sarukkai et al., 2024 )	SDXS ( Song et al., 2024 )	CB ( Cohen et al., 2025 )	Flux ( Black Forest Labs, 2024 )	SF ( Avrahami et al., 2025 )	Ours	S-a-S ( Sarukkai et al., 2024 )	SDXS ( Song et al., 2024 )	CB ( Cohen et al., 2025 )	Flux ( Black Forest Labs, 2024 )	SF ( Avrahami et al., 2025 )	Ours
FID-I $\downarrow$	195.1	39.8	198.4	37.0	36.5	35.8	202.9	40.2	193.4	38.5	38.3	36.4	210.7	44.0	203.4	52.5	43.6	36.1	212.6	45.7	208.5	58.0	47.6	36.7
FID-C $\downarrow$	0.442	0.163	0.406	0.163	0.190	0.158	0.433	0.165	0.396	0.166	0.198	0.144	0.465	0.177	0.398	0.212	0.203	0.142	0.470	0.180	0.421	0.229	0.209	0.141
DINOv2 $\uparrow$	0.288	0.333	0.331	0.379	0.392	0.402	0.281	0.336	0.349	0.383	0.387	0.409	0.290	0.335	0.319	0.381	0.378	0.415	0.289	0.334	0.313	0.375	0.370	0.415
CLIPScore $\uparrow$	0.256	0.299	0.292	0.294	0.300	0.304	0.266	0.298	0.291	0.291	0.297	0.305	0.266	0.295	0.291	0.278	0.294	0.304	0.252	0.294	0.289	0.274	0.291	0.304
T5-Cos $\uparrow$	0.630	0.648	0.644	0.648	0.656	0.658	0.623	0.648	0.639	0.646	0.656	0.657	0.629	0.646	0.644	0.642	0.654	0.656	0.629	0.646	0.630	0.641	0.654	0.655

Appendix

Table 6: Extended comparison at intermediate sketch completion levels on FS-COCO. Results at

\{10,50,100\}\%

are in the main paper ( Table 1 ).

				10% Completion					20% Completion					30% Completion					70% Completion					80% Completion
Variant	PCM	KV	BAC	FID-I $\downarrow$	FID-C $\downarrow$	DINO $\uparrow$	CLIP $\uparrow$	T5 $\uparrow$	FID-I $\downarrow$	FID-C $\downarrow$	DINO $\uparrow$	CLIP $\uparrow$	T5 $\uparrow$	FID-I $\downarrow$	FID-C $\downarrow$	DINO $\uparrow$	CLIP $\uparrow$	T5 $\uparrow$	FID-I $\downarrow$	FID-C $\downarrow$	DINO $\uparrow$	CLIP $\uparrow$	T5 $\uparrow$	FID-I $\downarrow$	FID-C $\downarrow$	DINO $\uparrow$	CLIP $\uparrow$	T5 $\uparrow$
Baseline	✗	✗	✗	32.9	0.149	0.375	0.298	0.651	37.0	0.163	0.379	0.294	0.648	38.5	0.166	0.383	0.291	0.646	52.5	0.212	0.381	0.278	0.642	58.0	0.229	0.375	0.274	0.641
+ PCM	✓	✗	✗	36.5	0.152	0.400	0.306	0.657	35.0	0.148	0.407	0.306	0.656	34.3	0.147	0.413	0.305	0.656	34.5	0.146	0.422	0.303	0.654	34.7	0.147	0.421	0.303	0.653
+ LS-KVI	✓	✓	✗	36.5	0.152	0.400	0.306	0.657	37.0	0.163	0.402	0.304	0.656	36.4	0.163	0.408	0.303	0.655	36.2	0.159	0.419	0.302	0.653	36.5	0.160	0.418	0.301	0.653
Full (+ BAC)	✓	✓	✓	36.5	0.152	0.400	0.306	0.657	35.8	0.158	0.402	0.304	0.658	36.4	0.144	0.409	0.305	0.657	36.1	0.142	0.415	0.304	0.656	36.7	0.141	0.415	0.304	0.655

Appendix

Table 7: Extended ablation at intermediate completion levels on FS-COCO. Results at

\{50,100\}\%

are in the main paper ( Table 4 ).

Figure 10: Extended qualitative comparison of vital-layer vs. non-vital-layer K/V injection across four FS-COCO samples at progressive sketch completions ( $20\%$ , $50\%$ , $80\%$ , $100\%$ ). Vital-layer ( ( Avrahami et al., 2025 ) ) injection progressively corrupts appearance, shifting illumination (clock tower darkens), altering identity (horse rider changes clothing and pose), and inducing color drift (banana flower saturates), while non-vital-layer ( $\{5,6,7,10,13,14\}$ ) injection preserves consistent appearance, lighting, and identity throughout the progression. This extends the single-sample comparison in Fig. 5 of the main paper.

LS-KVI state	Return LPIPS $\downarrow$	Recovery $\uparrow$	DINOv2 $\uparrow$	Erased-region L1 $\downarrow$	Outside-change L1 $\downarrow$
Retain previous K/V	$0.295$	$0.019$	$0.854$	$0.121$	$\mathbf{0.096}$
Invalidate previous K/V	$\mathbf{0.204}$	$\mathbf{0.303}$	$\mathbf{0.885}$	$\mathbf{0.085}$	$0.123$

Appendix

Table 8: Erasure-specific LS-KVI ablation on 100 FS-COCO sequences.

Figure 11: Layerwise relative change in the measured control-induced perturbation across progressive sketch transitions at solver steps 0 and 1. The response varies systematically with backbone depth and solver step.

Figure 12: Comparison of BAC correction-factor variants on an out-of-distribution sketch from Pixelor ( Bhunia et al., 2020 ) . We evaluate cosine similarity, KL divergence, Jensen–Shannon divergence (JSD), and channel-wise JSD. The first column (W/o BAC) applies no correction factor. KL and JSD introduce visible artifacts at full completion, while disabling correction altogether retains a solid background with dark sketching artifacts. Compared to cosine similarity, our channel-wise JSD comparison produces smoother modulation without imprinting strong sketch-edge details into the generated image.

Transition	FLUX (5 layers)	SD3.5 Canny (19)	SD3.5 Blur (19)
10 $\to$ 20%	3.2 $\times$	3.3 $\times$	2.7 $\times$
20 $\to$ 30%	2.0 $\times$	2.0 $\times$	2.7 $\times$
30 $\to$ 50%	2.2 $\times$	2.2 $\times$	2.3 $\times$
50 $\to$ 70%	2.3 $\times$	2.0 $\times$	2.4 $\times$
80 $\to$ 100%	2.2 $\times$	1.8 $\times$	2.6 $\times$

Appendix

Table 9: Late/early JSD ratio across FLUX and SD3.5. Ratios above one indicate stronger redistribution in later ControlNet layers.

Control branch	Step 0	Step 1	Step 2	Step 3	Last/First	Last higher
SD3.5 Canny	0.005	0.007	0.009	0.021	3.97 $\times$	99.3%
SD3.5 Blur	0.118	0.147	0.168	0.212	1.79 $\times$	100%
FLUX (2-step)	0.134	0.206	–	–	1.54 $\times$	97.0%

Appendix

Table 10: Step asymmetry at the

10{\to}20\%

transition. SD3.5 exhibits progressive amplification across its four solver steps.

Dataset	10 $\to$ 20	20 $\to$ 30	30 $\to$ 50	50 $\to$ 70	70 $\to$ 80	80 $\to$ 100
FS-COCO	3.2 $\times$	2.0 $\times$	2.2 $\times$	2.3 $\times$	2.3 $\times$	2.2 $\times$
FrISS	3.4 $\times$	2.0 $\times$	2.1 $\times$	2.1 $\times$	2.1 $\times$	2.1 $\times$
Photo-Sketching	3.2 $\times$	2.0 $\times$	2.1 $\times$	2.1 $\times$	2.1 $\times$	2.2 $\times$
Sketchy	3.3 $\times$	1.8 $\times$	2.2 $\times$	2.0 $\times$	1.9 $\times$	2.0 $\times$

Appendix

Table 11: Late/early JSD ratio at solver step 0 across four sketch domains. The same band asymmetry persists at every evaluated transition.

Dataset	Step 1 / Step 0	Samples with step 1 higher
FS-COCO	1.54 $\times$	97.0%
FrISS	1.59 $\times$	97.7%
Photo-Sketching	1.44 $\times$	93.8%
Sketchy	2.15 $\times$	96.3%

Appendix

Table 12: Step-1/step-0 JSD amplification at the

10{\to}20\%

transition. The existence of step asymmetry is stable across domains, while its magnitude remains content-dependent.

Rank	FS-COCO	FrISS	Photo-Sketching	Sketchy
1	20 $\to$ 30 (.362)	20 $\to$ 30 (.365)	20 $\to$ 30 (.371)	20 $\to$ 30 (.390)
2	80 $\to$ 100 (.247)	80 $\to$ 100 (.241)	80 $\to$ 100 (.241)	80 $\to$ 100 (.256)
3	30 $\to$ 50 (.238)	30 $\to$ 50 (.228)	30 $\to$ 50 (.232)	30 $\to$ 50 (.256)
4	50 $\to$ 70 (.225)	50 $\to$ 70 (.217)	50 $\to$ 70 (.214)	50 $\to$ 70 (.234)
5	70 $\to$ 80 (.193)	70 $\to$ 80 (.203)	70 $\to$ 80 (.198)	70 $\to$ 80 (.212)
6	10 $\to$ 20 (.134)	10 $\to$ 20 (.118)	10 $\to$ 20 (.128)	10 $\to$ 20 (.104)

Appendix

Table 13: Transition ranking by mean JSD across datasets (rank 1 indicates the largest redistribution). The ordering is preserved across all four domains.

Figure 13: Progressive sketch-to-image generation, extension of Fig. 8 . Prompt: In this scene the cat sitting. StableFlow shows stronger stroke-region amplification, while SDXS contains visible artifacts. The baseline forms a cat but changes noticeably in appearance and structure across turns. Our method remains responsive to stroke additions while preserving the sketch structure and overall appearance.

Figure 14: Progressive sketch-to-image generation. Prompt: farm with trees in surrounding . StableFlow shows stronger green saturation, while the baseline contains visible artifacts. SDXS captures the overall structure but retains less fine detail. Our approach adapts to sketch changes while preserving scene appearance and details.

Figure 15: Progressive sketch-to-image generation. Prompt: a train moving through the woods. Both the baseline and SDXS contain visible artifacts, while StableFlow shows stronger green saturation. Our method adapts to the final tree-branch updates while retaining a consistent scene appearance.

Figure 16: Progressive sketch-to-image generation. Prompt: a giraffe is standing near a tree . This is a challenging scenario where the evolving sketch initially differs from the prompt’s general subject. StableFlow remains close to the initial structure, while SDXS shows limited adherence to later sketch updates and the baseline contains visible artifacts. Our model progressively adapts to the structural changes, following the initial tree strokes before forming the giraffe structure as additional evidence arrives.

Figure 17: Progressive sketch-to-image generation. Prompt: A train on a rail track . StableFlow shows a reddish triangular artifact together with strong persistence of the initial structure and color palette. The baseline follows the sketch closely but produces a flat appearance, while SDXS preserves the main structure with limited detail. Our method progressively integrates specific details (e.g., the pantograph at 70% completion) and preserves them across subsequent steps.

Figure 18: Progressive sketch-to-image generation. Prompt: A bird is sitting on the branch of a tree eating a fruit. The baseline shows visible artifacts around imperfect strokes, while SDXS introduces extraneous elements. StableFlow exhibits stronger visual amplification, whereas our method balances the evolving sketch structure and prompt semantics while retaining the intended visual content.

Figure 19: Progressive sketch-to-image generation. Prompt: a train on the tracks. Our method maintains a coherent progression across sketch completion levels while adapting the top-down view to later strokes. SDXS and the baseline show more visible artifacts, while StableFlow preserves the initial structure strongly and retains the saturated colors established at the 10% stage.

Figure 20: Progressive sketch-to-image generation. Prompt: A dog sitting and eagerly waiting to catch the frisbee . StableFlow shows stronger background saturation and visible artifacts at 80% and 100% progression. SDXS produces a high-quality image with some flawed details. Compared to the baseline, our method shows stronger sketch adherence while preserving the dog’s facial structure.

Figure 21: Progressive sketch-to-image generation. Prompt: Jetplanes flying in the sky . StableFlow shows strong persistence of the initial structure together with color distortion and weaker adherence to later strokes. SDXS and the baseline contain visible artifacts under the evolving sketch condition. Our method adapts to the evolving sketch while preserving the emerging contrails and reducing visible stroke artifacts.

Figure 22: Progressive sketch-to-image generation. Prompt: sheep standing in a field. StableFlow remains focused on a single, greenish sheep, while the baseline follows the rough sketch structure but produces an unrealistic animal. SDXS produces a high-quality image with weaker preservation of the underlying sketch structure. Our method maintains sketch fidelity, introducing the second sheep at 70% completion and refining the scene across later stages.

Figure 23: Progressive sketch-to-image generation. Prompt: player playing a tennis game . StableFlow shows strong persistence of structure and color together with visible artifacts, while SDXS follows the sketch more loosely and contains image-level artifacts. The baseline also shows substantial artifacts. Our method deviates slightly from the exact pose in the ground truth but retains the overall sketch structure with fewer visible artifacts.

Figure 24: Progressive sketch-to-image generation. Prompt: Few birds are flying over the mountains . Our method forms birds from rough strokes while suppressing many of the initial stroke artifacts. The baseline reflects the sketch structure but yields a less photorealistic interpretation. StableFlow shows stronger color saturation and weaker agreement with the final sketch, while SDXS contains substantial image-level artifacts.

Figure 25: Progressive sketch-to-image generation. Prompt: Industrial belt in the city . This represents a challenging scenario where the prompt’s open-ended nature makes interpretation difficult. Our method interprets the circular symbol and box-like structures consistently across the progression. The competing outputs show different limitations, including stronger saturation for StableFlow and visible artifacts for the baseline and SDXS.

Figure 26: Progressive sketch-to-image generation. Prompt: aeroplanes are flying in the sky . StableFlow shows stronger color saturation, while the baseline contains visible imperfections around sketch-aligned regions. SDXS changes noticeably across turns and retains tree-like artifacts. Our method translates the evolving strokes into jets while gradually reducing the early tree-like structures.

Figure 27: Progressive sketch-to-image generation. Prompt: a train moving on tracks. The baseline contains cartoon-like artifacts, while StableFlow strongly preserves the initial structure and shows limited later adaptation together with color distortion. SDXS shows weak sketch adherence. Our method provides smoother progression while balancing the evolving sketch with scene realism.

Figure 28: Progressive sketch-to-image generation. Prompt: signal and direction pole on road . In this scene-level generation, StableFlow strongly preserves the initial structure and shows color distortion as details are added. The baseline reflects several sketch imperfections in the final image, while SDXS contains limited fine detail. Our method incorporates progressive details while maintaining a coherent road scene with fewer visible artifacts.

Figure 29: Progressive sketch-to-image generation. Prompt: People are crossing the busy streets. In this complex, multi-element scene, the baseline becomes more cartoon-like and contains visible artifacts under rough strokes. SDXS shows distorted figures, while StableFlow has stronger color saturation and adapts less to later strokes. Our method maintains a coherent traffic scene with central figures that remain aligned with the evolving strokes.

Figure 30: Progressive sketch-to-image generation. Prompt: Two planes flying in sky where a coconut tree is also in the frame. StableFlow exhibits color distortion while maintaining the broad sketch structure, the baseline shows artifacts in the tree leaves, and SDXS shows weaker sketch adherence. Our method progresses from a single airplane to multiple planes as the corresponding strokes are introduced.

Figure 31: Progressive sketch-to-image generation. Prompt: a park with clock tower and a streetlamp nearby This example features a vague description of a park; as the user draws trees (despite the prompt focusing on a park and clock), competing methods drift and generate visible artifacts under the evolving sketch condition. Our method demonstrates flexibility by interpreting rough tree strokes as lamp-like structures appropriate for a park scene, while smoothly incorporating user details added at the 70% stage.

Figure 32: Progressive sketch-to-image generation. Prompt: A mini truck on road . This is a challenging task because the upper section of the truck is only partially specified. Our method interprets the incomplete strokes as a plausible vehicle structure and updates it as more evidence arrives. The competing outputs show different behaviors: the baseline contains substantial artifacts, StableFlow strongly preserves the initial structure, and SDXS preserves the overall intent with less fine detail. Our approach updates the structure as new strokes are provided.

Figure 33: Progressive sketch-to-image generation. Prompt: a train on a track. StableFlow strongly preserves the initial structure and exhibits substantial color amplification. SDXS retains the broad structure but shows limited response to small corrections, while the baseline contains heavy artifacts. Our method forms a coherent train structure and extends the sky when additional cloud strokes are introduced.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 469 papers

Jul 26, 2026

SketchMamba: A Lightweight State-Space Model for Joint Progressive Sketch Classification and Stroke Auto-Completion

Existing vector-sketch models treat recognition and generation as separate tasks, leaving a gap for streaming interfaces that must understand a drawing as it is being made. We present SketchMamba, a single causal sequence model that continuously classifies a sketch from any partial prefix while simultaneously generating its continuation. We achieve this by applying a dense per-step classification loss to a selective state-space backbone. Evaluated on a 58-class subset of the Quick, Draw! dataset, SketchMamba yields 94.93% final-step accuracy and a progressive-accuracy Area Under the Curve (AUC) of 0.706, crossing 90% of its final accuracy by the time 70% of the strokes are drawn. In a matched-budget comparison, the 1.55 million-parameter backbone ties a causal Transformer while outperforming recurrent and convolutional baselines. Ablations confirm that the dense supervision regime, rather than the architecture alone, drives the early-prediction capability. The results demonstrate that a single causal hidden state can unify progressive recognition and autoregressive generation without auxiliary encoders or task-specific branching.

Kavish Jhaveri, Arya Shah

The National High School, Ahmedabad, India · Indian Institute of Technology, Gandhinagar, India

Autoregressive Generation Image Classification

Ask PDF

Aug 14, 2026

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance

Nikolai Röhrich, Isabell Hans, Felix Krause +1

LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany

Diffusion Model Guidance T2I Generation Controllable Image Generation Diffusion Guidance Diffusion Models

Ask PDF

Oct 13, 2026

APT: Feedback-Driven Agentic Post-Training for Text-to-Image Generation

Post-training is crucial for improving text-to-image models, yet existing pipelines typically rely on manually curated training data and fixed training distributions. We introduce APT (Agentic Post-Training), a fully automated framework that adaptively post-trains a base model according to its evolving capability profile. APT coordinates three agents to construct evaluation sets from user requirements, diagnose model-specific strengths and failure modes, and generate reinforcement learning data with an adaptive prompt distribution that emphasizes underperforming capabilities. By repeatedly evaluating, diagnosing, and updating the generator, APT lets the current model's behavior shape its next training tasks, forming a feedback loop inspired by recursive self-improvement. For general-purpose settings without user-specified evaluation scenarios, we further develop an evaluation harness that enables broad and diverse evaluation set generation. Experiments across multiple base models, benchmarks, and generation architectures demonstrate that APT is model-agnostic, benchmark-agnostic, architecture-agnostic, and fully automated.

Sirui Zhang, Fangyun Wei, Dong Chen +5

University of Science and Technology of China · Microsoft Research Asia · University of Sydney +2

T2I Generation RL for Image Generation

Ask PDF

Abstract

Image Generation

Explore similar work

SketchMamba: A Lightweight State-Space Model for Joint Progressive Sketch Classification and Stroke Auto-Completion

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

APT: Feedback-Driven Agentic Post-Training for Text-to-Image Generation