Image steering aims to provide fine-grained control over concepts in generated images, enabling targeted and continuous manipulation without broadly altering the generation process. Recent steering methods for diffusion models primarily focus on the text-conditioning space or rely on costly auxiliary training. Deviating from that, we here propose a novel steering approach that extracts visual concept directions directly from the image-side activations of Diffusion Transformers. Specifically, we showcase few-shot concept extraction in the activation space using a simple difference-of-means estimator over intermediate residuals. We observe (a) scene-level concepts such as lighting, weather, and style are distributed across image tokens, as well as (b) local concepts such as object colors, textures, materials, and facial attributes are concentrated in specific regions. Towards controlling both concepts, we introduce a new unified training-free framework that uses global difference-of-means for (a) distributed concepts and masked difference-of-means for (b) local concepts, preventing local signals from being diluted by global averaging. Notably, using only a handful of positive and negative prompt pairs, our method identifies and extracts reusable concept directions that can be injected at inference for continuous steering. Quantitatively, our method outperforms text-encoder steering and training-free activation-editing baselines, as well as is competitive with specialized editing models. Our results suggest that as concept directions can be applied at patch level, our method enables precise targeted and compositional steering, manipulating multiple concepts in the same image.
Figures & tables
Figure 1: Overview of our few-shot concept extraction and steering pipeline. (1) We start from a small set of negative and positive prompt pairs, optionally with masks for localized concepts extraction. (2) We extract residual activations from a selected Diffusion Transformer block and estimate a concept direction using difference-of-means between positive and negative residuals. For global concepts, this is computed over all image tokens and for local concepts, it is restricted to the masked region patches. (3) At inference time, the resulting concept vector is injected into transformer blocks to steer generation, either globally or within a selected region.
Figure 2: Qualitative examples of concept steering. We show representative examples of our method across different concept families. Each row corresponds to a held-out prompt and visualizes the effect of increasing the steering strength α . The examples illustrate that extracted concept directions can produce smooth steering while preserving the main content of the image.
Figure 3: Compositional and targeted steering. Examples of different concept directions being applied to different regions in the same image. By selecting the spatial area to steer, our method enables targeted steering such as changing two different objects with two different colors or textures, as well as modifying different attributes of two persons. Notably, global concepts can be combined, as shown with snowy + anime in a single inference steering pass.
Editing models
Training-free
Ours
(reference-conditioned)
(concept steering)
Dev
Schnell
Metric
Ay SliderEdit
Ay K. Kontext
Ay SAEdit
Ay FluxSpace
5 pairs
5 pairs
10 pairs
20 pairs
Δ VQA ↑
0.35
0.36
0.19
0.17
0.29
0.32
0.32
0.31
LPIPS ↓
0.44
0.32
0.28
0.21
0.34
0.29
0.29
0.29
Table 1: Quantitative comparison across methods. Higher Δ VQA indicates stronger agreement with the requested edit, while lower LPIPS indicates better preservation of the original image. Methods are grouped by class. Ay Editing models are trained image-editing models conditioned on the reference image and serve as an upper reference. Ay Training-free baselines use the FLUX.1-dev backbone and are directly comparable to our Dev variant. We report our method on FLUX.1-dev with 5 positive/negative prompt pairs, and on FLUX.1-schnell with 5, 10, and 20 pairs.
Figure 4: Edit success versus content preservation. We compare methods using Δ VQA to measure whether the requested edit is achieved and LPIPS to measure visual drift from the original image. Better methods appear toward the upper-left, indicating stronger edit success with less content drift. We note that Δ VQA slightly decreases at steering level 5, as some concepts appear at lower strengths and stronger interventions can lead to oversteering.
Figure 6Table 7
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Qualitative ablation samples – face concept: beard . Top row of each panel: extraction-timestep ablation. Bottom row: extraction-layer ablation.
Figure 8: Qualitative ablation samples – object concept: blue . Top row of each panel: extraction-timestep ablation. Bottom row: extraction-layer ablation.
Figure 9: Qualitative ablation samples – global concept: snowy . Top row of each panel: extraction-timestep ablation. Bottom row: extraction-layer ablation.
Category
Δ VQA ↑
LPIPS ↓
Face
0.213
0.126
Object
0.303
0.189
Global
0.518
0.510
Average
0.345
0.275
Appendix
Table 4: Prompt-pair sensitivity ablation. We evaluate our method using substantially simplified positive/negative prompt pairs while keeping all extraction and inference settings fixed.
Figure 10: Representative failure cases. We show examples of semantic mismatch, spatial drift, and concept entanglement. In each row, increasing the steering strength introduces the target concept to different degrees, but may also produce weak edits, unwanted geometry changes, or correlated visual attributes.
Concept
Family
Extraction pairs
Extraction time
Generation time
Total time
blue
Object
5
14.26s
11.73s
26.00s
cartoon
Global
5
10.52s
10.73s
21.25s
smile
Face
5
12.50s
11.75s
24.25s
Appendix
Table 5: Runtime for concept extraction and generation. For each concept, we report the total time to extract the concept direction from five positive-negative prompt pairs, the total time to generate five original images and five steered images, and the combined runtime. Generation time includes the steering-mask computation when applicable.
Method
Face Outside mask LPIPS ↓
Object Outside mask LPIPS ↓
Ours
0.134
0.144
FluxSpace
0.055
0.124
SAEdit
0.153
0.250
SliderEdit
0.398
0.288
Kontinuous Kontext
0.290
0.202
Appendix
Table 6: Outside mask LPIPS evaluation. Lower values indicate less change outside the target region.
Extraction-mask perturbation
Face Δ VQA ↑
Face LPIPS ↓
Object Δ VQA ↑
Object LPIPS ↓
Erosion – small
0.197
0.267
0.294
0.179
Erosion – medium
0.134
0.228
0.280
0.169
Dilation – small
0.188
0.266
0.301
0.185
Dilation – medium
0.191
0.256
0.305
0.190
Shift – small
0.194
0.241
0.290
0.180
Shift – medium
0.189
0.233
0.288
0.181
Appendix
Table 7: Robustness to extraction-mask perturbations. We simulate segmentation errors through erosion, dilation, spatial shifts, and false-positive regions. Higher Δ VQA indicates stronger editing, while lower LPIPS indicates better preservation.
Table 16
Figure 11: Real-image steering with FLUX.2. Concept directions extracted exclusively from synthetic prompt pairs transfer to real reference images across multiple concepts. For each example, increasing the steering strength α progressively increases the presence of the target concept while preserving the reference image. In contrast to FLUX.2 direct prompt editing, which produces a single edit for a given instruction, our approach provides continuous and progressive control over edit strength.
Figure 12: Qualitative examples of concept steering. We present more qualitative samples across concepts.