Image steering aims to provide fine-grained control over concepts in generated images, enabling targeted and continuous manipulation without broadly altering the generation process. Recent steering methods for diffusion models primarily focus on the text-conditioning space or rely on costly auxiliary training. Deviating from that, we here propose a novel steering approach that extracts visual concept directions directly from the image-side activations of Diffusion Transformers. Specifically, we showcase few-shot concept extraction in the activation space using a simple difference-of-means estimator over intermediate residuals. We observe (a) scene-level concepts such as lighting, weather, and style are distributed across image tokens, as well as (b) local concepts such as object colors, textures, materials, and facial attributes are concentrated in specific regions. Towards controlling both concepts, we introduce a new unified training-free framework that uses global difference-of-means for (a) distributed concepts and masked difference-of-means for (b) local concepts, preventing local signals from being diluted by global averaging. Notably, using only a handful of positive and negative prompt pairs, our method identifies and extracts reusable concept directions that can be injected at inference for continuous steering. Quantitatively, our method outperforms text-encoder steering and training-free activation-editing baselines, as well as is competitive with specialized editing models. Our results suggest that as concept directions can be applied at patch level, our method enables precise targeted and compositional steering, manipulating multiple concepts in the same image.
Figures & tables
Figure 1: Overview of our few-shot concept extraction and steering pipeline. (1) We start from a small set of negative and positive prompt pairs, optionally with masks for localized concepts extraction. (2) We extract residual activations from a selected Diffusion Transformer block and estimate a concept direction using difference-of-means between positive and negative residuals. For global concepts, this is computed over all image tokens and for local concepts, it is restricted to the masked region patches. (3) At inference time, the resulting concept vector is injected into transformer blocks to steer generation, either globally or within a selected region.
Figure 2: Qualitative examples of concept steering. We show representative examples of our method across different concept families. Each row corresponds to a held-out prompt and visualizes the effect of increasing the steering strength α . The examples illustrate that extracted concept directions can produce smooth steering while preserving the main content of the image.
Figure 3: Compositional and targeted steering. Examples of different concept directions being applied to different regions in the same image. By selecting the spatial area to steer, our method enables targeted steering such as changing two different objects with two different colors or textures, as well as modifying different attributes of two persons. Notably, global concepts can be combined, as shown with snowy + anime in a single inference steering pass.
Editing models
Training-free
Ours
(reference-conditioned)
(concept steering)
Dev
Schnell
Metric
Ay SliderEdit
Ay K. Kontext
Ay SAEdit
Ay FluxSpace
5 pairs
5 pairs
10 pairs
20 pairs
Δ VQA ↑
0.35
0.36
0.19
0.17
0.29
0.32
0.32
0.31
LPIPS ↓
0.44
0.32
0.28
0.21
0.34
0.29
0.29
0.29
Table 1: Quantitative comparison across methods. Higher Δ VQA indicates stronger agreement with the requested edit, while lower LPIPS indicates better preservation of the original image. Methods are grouped by class. Ay Editing models are trained image-editing models conditioned on the reference image and serve as an upper reference. Ay Training-free baselines use the FLUX.1-dev backbone and are directly comparable to our Dev variant. We report our method on FLUX.1-dev with 5 positive/negative prompt pairs, and on FLUX.1-schnell with 5, 10, and 20 pairs.
Figure 4: Edit success versus content preservation. We compare methods using Δ VQA to measure whether the requested edit is achieved and LPIPS to measure visual drift from the original image. Better methods appear toward the upper-left, indicating stronger edit success with less content drift. We note that Δ VQA slightly decreases at steering level 5, as some concepts appear at lower strengths and stronger interventions can lead to oversteering.
Figure 6Table 7
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Qualitative ablation samples – face concept: beard . Top row of each panel: extraction-timestep ablation. Bottom row: extraction-layer ablation.
Figure 8: Qualitative ablation samples – object concept: blue . Top row of each panel: extraction-timestep ablation. Bottom row: extraction-layer ablation.
Figure 9: Qualitative ablation samples – global concept: snowy . Top row of each panel: extraction-timestep ablation. Bottom row: extraction-layer ablation.
Category
Δ VQA ↑
LPIPS ↓
Face
0.213
0.126
Object
0.303
0.189
Global
0.518
0.510
Average
0.345
0.275
Appendix
Table 4: Prompt-pair sensitivity ablation. We evaluate our method using substantially simplified positive/negative prompt pairs while keeping all extraction and inference settings fixed.
Figure 10: Representative failure cases. We show examples of semantic mismatch, spatial drift, and concept entanglement. In each row, increasing the steering strength introduces the target concept to different degrees, but may also produce weak edits, unwanted geometry changes, or correlated visual attributes.
Concept
Family
Extraction pairs
Extraction time
Generation time
Total time
blue
Object
5
14.26s
11.73s
26.00s
cartoon
Global
5
10.52s
10.73s
21.25s
smile
Face
5
12.50s
11.75s
24.25s
Appendix
Table 5: Runtime for concept extraction and generation. For each concept, we report the total time to extract the concept direction from five positive-negative prompt pairs, the total time to generate five original images and five steered images, and the combined runtime. Generation time includes the steering-mask computation when applicable.
Method
Face Outside mask LPIPS ↓
Object Outside mask LPIPS ↓
Ours
0.134
0.144
FluxSpace
0.055
0.124
SAEdit
0.153
0.250
SliderEdit
0.398
0.288
Kontinuous Kontext
0.290
0.202
Appendix
Table 6: Outside mask LPIPS evaluation. Lower values indicate less change outside the target region.
Extraction-mask perturbation
Face Δ VQA ↑
Face LPIPS ↓
Object Δ VQA ↑
Object LPIPS ↓
Erosion – small
0.197
0.267
0.294
0.179
Erosion – medium
0.134
0.228
0.280
0.169
Dilation – small
0.188
0.266
0.301
0.185
Dilation – medium
0.191
0.256
0.305
0.190
Shift – small
0.194
0.241
0.290
0.180
Shift – medium
0.189
0.233
0.288
0.181
Appendix
Table 7: Robustness to extraction-mask perturbations. We simulate segmentation errors through erosion, dilation, spatial shifts, and false-positive regions. Higher Δ VQA indicates stronger editing, while lower LPIPS indicates better preservation.
Table 16
Figure 11: Real-image steering with FLUX.2. Concept directions extracted exclusively from synthetic prompt pairs transfer to real reference images across multiple concepts. For each example, increasing the steering strength α progressively increases the presence of the target concept while preserving the reference image. In contrast to FLUX.2 direct prompt editing, which produces a single edit for a given instruction, our approach provides continuous and progressive control over edit strength.
Figure 12: Qualitative examples of concept steering. We present more qualitative samples across concepts.
Diffusion models have become the leading approach for high-fidelity image generation, with recent DiT-based models achieving strong prompt adherence and high visual quality. However, removing unwanted concepts from these models typically requires expensive gradient-based fine-tuning, which is particularly prohibitive for multi-billion-parameter Diffusion Transformers and incompatible with distilled variants. We propose SHIFT, a simple model-weight-update-free inference-time framework for concept suppression in DiT models via targeted manipulation of intermediate activations at inference time, inspired by activation steering in LLMs. SHIFT learns steering vectors from contrastive prompts and applies them dynamically. We demonstrate that a single time-independent steering vector generalizes across all denoising timesteps and that vectors computed on a distilled model transfer effectively to its non-distilled counterpart without recomputation. SHIFT achieves strong concept suppression while preserving prompt fidelity and image quality. The same mechanism also enables flexible control -- adding, removing, or changing objects and styles without retraining. SHIFT targets hosted inference settings: it does not remove the concept from the model weights and is therefore complementary to weight-level unlearning.
Concept erasure aims to remove a target concept, such as a copyrighted style, a recognizable character, or unsafe content, from a pretrained text-to-image diffusion model while preserving its ability to generate other content. Existing activation steering methods build an erasure direction mainly from the target concept and adjust model activations along it at inference time. However, target and retained concepts often overlap in the model's representation space, so this direction also contains shared components that retained concepts rely on. Steering directly along this direction can therefore suppress retained concepts and harm the generation of non-target content. To address this issue, we propose Retain-aware Activation Steering (RASteer), a training-free method. RASteer first builds a retain subspace from the concepts to preserve. Retain-Orthogonal Steering (ROS) then removes components aligned with this subspace from the erasure direction, making steering more specific to the target. Since fully removing the shared components can weaken erasure, we further introduce Overlap-Adaptive Calibration (OAC). At each layer and denoising step, OAC uses the overlap between the erasure direction and the retain subspace to control how much of each shared component is removed, balancing target erasure and concept preservation. Experiments on unsafe-content, instance, and artistic-style erasure across multiple backbones and benchmarks show that RASteer matches or outperforms the activation steering and weight editing baselines we evaluate, achieving a better balance between erasure and preservation.
Yongliang Wu, Haori Lu, Yulun Wu +3
University of Illinois Urbana-Champaign · University of Pennsylvania · National University of Singapore
Text-to-image diffusion transformers (DiTs) are powerful generators, yet direct prompting provides limited control interface for style intensity and can fail to suppress unwanted concepts. To enable these controls, we introduce AcFlow, an inference-time controller that transports intermediate layer image-token activations through a learned concept-conditioned velocity field while keeping the base DiT frozen. A textual concept description specifies the desired intervention, while the integration horizon provides a continuous control parameter. The field produces token-varying, activation-dependent updates. With parameters shared across concepts within each task family, one field covers over 15,000 style descriptions or over 1,000 suppression concepts, and generalizes to concepts unseen during training without per-concept fitting. On style control, AcFlow achieves the best style--content trade-off among the evaluated baselines in the high-style-alignment regime. At a fixed operating point, AcFlow attains style--content alignment of 0.5365/0.2860, compared with 0.4397/0.2684 for the baseline with the highest style alignment. On concept suppression, AcFlow reduces the fraction of images showing the concept from 95.3%/82.1% to 41.6%/40.5% on held-in/held-out concepts, including cases where deleting them from the prompt fails to remove them. Our analyses support the learned velocity field as an adaptive control mechanism, with update directions varying across tokens and depending on their activation states. Our code is available at https://github.com/Nove1yst/AcFlow.
Junran Wang, Zehao Jin, Tianyu Luan +3
Georgia Institute of technology · South China University of Technology