cs.CVNov 14, 2026

Few-Shot Visual Concept Extraction for Steering Diffusion Transformers

Authors: Nabyl Quignon, Antitza Dantcheva

Organizations: Inria Centre at Université Côte d’Azur

Abstract

Image steering aims to provide fine-grained control over concepts in generated images, enabling targeted and continuous manipulation without broadly altering the generation process. Recent steering methods for diffusion models primarily focus on the text-conditioning space or rely on costly auxiliary training. Deviating from that, we here propose a novel steering approach that extracts visual concept directions directly from the image-side activations of Diffusion Transformers. Specifically, we showcase few-shot concept extraction in the activation space using a simple difference-of-means estimator over intermediate residuals. We observe (a) scene-level concepts such as lighting, weather, and style are distributed across image tokens, as well as (b) local concepts such as object colors, textures, materials, and facial attributes are concentrated in specific regions. Towards controlling both concepts, we introduce a new unified training-free framework that uses global difference-of-means for (a) distributed concepts and masked difference-of-means for (b) local concepts, preventing local signals from being diluted by global averaging. Notably, using only a handful of positive and negative prompt pairs, our method identifies and extracts reusable concept directions that can be injected at inference for continuous steering. Quantitatively, our method outperforms text-encoder steering and training-free activation-editing baselines, as well as is competitive with specialized editing models. Our results suggest that as concept directions can be applied at patch level, our method enables precise targeted and compositional steering, manipulating multiple concepts in the same image.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SHIFT: Steering Hidden Intermediates in Flow Transformers

    Apr 10, 2026Nina Konovalova, Ibragim Idrisov, Aibek AlanovControllable Image GenerationConcept Erasure

  2. RASteer: Retain-Aware Activation Steering for Concept Erasure in Diffusion Models

    Oct 1, 2026Yongliang Wu, Haori Lu, Yulun Wu +3Diffusion Model UnlearningActivation Steering

  3. AcFlow: Controlling Text-to-Image Diffusion Transformers via Learned Conditional Activation Flow

    Sep 9, 2026Junran Wang, Zehao Jin, Tianyu Luan +3Diffusion TransformerT2I Generation