SHIFT: Steering Hidden Intermediates in Flow Transformers

Nina Konovalova; Andrey Kuznetsov; Aibek Alanov

Figure 1: Double-stream block.

Figure 2: Steering pipeline overview with three stages: (1) dataset construction from contrastive prompt pairs, (2) steering vector computation based on mean difference and separation plane, and (3) application of the vector during inference, (4) steering inside diffusion transformer.

Figure 3: Flux.1[dev] origin and steered generation. * denotes that we use steering vector from Flux.1[schnell] activations to steer Flux.1[dev]

Figure 4: Qualitative Comparison with Baselines: the first row – original generation of FLUX.1[schnell], from top to bottom – UCE, CA and SHIFT (Ours)

Method	Detected Nudity (Quantity)				MS-COCO 5K
	Common	Female	Male	Total $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
ESD	280	121	25	426	31.88	30.86
EAP	373	175	10	558	32.02	31.51
CA	234	98	10	342	31.81	31.25
UCE	232	127	5	364	31.65	31.55
SHIFT (ours, strength 250)	87	32	3	122	33.9	31.22

Table 1: (Left) Quantity of explicit content detected using the NudeNet detector on the I2P benchmark for Flux.1 [schnell]. (Right) Comparison of FID and CLIP on MS-COCO 5K sampled images.

Method	Detected Nudity (Quantity)
	Common	Female	Male	Total $\downarrow$
CA	253	65	26	344
ESD	329	145	32	506
UCE	122	39	12	173
MACE	173	55	28	256
EAP (Bui et al., 2024)	287	86	13	386

Table 2: Quantity of explicit content detected using the NudeNet detector on the I2P benchmark for Flux.1 [dev]

Figure 5: Qualitative Comparison with Baselines for Flux.1[dev] (left) and Flux.1[schnell] (right). * denotes that we use steering vector from Flux.1[schnell] activations to steer Flux.1[dev]

	Snoopy		Mickey		Spongebob		Pikachu		Dog		Legislator
Method	CS $\downarrow$	FID	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$
CA	22.49	168.37	24.25	129.91	28.25	131.78	27.44	129.54	22.86	96.94	21.18	83.83
SHIFT (ours)	18.57	136.20	26.06	55.56	27.35	63.27	26.25	74.24	24.18	56.43	21.77	47.08
FLUX.1 [schnell]	28.01	–	26.72	–	27.94	–	27.15	–	24.62	–	21.89	–

Table 3: Quantitative evaluation of concrete object erasure for Flux.1[schnell] model.

Figure 6: Steering to remove small concrete objects: glasses, hat, red lipstick; first row: origin image, second: steered

Figure 7: Steering to remove concrete concept, ablations for 3 different close concepts: Mickey, Snoopy and Spongebob. The * in the last row indicates step-specific steering vectors. DT – indicates steering strength, CLIP indicates pooled text encoder strength.

Method	Strength	Detected Nudity (Quantity)				MS-COCO 1K
		Common	Female	Male	Total $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
SHIFT (no cls)	250	149	65	16	230	71.09	31.45
SHIFT (with cls)	250	87	32	3	122	68.72	31.54
SHIFT (with cls)	500	73	23	1	97	69.60	31.37
FLUX.1 [schnell]	–	406	161	38	605	66.81	31.92

Table 4: Ablation on steering strength and classifier for regularization usage. (Left) Quantity of explicit content detected using the NudeNet detector on the I2P benchmark. (Right) Comparison of FID and CLIP on MS-COCO 1K sampled images.

Blocks	Steps	Legislator	Mickey	Pikachu	Spongebob	Dog	Snoopy
		CLIP $\uparrow$ FID $\downarrow$	CLIP $\uparrow$ FID $\downarrow$	CLIP $\uparrow$ FID $\downarrow$	CLIP $\uparrow$ FID $\downarrow$	CLIP $\uparrow$ FID $\downarrow$	CLIP $\downarrow$ FID $\uparrow$
0	0	22.6 –	27.4 –	26.5 –	28.4 –	25.1 –	28.3 –
0–4	all	22.3 58.1	27.0 105.0	26.8 73.2	28.6 119.5	25.0 53.8	16.5 272.0
0–9	all	22.7 60.9	26.8 110.7	26.9 80.7	28.4 124.7	25.0 62.8	16.4 275.6
10–19	all	22.9 56.1	26.8 95.7	26.1 86.6	27.8 123.8	24.9 62.2	26.6 136.7
all	0	22.9 68.7	26.6 112.1	26.3 94.0	27.9 143.3	24.9 76.8	16.0 278.6

Table 5: Ablation of steering across block ranges and timestep schedules. Each cell reports CLIP

\uparrow

and FID

\downarrow

. The * indicates step-specific steering vectors.

Str	Txt Str	Type	Legislator		Mickey		Pikachu		Spongebob		Dog		Snoopy
			CLIP $\uparrow$	FID $\downarrow$	CLIP $\uparrow$	FID $\downarrow$	CLIP $\uparrow$	FID $\downarrow$	CLIP $\uparrow$	FID $\downarrow$	CLIP $\uparrow$	FID $\downarrow$	CLIP $\downarrow$	FID $\uparrow$
0	0	–	22.59	–	27.36	–	26.49	–	28.39	–	25.13	–	28.29	–
500	6	diff sep	22.9	68.7	26.6	112.0	26.3	93.9	27.8	143.3	24.8	76.8	15.9	278.5
500	6	diff mean	22.6	59.9	26.9	99.7	26.4	90.6	28.4	126.9	25.1	70.9	22.8	191.4
500	6	normals	22.5	60.4	27.0	97.3	26.1	92.1	28.4	118.4	25.1	73.6	23.9	187.4

Table 6: Ablation across steering-vector parameterization types. For each concept, we report CLIP and FID .

Str	Txt	Cls	Norm	Cos	Legislator	Mickey	Pikachu	Spongebob	dog	Snoopy
	Str				CLIP $\uparrow$	CLIP $\uparrow$	CLIP $\uparrow$	CLIP $\uparrow$	CLIP $\uparrow$	CLIP $\downarrow$
0	0	–	–	–	22.59	27.36	26.49	28.39	25.13	28.29
0	6	–	–	–	22.56	27.12	26.47	28.24	25.22	26.35
100	6	✓	✓	✓	22.81	27.14	26.37	28.45	24.88	20.05
500	6	–	✓	✓	20.11	24.95	25.88	27.80	24.24	13.26
500	6	✓	✓	✓	22.90	26.63	26.33	27.89	24.87	15.99

Table 7: CLIP ablation of steering strength and regularization settings on concrete concept erasure.

Method	Strength	Detected Nudity (Quantity)				MS-COCO 1K
		Common	Female	Male	Total $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
no cls
SHIFT (schnell acts)	100	149	65	16	230	71.63	30.78
SHIFT (schnell acts)	250	163	72	16	251	74.74	30.50
SHIFT (schnell acts)	500	134	43	18	195	77.71	29.53
SHIFT (dev acts)	100	179	70	19	268	72.59	30.69

Table 8: Quantitative comparison of Flux.1[dev] steering with different strength and with/without classifier for regularization. We additionally tested steering vector from distilled model applied to origin.

Str	Txt Str	Cls	Norm	Cos	Legislator	Mickey	Pikachu	Spongebob	dog	Snoopy
					FID $\downarrow$	FID $\downarrow$	FID $\downarrow$	FID $\downarrow$	FID $\downarrow$	FID $\uparrow$
0	0	–	–	–	–	–	–	–	–	–
0	6	✓	✓	✓	13.76	55.93	26.38	39.46	15.35	144.89
100	6	✓	✓	✓	47.08	91.09	73.63	106.95	44.21	227.29
500	6	–	✓	✓	88.30	153.99	103.37	168.46	124.03	290.35
500	6	✓	✓	✓	68.68	112.05	93.99	143.30	76.82	278.59

Table 9: FID ablation for steering strength and regularization settings for Flux.1[schnell].

	Snoopy		Mickey		Spongebob		Pikachu		Dog		Legislator
Method	CS $\downarrow$	FID	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$
ESD	26.77	15.44	26.18	6.56	27.09	9.51	26.81	7.87	24.16	8.76	20.34	8.52
SHIFT, our (dev acts)	11.98	181	22.70	88.70	26.13	67.19	25.14	51.12	22.98	56.00	17.90	72.95
FLUX.1 [dev]	27.43	–	26.20	–	27.14	–	18.81	–	24.19	–	20.51	–

Table 10: Quantitative evaluation of concrete object erasure for Flux.1[schnell] model.

Figure 8: Examples of image generations on the COCO dataset using Flux.1[dev], Flux.1[schnell], and steered Flux.1[dev]*, Flux.1[dev] and Flux.1[schnell] generations, where * denotes steering Flux.1[dev] with the activation vector derived from Flux.1[schnell].

Method	Strength	Detected Nudity (Quantity)				MS-COCO 5K
		Common	Female	Male	Total $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
with cls
SHIFT (schnell acts)	250	106	31	15	152	37.90	30.69
SHIFT (dev acts)	250	155	43	17	215	38.50	30.69
SHIFT (schnell acts)	500	98	28	12	138	38.06	30.40
FLUX.1 [dev]	–	406	161	38	605	35.83	30.91

Table 11: Quantitative evaluation of Flux.1[dev] steering for nudity concept erasure and overall image quality metrics (FID and CLIP) on the COCO dataset using 5k prompts.

Figure 9: Qualitative results for Van Gogh style erasure. First row: original Flux.1[dev] and Flux.1[schnell] generations. Second row: UCE and ESD competitors. Last rows: our steered generations, where * denotes steering Flux.1[dev] with the activation vector derived from Flux.1[schnell].

Figure 10: Qualitative results demonstrating preservation of other artists’ styles (Picasso, Rembrandt, Warhol, Caravaggio) while erasing Van Gogh’s style using our steered model (SHIFT).

Method	Old artists		Modern artists
	$\mathbf{Acc_{E}}$	$\mathbf{Acc_{U}}$	$\mathbf{Acc_{E}}$	$\mathbf{Acc_{U}}$
ESD	20.00%	63.75%	68.33%	67.08%
SHIFT [dev]	30%	63.75%	15%	68.75%
SHIFT [schnell]	20.00%	67.50 %	13.33	70.41
FLUX.1 [dev]	68.33%	79.58%	88.33%	81.25%
UCE	23.33%	75.41%	61.67	64.16

Table 12: Style-erasure accuracy on old and modern artist groups.

\mathrm{Acc}_{E}

denotes accuracy on erased-target prompts, and

\mathrm{Acc}_{U}

denotes accuracy on unaffected prompts.

Figure 11: Dataset examples for constructing steering vectors for the "add concept" task. Provided images are for target concepts: "red lipstick", "hat", and "smile".

Figure 12: Ablation of number of steered DiT blocks for adding "hat" using Flux.1[schnell]. Rows (top to bottom): origin, 7, 9, 12 all blocks. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 13: Ablation of number of steered DiT blocks for adding "smile" using Flux.1[schnell]. Rows (top to bottom): origin, 7, 9, 12 all blocks. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 14: Ablation of number of steered DiT blocks for adding "red lipstick" using Flux.1[schnell]. Rows (top to bottom): origin, 7, 9, 12 all blocks. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 15: Ablation of number of steered DiT blocks for adding "apple" using Flux.1[schnell]. Rows (top to bottom): origin, 7, 9, 12 all blocks. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 16: Ablation of number of timesteps for adding "smile" (left), "apple" (right) using Flux.1[schnell]. Rows (top to bottom): origin, vector from 0 step applied to all steps; all steps; 0 step; 0-1 steps and 2-3 steps steering. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 17: Ablation of number of timesteps for adding "red lipstick" (left), "hat" (right) using Flux.1[schnell]. Rows (top to bottom): origin, vector from 0 step applied to all steps; all steps; 0 step; 0-1 steps and 2-3 steps steering. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 18: Dataset examples for constructing steering vectors for the "add concept" task. Provided images are for target concepts: "banana

\rightarrow

apple, young

\rightarrow

old".

Figure 19: Ablation of number of steered DiT blocks for switching concepts "young

\rightarrow

old" using Flux.1[schnell]. Rows (top to bottom): origin, all blocks, 7, 9, 12. DiT strength:

1000

; text encoder strength:

3

.

Figure 20: Ablation of number of steered DiT blocks for switching concepts "banana

\rightarrow

apple" using Flux.1[schnell]. Rows (top to bottom): origin, all blocks, 7, 9, 12. DiT strength:

1000

; text encoder strength:

3

.

Figure 21: Ablation of number of steered DiT blocks for switching concepts "dog

\rightarrow

cat" using Flux.1[schnell]. Rows (top to bottom): origin, all blocks, 7, 9, 12. DiT strength:

1000

; text encoder strength:

3

.

Figure 22: Ablation of number of steered DiT blocks for switching concepts "bicycle

\rightarrow

car" using Flux.1[schnell]. Rows (top to bottom): origin, all blocks, 7, 9, 12. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 23: Ablation of number of timesteps for switching "banana

\rightarrow

apple" (left), "dog

\rightarrow

cat" (right) using Flux.1[schnell]. Rows (top to bottom): origin, vector from 0 step applied to all steps; all steps; 0 step; 0-1 steps and 2-3 steps steering. DiT strength:

1000

; text encoder strength:

3

.

Figure 24: Ablation of number of timesteps for switching "young

\rightarrow

old" (left), "bicycle

\rightarrow

car" (right) using Flux.1[schnell]. Rows (top to bottom): origin, vector from 0 step applied to all steps; all steps; 0 step; 0-1 steps and 2-3 steps steering. DiT strength:

1000

; text encoder strength:

3/1.5

.

Figure 25: Dataset examples for constructing steering vectors for the "add concept" task. Provided images are for target concepts: "cyberpunk", "sketch" and "impressionism" style.

Figure 26: Ablation of number of steered DiT blocks for adding style "cyberpunk" using Flux.1[schnell]. Rows (top to bottom): origin, 7, 9, 12 all blocks. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 27: Ablation of number of steered DiT blocks for adding style "impressionism" using Flux.1[schnell]. Rows (top to bottom): origin, 7, 9, 12 all blocks. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 28: Ablation of number of steered DiT blocks for adding style "sketch" using Flux.1[schnell]. Rows (top to bottom): origin, 7, 9, 12 all blocks. DiT strength:

1000

; text encoder strength:

1.5

.

Figure 29: Ablation of steering-vector transfer from Flux.1[schnell] to Flux.1[dev] for the "switch concept" task. Rows show (top to bottom): base generation (neutral prompt), steered result with "dog

\rightarrow

cat", steered result with "banana

\rightarrow

apple", steered result with "young

\rightarrow

old". DiT steering strength:

1000

; text encoder strength:

3

.

Figure 30: Ablation of steering-vector transfer from Flux.1[schnell] to Flux.1[dev] for the "add concept" task. Rows show (top to bottom): steered result with "hat", steered result with "smile", steered result with "glasses". DiT steering strength:

1000

; text encoder strength:

1.5

.

Figure 31: Steering results for unconditional (empty-prompt) generations. Top row: add-concept task; middle row: switch-concept task; bottom row: stylization task.

Figure 32: Classifier scores for different blocks and steps. Left: steering tasks (stylization

\rightarrow

switch

\rightarrow

add concept); right: classifiers for the switch task (linear SVM

\rightarrow

logistic regression

\rightarrow

RBF SVM).

SHIFT: Steering Hidden Intermediates in Flow Transformers

Abstract

Figures & tables

Controllable Image Generation

AcFlow: Controlling Text-to-Image Diffusion Transformers via Learned Conditional Activation Flow

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds

Abstract

Controllable Image Generation

Explore similar work

AcFlow: Controlling Text-to-Image Diffusion Transformers via Learned Conditional Activation Flow

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds