Cross-modal Steering Vectors Exist Across Both Image and Text Generation

Donghao Zeng; Nishant Subramani; Tamar Rott Shaham; Benno Krojer

cs.CVOct 26, 2026

Cross-modal Steering Vectors Exist Across Both Image and Text Generation

Authors: Donghao Zeng, Nishant Subramani, Tamar Rott Shaham, Benno Krojer

Organizations: Mila, McGill University · Carnegie Mellon University · MIT CSAIL · Northeastern University

Abstract

Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We study cross-modal steering in several unified multimodal models (UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show that steering vectors transfer zero-shot in both directions (text-to-image & image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language), steering vectors often induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature too: despite a clear modality gap separating text and image representations, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept. Our findings shed light on how vision and language interact inside unified multimodal models and point towards more general, modality-agnostic approaches for controlling their behavior.

Figures & tables

Figure 1: Cross-modal steering vectors transfer zero-shot in both directions. A steering vector $\mathbf{v}$ extracted entirely from one modality can steer generation both within that modality and, without any adaptation, in the other.

Figure 2: Concept rate on UniAR ( Peng et al., 2026 ) at middle layer range. Concept rate is the fraction of steered outputs in which an LLM judge finds the concept shifted toward the steered pole ( Section 3.1 ), plotted against normalized steering strength. Black curves show intra-modality steering (top: img

\to

img , bottom: txt

\to

txt ). The colored areas indicate the gap between the two, either when transfer performs better (green) or worse (red). Recovery in Figures 3 and 3 quantifies this comparison. For most concepts, the transfer curve comes close to the intra-modality curve.

Panel of Figure 4 — Figure 4: Cross-modal and intra-modal steering in UniAR (middle layer range). We show examples for emotion, distance and cleanness steering. In both modalities the transferred vector produces a similar shift as the intra-modality vector.

Figure 5: Cross-modal cosine similarity vs. cross-modal steering success. Each point is one (concept, layer range):

x

is cosine between text- and image-derived steering vectors;

y

is our recovery metric ( Section 3.1 ) for the given transfer direction (left: txt

\to

img , right: img

\to

txt ). Spatial is excluded because it does not steer in most settings;

\rho

: Spearman,

r

: Pearson; ∗ marks

p<0.05

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Concept	Example subjects	$-$ pole	$+$ pole
emotion	teacher, chef, coach	a teacher is sad	a teacher is happy
age	engineer, cook, merchant	an engineer is young	an engineer is old
cleanness	oven, fork, glove	an oven is clean	an oven is dirty
chaos	street, hospital, party	a street is peaceful	a street is chaotic
size	flower, pen, statue	a flower is small	a flower is large
depth	flower, pen, statue	a flower is far away	a flower is nearby

Appendix

Table 2: Example text minimal pairs. Each subject gives one pair that differs only in the pole word.

Figure 6: One image pair per concept, with the description used to generate its baseline image.

Concept	Image description	Text subject
emotion	A head-and-shoulders portrait of a person with short cropped hair , wearing a plain crew-neck shirt and holding a relaxed, neutral expression. They stand in front of a weathered red brick wall in soft natural daylight. It is shot as a sharp, realistic photograph with an even, unremarkable mood.	a person with short cropped hair
cleanness	A photograph of a modern kitchen with wooden cabinets , stone countertops, and a stainless steel sink. Even natural daylight from a window falls across the counters, backsplash, and floor. It is a plain, realistic interior photograph with no people.	a modern kitchen with wooden cabinets
size	A single traffic cone in the center of a wide empty asphalt parking lot with faded painted markings, under flat even daylight.	a traffic cone
color	A photograph of a car parked alone on a wide empty asphalt lot, the car centred and seen from the side, under soft even daylight and a pale overcast sky. A plain, realistic photograph with no people.	a parked car

Appendix

Table 3: Example evaluation prompts. Image generation uses the full description, and text generation uses only its subject, shown in bold.

Model	Setting	$\alpha$ grid	Decoding
UniAR	txt $\to$ txt , img $\to$ txt	$\pm 100$ , step $20$	greedy, $96$ tokens
	img $\to$ img txt $\to$ img	$\pm 10$ , step $2$ $\pm 15$ , step $3$	$16\times 16$ tokens, guidance $1.5$ , temperature $1.0$ , SD3 decoder with $16$ steps and guidance $1.5$ , $544\times 544$ image
Emu3.5	txt $\to$ txt , img $\to$ txt	$\pm 25$ , step $5$	greedy, $96$ tokens
	img $\to$ img txt $\to$ img	$\pm 1.5$ , step $0.3$ $\pm 5$ , step $1$	$32\times 32$ tokens, guidance $2.0$ , top- $k$ $10{,}240$ , top- $p$ $1.0$ , temperature $1.0$ , VQ decoder, $512\times 512$ image
Liquid	txt $\to$ txt , img $\to$ txt	$\pm 25$ , step $5$	greedy, $96$ tokens
	img $\to$ img txt $\to$ img	$\pm 0.5$ , step $0.1$ $\pm 1.5$ , step $0.3$	$32\times 32$ tokens, guidance $7.0$ , top- $k$ $4{,}096$ , top- $p$ $0.96$ , temperature $0.99$ , VQ decoder, $512\times 512$ image

Appendix

Table 4: The strength grid is set per model and setting ( Section A.4 ). We keep each model’s default decoding settings other than the number of decoder steps and the image resolution for practical reasons, and fix the seed at

42

		Single layer			Contiguous layer range
Model	Depth	early	middle	late	early	middle	late
UniAR	36	9	18	27	7–10	16–19	25–28
Emu3.5	64	16	32	48	12–19	28–35	44–51
Liquid	28	7	14	21	5–8	12–15	19–22

Appendix

Table 5: Layer configurations for each model (

0

-indexed). Single layers are evenly spaced in depth, each contiguous layer range spans the layers around its matching single layer, and the seventh configuration steers all layers.

Figure 7: DreamSim pole score for image-derived (left) and text-derived (right) vectors.

Figure 8: Concept rate (solid) and output quality (dashed) against normalized strength at the middle layer range, averaged over models, concepts, prompts and both signs of

\alpha

. Quality is text fluency (left) and image coherence (right), relative to the unsteered outputs.

Figure 9: The annotation interface shown to human reviewers. It names the asked pole and its opposite. Two model outputs, the unsteered baseline and the steered output, are shown as Candidate A and B in random order. The human reviewer answers A, B or Tie.

Model	Layers	Direction	emotion	age	cleanness	chaos	size	depth	spatial	color	mean
UniAR	early	txt $\to$ img	0.98	1.26	0.90	0.81	0.81	0.74	0.09	1.00	0.82
		img $\to$ txt	1.14	0.91	1.01	1.02	0.25	1.19	–	1.21	0.96
	middle	txt $\to$ img	1.05	1.29	0.94	0.77	0.73	0.85	0.07	1.00	0.84
		img $\to$ txt	0.99	0.74	0.92	0.90	0.18	0.96	–	0.47	0.74
	late	txt $\to$ img	0.92	1.22	0.78	0.46	0.61	0.62	–	1.01	0.80
		img $\to$ txt	0.87	0.91	1.13	0.97	0.22	0.98	–	0.71	0.83

Appendix

Table 6: Cross-modal recovery : How much of intra-modality steering is recovered by the transferred vector, defined as ratio of transfer AUC to the intra-modality AUC of concept-rate curve (see Figures 2 and 3.1 ):

1.0

means the transferred vector steers as well as intra-modality,

0

no transfer; values above

1.0

mean transferred vector steers better than intra-modality. Cells are blank where intra-modality steering itself barely works (intra-modality AUC

<0.2

), so the ratio is not meaningful. Gray marks weak transfer (recovery

<0.5

); bold marks the mean of the best layer range, per model and setting.

Figure 10: UniAR at the middle layer range, same as Figure 2 , repeated for comparison. The black curve is the intra-modality vector and the colored curve the transferred one. The shaded area between them is their difference in AUC, green where the transferred vector steers more and red where it steers less.

Figure 11: Emu3.5 at the middle layer range, same layout as Figure 2 .

Figure 12: Liquid at the middle layer range, same layout as Figure 2 .

Figure 13: UniAR, every concept (rows) and setting (columns), one line per layer configuration. Single layers are light and dashed, layer ranges solid, one hue per depth (blue early, orange middle, green late), and all layers is the thick black line. Every curve starts at rate

0

\alpha=0

Figure 14: Same as Figure 13 for Emu3.5.

Figure 15: Same as Figure 13 for Liquid.

		vs. recovery			vs. AUC of transferred vector
Model	Direction	$\rho$	$r$	$p$	$\rho$	$p$
UniAR	txt $\to$ img	$+0.76$	$+0.58$	$<0.001$	$+0.80$	$<0.001$
UniAR	img $\to$ txt	$+0.17$	$+0.19$	$0.46$	$+0.17$	$0.46$
Emu3.5	txt $\to$ img	$-0.27$	$-0.19$	$0.23$	$+0.42$	$0.06$
Emu3.5	img $\to$ txt	$+0.47$	$+0.62$	$0.03$	$+0.04$	$0.88$
Liquid	txt $\to$ img	$-0.12$	$-0.13$	$0.60$	$+0.32$	$0.16$

Appendix

Table 7: Correlation between the cosine similarity of the text and image steering vectors and cross-modal steering, per model and transfer direction. Each cell uses

21

points (

7

concepts

\times

3

layer ranges; spatial excluded because it does not steer in most settings). Left: against recovery, as in Figure 5 . Right: against the concept-rate AUC of the transferred vector alone, i.e. the numerator of recovery.

	raw			unit length, centered
Model	text–text	image–image	image–text	text–text	image–image	image–text
UniAR	$0.75$	$0.74$	$0.65$	$0.17$	$0.16$	$-0.17$
Emu3.5	$0.36$	$0.45$	$0.17$	$0.15$	$0.18$	$-0.16$
Liquid	$0.51$	$0.50$	$0.42$	$0.09$	$0.07$	$-0.08$

Appendix

Table 8: Anisotropy at the middle layers: mean cosine similarity between random pairs of single-token activations, raw and after centering (unit length, midpoint subtracted). Samples:

64

tokens from each of

5

images per concept, and every token but the first of all minimal-pair sentences.

	UniAR			Emu3.5			Liquid
Concept	pole cos	ratio	dir. cos ( $z$ )	pole cos	ratio	dir. cos ( $z$ )	pole cos	ratio	dir. cos ( $z$ )
emotion	$-0.34$	$53\times$	$0.47$ (30)	$-0.62$	$203\times$	$0.09$ (6)	$-0.33$	$210\times$	$0.02$ (1)
age	$-0.49$	$25\times$	$0.31$ (20)	$-0.65$	$33\times$	$0.07$ (5)	$-0.41$	$25\times$	$0.05$ (3)
cleanness	$-0.46$	$10\times$	$0.20$ (13)	$-0.65$	$12\times$	$0.01$ (1)	$-0.36$	$9\times$	$0.08$ (5)
chaos	$-0.42$	$12\times$	$0.14$ (9)	$-0.61$	$23\times$	$0.00$ (0)	$-0.39$	$17\times$	$0.07$ (4)
size	$-0.59$	$15\times$	$-0.02$ ( $-1$ )	$-0.76$	$25\times$	$-0.02$ ( $-1$ )	$-0.59$	$11\times$	$-0.01$ ( $-1$ )

Appendix

Table 9: Same concept pole vs. same direction at the middle layers, per concept and model. pole cos : cosine, after centering, between the positive-pole centroid computed from images and from sentences. ratio : distance from the positive-image centroid to the positive-text centroid, divided by its distance to the negative-image centroid (independent of centering). dir. cos : cosine between the text and image steering vectors, with

z

= cosine divided by the baseline standard deviation.

Figure 16: (a) Per-layer cosine similarity between each concept’s textual and visual steering vectors in UniAR; the gray band is

\pm 2\sigma

for random directions in

d{=}4096

(

\sigma{=}0.016

). (b) Mean cosine similarity between random pairs of single-token activations across layers of UniAR, within text, within image, and across modalities, after centering (midpoint of the two modality means subtracted). The cross-modal cosine is negative after centering: the modality gap of Section 3.4 .

Concept	Poles	Pairs	Outcome
Language-native
formality	colloquial / formal	text template	works in both modalities
language identity	English / French	parallel translations	works in both modalities
language identity	English / Chinese	parallel translations	works in text; in images mostly adds lettering
Vision-native
motion blur	sharp / blurred	directional blur of one image	works in both modalities

Appendix

Table 10: The concepts we built minimal pairs for, and what came of each.

Figure 17: Formality, text $\to$ image. Rows: a person at a desk, a basket of laundry, one person pouring water for another, a pair of shoes, a menu. None of the prompts mentions clothing or register. Toward formal: suits, documents, neatly stacked laundry, dress shoes, a plain typeset menu. Toward colloquial the scenes become warmer and more cluttered, and beyond $-0.6$ they start to look like illustrations; beyond $+0.6$ they wash out.

Figure 18: English vs. French, text $\to$ image. Rows: a house, a town square, the front page of a newspaper, a poster. Toward French: a mansard roof, a Beaux-Arts square, mastheads and posters in French-looking but invented words. Toward English: vinyl siding and a garage, plain paving, English headlines and slogans.

Figure 19: The same direction on everyday objects. Rows: a breakfast, a parked car, a street sign, a station platform, a small shop. At $+0.4$ the objects are French while the image is still photographic (pastries, a vintage car, an enamel street sign, an iron station canopy, a Haussmann shopfront); at $+0.8$ the images turn golden and painterly.

Figure 20: Camera height, image $\to$ image. Rows: a statue, an apartment tower, a staircase, a person. Toward one pole the camera rises until it looks straight down on the subject; toward the other it drops to near ground level. The subject stays the same.

Figure 21: Motion blur, image $\to$ image. Rows: a cyclist, a dog, a train, a sports car. Toward blurred, the background first streaks as in a panning shot while the subject stays readable, then the whole frame blurs. Toward sharp, the scene fills with extra detail.

must include	unsteered	toward low resolution
a street with shops	“I strolled down the bustling street, where the scent of fresh bread and coffee filled the air. I passed by a cozy café, its inviting aroma drawing me in. […]”	“I’m walking down a busy street in the city. I see a shop with a bright red sign.”
a view from a window	“I wake up to the soft hum of the city, the distant clatter of traffic, and the faint scent of coffee and new beginnings. […]”	“I’m sitting in a window with a blurry view of the city. It’s a bright day, and I can see some buildings in the background. I’m not sure what I’m doing , but I’m just sitting there.”
a bridge over the river	“I wake up to the soft hum of the city, the distant sound of traffic and the gentle lapping of water. As I step out onto the bridge, […]”	“I was walking through the city, and I saw a bridge over the river. I was walking on the bridge, and I was looking at the city.”
a hotel lobby	“I wake up in the hotel lobby, the soft hum of the alarm clock filling the air. The lobby is quiet, with a few guests lingering in the background. […]”	“I’m in a hotel lobby, and I’m trying to find something. I see a pixelated object that looks like a pixelated character. I can’t tell what it is , but I’m trying to understand it.”
a construction site	“I wake up to the soft hum of the city, the distant clatter of traffic, and the faint scent of coffee. Today, I’m heading to the construction site, where the early morning light paints the steel and concrete in hues of gold and silver. […]”	“I’m working on a project at a construction site. I’m trying to fix something, but I can’t see what I’m fixing .”
a small alley	“I strolled through the city, the air thick with the scent of coffee and the distant hum of traffic. As I walked down a narrow alley, the walls on either side were lined with small shops and cafes. […]”	“I was walking through a small alley when I saw a person who looked like a ghost .”

Appendix

Table 11: Six examples of applying the high–low resolution steering vector to text generation, for the prompt “Describe your day in the city. Your description must include [X]” (

\alpha=+32

Figure 22: Resolution, image $\to$ image. Rows: a breakfast, a bicycle, a postbox, a poster, a living room. Toward high resolution scenes gain detail and clutter; toward low resolution they first become simpler and emptier, then turn into pixel blocks, and collapse at $+0.8$ .

Steering vector	Per-pole max	Norm. AUC
$\hat{v}_{\text{txt}}$ ( $\lambda\!=\!1$ )	0.76	0.56
$\hat{v}_{\text{img}}$ ( $\lambda\!=\!0$ )	0.70	0.47
Average, $\lambda\!=\!0.75$	0.76	0.58
Average, $\lambda\!=\!0.50$	0.73	0.55
Average, $\lambda\!=\!0.25$	0.71	0.49
Random-direction control	0.72	0.50

Appendix

Table 12: Averaging the two modalities’ steering vectors, text target, middle layer range: concept rate summarized as per-pole maximum and range-normalized AUC over

\alpha\in[-60,60]

, averaged over the six transfer concepts. No average improves over the better single-modality vector beyond judge noise.

Cross-Modal Representation Transfer

See all 30 papers

Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering
Jul 29, 2026Yu Wang, Sharon LiUnified Multimodal Models Cross-Modal Alignment
Department of Computer Sciences, University of Wisconsin-Madison
Steering Visual Generation in Unified Multimodal Models with Understanding Supervision
May 7, 2026Zeyu Liu, Zanlin Ni, Yang Yue +5Unified Multimodal Models Image Generation
1Tsinghua University · 2Kolors Team, Kuaishou Technology
Transferability Between Understanding and Generation in Unified Multimodal Models
Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5Unified Multimodal Models Multimodal Generation
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea

Jul 29, 2026J/K move · Enter open · S save

Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering

Yu Wang, Sharon Li

Department of Computer Sciences, University of Wisconsin-Madison

Unified Multimodal Models Cross-Modal Alignment Multimodal Generation Multimodal Model Interpretability Activation Steering

Open paper AskPDF arXiv

Unified multimodal models (UMMs) aim to integrate understanding and generation within a single architecture, yet it remains unclear whether these capabilities share a unified and transferable semantic space. This question is fundamentally challenging, as the two branches operate over heterogeneous representations (text tokens vs.\ visual latents) and distinct training objectives, making direct comparison difficult. To address this, we introduce cross-branch semantic steering, an intervention-based framework that extracts semantic directions from one branch and applies them to the other. We show that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness. In contrast, the reverse direction consistently shows limited effectiveness. Our analysis suggests that this asymmetry may be related to a practical representational mismatch: understanding-derived vectors capture transferable, object-centric semantics, while generation-derived vectors primarily encode low-level appearance features. Our results reveal that architectural unification does not guarantee semantic alignment, and establish cross-branch steering as a practical tool for probing multimodal representations.

Abstract

Cross-Modal Representation Transfer

Explore similar work

Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering

Steering Visual Generation in Unified Multimodal Models with Understanding Supervision

Transferability Between Understanding and Generation in Unified Multimodal Models