Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We study cross-modal steering in several unified multimodal models (UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show that steering vectors transfer zero-shot in both directions (text-to-image & image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language), steering vectors often induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature too: despite a clear modality gap separating text and image representations, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept. Our findings shed light on how vision and language interact inside unified multimodal models and point towards more general, modality-agnostic approaches for controlling their behavior.
Figures & tables
Figure 1: Cross-modal steering vectors transfer zero-shot in both directions. A steering vector v extracted entirely from one modality can steer generation both within that modality and, without any adaptation, in the other.
Figure 2: Concept rate on UniAR ( Peng et al., 2026 ) at middle layer range. Concept rate is the fraction of steered outputs in which an LLM judge finds the concept shifted toward the steered pole ( Section 3.1 ), plotted against normalized steering strength. Black curves show intra-modality steering (top: img → img , bottom: txt → txt ). The colored areas indicate the gap between the two, either when transfer performs better (green) or worse (red). Recovery in Figures 3 and 3 quantifies this comparison. For most concepts, the transfer curve comes close to the intra-modality curve.
Figure 3
Figure 4: Cross-modal and intra-modal steering in UniAR (middle layer range). We show examples for emotion, distance and cleanness steering. In both modalities the transferred vector produces a similar shift as the intra-modality vector.
Figure 5: Cross-modal cosine similarity vs. cross-modal steering success. Each point is one (concept, layer range): x is cosine between text- and image-derived steering vectors; y is our recovery metric ( Section 3.1 ) for the given transfer direction (left: txt → img , right: img → txt ). Spatial is excluded because it does not steer in most settings; ρ : Spearman, r : Pearson; ∗ marks p<0.05 .
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Concept
Example subjects
− pole
+ pole
emotion
teacher, chef, coach
a teacher is sad
a teacher is happy
age
engineer, cook, merchant
an engineer is young
an engineer is old
cleanness
oven, fork, glove
an oven is clean
an oven is dirty
chaos
street, hospital, party
a street is peaceful
a street is chaotic
size
flower, pen, statue
a flower is small
a flower is large
depth
flower, pen, statue
a flower is far away
a flower is nearby
Appendix
Table 2: Example text minimal pairs. Each subject gives one pair that differs only in the pole word.
Figure 6: One image pair per concept, with the description used to generate its baseline image.
Concept
Image description
Text subject
emotion
A head-and-shoulders portrait of a person with short cropped hair , wearing a plain crew-neck shirt and holding a relaxed, neutral expression. They stand in front of a weathered red brick wall in soft natural daylight. It is shot as a sharp, realistic photograph with an even, unremarkable mood.
a person with short cropped hair
cleanness
A photograph of a modern kitchen with wooden cabinets , stone countertops, and a stainless steel sink. Even natural daylight from a window falls across the counters, backsplash, and floor. It is a plain, realistic interior photograph with no people.
a modern kitchen with wooden cabinets
size
A single traffic cone in the center of a wide empty asphalt parking lot with faded painted markings, under flat even daylight.
a traffic cone
color
A photograph of a car parked alone on a wide empty asphalt lot, the car centred and seen from the side, under soft even daylight and a pale overcast sky. A plain, realistic photograph with no people.
a parked car
Appendix
Table 3: Example evaluation prompts. Image generation uses the full description, and text generation uses only its subject, shown in bold.
Model
Setting
α grid
Decoding
UniAR
txt → txt , img → txt
±100 , step 20
greedy, 96 tokens
img → img txt → img
±10 , step 2±15 , step 3
16×16 tokens, guidance 1.5 , temperature 1.0 , SD3 decoder with 16 steps and guidance 1.5 , 544×544 image
Table 4: The strength grid is set per model and setting ( Section A.4 ). We keep each model’s default decoding settings other than the number of decoder steps and the image resolution for practical reasons, and fix the seed at 42 .
Single layer
Contiguous layer range
Model
Depth
early
middle
late
early
middle
late
UniAR
36
9
18
27
7–10
16–19
25–28
Emu3.5
64
16
32
48
12–19
28–35
44–51
Liquid
28
7
14
21
5–8
12–15
19–22
Appendix
Table 5: Layer configurations for each model ( 0 -indexed). Single layers are evenly spaced in depth, each contiguous layer range spans the layers around its matching single layer, and the seventh configuration steers all layers.
Figure 7: DreamSim pole score for image-derived (left) and text-derived (right) vectors.
Figure 8: Concept rate (solid) and output quality (dashed) against normalized strength at the middle layer range, averaged over models, concepts, prompts and both signs of α . Quality is text fluency (left) and image coherence (right), relative to the unsteered outputs.
Figure 9: The annotation interface shown to human reviewers. It names the asked pole and its opposite. Two model outputs, the unsteered baseline and the steered output, are shown as Candidate A and B in random order. The human reviewer answers A, B or Tie.
Model
Layers
Direction
emotion
age
cleanness
chaos
size
depth
spatial
color
mean
UniAR
early
txt → img
0.98
1.26
0.90
0.81
0.81
0.74
0.09
1.00
0.82
img → txt
1.14
0.91
1.01
1.02
0.25
1.19
–
1.21
0.96
middle
txt → img
1.05
1.29
0.94
0.77
0.73
0.85
0.07
1.00
0.84
img → txt
0.99
0.74
0.92
0.90
0.18
0.96
–
0.47
0.74
late
txt → img
0.92
1.22
0.78
0.46
0.61
0.62
–
1.01
0.80
img → txt
0.87
0.91
1.13
0.97
0.22
0.98
–
0.71
0.83
Appendix
Table 6: Cross-modal recovery : How much of intra-modality steering is recovered by the transferred vector, defined as ratio of transfer AUC to the intra-modality AUC of concept-rate curve (see Figures 2 and 3.1 ): 1.0 means the transferred vector steers as well as intra-modality, 0 no transfer; values above 1.0 mean transferred vector steers better than intra-modality. Cells are blank where intra-modality steering itself barely works (intra-modality AUC <0.2 ), so the ratio is not meaningful. Gray marks weak transfer (recovery <0.5 ); bold marks the mean of the best layer range, per model and setting.
Figure 10: UniAR at the middle layer range, same as Figure 2 , repeated for comparison. The black curve is the intra-modality vector and the colored curve the transferred one. The shaded area between them is their difference in AUC, green where the transferred vector steers more and red where it steers less.
Figure 11: Emu3.5 at the middle layer range, same layout as Figure 2 .
Figure 12: Liquid at the middle layer range, same layout as Figure 2 .
Figure 13: UniAR, every concept (rows) and setting (columns), one line per layer configuration. Single layers are light and dashed, layer ranges solid, one hue per depth (blue early, orange middle, green late), and all layers is the thick black line. Every curve starts at rate 0 at α=0 .
Figure 14: Same as Figure 13 for Emu3.5.
Figure 15: Same as Figure 13 for Liquid.
vs. recovery
vs. AUC of transferred vector
Model
Direction
ρ
r
p
ρ
p
UniAR
txt → img
+0.76
+0.58
<0.001
+0.80
<0.001
UniAR
img → txt
+0.17
+0.19
0.46
+0.17
0.46
Emu3.5
txt → img
−0.27
−0.19
0.23
+0.42
0.06
Emu3.5
img → txt
+0.47
+0.62
0.03
+0.04
0.88
Liquid
txt → img
−0.12
−0.13
0.60
+0.32
0.16
Appendix
Table 7: Correlation between the cosine similarity of the text and image steering vectors and cross-modal steering, per model and transfer direction. Each cell uses 21 points ( 7 concepts ×3 layer ranges; spatial excluded because it does not steer in most settings). Left: against recovery, as in Figure 5 . Right: against the concept-rate AUC of the transferred vector alone, i.e. the numerator of recovery.
raw
unit length, centered
Model
text–text
image–image
image–text
text–text
image–image
image–text
UniAR
0.75
0.74
0.65
0.17
0.16
−0.17
Emu3.5
0.36
0.45
0.17
0.15
0.18
−0.16
Liquid
0.51
0.50
0.42
0.09
0.07
−0.08
Appendix
Table 8: Anisotropy at the middle layers: mean cosine similarity between random pairs of single-token activations, raw and after centering (unit length, midpoint subtracted). Samples: 64 tokens from each of 5 images per concept, and every token but the first of all minimal-pair sentences.
UniAR
Emu3.5
Liquid
Concept
pole cos
ratio
dir. cos ( z )
pole cos
ratio
dir. cos ( z )
pole cos
ratio
dir. cos ( z )
emotion
−0.34
53×
0.47 (30)
−0.62
203×
0.09 (6)
−0.33
210×
0.02 (1)
age
−0.49
25×
0.31 (20)
−0.65
33×
0.07 (5)
−0.41
25×
0.05 (3)
cleanness
−0.46
10×
0.20 (13)
−0.65
12×
0.01 (1)
−0.36
9×
0.08 (5)
chaos
−0.42
12×
0.14 (9)
−0.61
23×
0.00 (0)
−0.39
17×
0.07 (4)
size
−0.59
15×
−0.02 ( −1 )
−0.76
25×
−0.02 ( −1 )
−0.59
11×
−0.01 ( −1 )
Appendix
Table 9: Same concept pole vs. same direction at the middle layers, per concept and model. pole cos : cosine, after centering, between the positive-pole centroid computed from images and from sentences. ratio : distance from the positive-image centroid to the positive-text centroid, divided by its distance to the negative-image centroid (independent of centering). dir. cos : cosine between the text and image steering vectors, with z = cosine divided by the baseline standard deviation.
Figure 16: (a) Per-layer cosine similarity between each concept’s textual and visual steering vectors in UniAR; the gray band is ±2σ for random directions in d=4096 ( σ=0.016 ). (b) Mean cosine similarity between random pairs of single-token activations across layers of UniAR, within text, within image, and across modalities, after centering (midpoint of the two modality means subtracted). The cross-modal cosine is negative after centering: the modality gap of Section 3.4 .
Concept
Poles
Pairs
Outcome
Language-native
formality
colloquial / formal
text template
works in both modalities
language identity
English / French
parallel translations
works in both modalities
language identity
English / Chinese
parallel translations
works in text; in images mostly adds lettering
Vision-native
motion blur
sharp / blurred
directional blur of one image
works in both modalities
Appendix
Table 10: The concepts we built minimal pairs for, and what came of each.
Figure 17: Formality, text → image. Rows: a person at a desk, a basket of laundry, one person pouring water for another, a pair of shoes, a menu. None of the prompts mentions clothing or register. Toward formal: suits, documents, neatly stacked laundry, dress shoes, a plain typeset menu. Toward colloquial the scenes become warmer and more cluttered, and beyond −0.6 they start to look like illustrations; beyond +0.6 they wash out.
Figure 18: English vs. French, text → image. Rows: a house, a town square, the front page of a newspaper, a poster. Toward French: a mansard roof, a Beaux-Arts square, mastheads and posters in French-looking but invented words. Toward English: vinyl siding and a garage, plain paving, English headlines and slogans.
Figure 19: The same direction on everyday objects. Rows: a breakfast, a parked car, a street sign, a station platform, a small shop. At +0.4 the objects are French while the image is still photographic (pastries, a vintage car, an enamel street sign, an iron station canopy, a Haussmann shopfront); at +0.8 the images turn golden and painterly.
Figure 20: Camera height, image → image. Rows: a statue, an apartment tower, a staircase, a person. Toward one pole the camera rises until it looks straight down on the subject; toward the other it drops to near ground level. The subject stays the same.
Figure 21: Motion blur, image → image. Rows: a cyclist, a dog, a train, a sports car. Toward blurred, the background first streaks as in a panning shot while the subject stays readable, then the whole frame blurs. Toward sharp, the scene fills with extra detail.
must include
unsteered
toward low resolution
a street with shops
“I strolled down the bustling street, where the scent of fresh bread and coffee filled the air. I passed by a cozy café, its inviting aroma drawing me in. […]”
“I’m walking down a busy street in the city. I see a shop with a bright red sign.”
a view from a window
“I wake up to the soft hum of the city, the distant clatter of traffic, and the faint scent of coffee and new beginnings. […]”
“I’m sitting in a window with a blurry view of the city. It’s a bright day, and I can see some buildings in the background. I’m not sure what I’m doing , but I’m just sitting there.”
a bridge over the river
“I wake up to the soft hum of the city, the distant sound of traffic and the gentle lapping of water. As I step out onto the bridge, […]”
“I was walking through the city, and I saw a bridge over the river. I was walking on the bridge, and I was looking at the city.”
a hotel lobby
“I wake up in the hotel lobby, the soft hum of the alarm clock filling the air. The lobby is quiet, with a few guests lingering in the background. […]”
“I’m in a hotel lobby, and I’m trying to find something. I see a pixelated object that looks like a pixelated character. I can’t tell what it is , but I’m trying to understand it.”
a construction site
“I wake up to the soft hum of the city, the distant clatter of traffic, and the faint scent of coffee. Today, I’m heading to the construction site, where the early morning light paints the steel and concrete in hues of gold and silver. […]”
“I’m working on a project at a construction site. I’m trying to fix something, but I can’t see what I’m fixing .”
a small alley
“I strolled through the city, the air thick with the scent of coffee and the distant hum of traffic. As I walked down a narrow alley, the walls on either side were lined with small shops and cafes. […]”
“I was walking through a small alley when I saw a person who looked like a ghost .”
Appendix
Table 11: Six examples of applying the high–low resolution steering vector to text generation, for the prompt “Describe your day in the city. Your description must include [X]” ( α=+32 ).
Figure 22: Resolution, image → image. Rows: a breakfast, a bicycle, a postbox, a poster, a living room. Toward high resolution scenes gain detail and clutter; toward low resolution they first become simpler and emptier, then turn into pixel blocks, and collapse at +0.8 .
Steering vector
Per-pole max
Norm. AUC
v^txt ( λ=1 )
0.76
0.56
v^img ( λ=0 )
0.70
0.47
Average, λ=0.75
0.76
0.58
Average, λ=0.50
0.73
0.55
Average, λ=0.25
0.71
0.49
Random-direction control
0.72
0.50
Appendix
Table 12: Averaging the two modalities’ steering vectors, text target, middle layer range: concept rate summarized as per-pole maximum and range-normalized AUC over α∈[−60,60] , averaged over the six transfer concepts. No average improves over the better single-modality vector beyond judge noise.
Unified multimodal models (UMMs) aim to integrate understanding and generation within a single architecture, yet it remains unclear whether these capabilities share a unified and transferable semantic space. This question is fundamentally challenging, as the two branches operate over heterogeneous representations (text tokens vs.\ visual latents) and distinct training objectives, making direct comparison difficult. To address this, we introduce cross-branch semantic steering, an intervention-based framework that extracts semantic directions from one branch and applies them to the other. We show that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness. In contrast, the reverse direction consistently shows limited effectiveness. Our analysis suggests that this asymmetry may be related to a practical representational mismatch: understanding-derived vectors capture transferable, object-centric semantics, while generation-derived vectors primarily encode low-level appearance features. Our results reveal that architectural unification does not guarantee semantic alignment, and establish cross-branch steering as a practical tool for probing multimodal representations.
Yu Wang, Sharon Li
Department of Computer Sciences, University of Wisconsin-Madison
Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.
Zeyu Liu, Zanlin Ni, Yang Yue +5
1Tsinghua University · 2Kolors Team, Kuaishou Technology
Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate transferability in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.
Jiwon Kang, Heeji Yoon, Jaewoo Jung +5
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea