Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation

Zhengyang Luo; Xiaoyu Qiu; Zhiheng Tan; Yuchen Liang; Yingbin Liang; Liyue Shen

cs.AIOct 10, 2026

Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation

Authors: Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan, Yuchen Liang, Yingbin Liang, Liyue Shen

Organizations: University of Michigan, Ann Arbor · The Ohio State University · Indiana University Indianapolis

Abstract

Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood. We study this question from the perspective of conditional guidance: we probe every attention head to show the activation difference between two branches of classifier-free guidance (CFG) with different conditional guidance. On two unified VLMs of autoregressive and masked-diffusion paradigms, the probing reveals a subset of heads that are more sensitive to the change of condition while others are agnostic to it. This characteristic pattern holds in both T2I and I2T generation and is largely stable across data samples, demonstrating the internal mechanism of unified VLMs. Motivated by this, we propose a new inference-time intervention approach for unified multimodal generation, named HeadTuned-CFG, which uses the probed condition-change awareness to select and tune the conditional guidance at the attention-head level. Experiments show consistent benefits in both multimodal generation directions of unified VLMs without additional training, and in particular substantial improvements over strong baselines in T2I generation. These results further support the key finding of our probing analysis: the observed pattern of functional specialization is an intrinsic property of unified VLMs.

Figures & tables

Figure 1: Condition-change aware (green) and agnostic (brown) attention heads of Janus. Left: T2I generation ; each map shows where the generated image attends to a boxed keyword of the prompt. Right: I2T generation with the same image as input; each map shows where a boxed keyword of the generated caption attends in the input image. Middle: the generated (input) image and head-level condition-change awareness map of both T2I and I2T probing (layer by head, brighter means higher awareness), with the visualized heads marked. The Muddit counterpart is in Appendix A .

Figure 2: Layer-by-head condition-change awareness $\delta^{(l,h)}$ of Muddit and Janus, averaged over $|\mathcal{P}|=500$ probing samples (and over steps for Muddit). Left of the dashed line: T2I generation; right: I2T generation.

Figure 3: Proposed HeadTuned-CFG method on Muddit. Illustration of head-level probing, selection, and tuning process of the single-stream attention heads.

	animals	art	fashion	food	indoor	landscape	logo	people	plants	vehicles	full
CFG	33.91	42.82	38.79	39.53	43.82	40.93	40.27	43.85	49.93	36.89	20.90
PAG	52.31	57.31	53.30	64.44	61.24	65.27	51.48	60.09	76.86	54.04	34.37
PAG+CFG	38.21	44.40	40.03	42.27	43.74	45.92	39.46	45.28	56.44	38.59	22.39
HeadTuned-CFG	30.41	36.70	34.02	33.60	36.17	34.56	36.65	36.88	43.01	30.93	16.29

Table 1: Muddit T2I: FID per MJHQ category (

n=3{,}000

each) and on MJHQ-full (

n=30{,}000

); CFG (

w=3

), PAG (

\lambda=2.5

, single-stream blocks 9–10), PAG+CFG (

\lambda=1.5

on CFG with

w=3

), and HeadTuned-CFG (

w=9

k=50\%

s=0.4

Figure 4: Heads tuned by the four selection rules on Muddit (single-stream block by head), with FID on MJHQ-animals at $w=9$ , $k=50\%$ , $s=0.4$ . The probed and the top- $50\%$ selections change per denoising step; their panels show the fraction of the 64 steps in which a head is tuned.

	random		popular		adversarial		overall
	Acc	F1	Acc	F1	Acc	F1	Acc	F1
Regular	84.17	81.60	82.97	80.47	81.33	78.99	82.82	80.35
VCD	87.77	86.33	85.97	84.63	83.93	82.79	85.89	84.58
HeadTuned-VCD	88.80	87.76	86.43	85.54	84.13	83.50	86.45	85.60

Table 3: Janus on POPE, accuracy and F1 (%) per split (

n=3{,}000

each): Regular decoding, VCD (

w=8

\beta=0.2

T=500

), and HeadTuned-VCD (

w=8

k=20

heads,

s=1.3

	existence	count	position	color	total
Regular	138.33	56.67	90.00	93.33	378.33
VCD	170.00	81.67	91.67	123.33	466.67
HeadTuned-VCD	170.00	83.33	91.67	130.00	475.00

Table 4: Janus on the hallucination subset of MME, score (accuracy

+

image-level accuracy, out of 200) per subtask and in total: Regular decoding, VCD (

w=10

\beta=0.2

T=500

), and HeadTuned-VCD (

w=8

k=15

heads,

s=1.1

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Condition-change aware (green) and agnostic (brown) attention heads of Muddit, in the layout of Figure 1 : the T2I maps show the alignment of each head’s CFG difference with the keyword in value space, the I2T maps the attention of the decoded keyword to the image, and the middle column the generated image and the Muddit probes with the visualized heads marked.

	T2I	I2T
Spearman $\rho$	$0.970\pm 0.014$	$0.962\pm 0.023$

Appendix

Table 5: Mean pairwise Spearman rank correlation of the condition-change awareness ranking of Janus across

|\mathcal{P}|=500

probing samples.

Figure 6: Cross-sample stability of the condition-change awareness ranking in Muddit, per denoising step. Left: T2I generation. Right: I2T generation.

	FID $\downarrow$	CLIP $\uparrow$	IR $\uparrow$	HPS $\uparrow$	P/R $\uparrow$	D/C $\uparrow$	IS $\uparrow$	Aes. $\uparrow$
CFG	20.90	0.320	0.63	0.257	0.733/0.464	0.856/0.733	21.83	5.85
PAG	34.37	0.294	$-0.26$	0.217	0.597/0.382	0.577/0.508	16.45	5.66
PAG+CFG	22.39	0.318	0.64	0.264	0.720/0.431	0.851/0.712	20.77	5.91
HeadTuned-CFG	16.29	0.324	0.82	0.271	0.762 / 0.529	0.969 / 0.808	22.77	5.84

Appendix

Table 6: Muddit on MJHQ-full, overall (

n=30{,}000

): CFG (

w=3

), PAG (

\lambda=2.5

on single-stream blocks 9–10), PAG+CFG (

\lambda=1.5

on top of CFG with

w=3

), and HeadTuned-CFG (

w=9

k=50\%

s=0.4

). IR is ImageReward; HPS is human preference score; Aes. is aesthetic score; P/R is precision/recall; D/C is density/coverage.

	FID $\downarrow$	CLIP $\uparrow$	IR $\uparrow$	HPS $\uparrow$	P/R $\uparrow$	D/C $\uparrow$	IS $\uparrow$	Aes. $\uparrow$
CFG	10.51	0.321	0.59	0.251	0.660/ 0.666	0.660/0.797	24.62	5.53
HeadTuned-CFG	9.92	0.318	0.56	0.249	0.682 /0.660	0.702 / 0.811	24.51	5.52

Appendix

Table 7: Janus on MJHQ-full, overall (

n=30{,}000

): CFG (

w=4

) versus HeadTuned-CFG (

w=4

k=15\%

s=0.75

); abbreviations as in Table 6 .

Figure 7: Janus on MJHQ-full (

n=30{,}000

): FID against CLIP, ImageReward, HPS, aesthetic score, Inception Score, precision, recall, density, and coverage over the guidance scale

w

(labels), for vanilla CFG and HeadTuned-CFG (

k=15\%

s=0.75

unless noted).

Figure 8: Heads tuned by the four selection rules on Janus (layer by head; colored = tuned), with FID on MJHQ-animals at $w=4$ , $k=15\%$ , $s=0.75$ .

$w$	1.0	1.5	2.0	2.5	3.0	5.0	7.0	9.0
FID	77.96	41.95	36.17	34.51	33.91	34.05	35.35	37.40

Appendix

Table 8: Vanilla CFG on Muddit, MJHQ-animals (

n=3{,}000

), over the guidance scale

w

$k\backslash s$	0.2	0.3	0.4	0.5	0.6
40%	31.60	31.33	31.91	–	–
45%	–	31.11	30.83	31.51	–
50%	–	30.61	30.41	31.21	–
55%	–	–	31.63	31.01	31.86
60%	–	–	31.77	31.76	32.25

Appendix

Table 9: HeadTuned-CFG search on Muddit, MJHQ-animals,

w=9

: FID over the head fraction

k

(rows) and strength

s

(columns). Vanilla CFG at

w=9

: FID 37.40.

$k\backslash s$	0.6	0.7	0.8	0.9
15%	–	–	33.87	33.61
25%	–	34.09	33.70	33.80
35%	34.57	33.83	33.98	34.00

Appendix

Table 10: HeadTuned-CFG search on Muddit, MJHQ-animals,

w=3

: FID over the head fraction

k

(rows) and strength

s

(columns). Vanilla CFG at

w=3

: FID 33.91.

$w$	3.0	3.5	4.0	4.5	5.0	7.5
FID	27.45	28.07	27.32	27.57	27.92	28.87

Appendix

Table 11: Vanilla CFG on Janus, MJHQ-animals (

n=3{,}000

), over the guidance scale

w

$k\backslash s$	0.60	0.65	0.70	0.75	0.80
10%	26.51	26.57	26.34	27.09	–
15%	–	26.65	26.04	25.95	26.92
20%	–	28.12	27.08	26.32	27.19
25%	–	29.74	27.23	26.40	27.22
30%	–	–	28.88	26.95	27.35

Appendix

Table 12: HeadTuned-CFG search on Janus, MJHQ-animals,

w=4

: FID over the head fraction

k

(rows) and strength

s

(columns). Vanilla CFG at

w=4

: FID 27.32.

$k\backslash s$	0.70	0.75	0.80
10%	27.11	26.76	28.01
15%	26.95	26.48	27.97
20%	27.13	26.53	28.21

Appendix

Table 13: HeadTuned-CFG search on Janus, MJHQ-animals,

w=7.5

: FID over the head fraction

k

(rows) and strength

s

(columns). Vanilla CFG at

w=7.5

: FID 28.87.

$w$	2	4	6	8	10	12
Acc	87.10	87.60	87.70	87.77	87.77	87.77
F1	85.47	86.12	86.25	86.33	86.33	86.33

Appendix

Table 14: VCD on Janus, POPE random split (

n=3{,}000

): accuracy and F1 (%) over the guidance scale

w=1+\alpha

(

\beta=0.2

T=500

). Regular decoding: accuracy 84.17, F1 81.60.

	Acc.				F1
$k\backslash s$	1.2	1.3	1.4	1.5	1.2	1.3	1.4	1.5
15	88.47	88.70	88.60	88.73	87.28	87.63	87.57	87.76
20	88.63	88.80	88.73	88.70	87.50	87.76	87.74	87.76
25	88.53	88.53	88.27	88.00	87.40	87.46	87.22	87.01

Appendix

Table 15: HeadTuned-VCD search on Janus, POPE random split, on top of VCD at

w=8

(accuracy 87.77, F1 86.33): accuracy and F1 (%) over the number of tuned heads

k

(rows) and strength

s

(columns).

$w$	2	4	6	8	10	12
existence	145.00	160.00	165.00	165.00	170.00	170.00
count	58.33	76.67	76.67	76.67	81.67	81.67
position	80.00	93.33	93.33	91.67	91.67	91.67
color	115.00	130.00	130.00	123.33	123.33	123.33
total	398.33	460.00	465.00	456.67	466.67	466.67

Appendix

Table 16: VCD on Janus, hallucination subset of MME: score per subtask and in total over the guidance scale

w=1+\alpha

(

\beta=0.2

T=500

). Regular decoding: 138.33 / 56.67 / 90.00 / 93.33, total 378.33.

$k\backslash s$	1.05	1.10	1.15	1.20	1.25	1.30
10	461.67	466.67	458.33	460.00	468.33	445.00
15	465.00	471.67	460.00	470.00	465.00	458.33
20	468.33	463.33	466.67	468.33	461.67	455.00

Appendix

Table 17: HeadTuned-VCD search on Janus, hallucination subset of MME, on top of VCD at

w=6

: total score over the number of tuned heads

k

(rows) and strength

s

(columns). VCD at

w=6

: 465.00.

$k\backslash s$	1.05	1.10	1.15	1.20	1.25	1.30
10	465.00	458.33	461.67	460.00	461.67	443.33
15	473.33	475.00	460.00	465.00	463.33	458.33
20	466.67	466.67	463.33	468.33	460.00	450.00

Appendix

Table 18: HeadTuned-VCD search on Janus, hallucination subset of MME, on top of VCD at

w=8

: total score over the number of tuned heads

k

(rows) and strength

s

(columns). VCD at

w=8

: 456.67.

$k\backslash s$	1.05	1.10	1.15	1.20	1.25	1.30
10	470.00	463.33	461.67	460.00	461.67	443.33
15	468.33	470.00	465.00	465.00	458.33	448.33
20	466.67	466.67	458.33	463.33	460.00	445.00

Appendix

Table 19: HeadTuned-VCD search on Janus, hallucination subset of MME, on top of VCD at

w=10

: total score over the number of tuned heads

k

(rows) and strength

s

(columns). VCD at

w=10

: 466.67.

Explore similar work

Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
Jul 28, 2026Weiming Zhuang, Jiabo Huang, Jingtao Li +4Unified Multimodal Models Visual Tokenization
Sony AI
Gaze Heads: How VLMs Look at What They Describe
Jun 12, 2026Rohit Gandikota, David BauImage Captioning Visual Attention
Northeastern University
UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation
May 12, 2026Yiyan Xu, Qiulin Wang, Wenjie Wang +5Subject-Driven Image Generation Image Generation
University of Science and Technology of China · Kling Team, Kuaishou Technology

Jul 28, 2026J/K move · Enter open · S save

Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

Weiming Zhuang, Jiabo Huang, Jingtao Li, Zhizhong Li+3

Sony AI

Unified Multimodal Models Visual Tokenization Multimodal Generation Multimodal Understanding

Open paper AskPDF arXiv

Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities. To address these challenges, we present Argus-Unified, a compact, effective and unified multimodal model built with low demand on computation and data. Instead of aligning modalities from scratch, Argus-Unified effectively leverages pretrained vision-language models (VLMs) that provide strong multimodal priors. Specifically, we introduce hybrid visual tokens that preserve continuous tokens for understanding while learning discrete tokens for generation from a frozen unified vision encoder. Our training pipeline includes two stages: the first stage learns a quantizer and image decoder on top of the frozen vision encoder, the second stage trains the LLM initialized from a pretrained VLM for the unified multimodal modeling. Using by far the least amount of data (15.6M) and the lowest cost (~$2,000), we demonstrate that unified multimodal models can be trained economically while achieving strong performance in both understanding and generation. Notably, our model attains state-of-the-art multimodal understanding on GQA, POPE, and VQAv2, and competitive generation quality compared to models with dedicated vision encoders (e.g., Janus, Janus-Pro), all at ~10x lower cost and with ~5x less data. We envision Argus-Unified as a useful baseline that lowers the development barrier for unified models.

Abstract

Explore similar work

Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

Gaze Heads: How VLMs Look at What They Describe

UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation