Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervision

Jiyeong Kim; Yerim So; Hyesong Choi; Uiwon Hwang; Dongbo Min

cs.CVMar 20, 2026

Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervision

Authors: Jiyeong Kim, Yerim So, Hyesong Choi, Uiwon Hwang, Dongbo Min

Organizations: Ewha Womans University, South Korea · Soongsil University, South Korea

Abstract

Unified Multimodal Models (UMMs) have emerged as a promising paradigm that integrates multimodal understanding and generation within a unified modeling framework. However, current generative training paradigms suffer from inherent limitations. We present Semantically-Grounded Supervision (SeGroS), a fine-tuning framework designed to resolve the granularity mismatch and supervisory redundancy in UMMs. At its core, we propose a novel visual grounding map to construct two complementary supervision signals. First, we formulate semantic Visual Hints to compensate for the sparsity of text prompts. Second, we generate a semantically-grounded Corrupted Input to explicitly enhance the supervision of masking-based UMMs by restricting the reconstruction loss to core text-aligned regions. Extensive evaluations on GenEval, DPGBench, and CompBench demonstrate that SeGroS significantly improves generation fidelity and cross-modal alignment across various UMM architectures.

Figures & tables

Figure 1: Generative Training Paradigms in UMMs. (a) Text-conditioned training in UMMs [ 39 , 44 ] uses the text prompt as the semantic specification and applies random masking to form the corrupted input for token-level reconstruction. (b) Image reconstruction-based approach, Reca [ 43 ] , uses image prompts to reconstruct corrupted input, yet still relies on random masking. (c) Our approach, Se mantically- Gro unded S upervision ( SeGroS ), constructs its training signal by conditioning on text and explicitly text-aligned visual prompts. Furthermore, we selectively mask the core semantic regions to construct the corrupted input. This provides highly efficient supervision that concentrates model capacity on core structures rather than irrelevant regions.

Figure 2: Overview of SeGroS during training. (a) The SeGroS Map computes a grounding score for each visual token based on its semantic similarity to discriminative text tokens and constructs two complementary training signals. The Visual Hints preserve high-groundedness tokens as additional conditioning signals (yellow patches). The Corrupted Input retains low-groundedness tokens as unmasked context (green patches) while masking the remaining tokens, which are reconstructed by the unified autoregressive Transformer. (b) Discriminative Text Token Filtering identifies visually relevant text tokens via intra- and inter-modal affinities, which are then used to compute the Visual Grounding Map. At inference time, SeGroS Map generation is omitted, allowing the original UMMs to generate images without architectural modifications.

Fine-Tuning Dataset: MidjourneyV6 Dataset
UMMs	Method	GenEval $\uparrow$							DPGBench $\uparrow$	CompBench $\uparrow$
UMMs	Method	Single	Two	Count	Color	Position	Attr.	Overall	Overall	Overall
Show-o-256 [ 44 ]	w/o SFT	97.4	63.3	52.1	82.3	14.2	30.3	56.6	70.65	73.58
	SFT*	97.8	63.4	53.4	80.3	16.5	33.5	57.5	74.6	75.37
	Reca	97.4	73.6	56.0	83.8	20.3	40.2	61.9	75.7	77.5*
	Ours	98.1	72.5	54.4	86.4	21.0	41.0	62.22	76.52	78.30

Table 1: Comparison on text-to-image generation across UMMs. We compared the proposed method with the Supervised Fine-Tuning (SFT) and Reca [ 43 ] . ‘*’ indicates results reproduced using the official implementations.

Method	MME	POPE ${}_{(\texttt{Acc})}$	POPE ${}_{(\texttt{F1})}$	GQA	MMMU	SEED
w/o SFT	1195	83.8	83.9	58.8	34.7	65.2
SFT	1210	84.0	84.1	58.5	35.3	65.1
Reca	1190	83.9	83.4	58.5	34.9	65.1
Ours	1217	84.5	84.2	58.7	36.0	65.5

Table 2: Visual understanding performance.

Figure 3: Visualization of text token affinity and derived visual hints. (a) The original input image with text description. (e–h) The extracted visual hints are represented by the image prompt. For (f–h), we project the text scores from (b–d) to compute the visual grounding map, retaining only the top 30% of visual tokens with the highest groundedness scores. (e) shows the image prompt generated using all text words without discriminative filtering. Text scores exclude non-semantic functional stopwords.

Figure 4: Qualitative comparison of T2I results using Harmon-1.5B [ 39 ] fine-tuned on the BLIP3o-60k [ 6 ] dataset. Baseline indicates the model before SFT. More visualizations for other backbones are provided in the supplementary material.

Table 7

Method	Visual Hints	Masking ratio $\gamma(t)$	Loss targets	GenEval $\uparrow$	DPGBench $\uparrow$
SFT	0%	$\mathcal{U}[0.7,1.0)$	All Masked Regions	72.1	82.5
Ours	30%	$\mathcal{U}[0.7,1.0)$	All Masked Regions	81.5	85.4
Ours (drop-loss)	30%	$\mathcal{U}[0.7,1.0)$	Top- $30\%$ Masked Regions	82.1	84.7

Table 6: Ablation on supervision allocation: We form the corrupted input with a masking ratio

\gamma(t)\sim\mathcal{U}[0.7,1.0)

, i.e . the ratio of visible tokens is

1-\gamma(t)

. While SFT employs random masking, our method adopts adaptive masking guided by the visual grounding map. In ‘Ours (drop-loss)’, supervision for the reconstruction loss during training is restricted to the Top-30% grounded targets.

Figure 5: Qualitative T2I generation results of SeGroS with OpenUni-3.6B [ 38 ] . Refer to the supplementary material for the corresponding text prompts.

Optimization
	Show-o [ 44 ]		Harmon [ 39 ]		OpenUni [ 38 ]
Configuration	$\mathbf{256^{2}}$	$\mathbf{512^{2}}$	0.5B	1.5B	1.6B	3.6B
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW	AdamW
Learning Rate	5e-7	5e-7	1e-5	1e-5	1e-5	1e-5
Weight Decay	0.01	0.01	0.01	0.01	0.01	0.01
Optimizer Momentum ( $\beta_{1},\beta_{2}$ )	(0.9, 0.999)		(0.9, 0.95)		(0.9, 0.95)

Table 7: Hyperparameter settings for fine-tuning with SeGroS. We provide the detailed configuration for reproducing our results across different UMM architectures.

Figure 6: Attention mask adopted in SeGroS. Dark cells denote blocked attention, whereas colored cells denote allowed attention. Rows correspond to query tokens and columns to key tokens.

Method	Noise Scale ( $\alpha$ )	GenEval	DPGBench
w/o SFT	None	72.9	80.9
SFT	None	85.0	85.6
Ours	0 (None)	87.9	86.2
Ours	0.5	88.7	86.6
Ours	1.0	87.8	86.2

Table 8: Ablation study on the noise scale

\alpha

for the visual grounding map, where the noise vector is sampled as

\boldsymbol{\xi}\sim\mathcal{U}([0,\alpha]^{N_{I}})

in Eq. (12). Experiments are conducted using the Harmon-1.5B [ 39 ] backbone on BLIP3o-60k [ 6 ] .

Method	Text	Img.	Hint Selection Rule	Visual Hint Ratio	GenEval	DPG
Standard T2I	✓	×	–	0%	72.08	82.45
Reca [ 43 ]	×	✓	All visual hints	100%	78.7	84.67
Reca + SeGroS hint filtering	×	✓	Top-30% by SeGroS grounding map	30%	79.2	85.2
Ours (SeGroS)	✓	✓	Top-30% by SeGroS grounding map	30%	81.5	85.4

Table 9: Ablation on visual hint redundancy and transferability of our visual grounding map. Applying the proposed visual grounding map to prune Reca’s dense visual hints from 100% to the top 30% improves over vanilla Reca.

Category	Show-o-512		Harmon-1.5B		OpenUni-3.6B
Category	Baseline	Ours	Baseline	Ours	Baseline	Ours
Color	88.70	90.22	89.55	93.83	84.47	91.88
Shape	77.32	82.19	79.09	84.60	75.22	86.60
Texture	84.55	88.39	86.28	91.18	82.87	89.75
Spatial	83.96	87.05	84.98	93.19	78.29	86.95
Non-spatial	83.94	85.49	82.77	88.00	83.10	87.25

Table 10: Detailed results on T2I-CompBench. Baseline denotes the original backbone models, and Ours denotes the proposed SeGroS framework. The experimental setup is identical to Tab. 1 in the main paper.

Figure 7: Qualitative GenEval results on OpenUni-3.6B. We compare samples generated by Reca and SeGroS using the OpenUni-3.6B backbone under identical GenEval prompts.

Figure 8: Qualitative results on Show-o-512. We compare samples generated by Reca and SeGroS using the Show-o-512 backbone under identical T2I-CompBench prompts.

Figure 9: Qualitative results on Harmon 1.5B trained with SeGroS.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 290 papers

Semantic Generative Tuning for Unified Multimodal Models
May 18, 2026Songsong Yu, Yuxin Chen, Ying Shan +1Image Segmentation Unified Multimodal Models
Shanghai Jiao Tong University · Tencent ARCLab
Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models
Mar 6, 2026Jiadong Pan, Liang Li, Yuxin Peng +6Unified Multimodal Models T2I Generation
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Peking University +2
Reconstruction Alignment Improves Unified Multimodal Models
Sep 8, 2025Ji Xie, Trevor Darrell, Luke Zettlemoyer +1Unified Multimodal Models Cross-Modal Alignment
1UC Berkeley · University of Washington · 3Duke University

May 18, 2026J/K move · Enter open · S save

Semantic Generative Tuning for Unified Multimodal Models

Songsong Yu, Yuxin Chen, Ying Shan, Yanwei Li

Shanghai Jiao Tong University · Tencent ARCLab

Image Segmentation Unified Multimodal Models Cross-Modal Alignment Multimodal Generation Multimodal Understanding

Open paper AskPDF arXiv

Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture. However, prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives. Such a decoupled strategy yields misaligned representation spaces, isolating visual understanding from generation and hindering their mutual reinforcement. This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs. Our empirical investigation reveals that high-level semantic tasks, particularly image segmentation, serve as optimal proxies. Unlike low-level tasks that distract models with texture details, segmentation provides structural semantics that significantly enhance both vision-centric perception and generative layout fidelity. Building upon these insights, we introduce Semantic Generative Tuning (SGT), a novel paradigm that leverages segmentation as a generative proxy to align and synergize multimodal capabilities. Mechanistic analyses further demonstrate that SGT fundamentally improves feature linear separability and optimizes visual-textual attention allocation pattern. Extensive evaluations show that SGT consistently improves both multimodal comprehension and generative fidelity across mainstream benchmarks. Our code is available on the https://song2yu.github.io/SGT/.

Abstract

Unified Multimodal Models

Explore similar work

Semantic Generative Tuning for Unified Multimodal Models

Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models

Reconstruction Alignment Improves Unified Multimodal Models