GMAIL: Generative Modality Alignment for generated Image Learning

Shentong Mo; Sukmin Yun

cs.CVFeb 17, 2026

GMAIL: Generative Modality Alignment for generated Image Learning

Authors: Shentong Mo, Sukmin Yun

Organizations: Department of Machine Learning, CMU, USA · Department of Machine Learning, MBZUAI, UAE · Department of Artificial Intelligence, Hanyang University ERICA, South Korea

Abstract

Generative models have made it possible to synthesize highly realistic images, potentially providing an abundant data source for training machine learning models. Despite the advantages of these synthesizable data sources, the indiscriminate use of generated images as real images for training can even cause mode collapse due to modality discrepancies between real and synthetic domains. In this paper, we propose a novel framework for discriminative use of generated images, coined GMAIL, that explicitly treats generated images as a separate modality from real images. Instead of indiscriminately replacing real images with generated ones in the pixel space, our approach bridges the two distinct modalities in the same latent space through a multi-modal learning approach. To be specific, we first fine-tune a model exclusively on generated images using a cross-modality alignment loss and then employ this aligned model to further train various vision-language models with generated images. By aligning the two modalities, our approach effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks. Our framework can be easily incorporated with various vision-language models, and we demonstrate its efficacy throughout extensive experiments. For example, our framework significantly improves performance on image captioning, zero-shot image retrieval, zero-shot image classification, and long caption retrieval tasks. It also shows positive generated data scaling trends and notable enhancements in the captioning performance of the large multimodal model, LLaVA.

Figures & tables

Figure 1: Illustration of the proposed framework for vision-language tuning with Gen-Real alignment from diffusion models. We propose a method that explicitly aligns a CLIP trained on real images with another CLIP model trained on generated images and then leverages the aligned CLIP to train state-of-the-art vision-language models on generated images ( i.e., Gen-CLIP Flow). For inference with real images, the original CLIP is used to process real images, thereby avoiding discrepancies between real and generated modalities.

Method	B@4 (↑)	METEOR(↑)	CIDEr (↑)	SPICE (↑)	ROUGE-L (↑)	WMD (↑)
ClipCap ( Mokady et al., 2021 )	32.15	27.10	108.35	20.12	–	–
ClipCap + GMAIL (ours)	38.12	31.67	119.53	23.75	56.27	62.16
IFCap ( Lee et al., 2024 )	33.25	28.60	115.27	21.58	51.35	56.72
IFCap + GAMIL (ours)	39.32	32.07	127.86	23.98	59.83	63.51
LLaVA ( Liu et al., 2023 )	39.67	32.38	134.29	24.17	61.36	65.78
LLaVA + GMAIL (ours)	43.26	34.89	146.38	27.23	65.25	71.39

Table 1: Image captioning. We perform fine-tuning on pre-trained ClipCap, IFCap, LLaVA, and LLaMA-3 for image captioning on COCO. We report the standard metrics to evaluate the quality of generated captions. The best results are indicated in bold .

Method	Image-to-Text			Text-to-Image
Method	R@1 (↑)	R@5 (↑)	R@10 (↑)	R@1 (↑)	R@5 (↑)	R@10 (↑)
CLIP ( Radford et al., 2021 )	51.8	76.8	84.3	32.7	57.7	68.2
CLIP + GMAIL (ours)	56.8	80.1	87.2	37.5	62.7	73.2
Long-CLIP ( Zhang et al., 2024 )	57.2	80.8	87.8	40.4	65.9	75.7
Long-CLIP + GMAIL (ours)	62.3	84.1	91.2	45.6	69.8	79.5

Table 2: Zero-shot image retrieval on COCO. We perform zero-shot retrieval on pre-trained CLIP models for image retrieval on the COCO benchmark. We report the image-to-text and text-to-image Recall@1,5,10 metrics to evaluate the quality of retrieved images.

Method	Image-to-Text			Text-to-Image
Method	R@1 (↑)	R@5 (↑)	R@10 (↑)	R@1 (↑)	R@5 (↑)	R@10 (↑)
CLIP ( Radford et al., 2021 )	44.1	68.2	77.0	24.7	45.1	54.6
CLIP + GMAIL (ours)	47.1	71.2	79.6	30.2	50.3	60.5
Long-CLIP ( Zhang et al., 2024 )	47.2	71.5	80.0	33.1	55.6	64.9
Long-CLIP + GMAIL (ours)	51.6	75.3	83.6	39.3	61.5	71.8

Table 3: Zero-shot image retrieval on Flickr30k. We perform zero-shot retrieval on pre-trained CLIP models for image retrieval on the Flickr30k benchmark. We report the image-to-text and text-to-image Recall@1,5,10 metrics to evaluate the quality of retrieved images.

Method	DTD	Stanford Cars	SUN397	Food 101	Aircraft	Oxford Pets	Caltech 101	ImageNet
CLIP ( Radford et al., 2021 )	55.20	77.53	69.31	93.08	32.88	93.33	93.24	75.54
CLIP + GMAIL (ours)	65.26	81.32	75.53	95.21	37.85	95.23	95.57	77.68
SynCLR ( Tian et al., 2024 )	79.90	93.80	76.20	91.60	81.70	93.60	95.30	85.80 (ft)
SynCLR + GMAIL (ours)	83.67	96.56	81.25	96.38	86.75	95.70	98.35	87.95 (ft)

Table 4: Zero-shot image classification. We perform a zero-shot evaluation on pre-trained CLIP models for image classification on eight benchmarks. We report the top-1 accuracy to evaluate the quality of learned representations. The best results are indicated in bold .

Method	Image-to-Text	Text-to-Image
CLIP ( Radford et al., 2021 )	78.2	79.6
CLIP + GMAIL (ours)	85.2	86.7
Long-CLIP ( Zhang et al., 2024 )	94.6	93.3
Long-CLIP + GMAIL (ours)	97.2	96.1

Table 5: Long caption retrieval on ShareGPT4V. We report the image-to-text and text-to-image Recall@1 to evaluate the quality of retrieved images. The best results are indicated in bold .

Alignment	B@4 (↑)	METEOR(↑)	CIDEr (↑)	SPICE (↑)	ROUGE-L (↑)	WMD (↑)
✗	36.15	30.32	115.35	22.95	55.12	61.08
✓	38.12	31.67	119.53	23.75	56.27	62.16

Table 6: Ablation study on Gen-Real Alignment. We perform ablation studies on image captioning from pre-trained CLIP on generated images. The best results are indicated in bold .

Train Data	Image-to-Text			Text-to-Image
Train Data	R@1 (↑)	R@5 (↑)	R@10 (↑)	R@1 (↑)	R@5 (↑)	R@10 (↑)
COCO	47.1	71.2	79.6	30.2	50.3	60.5
CC3M	48.6	73.6	82.2	32.6	52.6	62.3
CC12M	50.9	75.3	84.6	34.9	54.7	64.8

Table 7: Scaling trend of Gen-Real alignment on zero-shot image retrieval on Flickr30k. We perform zero-shot retrieval on models trained from COCO, CC3M, and CC12M on Flickr30k. We report the Recall@1,5,10 metrics to evaluate the quality of retrieved images.

Figure 2: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.

Method	B@4 (↑)	CIDEr (↑)
SigLIP	37.51	117.82
SigLIP + GMAIL (ours)	42.35	125.68

Table 8: Comparison with SigLIP on COCO captioning. Our GMAIL significantly improves SigLIP by effectively addressing the synthetic-real discrepancy. The best results are bold .

Dual Projection	Alignment	Fine-tuning Data	B@4 (↑)	CIDEr (↑)	SPICE (↑)
✗	✗	✗	32.15	108.35	20.12
✗	✗	Synthetic	36.15	115.35	22.95
✓	✓	Synthetic	38.12	119.53	23.75
✗	✗	Real	38.24	119.78	23.86
✓	✓	Real	38.37	119.95	23.98

Table 9: Comparison with models trained on real images. We perform experiments on image captioning from pre-trained CLIP on generated and real images. The best results are indicated in bold .

Method	Accuracy (%)
LLaVA	85.2
LLaVA + GMAIL (ours)	87.6
LLaMA-3	88.5
LLaMA-3 + GMAIL (ours)	91.2

Table 10: Visual question answering on ScienceQA. We report the average accuracy on questions with the image context from the ScienceQA benchmark. The best results are bold .

Method	B@4 (↑)	CIDEr (↑)	SPICE (↑)
LoRA (rank = 2)	36.85	117.62	23.10
LoRA (rank = 4)	38.12	119.53	23.75
LoRA (rank = 6)	37.96	119.12	23.60
Full fine-tuning	37.50	118.95	23.50

Table 11: Ablation study on LoRA rank and full fine-tuning. We perform experiments on image captioning from pre-trained CLIP on generated images. The best results are indicated in bold .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{L}_{align}=-\frac{1}{|\mathcal{B}|}\sum_{(x_{g},x_{r})\in\mathcal{B}}\log\frac{\exp(\text{sim}(f_{g}(x_{g}),f_{r}(x_{r}))/\tau)}{\sum_{x_{r}^{\prime}\in\mathcal{B}}\exp(\text{sim}(f_{g}(x_{g}),f_{r}(x_{r}^{\prime}))/\tau)}

Appendix

Algorithm 1 GMAIL Algorithm: Training and Inference on Generated and Real Images

Dual Projection	Alignment	Synthetic	Training	Training	Memory	FLOPs
Dual Projection	Alignment	Data	Time (hrs)	Steps	Usage (GB)	(G)
✗	✗	✗	8	50k	24	70.2
✗	✗	✓	12	70k	26	85.5
✓	✓	✓	10	60k	28	85.5

Appendix

Table 12: Computational costs comparisons on COCO training. Our GMAIL introduces a slight increase in memory usage but remains more efficient on the convergence training time and steps than the baseline of indiscriminate mixing (gen+real) without alignment.

Dual Projection	Alignment	Synthetic Data	B@4 (↑)	CIDEr (↑)	SPICE (↑)
✗	✗	✗	32.15	108.35	20.12
✗	✗	✓	35.76	113.42	22.63
✓	✓	✓	37.92	117.6	23.42

Appendix

Table 13: Comparison with the same training steps. We compare CLIP trained on real images, fine-tuned CLIP on generated images without alignment, and ours under COCO training with the same training steps. The best results are indicated in bold .

Alignment	Cosine Similarity (↑)
✗	0.52
✓	0.89

Appendix

Table 14: Quantitative similarity metrics comparisons on COCO. We computed the cosine similarity between paired real and generated image embeddings without and with alignment on COCO dataset.

Method	Accuracy (%)
LLaVA	44.7
LLaVA + GMAIL (ours)	48.3

Appendix

Table 15: Multi-modal reasoning and visual-language alignment on MMMU. We evaluated GMAIL’s performance on MMMU benchmark when integrated with LLaVA. We report the average accuracy on questions with the image context. The best results are bold .

Method	B@4 (↑)	CIDEr (↑)	SPICE (↑)
Task2Sim	36.65	115.72	23.02
GMAIL (ours)	38.12	119.53	23.75

Appendix

Table 16: Comparison to Task2Sim. We report a comparison to Task2Sim ( Mishra et al., 2022 ) , which introduced a mixed training approach (joint usage of real and synthetic data). The best results are indicated in bold .

Method	B@4 (↑)	CIDEr (↑)	SPICE (↑)
FLUX (without alignment)	37.20	117.82	23.40
FLUX + GMAIL (ours)	39.54	122.36	24.15

Appendix

Table 17: Image generation with FLUX. We replace a generative model from Stable Diffusion v2 to FLUX and perform experiments on the image captioning task. The best results are indicated in bold .

Figure 3: Qualitative Visualizations of embeddings of real and synthetic images without (Left) and with (Right) alignment. Blue and red dots denote the embeddings for real and synthetic images, respectively. Our GMAIL with alignment significantly reduced the gap between real and synthetic images, with both modalities aligning closely in the latent space.

Figure 4: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.

Figure 5: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.

Figure 6: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.

Figure 7: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.

Figure 8: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.

Figure 9: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards
Jun 25, 2026Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar +5Unified Multimodal Models Multimodal Generation
1Mohamed bin Zayed University of Artificial Intelligence · 4Linköping University · 3Australian National University +1
Let ViT Speak: Generative Language-Image Pre-training
May 1, 2026Yan Fang, Mengcheng Lan, Zilong Huang +7Cross-Modal Learning Vision Transformer
1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University
Unified Text-Image Generation with Weakness-Targeted Post-Training
Jan 7, 2026Jiahui Chen, Philippe Hansen-Estruch, Xiaochuang Han +7Unified Multimodal Models T2I Generation
FAIR at Meta · University of Texas, Austin · Work done at Meta +2

Jun 25, 2026J/K move · Enter open · S save

Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards

Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar, Abdelrahman Shaker+4

1Mohamed bin Zayed University of Artificial Intelligence · 4Linköping University · 3Australian National University+1

Unified Multimodal Models Multimodal Generation Multimodal Foundation Models Image Generation Evaluation Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

Most unified large multimodal models (LMMs) that support both visual understanding and image generation still rely on curated post-training supervision, such as human annotations, preference labels, or external reward models. We ask whether a unified LMM can improve both abilities autonomously using only unlabeled images. We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a Solver that answers and evaluates them, and a Generator that synthesizes images. Training uses only self-derived consistency signals, without human annotations, preference labels, or task-trained external reward/judge models. To stabilize learning, we introduce Solver Token Entropy (STE), a continuous difficulty signal based on token-level prediction uncertainty that remains useful even when sample-level consistency becomes unreliable. For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning. This creates a solver-mediated coupling, where better visual understanding enables more reliable generation assessment and stronger internal training signals. The framework preserves the same role decomposition, reward logic, and training schedule across diffusion-based BLIP3o, rectified-flow BAGEL, and autoregressive VARGPT-v1.1 architectures, requiring only each backbone's native prompting and generation interface. Across eight understanding metrics, our method consistently improves over the corresponding base models. On BAGEL, it achieves a

+3.5\%

absolute gain on MMMU and improves GenEval image generation performance from

82\%

85\%

. Code and models are publicly released.

Abstract

Cross-Modal Alignment

Explore similar work

Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards

Let ViT Speak: Generative Language-Image Pre-training

Unified Text-Image Generation with Weakness-Targeted Post-Training