Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning

Riad Ahmed Anonto; Sardar Md. Saffat Zabin; M. Saifur Rahman

cs.CVSep 22, 2025

Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning

Authors: Riad Ahmed Anonto, Sardar Md. Saffat Zabin, M. Saifur Rahman

Organizations: Bangladesh University of Engineering and Technology (BUET), Dhaka, Bangladesh

Abstract

Adapting vision-language models in low-resource languages remains challenging, as they often produce fluent text about the wrong objects. This stems from scarce paired data, translation pivots that break alignment, and English-centric pretraining that ignores target-language semantics. We address this with a compute-aware Bengali captioning pipeline trained on LaBSE-verified EN-BN pairs and 110k bilingual-prompted synthetic images. A frozen MaxViT yields stable visual patches, a Bengali-native mBART-50 decodes, and a lightweight bridge links the modalities. Our core novelty is a tri-loss objective: Patch-Alignment Loss (PAL) aligns real and synthetic patch descriptors using decoder cross-attention, InfoNCE enforces global real-synthetic separation, and Sinkhorn-based OT ensures balanced fine-grained patch correspondence. This PAL+InfoNCE+OT synergy tightens real-synthetic patch alignment, reduces spurious matches, and drives strong gains on Flickr30k-1k (BLEU-4 12.29, METEOR 27.98, BERTScore-F1 71.20) and MSCOCO-1k (BLEU-4 12.00, METEOR 28.14, BERTScore-F1 75.40), outperforming strong CE baselines and narrowing the real-synthetic centroid gap by 41%.

Figures & tables

Figure 1: Pipeline. MSCOCO captions are translated to Bengali and LaBSE-verified, then used to render a synthetic image per caption, forming (real, synthetic, BN caption) triplets. A frozen MaxViT feeds a Linear+LN bridge into an mBART-50 decoder. Last- $k$ cross-attention provides text-conditioned patch weights for pooling. Training uses CE on real images and PAL on pooled real–synthetic descriptors, with InfoNCE and Sinkhorn OT as lightweight regularizers; only the bridge and decoder are updated.

Method	B1	B2	B3	B4	METEOR	BERT-F1
Our model	49.14	29.64	18.11	12.29	27.98	71.20
ViT + Bangla GPT-2 [ 17 ]	26.84	12.32	4.68	3.21	26.62	57.34
CNN + LSTM [ 9 ]	28.03	15.45	6.51	3.98	22.57	70.98
Google PaliGemma (3B) [ 2 ]	35.40	22.19	15.81	11.28	32.01	77.96
mBLIP mT0-XL [ 12 ]	36.73	24.17	14.45	9.53	28.92	76.92
InternVL 2.5 (4B) [ 4 , 5 , 6 , 11 ]	18.71	5.64	3.14	2.13	23.35	69.98

Table 1: Flickr30k-1k (out-of-domain): Comparison of our model (PAL+InfoNCE+OT) with established captioners. B1–B4 are BLEU-1 to BLEU-4. Higher is better.

Dataset	B1	B2	B3	B4	METEOR	BERT-F1
MSCOCO-1k	38.61	25.11	17.43	12.00	28.14	75.40
Flickr30k-1k	49.14	29.64	18.11	12.29	27.98	71.20

Table 2: Single-model performance (PAL + InfoNCE + OT) on MSCOCO-1k (in-domain) and Flickr30k-1k (out-of-domain). B1–B4 are BLEU-1 to BLEU-4.

Method	B1	B2	B3	B4	METEOR	BERT-F1
CE (Real-only)	36.57	23.81	12.37	5.80	24.96	68.38
CE (Real+Syn)	40.34	24.46	12.93	5.91	24.51	68.69
CE + InfoNCE	43.60	25.67	11.98	7.50	24.99	69.70
CE + InfoNCE + OT	42.56	24.00	11.85	7.52	26.22	70.21
PAL	46.53	27.80	16.30	10.19	27.29	70.97
PAL + InfoNCE	46.86	27.96	16.57	10.34	27.49	70.69

Table 3: Ablation results on Flickr30k-1k. Higher is better.

$\lambda$	$\tau$	$\rho$	BLEU-4	METEOR	BERT-F1
0.3	0.7	0.5	2.77	12.8	62.2
0.5	0.7	0.1	2.50	17.5	63.7
0.5	1.0	0.5	2.37	10.9	61.3
0.5	1.3	0.5	2.42	15.8	61.9

Table 4: PAL hyperparameter sensitivity on a reduced-data 6-epoch run. Relative trends guide our chosen configuration.

Figure 2: PAL improves optimization and alignment. Left: PAL reduces total and CE loss. Right: PAL reduces the real–synthetic centroid gap from 0.361 to 0.213 and MMD-RBF from 0.013 to 0.002.

Image Captioning

Image captioning generates natural-language descriptions of visual content, with research focused on accuracy, coverage, grounding, and evaluation.

See all 94 papers

TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
Apr 13, 2026Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis +16Vision-Language Pretraining Vision-Language Alignment
Google DeepMind · xAI · Epsilon Health +1
Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning
Jul 3, 2026Chengzhen Yu, Canran Xiao, Siyuan Ma +1VLM Robustness Contrastive Learning
1Nanyang Technological University (NTU) · 2Shenzhen Campus of Sun Yat-sen University
PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models
Aug 22, 2026Jihyung Ko, Eunji Jung, Hyeongsub Kim +4Image Captioning Vision-Language Models
Seoul National University, Korea · LG CNS, Korea · Konkuk University, Korea +1

Apr 13, 2026J/K move · Enter open · S save

TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis, Kaifeng Chen+15

Google DeepMind · xAI · Epsilon Health+1

Vision-Language Pretraining Vision-Language Alignment Masked Image Modeling VLM Distillation

Open paper AskPDF arXiv

Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment -- surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/ .

Abstract

Image Captioning

Explore similar work

TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models