A child learns which words name which things from roughly 100 million words of input. Vision-language models trained at a comparable scale bind words to referents far less reliably, and we argue that part of the gap is structural: the usual next-token captioning objective has no variable standing for "this word refers to that region," so training applies no direct pressure to ground words. We add such a variable by treating each scene as a small assignment problem between the content words heard and the regions seen, solved as entropic optimal transport under two constraints. First, a word may be assigned to a special NOTHING column instead of any region, which matters because most content words in caregiver speech do not name anything visible. Second, no region may receive more than a bounded share of the assignment mass, a capacity constraint that encodes mutual exclusivity. The alignment step costs about eight Sinkhorn iterations, small next to the vision encoder. In controlled simulation, the naive version of the idea, a row-wise softmax that forces every word onto some region, collapses to 0.000 word-to-picture accuracy under realistic non-reference rates, below the 0.058 of captioning alone, while our two constraints reach 0.075 in that regime and 0.700 on clean input. A one-parameter version of the model reproduces adult cross-situational learning accuracy across three ambiguity conditions (RMSE 0.024). On the SAYCam infant video corpus, adding the alignment loss to a next-token captioning model raises word-to-picture accuracy from 32.4% to 48.7% and localization IoU from 0.18 to 0.38. A contrastive model trained on the same child's data remains stronger on its own benchmark, so our claim is that the loss improves a captioning objective under high non-reference rates, not that it sets a new best.
Figures & tables
Utterance
Content words
What should happen
“look at the ball”
ball
ball → region containing the ball
“are you hungry”
hungry
hungry → Nothing
“oh look, are you hungry, there is the ball”
hungry, ball
hungry → Nothing ; ball → ball region
“where did the ball go”
ball
ball → Nothing (just left frame)
“put the cup on the table”
put, cup, table
cup, table → regions; put → Nothing
“there we go, okay, come on”
come
come → Nothing (filler)
Table 1: Motivating examples: most heard content words do not name anything visible in the paired scene.
Figure 1: The GAVAGAI alignment module. Region proposals and filtered content words form a cost matrix C with an added Nothing column. Capped Sinkhorn iterations give the plan P⋆ , which updates the cross-scene surprisal prior and drives the auxiliary loss added to next-token captioning.
Condition
Clean
Moderate
Realistic
Captioning only
0.392
0.242
0.058
+ Naive alignment (row-softmax)
0.650
0.358
0.000
+ Nothing column only
0.658
0.358
0.017
+ Nothing + exclusivity (ours)
0.700
0.400
0.075
Table 2: Word-to-picture accuracy under controlled simulation, by noise regime (mean of 3 seeds). Under the realistic regime the naive baseline collapses to chance; the full method is the only setting that beats captioning in every regime.
Figure 4Table 5
Figure 4: Schematic illustration of three assignment behaviors: correct referral, non-reference routing to Nothing , and an adjective-modifier failure. These are drawn schematics, not SAYCam frames, since the corpus is access-restricted under Databrary and frames cannot be reproduced here.
Hand
Floor
Ball
Table
Nothing
ball
0.01
0.02
0.88
0.05
0.04
hungry
0.02
0.01
0.01
0.01
0.95
Figure 5: Single-scene soft assignment P for “oh look, are you hungry, there’s the ball”. The target word ball maps to the ball region; the non-referential hungry routes to Nothing .
Variant
W2P acc.
IoU
(a) Capacity schedule bj
Uniform
0.431
0.32
Plausibility-weighted
0.487
0.38
(b) Cross-scene tally scoring
Raw co-occurrence
0.398
0.28
Surprisal-corrected
0.487
0.38
(c) Content-word filter
Closed-class stop-list
0.442
0.33
Learned classifier
0.487
0.38
Table 5: Ablations on SAYCam. Bold marks the default configuration.
Adapting vision-language models in low-resource languages remains challenging, as they often produce fluent text about the wrong objects. This stems from scarce paired data, translation pivots that break alignment, and English-centric pretraining that ignores target-language semantics. We address this with a compute-aware Bengali captioning pipeline trained on LaBSE-verified EN-BN pairs and 110k bilingual-prompted synthetic images. A frozen MaxViT yields stable visual patches, a Bengali-native mBART-50 decodes, and a lightweight bridge links the modalities. Our core novelty is a tri-loss objective: Patch-Alignment Loss (PAL) aligns real and synthetic patch descriptors using decoder cross-attention, InfoNCE enforces global real-synthetic separation, and Sinkhorn-based OT ensures balanced fine-grained patch correspondence. This PAL+InfoNCE+OT synergy tightens real-synthetic patch alignment, reduces spurious matches, and drives strong gains on Flickr30k-1k (BLEU-4 12.29, METEOR 27.98, BERTScore-F1 71.20) and MSCOCO-1k (BLEU-4 12.00, METEOR 28.14, BERTScore-F1 75.40), outperforming strong CE baselines and narrowing the real-synthetic centroid gap by 41%.
Riad Ahmed Anonto, Sardar Md. Saffat Zabin, M. Saifur Rahman
Bangladesh University of Engineering and Technology (BUET), Dhaka, Bangladesh
Vision-language alignment powers open-vocabulary recognition, retrieval, and LVLM grounding, yet natural captions are often underspecified, making similarity brittle and overly confident under paraphrase and omitted details. We aim to learn representations whose matching is stable across caption views and whose confidence reflects how strongly text constrains an image. We propose Text as Partial Constraint (TPC), a core-residual alignment framework that treats multi-view captions as incomplete supervision. It distills a consensus semantic core as the alignment target, learns a single-view core predictor for standard inference with one query, and explicitly discourages vision-language similarity from depending on the orthogonal unsaid residual. An uncertainty-aware contrastive objective further softens alignment when caption views disagree, reducing overconfident updates under weak language constraints. Across zero-shot recognition and adversarial robustness, TPC achieves 81.42/64.05 Top-1 clean/robust accuracy on ImageNet and 76.19/52.03 on an Avg-14 transfer suite, while improving LVLM transfer with 85.16 POPE F1 and 59.57 OKVQA accuracy under an LLaVA-1.5-7B stack. These results suggest that modeling text as a partial constraint is a practical and principled route to more reliable vision-language representations under underspecified language supervision.
Chengzhen Yu, Canran Xiao, Siyuan Ma +1
1Nanyang Technological University (NTU) · 2Shenzhen Campus of Sun Yat-sen University
Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing accounts connect this modality gap to initialization, contrastive dynamics, and information imbalance, while its distributional and pairwise contributions to retrieval remain unresolved. We introduce UOT-Gap, a training-free variational diagnostic that models frozen image and text embeddings with unbalanced entropic optimal transport (UOT). The UOT optimum separates transport, coupling complexity, and marginal mass variation; a complementary pair-aware residual compares observed image-caption pairs with the UOT soft matching. On Flickr8K and COCO-1K with frozen CLIP, OpenCLIP, and SigLIP encoders, caption degradation reduces Flickr8K Recall@1 from 0.559 to 0.003. Across six dataset-model conditions, the pair-aware residual tracks retrieval degradation with mean absolute Spearman 0.973, compared with 0.392 for the mean gap. The association remains stable across five random COCO-1K subsets at 0.954±0.026, with a minimum of 0.943. UOT barycentric updates reduce the transport objective while degrading retrieval, distinguishing geometric objective descent from task improvement. These results establish UOT-Gap as a diagnostic for caption quality, modality alignment, and retrieval robustness.