cs.CLOct 30, 2026

GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

Authors: Rajdeep Pal, Jotiraditya Banerjee, Saptarshi Pani

Organizations: St Thomas’ College of Engineering and Technology Kolkata, India · Jadavpur University Kolkata, India

Abstract

A child learns which words name which things from roughly 100 million words of input. Vision-language models trained at a comparable scale bind words to referents far less reliably, and we argue that part of the gap is structural: the usual next-token captioning objective has no variable standing for "this word refers to that region," so training applies no direct pressure to ground words. We add such a variable by treating each scene as a small assignment problem between the content words heard and the regions seen, solved as entropic optimal transport under two constraints. First, a word may be assigned to a special NOTHING column instead of any region, which matters because most content words in caregiver speech do not name anything visible. Second, no region may receive more than a bounded share of the assignment mass, a capacity constraint that encodes mutual exclusivity. The alignment step costs about eight Sinkhorn iterations, small next to the vision encoder. In controlled simulation, the naive version of the idea, a row-wise softmax that forces every word onto some region, collapses to 0.000 word-to-picture accuracy under realistic non-reference rates, below the 0.058 of captioning alone, while our two constraints reach 0.075 in that regime and 0.700 on clean input. A one-parameter version of the model reproduces adult cross-situational learning accuracy across three ambiguity conditions (RMSE 0.024). On the SAYCam infant video corpus, adding the alignment loss to a next-token captioning model raises word-to-picture accuracy from 32.4% to 48.7% and localization IoU from 0.18 to 0.38. A contrastive model trained on the same child's data remains stronger on its own benchmark, so our claim is that the loss improves a captioning objective under high non-reference rates, not that it sets a new best.

Figures & tables

Explore similar work

CardsList
  1. Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning

    Sep 22, 2025Riad Ahmed Anonto, Sardar Md. Saffat Zabin, M. Saifur RahmanImage CaptioningVLM Adaptation

  2. Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

    Jul 3, 2026Chengzhen Yu, Canran Xiao, Siyuan Ma +1VLM RobustnessContrastive Learning