GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport
Organizations: St Thomas’ College of Engineering and Technology Kolkata, India · Jadavpur University Kolkata, India
Abstract
A child learns which words name which things from roughly 100 million words of input. Vision-language models trained at a comparable scale bind words to referents far less reliably, and we argue that part of the gap is structural: the usual next-token captioning objective has no variable standing for "this word refers to that region," so training applies no direct pressure to ground words. We add such a variable by treating each scene as a small assignment problem between the content words heard and the regions seen, solved as entropic optimal transport under two constraints. First, a word may be assigned to a special NOTHING column instead of any region, which matters because most content words in caregiver speech do not name anything visible. Second, no region may receive more than a bounded share of the assignment mass, a capacity constraint that encodes mutual exclusivity. The alignment step costs about eight Sinkhorn iterations, small next to the vision encoder. In controlled simulation, the naive version of the idea, a row-wise softmax that forces every word onto some region, collapses to 0.000 word-to-picture accuracy under realistic non-reference rates, below the 0.058 of captioning alone, while our two constraints reach 0.075 in that regime and 0.700 on clean input. A one-parameter version of the model reproduces adult cross-situational learning accuracy across three ambiguity conditions (RMSE 0.024). On the SAYCam infant video corpus, adding the alignment loss to a next-token captioning model raises word-to-picture accuracy from 32.4% to 48.7% and localization IoU from 0.18 to 0.38. A contrastive model trained on the same child's data remains stronger on its own benchmark, so our claim is that the loss improves a captioning objective under high non-reference rates, not that it sets a new best.
Figures & tables
| Utterance | Content words | What should happen |
|---|---|---|
| “look at the ball” | ball | ball region containing the ball |
| “are you hungry” | hungry | hungry Nothing |
| “oh look, are you hungry, there is the ball” | hungry, ball | hungry Nothing ; ball ball region |
| “where did the ball go” | ball | ball Nothing (just left frame) |
| “put the cup on the table” | put, cup, table | cup, table regions; put Nothing |
| “there we go, okay, come on” | come | come Nothing (filler) |
| Condition | Clean | Moderate | Realistic |
|---|---|---|---|
| Captioning only | 0.392 | 0.242 | 0.058 |
| + Naive alignment (row-softmax) | 0.650 | 0.358 | 0.000 |
| + Nothing column only | 0.658 | 0.358 | 0.017 |
| + Nothing + exclusivity (ours) | 0.700 | 0.400 | 0.075 |
| Hand | Floor | Ball | Table | Nothing | |
|---|---|---|---|---|---|
| ball | 0.01 | 0.02 | 0.88 | 0.05 | 0.04 |
| hungry | 0.02 | 0.01 | 0.01 | 0.01 | 0.95 |
| Variant | W2P acc. | IoU | |
|---|---|---|---|
| (a) Capacity schedule | Uniform | 0.431 | 0.32 |
| Plausibility-weighted | 0.487 | 0.38 | |
| (b) Cross-scene tally scoring | Raw co-occurrence | 0.398 | 0.28 |
| Surprisal-corrected | 0.487 | 0.38 | |
| (c) Content-word filter | Closed-class stop-list | 0.442 | 0.33 |
| Learned classifier | 0.487 | 0.38 |
Vision-Language Alignment
Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.
See all 370 papers