Scaling cross-situational word learning on children's naturalistic experiences
Authors: Michael C. Frank, Alvin W. M. Tan, Khai Loong Aw, Robert Z. Sparks, Victor Moreno, Grace Keene-Lee, Meesha R. Ryan, Tarun Sepuri, Jane Yang, Virginia A. Marchman, Daniel L. K. Yamins, Bria Long
Organizations: Department of Psychology, Stanford University · Department of Computer Science, Stanford University · Department of Psychology, Harvard University · Department of Psychology, University of Wisconsin, Madison · Department of Psychology, University of Oregon · Department of Psychology, University of California, San Diego
How do children learn the meanings of the simple, concrete words that form a majority of their early vocabulary? Statistical information about many word meanings can be recovered from the gradual accumulation of information about the co-occurrence of words with their referents, a proposal known as ``cross-situational learning''. Contrastive vision--language models allow this learning strategy to be tested on egocentric video data capturing children's early real-life experiences with visual and linguistic inputs. Initial successes of this method on small datasets provide a proof-of-concept, but vision models struggle to learn robust visual representations when trained on these same egocentric videos. How dependent is cross-situational word learning performance on the quality of the model's visual representations? And can cross-situational models scale to larger and more diverse datasets? To address these questions, we trained models on the BabyView corpus, including 2,441 hours of longitudinal egocentric video from 48 children. Models trained on this dataset learned object--label mappings, but performance was far higher when they used internet-pretrained visual representations as opposed to when visual representations were learned solely from the developmental video data. Further, the majority of the learning signal was carried by a small proportion of utterances that described a visible referent. Models learned lexical representations that showed recognizable human-like semantic structure, but model performance showed substantially different trajectories over training experiences than those observed in children. Cross-situational learning is a viable word learning strategy for early noun vocabulary, but constructing developmentally plausible models will require understanding how to learn high-quality visual representations from naturalistic data.
Figures & tables
Figure 1: Our training pipeline. (A) The 2026.1 release of the BabyView corpus (N=51) was transcribed and downsampled to a set of frame–utterance pairs, then filtered on language (N=48). (B) Our VLM pipeline learned correspondences between the words in each utterance and the visual representation of subregions of the frame (as described by the visual encoding model). Primary evaluations were 4-alternative forced-choice trials constructed from an independent set of photos [ 31 ] . (C) Each pair was automatically annotated for referential alignment (whether the utterance referred to something visible in the frame).
Figure 2: Model performance on the Konkle 4AFC evaluation, plotted by number of frame–utterance training pairs (log scale). (A) Performance split by vision encoder. OTS refers to models trained on internet-scale data, while BV refers to models trained on BabyView frames; numbers refer to model parameters. Points (mean ± 1 SD) are computed across model runs with different random seeds and subsamples of data. Sigmoid fits are 3 parameter logistic fits (lower asymptote fixed at 25%), with 95% confidence bands computed by Monte Carlo simulation. Black diamonds show data from Vong et al. (2024), open, and Vong & Lake (2026), filled. (B) Scaling data for the OTS-304M and BV-304M models, trained only on filtered, referentially-aligned pairs. Blue and orange reproduce the unfiltered scaling curves from panel A. Arrows show how many times more unfiltered data are necessary to match the filtered model.
Figure 3: Assessing the lexicon learned by the top performing model (using the OTS-304M encoder). (A) t-distributed stochastic neighbor embedding (t-SNE) plot of lexical representations. Dots represent the set of 921 nouns whose top-3 nearest-neighbor similarity exceeds a threshold and that appeared 50 or more times in the training data. Object nouns appearing on the MB-CDI are colored by their semantic category on the instrument; words appearing on the Konkle evaluation are circled. (B) Mean category cohesion gap for each MB-CDI category: mean within-category cosine minus mean cosine to all other nouns. Gray bars show the 2.5%-97.5% band for a label-permuted null distribution. Error bars on points show between-seed SD. (C) Spearman correlation between model relatedness (cosine similarity) and human relatedness judgments for noun-noun pairs, plotted by training pairs (log scale). Correlations are computed across vocabulary present in the vocabulary at each scale. Lines show the OTS-304M model (solid) vs. a language-only word2vec model trained on the same data (dashed). Dotted line shows the partial correlation of the model, controlling for the word2vec correlation.
Figure 4: Scaling performance of models by developmental exposure time (years), assuming 4000 waking hours/year; faded lines indicate extrapolation. (A) Konkle evaluation fits and imputed proportion correct for children based on data from English-learning children in Wordbank. Evaluation is on the 40 words overlapping between the MB-CDI and the Konkle set. Error bar below the chance line is the 95% confidence interval on the per-child median speech rate, indicating uncertainty on the input quantity to time mapping (horizontal) based on cross-child variation. (B) Model curves for the LEVANTE vocabulary evaluation, overlaid with LEVANTE children’s proportion correct on these same words.
Table S1: Training runs underlying all figures. Every run uses a frozen encoder, region-MIL head (unless noted), InfoNCE, 20 epochs, and dev-117 epoch selection; quantities are designed training pairs (full corpus = 1,686,105).
Figure S1: Histogram of the number of hours recorded by each child included in the full 2026.1 BabyView release.
Figure S2: (A) Histogram of the target child’s age at the time of each recording. (B) Cumulative hours of recording per child for each child in the corpus.
Figure S3: (A) Histogram of total utterances per hour. (B) Utterances per hour plotted by child’s median age.
Figure S4: Comparison of models with and without multiple-instance learning (MIL) across sub-regions of the frame. (A) Mean-pooled vs. MIL accuracies for all encoders and three training data scales. (B) Difference scores for each model, scale and seed.
Figure S5: (A) Similarity of target images to their category prototype for each test set (Konkle, held-out BabyView frames) and encoder; similarity is measured by accuracy in categorization judgments in a 4AFC. (B) Word learning performance for the full corpus plotted by prototype 4AFC.
Figure S6: In-domain evaluation on held-out BabyView frames (from non-English videos). (A) Accuracy for each encoder plotted by number of training pairs. Error bars show between-seed standard deviation. (B) Advantage for OTS vs. BV-trained models (average across model sizes) for both Konkle and held-out BabyView frames.
Figure S7: Konkle evaluation performance by dataset diversity. Results come from OTS-304M runs with matched total number of frame-utterance pairs, but with different numbers of children’s data contributing. Error bars show between-seed standard deviation.
Figure S8: Comparison of models with and without multiple-instance learning (MIL) across temporal windows ( ±5 s, for a total of 11 frames). (A) Midpoint frame vs. MIL Konkle accuracies for all encoders and four training data scales. (B) Difference scores for each model, scale and seed.
Figure S9: (A) Log count histogram of Gemini referential alignment ratings, with retained pairs shown in purple. (B) Proportion aligned pairs by child, ordered by rate.
Figure S10: Full results for aligned data across all encoders. Plotting conventions are as in Figure 2 .
Figure S11: Accuracy for each encoder, plotted by data subsample. Whole corpus accuracy is compared to: accuracy on the aligned subsample alone; a size-matched random subsample; a size-matched subsample of unaligned pairs, matched to the aligned set’s joint distribution of target-noun and utterance-length; the full corpus with the aligned subsample removed; and the full corpus with the matched unaligned subsample removed. Aligned scaling arms train on the N highest-rated pairs; the alignment-selection controls train on the full set.
Figure S12: tSNE results for all six encoders. Plotting conventions are as in Figure 3 A.
Figure S13: Spearman correlation with human relatedness judgments (nouns only) from models trained on only referentially-aligned pairs (solid lines) compared with all pairs (dashed lines).
Figure S14: (A) Spearman correlation with human relatedness judgments for noun-noun pairs; results for all encoders. (B) Results across all words. Plotting conventions are as in Figure 3 C.
Figure S15: (A) Alignment (negative correlation) between children’s item difficulty and OTS-304M model item accuracy, across model training scales. Alignments are computed on the 74/177 Konkle items in the test and dev set with a corresponding MB-CDI item, for production and comprehension. Child difficulties are derived from a Rasch model. Because model accuracy reaches ceiling, correlations decline at larger scales. Numbers show the total number of items in the model vocabulary at each scale. The gray dashed line shows the proportion of items with model accuracy >90% . (B) Model accuracy correlation with comprehension difficulty at 300k training pairs. (C) Model accuracy correlation with production difficulty at 300k training pairs.
Figure S16: Full results on the LEVANTE-bench vocabulary assessment for all embedding models. Solid lines show results across all items; dashed lines show results from only those items that were in the models’ vocabulary for each training dataset size.
Figure S17: Comparison of accuracy for the OTS-304M model on the corpus with key-child utterances removed (blue) and a matched random draw (gray).
Children acquire language grounding with remarkable robustness from limited visuo-linguistic input in ways that surpass today's best large multimodal models. Recent research suggests current vision-language models (VLMs) trained on curated web data fail to generalize to the sparse, weakly-aligned egocentric streams produced by wearable devices, embodied agents, and infant head-cams -- and no fixed evaluation pipeline exists for measuring progress on this regime. We train VLMs on datasets with varying degrees of semantic alignment between visual and linguistic inputs, including naturalistic infant and adult egocentric videos, and evaluate them with a comprehensive suite spanning multimodal language grounding and unimodal vision and language tasks. At the core of this suite is Machine-DevBench, a corpus-grounded benchmark of lexical and grammatical competence, automatically generated from the model's training vocabulary across logarithmic frequency bins to eliminate the train/eval mismatch and low statistical power of prior developmental benchmarks. Our results show that current VLM paradigms hinge on the tight semantic alignment of curated data and fail to exploit the weakly-aligned signal that dominates naturalistic egocentric input -- the very regime in which humans thrive. To motivate progress, we introduce the EgoBabyVLM Challenge to drive the development of models capable of grounded language learning from the kind of naturalistic data that human infants experience.
Dongyan Lin, Phillip Rust, Angel Villar Corrales +19
Meta Superintelligence Labs · Stanford University · Meta Reality Labs +2
Children learn the meanings of words from a continuous, temporally structured stream of egocentric experience. Recent work shows that neural networks can also learn word-referent mappings from a child's egocentric video recordings, but they cycle through the shuffled data for hundreds of epochs, contrasting with how children actually encounter their environment. We introduce BabyCL, a continual multimodal learning framework that processes the SAYCam dataset in a single chronological pass, combining streaming visual representation learning with an image-text contrastive objective. BabyCL combines a multi-stage temporal segmentation of the stream with a dual replay buffer that independently manages visual and multimodal histories, and it is jointly trained with three contrastive losses on a shared backbone. Under a matched optimization budget, BabyCL outperforms streaming learning baselines on the SAYCam Labeled-S 4AFC benchmark, substantially narrowing the gap to an upper bound of offline training. Ablations show that the gains are robust to the length of the online temporal segmentation window and the eviction rule of the replay buffer. Together, these results show that meaningful word-referent mappings can emerge under training conditions much closer to a child's actual experience.
Xiaoyang Jiang, Yanlai Yang, Kenneth A. Norman +2
Agentic Learning AI Lab, New York University · Department of Psychology, Princeton University
Learning grounded word meaning from natural experience requires resolving two ambiguities in infant-view recordings: when the named referent appears and where it is in a cluttered frame. In SAYCam-style data, caregiver speech is sparse and weakly synchronized with egocentric video, so single-frame contrastive pairing yields noisy positives in which the intended object is absent or entangled with distractors. We propose BabyMind, an object-first bias for child-view contrastive learning under sparse, noisy supervision. BabyMind extracts candidate object embeddings using an offline mask-based region interface, links candidates across a short utterance-centered window into lightweight object files via tracking, and aligns utterances to bags of object files with a prototype-space multiple-instance contrastive objective. Track-coherence and global-object agreement regularizers stabilize learning and transfer object-file structure into the global frame embedding used at evaluation. On SAYCam-S, BabyMind improves Labeled-S 15 forced-choice accuracy by +2.6 points over CVCL and yields consistent gains on in-vocabulary out-of-distribution benchmarks. Code is available at https://github.com/sathiiii/BabyMind.
Sathira Silva, Abrham Kahsay Gebreselasie, Muhammad Umer Sheikh +3
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Weizmann Institute of Science, Rehovot, Israel