cs.CVNov 9, 2026

Before Words, Beyond Speech: Evaluating Nonverbal Social Reasoning in Early Childhood

Authors: Marie Amale Huynh, Laura Bravo-Sánchez, Lauren Klein, Nick Haber, Philip Fisher, Serena Yeung-Levy

Organizations: Stanford University

Abstract

Children's earliest years are a period of intense social learning, during which meaning is built through gaze, gesture, and physical contact. Yet, this critical aspect of social interaction remains under-evaluated in multimodal video understanding, as existing benchmarks prioritize the conversation-driven interactions of adults. We address this gap with \bench (N\textbf{N}aturalistic E\textbf{E}arly-childhood S\textbf{S}ocial inT\textbf{T}eractions), the first child-centric benchmark for nonverbal social reasoning. \bench comprises \totalvids manually annotated clips (∼\sim12s) curated for diversity across naturalistic settings, cultural contexts, and developmental stages. Beyond a lack of data, targeted model improvement in this domain is hindered by coarse evaluation schemas that conflate basic perception with higher-order reasoning. To resolve this, \bench introduces a compositional schema that builds atomic interaction cues into progressive levels of social reasoning: contextual, behavioral, interpersonal, and open-ended descriptions. Extensive analysis of state-of-the-art Vision-Language Models (VLMs) reveals a stark performance collapse: despite strong contextual recognition, models exhibit substantial gaps in decoding precise behavioral and interpersonal cues. We trace these failures to specific biases, including poor consequence grounding and temporal sensitivity. Finally, we ground \bench in real-world applications by evaluating VLMs using domain adaptation strategies. Ultimately, \bench provides a rigorous testbed for advancing socially grounded artificial intelligence in early childhood. Project page: https://laubravo.github.io/nest_website/.

Figures & tables

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?

    Aug 13, 2026Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera +1Supervised Fine-TuningMultimodal Large Language Models

  2. SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos

    Aug 31, 2026Zheyu Huang, Zijing Shi, Haozhe Luo +4Video Question AnsweringMultimodal Reasoning

  3. GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions

    May 15, 2026Junho Kim, Xu Cao, Houze Yang +6Multimodal QAMultimodal Model Evaluation