Before Words, Beyond Speech: Evaluating Nonverbal Social Reasoning in Early Childhood
Organizations: Stanford University
Abstract
Children's earliest years are a period of intense social learning, during which meaning is built through gaze, gesture, and physical contact. Yet, this critical aspect of social interaction remains under-evaluated in multimodal video understanding, as existing benchmarks prioritize the conversation-driven interactions of adults. We address this gap with \bench (aturalistic arly-childhood ocial ineractions), the first child-centric benchmark for nonverbal social reasoning. \bench comprises \totalvids manually annotated clips (12s) curated for diversity across naturalistic settings, cultural contexts, and developmental stages. Beyond a lack of data, targeted model improvement in this domain is hindered by coarse evaluation schemas that conflate basic perception with higher-order reasoning. To resolve this, \bench introduces a compositional schema that builds atomic interaction cues into progressive levels of social reasoning: contextual, behavioral, interpersonal, and open-ended descriptions. Extensive analysis of state-of-the-art Vision-Language Models (VLMs) reveals a stark performance collapse: despite strong contextual recognition, models exhibit substantial gaps in decoding precise behavioral and interpersonal cues. We trace these failures to specific biases, including poor consequence grounding and temporal sensitivity. Finally, we ground \bench in real-world applications by evaluating VLMs using domain adaptation strategies. Ultimately, \bench provides a rigorous testbed for advancing socially grounded artificial intelligence in early childhood. Project page: https://laubravo.github.io/nest_website/.
Figures & tables
| Dataset | # Videos | # Q | Source | Children | Non-verbal | Uniform schema | Q/Vid | Demog. |
| Social-IQ 2.0 [ 59 ] | 1,000 | 6,000 | YouTube | ✕ | ✕ | ✕ | 6.0 | ✕ |
| MMDB † [ 43 ] | 160 | – | Lab study | ✓ | ✓ | ✓ | – | ✓ |
| Social Genome [ 34 ] | 272 | 1,486 | YouTube | ✕ | ✕ | ✕ | 5.5 | ✕ |
| R 3 -VQA [ 36 ] | 316 | 5,156 | YouTube | ✕ | ✕ | ✕ | 16.3 | ✕ |
| SIV-Bench [ 22 ] | 2,792 | 8,792 | YouTube, TikTok | ✓ – | ✕ | ✓ – | 3.1 | ✕ |
| Motion2Mind [ 25 ] | 4,730 | – | TV, movies | ✕ | ✓ | ✓ – | – | ✕ |
| Model | Contextual | Behavioral | Interpersonal | All | Open Inter. | |||||||||||
| Size | age | loc. | main act. | Avg | object attn. | shared attn. | event at t | Avg | react. | cause | conseq. | Avg | Avg | sim. (0-3) | recall | |
| Human | – | 87.9 | 90.6 | 67.8 | 82.1 | 88.7 | 93.2 | 95.3 | 92.4 | 79.3 | 86.1 | 94.0 | 86.5 | 87.5 | – | – |
| Open-source | ||||||||||||||||
| LLaVA-NeXT-Video | 7B | 44.5 | 64.7 | 46.3 | 51.8 | 51.5 | 37.7 | 27.5 | 38.9 | 44.1 | 41.0 | 38.7 | 41.3 | 44.0 | 0.8 | 7.3 |
| LLaVA-OV | 7B | 69.4 | 82.6 | 66.1 | 72.7 | 68.4 | 50.2 | 38.4 | 52.3 | 75.3 | 50.2 | 35.3 | 53.6 | 59.6 | 1.0 | 14.1 |
| openPangu-VL | 7B | 61.8 | 85.8 | 68.7 | 72.1 | 77.5 | 48.8 | 53.8 | 60.0 | 67.9 | 53.4 | 52.8 | 58.1 | 63.4 | 0.7 | 8.2 |
| Model | Sim before | Sim | Rec before | Rec |
| Qwen3.5-VL 4B | 1.00 | +0.11 | 0.11 | +0.08 |
| Qwen3.5-VL 9B | 1.06 | +0.15 | 0.11 | +0.12 |
| Qwen3.5-VL 27B | 1.26 | -0.08 | 0.24 | -0.01 |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| ChildPlay | CHILDES | FIND | |
| Segments / questions | 316 / 3,837 | 736 / 8,359 | 156 / 1,828 |
| Source videos / studies | 80 / 1 | 333 / 17 | 58 / 4 |
| Original collection | Public YouTube video, released as a research dataset | Research studies contributed to TalkBank | Parent-coaching intervention program |
| Consent & oversight at collection | ChildPlay dataset terms | Documented by contributing investigators; gated by TalkBank registration | Program’s IRB-approved protocol; caregiver consent at enrollment and for research purposes |
| Our secondary use | Stanford IRB 66624 | Stanford IRB 66624 | Stanford IRB 66624; accessed under the Intervention Program protocol |
| We redistribute | Annotations + metadata | Annotations + metadata | Nothing |
| Category | Artifact | Released | Access |
| Data | Questions + reference answers (12,196) | Full, public segments | Open |
| Multiple-choice options (generated distractors) | Yes | Open | |
| Annotation schema + protocol spreadsheet | Yes | Open | |
| Segment metadata (source video ID, 12 s window) | Yes | Open | |
| ChildPlay / CHILDES videos | No | YouTube / TalkBank registration | |
| Intervention video, 156 segments, 1,828 questions | No | Intervention program |
| Result | Data required | Public-only? |
| Table 2 — model comparison, all reasoning levels | Public subsets | Yes |
| Fig. 5 A — level-wise collapse (Sankey) | Public subsets + FIND | Partial |
| Table 13 and Figure 5 B — blind ablations | Public subsets | Yes |
| Figure 5 B and Figure 17(b) — audio ablations | Public subsets + FIND | Partial |
| Fig. 6 A — temporal grounding by action duration | Public subsets + FIND | Partial |
| Fig. 6 B — no-reaction false positives, dyad bias | Public subsets + FIND | Partial |
| Batch | Contextual | Behavioral | Interpersonal | Open Inter. | Total | ||||||
| age | location | main act. | shared attn. | object attn. | event attn. | conseq. | cause | react. | open inter. | Total | |
| ChildPlay | 657 | 316 | 312 | 275 | 154 | 736 | 378 | 378 | 315 | 316 | 3,837 |
| CHILDES | 1,539 | 737 | 704 | 640 | 352 | 1,592 | 649 | 681 | 729 | 736 | 8,359 |
| FIND | 388 | 154 | 134 | 139 | 56 | 346 | 150 | 150 | 155 | 156 | 1,828 |
| Total | 2,584 | 1,207 | 1,150 | 1,054 | 562 | 2,674 | 1,177 | 1,209 | 1,199 | 1,208 | 14,024 |
| Corpora | # Studies | # Videos | # Participants |
| FIND | 4 | 58 | 43 |
| CHILDES | 17 | 333 | 111 |
| ChildPlay | 1 | 80 | Unknown |
| Model group / reference | Human-written share (%) |
| All models | 22.1 |
| Qwen-family models | 21.0 |
| Other models | 22.9 |
| Uniform choice among distractors | 33.3 |
| Task | Accuracy | Fleiss’ | ||
| Contextual | ||||
| Age | 141 | 0.879 | – | – |
| Location | 64 | 0.906 | – | – |
| Main activity | 59 | 0.678 | – | – |
| Behavioral | ||||
| Shared attention | 59 | 0.932 | – | – |
| Model | Judge | Human [95% CI] | [95% CI] | |
| Recall (%), macro mean over responses | ||||
| Qwen3.5-27B | 31 | 17.4 | 24.7 [16.5, 33.5] | [ , 14.6] |
| GPT-5 mini | 31 | 47.2 | 49.6 [37.7, 61.8] | [ , 11.0] |
| All | 62 | 32.3 | 37.1 [29.2, 45.2] | [ , 10.7] |
| Similarity (0–3), macro mean over responses | ||||
| Qwen3.5-27B | 31 | 1.00 | 1.10 [0.89, 1.32] | [ , 0.42] |
| Model | Contextual | Behavioral | Interpersonal | All | Open Inter. | |||||||||||
| Size | age | loc. | main act. | Avg | object attn. | shared attn. | event at t | Avg | react. (bal.) | cause | conseq. | Avg | Avg | sim. (0-3) | recall | |
| Open-source | ||||||||||||||||
| LLaVA-NeXT-Video | 7B | 41.6 | 68.8 | 41.0 | 50.5 | 41.1 | 27.9 | 24.0 | 31.0 | 44.5 | 39.3 | 37.3 | 40.4 | 40.6 | 0.7 | 4.1 |
| LLaVA-OV | 7B | 63.6 | 87.0 | 59.0 | 69.9 | 53.6 | 43.4 | 38.2 | 45.0 | 72.8 | 46.0 | 36.0 | 51.6 | 55.5 | 1.0 | 8.8 |
| openPangu-VL | 7B | 55.9 | 89.6 | 69.4 | 71.6 | 73.2 | 47.7 | 46.0 | 55.6 | 69.9 | 56.7 | 59.3 | 62.0 | 63.1 | 1.1 | 10.0 |
| InternVL3.5 | 4B | 18.9 | 65.6 | 36.6 | 40.3 | 48.2 | 31.8 | 17.3 | 32.4 | 32.1 | 38.7 | 26.7 | 32.5 | 35.1 | 0.7 | 6.4 |
| Model | Contextual | Behavioral | Interpersonal | All | Open Inter. | |||||||||||
| Size | age | loc. | main act. | Avg | object attn. | shared attn. | event at t | Avg | react. (bal.) | cause | conseq. | Avg | Avg | sim. (0-3) | recall | |
| Open-source | ||||||||||||||||
| LLaVA-NeXT-Video | 7B | 45.4 | 24.7 | 21.3 | 30.5 | 20.5 | 35.9 | 17.0 | 24.4 | 44.6 | 38.7 | 33.9 | 39.1 | 31.3 | 0.6 | 2.3 |
| LLaVA-OV | 7B | 43.8 | 35.7 | 35.0 | 38.2 | 37.3 | 27.9 | 22.4 | 29.2 | 52.2 | 47.0 | 33.8 | 44.3 | 37.2 | 0.5 | 2.8 |
| openPangu-VL | 7B | 45.5 | 58.4 | 38.4 | 47.5 | 39.6 | 32.6 | 23.9 | 32.0 | 17.3 | 45.8 | 43.1 | 35.4 | 38.3 | 0.8 | 1.2 |
| InternVL3.5 | 4B | 43.1 | 50.5 | 37.3 | 43.6 | 35.1 | 28.9 | 16.1 | 26.7 | 52.9 | 40.1 | 39.6 | 44.2 | 38.2 | 0.9 | 0.5 |
| Model | Contextual | Behavioral | Interpersonal | All | Open Inter. | |||||||||||
| Size | age | loc. | main act. | Avg | object attn. | shared attn. | event at t | Avg | react. (bal.) | cause | conseq. | Avg | Avg | sim. (0-3) | recall | |
| Open-source | ||||||||||||||||
| LLaVA-NeXT-Video | 7B | 45.5 | 29.2 | 20.1 | 31.6 | 21.4 | 29.8 | 18.2 | 23.1 | 46.9 | 37.3 | 35.3 | 39.9 | 31.5 | 0.6 | 1.4 |
| LLaVA-OV | 7B | 43.8 | 35.1 | 31.3 | 36.7 | 33.9 | 33.8 | 21.1 | 29.6 | 46.5 | 46.0 | 35.3 | 42.6 | 36.3 | 0.4 | 1.9 |
| openPangu-VL | 7B | 44.7 | 35.1 | 32.8 | 37.5 | 44.6 | 37.7 | 23.7 | 35.4 | 16.3 | 45.3 | 44.0 | 35.2 | 36.0 | 0.9 | 0.1 |
| InternVL3.5 | 4B | 41.3 | 35.7 | 32.1 | 36.4 | 30.4 | 31.8 | 13.9 | 25.4 | 45.8 | 45.3 | 33.3 | 41.5 | 34.4 | 0.9 | 0.0 |
| Similarity | Recall | ||||||||||
| Model | Size (B) | Spearman | MAD | Qwen3 | GPT-5 | Spearman | MAD | Qwen3 | GPT-5 | ||
| Keye-VL 1.5 | 8 | 0.6 | 0.4 | 1.3 | 1.4 | -0.1 | 0.676 | 0.104 | 0.173 | 0.151 | 0.022 |
| Qwen3-VL | 32 | 0.3 | 0.7 | 0.9 | 1.5 | -0.6 | 0.712 | 0.113 | 0.212 | 0.173 | 0.039 |
| Qwen3.5 | 27 | 0.7 | 0.4 | 1.3 | 1.5 | -0.2 | 0.727 | 0.112 | 0.219 | 0.203 | 0.016 |
| Gemini 3.1 Flash Lite | - | 0.7 | 0.4 | 1.1 | 1.4 | -0.3 | 0.683 | 0.1 | 0.171 | 0.151 | 0.02 |
| Average | 0.6 | 0.5 | 1.2 | 1.5 | -0.3 | 0.700 | 0.107 | 0.194 | 0.170 | 0.024 | |
| Size | Source | Sim | Sim | Sim | Rec | Rec | Rec | |
| 4B | Overall | 683 | 1.000 | 1.113 | +0.113 | 0.111 | 0.193 | +0.082 |
| Childplay | 316 | 1.149 | 1.253 | +0.104 | 0.145 | 0.236 | +0.090 | |
| Childes | 357 | 0.880 | 0.994 | +0.115 | 0.081 | 0.156 | +0.074 | |
| FIND | 10 | 0.600 | 0.900 | +0.300 | 0.073 | 0.162 | +0.088 | |
| 9B | Overall | 683 | 1.060 | 1.211 | +0.151 | 0.113 | 0.230 | +0.118 |
| Childplay | 316 | 1.149 | 1.241 | +0.092 | 0.130 | 0.264 | +0.134 |