Children's earliest years are a period of intense social learning, during which meaning is built through gaze, gesture, and physical contact. Yet, this critical aspect of social interaction remains under-evaluated in multimodal video understanding, as existing benchmarks prioritize the conversation-driven interactions of adults. We address this gap with \bench (Naturalistic Early-childhood Social inTeractions), the first child-centric benchmark for nonverbal social reasoning. \bench comprises \totalvids manually annotated clips (∼12s) curated for diversity across naturalistic settings, cultural contexts, and developmental stages. Beyond a lack of data, targeted model improvement in this domain is hindered by coarse evaluation schemas that conflate basic perception with higher-order reasoning. To resolve this, \bench introduces a compositional schema that builds atomic interaction cues into progressive levels of social reasoning: contextual, behavioral, interpersonal, and open-ended descriptions. Extensive analysis of state-of-the-art Vision-Language Models (VLMs) reveals a stark performance collapse: despite strong contextual recognition, models exhibit substantial gaps in decoding precise behavioral and interpersonal cues. We trace these failures to specific biases, including poor consequence grounding and temporal sensitivity. Finally, we ground \bench in real-world applications by evaluating VLMs using domain adaptation strategies. Ultimately, \bench provides a rigorous testbed for advancing socially grounded artificial intelligence in early childhood. Project page: https://laubravo.github.io/nest_website/.
Figures & tables
Figure 1: Overview of the NEST benchmark. We introduce the first dataset (1,208 clips, 14k questions) for evaluating early-childhood non-verbal social reasoning. The example shows how an interaction description can be decomposed into NEST’s four-level schema (contextual, behavioral, interpersonal, open) to isolate the fine-grained details that define the exchange. By mapping this schema to task-specific video VQA questions, NEST overcomes the diagnostic ambiguity of standard aggregated metrics and disentangles a model’s underlying capabilities.
Dataset
# Videos
# Q
Source
Children
Non-verbal
Uniform schema
Q/Vid
Demog.
Social-IQ 2.0 [ 59 ]
1,000
6,000
YouTube
✕
✕
✕
6.0
✕
MMDB † [ 43 ]
160
–
Lab study
✓
✓
✓
–
✓
Social Genome [ 34 ]
272
1,486
YouTube
✕
✕
✕
5.5
✕
R 3 -VQA [ 36 ]
316
5,156
YouTube
✕
✕
✕
16.3
✕
SIV-Bench [ 22 ]
2,792
8,792
YouTube, TikTok
✓ –
✕
✓ –
3.1
✕
Motion2Mind [ 25 ]
4,730
–
TV, movies
✕
✓
✓ –
–
✕
Table 1: Comparison of NEST with the most directly related social and child-interaction video QA benchmarks. NEST is the only benchmark that combines child-centric, non-verbal social interaction with naturalistic multi-source video, reported demographics, and a hierarchical reasoning schema applied uniformly at high density across all videos. ✕ : absent, ✓ : present, ✓ – : partial. Children : extent of child–adult interaction content. Non-verbal : whether the benchmark targets non-verbal social cues without relying on dialogue. Uniform schema : whether the same fixed question schema is applied to every video (vs. questions generated per clip); ✓ – denotes a recurring task/dimension set that is not applied as a single dense schema. Q/Vid : mean questions per video, indicating annotation density over the schema. Demog. : whether participant demographics are reported.
Figure 2: (Left) NEST’s hierarchical task ontology for non-verbal social cues. (Right) Example of dense per-task VQA annotations (avg. ∼12 per clip) used to decompose model performance across the NEST schema.
Figure 3: Annotation procedure. From a large pool of video datasets featuring early childhood interactions, we select videos that support diversity across settings, countries, activities, ages, etc. We emphasize manual annotation (blue arrows) and quality control with semi-automatic steps (green) to arrive at NEST.
Figure 4: NEST Statistics. A) Child age distribution (677/1,208 clips with reported age). B) Recording country distribution (801/1,208 with reported origin). C) Duration distributions for actions (a behavior directed at another participant) and reactions (the responding behavior), covering both adult → child and child → adult pairs.
Model
Contextual
Behavioral
Interpersonal
All
Open Inter.
Size
age
loc.
main act.
Avg
object attn.
shared attn.
event at t
Avg
react.
cause
conseq.
Avg
Avg
sim. (0-3)
recall
Human
–
87.9
90.6
67.8
82.1
88.7
93.2
95.3
92.4
79.3
86.1
94.0
86.5
87.5
–
–
Open-source
LLaVA-NeXT-Video
7B
44.5
64.7
46.3
51.8
51.5
37.7
27.5
38.9
44.1
41.0
38.7
41.3
44.0
0.8
7.3
LLaVA-OV
7B
69.4
82.6
66.1
72.7
68.4
50.2
38.4
52.3
75.3
50.2
35.3
53.6
59.6
1.0
14.1
openPangu-VL
7B
61.8
85.8
68.7
72.1
77.5
48.8
53.8
60.0
67.9
53.4
52.8
58.1
63.4
0.7
8.2
Table 2: Model performance on the public subset of NEST (ChildPlay and CHILDES) . Open interactions are measured in similarity to ground truth (0-3) and recall (%), MCQs in accuracy (%) and balanced accuracy (%) when categorical. Best model and second best model. Darker color indicates higher performance.
Figure 5: A) Sankey plot of the best open-source model Qwen 3.5-27B, showing how correct (purple) and incorrect (gray) questions change across NEST’s levels. B) Model performance by input type relative to the visual input, the results highlight the non-verbal and visual nature of NEST.
Figure 6: A) Per-model accuracy on event-at-time- t questions binned by the question’s action duration (seconds). Gray dots are individual models. n = unique questions per bin. B) Recall and precision on ( Is Reaction? ) questions by aggregated initiator → responder dyad, averaged across all models ( ± SEM). nr = number of reaction segments; nnr = number of no-reaction segments.
Figure 7: Qwen3.5-VL 9B predictions before vs. after finetuning for two held-out NEST items (one ChildPlay, one CHILDES). Each row shows the keyframe, human ground truth, and predictions from the base and finetuned models.
Model
Sim before
Δ Sim
Rec before
Δ Rec
Qwen3.5-VL 4B
1.00
+0.11
0.11
+0.08
Qwen3.5-VL 9B
1.06
+0.15
0.11
+0.12
Qwen3.5-VL 27B
1.26
-0.08
0.24
-0.01
Table 4: Finetuning impact on the NEST-no-overlap open-interaction held-out set (n = 683). Subscript before denotes the un-finetuned base model; Δ columns report the change after finetuning (finetuned − base). We report similarity ( 0 - 3 ) and recall ( 0 - 1 ). A change of +0.12 corresponds to a gain of 12 percentage points.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
ChildPlay
CHILDES
FIND
Segments / questions
316 / 3,837
736 / 8,359
156 / 1,828
Source videos / studies
80 / 1
333 / 17
58 / 4
Original collection
Public YouTube video, released as a research dataset
Research studies contributed to TalkBank
Parent-coaching intervention program
Consent & oversight at collection
ChildPlay dataset terms
Documented by contributing investigators; gated by TalkBank registration
Program’s IRB-approved protocol; caregiver consent at enrollment and for research purposes
Our secondary use
Stanford IRB 66624
Stanford IRB 66624
Stanford IRB 66624; accessed under the Intervention Program protocol
We redistribute
Annotations + metadata
Annotations + metadata
Nothing
Appendix
Table 5: Provenance and oversight.
Category
Artifact
Released
Access
Data
Questions + reference answers (12,196)
Full, public segments
Open
Multiple-choice options (generated distractors)
Yes
Open
Annotation schema + protocol spreadsheet
Yes
Open
Segment metadata (source video ID, 12 s window)
Yes
Open
ChildPlay / CHILDES videos
No
YouTube / TalkBank registration
Intervention video, 156 segments, 1,828 questions
No
Intervention program
Appendix
Table 6: Release manifest. Annotations: CC BY-NC-SA 4.0.
Result
Data required
Public-only?
Table 2 — model comparison, all reasoning levels
Public subsets
Yes
Fig. 5 A — level-wise collapse (Sankey)
Public subsets + FIND
Partial
Table 13 and Figure 5 B — blind ablations
Public subsets
Yes
Figure 5 B and Figure 17(b) — audio ablations
Public subsets + FIND
Partial
Fig. 6 A — temporal grounding by action duration
Public subsets + FIND
Partial
Fig. 6 B — no-reaction false positives, dyad bias
Public subsets + FIND
Partial
Appendix
Table 7: What is reproducible from the public release.
Batch
Contextual
Behavioral
Interpersonal
Open Inter.
Total
age
location
main act.
shared attn.
object attn.
event attn.
conseq.
cause
react.
open inter.
Total
ChildPlay
657
316
312
275
154
736
378
378
315
316
3,837
CHILDES
1,539
737
704
640
352
1,592
649
681
729
736
8,359
FIND
388
154
134
139
56
346
150
150
155
156
1,828
Total
2,584
1,207
1,150
1,054
562
2,674
1,177
1,209
1,199
1,208
14,024
Appendix
Table 8: Question counts per data batch and task category. Categories: Contextual (age, loc., main act.), Behavioral (shared attn., object attn., event attn.), Interpersonal (conseq., cause, react.), and Open Inter. (open-ended).
Figure 8: Overview of dataset characteristics. A ) Number of segments by data source, totaling 1,208 segments across three corpora. B ) Distribution of the number of annotated people per video segment, with most segments containing two individuals (typically a child and caregiver). C ) Setting distribution.
Figure 9: Distribution of correct-answer letters per multiple-choice subtype, organized by question family. Open-ended subtypes are excluded.
Figure 10: Example NEST questions per task subtype. Each row shows the keyframe with bounding boxes overlaid on the people referenced by color, the question presented to the model, and the answer choices with the correct option highlighted. Subtypes are color-banded by task family: Contextual (purple — age, location, main activity), Behavioral (blue — shared attention, object of attention, event at time t) Interpersonal (tan — consequence, cause, reaction), and Open Inter. (pink — open ended).
Corpora
# Studies
# Videos
# Participants
FIND
4
58
43
CHILDES
17
333
111
ChildPlay
1
80
Unknown
Appendix
Figure 11: Data sources and participants.
Figure 12: Clip extraction from long-form videos. We use a gaze estimation model to sample candidate frames where the people are sharing each other’s focus. The clip length is set to 12s and frame rate between clips to 0.2 fps.
Figure 13: Video clip sample. Left: Bounding boxes were overlaid on a keyframe from the video clip so annotators could refer to individuals by color. Right: The full video clip played alongside the keyframe.
Figure 14: Inter-rater agreement for annotations in our dataset, measured on a subset of 64 video segments rated by 4 annotators. We report Krippendorff’s α for categorical variables (Age, Shared Attention) and cosine similarity for free-text descriptions (Location, Activity, Object of Focus, Description of Sharing Focus, Open Interactions).
Figure 15: Identity grounding. We study the two open models Qwen3-VL-8B and Mini-CPM-v4.5 under three conditions: (A) age category groupings, (B) prompting strategy, and (C) input source.
Model group / reference
Human-written share (%)
All models
22.1
Qwen-family models
21.0
Other models
22.9
Uniform choice among distractors
33.3
Appendix
Table 9: Human-written distractor share among incorrect responses. Each consequence question contains one human-written and two generated distractors. Uniform selection among incorrect options gives a human-written share of 33.3%.
Task
n
Accuracy
Fleiss’ κ
noverlap
Contextual
Age
141
0.879
–
–
Location
64
0.906
–
–
Main activity
59
0.678
–
–
Behavioral
Shared attention
59
0.932
–
–
Appendix
Table 10: Human evaluation results. For each (video, question) pair, we first average correctness across the raters who answered it, then average those per-question scores across questions within a task (macro-averaging at the question level), so each question contributes equally regardless of how many raters answered it. Inter-rater agreement (Fleiss’ κ ) is reported for the interpersonal tasks. n denotes the number of unique (video, question) pairs evaluated; noverlap is the size of the multi-rater subset used for κ . Reaction reports balanced accuracy under a binary framing.
Model
n
Judge
Human [95% CI]
Δ [95% CI]
Recall (%), macro mean over responses
Qwen3.5-27B
31
17.4
24.7 [16.5, 33.5]
+7.2 [ −0.4 , 14.6]
GPT-5 mini
31
47.2
49.6 [37.7, 61.8]
+2.5 [ −5.7 , 11.0]
All
62
32.3
37.1 [29.2, 45.2]
+4.8 [ −0.8 , 10.7]
Similarity (0–3), macro mean over responses
Qwen3.5-27B
31
1.00
1.10 [0.89, 1.32]
+0.10 [ −0.21 , 0.42]
Appendix
Table 11: Human validation of the open-interaction metrics. We report human and automated judge scores for Qwen3.5-27B and GPT-5 mini ( n=31 responses each), alongside combined results. Brackets indicate 95% confidence intervals; Δ denotes human − judge.
Figure 16: Comparing performance across models for cause and consequence questions.
Model
Contextual
Behavioral
Interpersonal
All
Open Inter.
Size
age
loc.
main act.
Avg
object attn.
shared attn.
event at t
Avg
react. (bal.)
cause
conseq.
Avg
Avg
sim. (0-3)
recall
Open-source
LLaVA-NeXT-Video
7B
41.6
68.8
41.0
50.5
41.1
27.9
24.0
31.0
44.5
39.3
37.3
40.4
40.6
0.7
4.1
LLaVA-OV
7B
63.6
87.0
59.0
69.9
53.6
43.4
38.2
45.0
72.8
46.0
36.0
51.6
55.5
1.0
8.8
openPangu-VL
7B
55.9
89.6
69.4
71.6
73.2
47.7
46.0
55.6
69.9
56.7
59.3
62.0
63.1
1.1
10.0
InternVL3.5
4B
18.9
65.6
36.6
40.3
48.2
31.8
17.3
32.4
32.1
38.7
26.7
32.5
35.1
0.7
6.4
Appendix
Table 12: Model performance on the FIND subset of NEST . Open interactions are measured in similarity to ground truth (0-3) and recall (%), all other tasks are in accuracy (%). Best model and second best model. Darker color indicates higher performance.
Model
Contextual
Behavioral
Interpersonal
All
Open Inter.
Size
age
loc.
main act.
Avg
object attn.
shared attn.
event at t
Avg
react. (bal.)
cause
conseq.
Avg
Avg
sim. (0-3)
recall
Open-source
LLaVA-NeXT-Video
7B
45.4
24.7
21.3
30.5
20.5
35.9
17.0
24.4
44.6
38.7
33.9
39.1
31.3
0.6
2.3
LLaVA-OV
7B
43.8
35.7
35.0
38.2
37.3
27.9
22.4
29.2
52.2
47.0
33.8
44.3
37.2
0.5
2.8
openPangu-VL
7B
45.5
58.4
38.4
47.5
39.6
32.6
23.9
32.0
17.3
45.8
43.1
35.4
38.3
0.8
1.2
InternVL3.5
4B
43.1
50.5
37.3
43.6
35.1
28.9
16.1
26.7
52.9
40.1
39.6
44.2
38.2
0.9
0.5
Appendix
Table 13: Model performance on the public subsets of NEST without visual input . Open interactions are measured in similarity to ground truth (0-3) and recall (%), all other tasks are in accuracy (%). Best model and second best model. Darker color indicates higher performance.
Model
Contextual
Behavioral
Interpersonal
All
Open Inter.
Size
age
loc.
main act.
Avg
object attn.
shared attn.
event at t
Avg
react. (bal.)
cause
conseq.
Avg
Avg
sim. (0-3)
recall
Open-source
LLaVA-NeXT-Video
7B
45.5
29.2
20.1
31.6
21.4
29.8
18.2
23.1
46.9
37.3
35.3
39.9
31.5
0.6
1.4
LLaVA-OV
7B
43.8
35.1
31.3
36.7
33.9
33.8
21.1
29.6
46.5
46.0
35.3
42.6
36.3
0.4
1.9
openPangu-VL
7B
44.7
35.1
32.8
37.5
44.6
37.7
23.7
35.4
16.3
45.3
44.0
35.2
36.0
0.9
0.1
InternVL3.5
4B
41.3
35.7
32.1
36.4
30.4
31.8
13.9
25.4
45.8
45.3
33.3
41.5
34.4
0.9
0.0
Appendix
Table 14: Model performance on the FIND subset of NEST without visual input . Open interactions are measured in similarity to ground truth (0-3) and recall (%), all other tasks are in accuracy (%). Best model and second best model. Darker color indicates higher performance.
Figure 17: Impact of audio on task performance. (a) Quantitative accuracy gaps across task subtypes. (b) Qualitative examples across modality conditions.
Similarity
Recall
Model
Size (B)
Spearman
MAD
Qwen3
GPT-5
Δ
Spearman
MAD
Qwen3
GPT-5
Δ
Keye-VL 1.5
8
0.6
0.4
1.3
1.4
-0.1
0.676
0.104
0.173
0.151
0.022
Qwen3-VL
32
0.3
0.7
0.9
1.5
-0.6
0.712
0.113
0.212
0.173
0.039
Qwen3.5
27
0.7
0.4
1.3
1.5
-0.2
0.727
0.112
0.219
0.203
0.016
Gemini 3.1 Flash Lite
-
0.7
0.4
1.1
1.4
-0.3
0.683
0.1
0.171
0.151
0.02
Average
0.6
0.5
1.2
1.5
-0.3
0.700
0.107
0.194
0.170
0.024
Appendix
Table 15: Open question model performance by judge. We measure inter-judge agreement for similarity and recall between Qwen3-VL-32B and GPT-5 using Spearman rank correlation, Mean Absolute Distance (MAD), and the overall delta.
Figure 18: Comparing performance across models by action-reaction dyad groups for cause and consequence interpersonal questions.
Figure 19: Accuracy stratified by MUSIQ [ 21 ] absolute. A higher MUSIQ score reflects a better perceived quality. It accounts for blur, noise, compression artifacts, resolution, etc.
Figure 20: Behavioral question (event at time t) accuracy by child age category. Visual condition (blue) substantially outperforms blind (purple) and random chance (25%) across all age groups, with no significant variation by age. Error bars show standard error of the mean (SEM) across models.
Figure 21: Model accuracy across task subtypes and geographic regions. Bars show average accuracy across models ( ± standard error of the mean (SEM)) for North America, Europe, and Asia, grouped by task category (Contextual, Behavioral, Interpersonal). The dashed line indicates chance level.
Figure 22: Model accuracy across task subtypes and recording settings. Bars show average accuracy across models ( ± SEM) for Home ( n=718 ), Classroom ( n=228 ), and Lab ( n=263 segments), grouped by task category; the dashed line indicates chance. Setting is largely nested within data source (Lab is 237/263 CHILDES 4) and the Classroom and Lab subsets are smaller, so differences are descriptive rather than causal.
Size
Source
n
Sim base
Sim ft
Δ Sim
Rec base
Rec ft
Δ Rec
4B
Overall
683
1.000
1.113
+0.113
0.111
0.193
+0.082
Childplay
316
1.149
1.253
+0.104
0.145
0.236
+0.090
Childes
357
0.880
0.994
+0.115
0.081
0.156
+0.074
FIND
10
0.600
0.900
+0.300
0.073
0.162
+0.088
9B
Overall
683
1.060
1.211
+0.151
0.113
0.230
+0.118
Childplay
316
1.149
1.241
+0.092
0.130
0.264
+0.134
Appendix
Table 16: Finetuning impact on the NEST-no-overlap open-interaction held-out set, broken out by data source. base is the un-finetuned base model and ft is the finetuned model. Sim is the MMBench-Video judge score (0-3); Rec is claim-level recall (0-1).
Figure 23: Prompts for the application of NEST on parent-coaching videos.
Training Multimodal Large Language Models for audio-visual social understanding is a crucial step toward embodied social intelligence. Chain-of-thought (CoT) reasoning has become the dominant approach, with HumanOmniV2 and its IntentBench benchmark as a prominent reference point. In this context, we report three findings. First, IntentBench is highly noisy: ∼7% of questions are broken and ∼23% are trivially answerable without the video input. We remove the affected questions and release Intentbench-Prime. Second, current reasoning approaches are expensive and surprisingly ineffective. A simple Vanilla SFT baseline matches or outperforms existing reasoning methods across three benchmarks at a fraction of the cost, establishing it as an essential baseline for evaluating novel fine-tuning techniques. Third, our analysis reveals that substantial priors can be learned solely from the text modality and that using a textual caption instead of the video yields performance on par with Vanilla SFT. These surprising findings reveal the limitations of current MLLMs when it comes to social understanding. IntentBench-Prime, Vanilla SFT model, and code are publicly available.
Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera +1
Inria at Univ. Grenoble Alpes, CNRS, LJK, France · University of Twente, The Netherlands
Recent advances in Large Multimodal Models (LMMs) have greatly improved video understanding, yet their ability to reason about human-centered social situations remains limited. Existing benchmarks typically rely on videos with a single observed trajectory, making it difficult to determine whether models truly understand social dynamics or merely exploit recurring narrative patterns. We introduce SocialReasonBench, a video multiple-choice QA benchmark for evaluating socially grounded reasoning in scenarios derived from interactive narratives. Built from gameplay videos of Detroit: Become Human, the benchmark leverages branching storylines where player decisions lead to alternative social outcomes that can be checked against the game's own script, flowchart, and recorded branches. We develop a multi-agent curation pipeline that localizes socially meaningful clips, grounds answer labels in game-state signals, and generates theory-guided questions with diagnostic distractors. SocialReasonBench covers seven reasoning dimensions, including intent recognition, emotional empathy, moral dilemma, counterfactual reasoning, and causal antecedent. Experiments on contemporary LMMs show that models perform reasonably well on basic social understanding but struggle with counterfactual and causal reasoning. Further ablation and diagnostic error analyses reveal that models often depend on incomplete modality cues and fall into reasoning traps such as visual shortcuts, highlighting a gap between observable event recognition and deeper reasoning over latent social states.
Zheyu Huang, Zijing Shi, Haozhe Luo +4
AAII, University of Technology Sydney · Northeastern University · Fudan University +1
Understanding social interactions requires reasoning over subtle non-verbal cues, yet current multimodal large language models (MLLMs) often fail to identify who interacts with whom in multi-person videos. We introduce GRASP, a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events. GRASP contains 290K question--answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze--gesture reasoning, together with GRASP-Bench for evaluation. Unlike prior resources that focus on either isolated cues or high-level social QA, GRASP builds questions from identity-consistent gaze trajectories, deictic gestures, and their joint compositions into social events. Moreover, we propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction. Experiments show that SGR improves performance on GRASP-Bench while maintaining zero-shot performance on related social video QA benchmarks.
Junho Kim, Xu Cao, Houze Yang +6
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Amazon AGI +1