ChildGaze: A Benchmark Dataset for Collaborative Behavior Understanding in Children
Authors: Sindhuja Penchala, Saketh Reddy Kontham, Prachi Bhattacharjee, S. Nima Mahmoodi, Daniel Fonseca, Sareh Karami, Mehdi Garemani, Sudip Mittal, Shahram Rahimi, Noorbakhsh Amiri Golilarz
Organizations: The University of Alabama, Tuscaloosa, AL, USA · Mississippi State University, Mississippi State, MS, USA
Understanding collaborative behavior in children is important for analyzing social participation, peer interaction, shared attention, and engagement during play and learning activities. Reliable recognition of these cues can support research in child development, educational analysis, and human-centered computer vision. However, estimating where a child is looking does not necessarily reveal whether the child is actively participating in a shared activity. To support this higher-level analysis, we introduce ChildGaze, a child-centered behavioral annotation dataset built on the ChildPlay video collection [1]. ChildGaze introduces two behavioral labels, collaborative and non-collaborative, assigned independently to each child within a frame. The dataset provides face, left-hand, and right-hand bounding boxes for children and adults and organizes the annotations at the row, person, and frame levels. The current release contains 27 annotated video files, 10,641 frames, and 73,268 body-part annotation rows. Annotation reliability was evaluated on 1,187 frames using independent annotations from two annotators. The collaboration labels achieved 93.16% raw agreement and a Cohen's kappa of 0.8631, while bounding-box annotations achieved an overall mean IoU of 0.808. Baseline experiments with pretrained ViT and Swin Transformer models achieved up to 97.44% child-person-level accuracy and 96.80% frame-level accuracy, respectively. These results show that ChildGaze provides a reliable benchmark for studying collaborative behavior in naturalistic child-adult and peer interactions.
Figures & tables
Fig. 1: Representative examples of the frame-level collaboration annotation scheme. Each child receives an individual collaboration label, and the frame label is assigned by majority vote across the children: Yes when the collaborative labels are in the majority, No when the non-collaborative labels are in the majority, and Undetermined when the two counts are equal.
Feature
ChildPlay [ 1 ]
ChildPlay-Hand [ 6 ]
ChildPlay-R [ 14 ]
SCB-Dataset3 [ 12 ]
MMASD [ 10 ]
ChildGaze (Ours)
Multiple children per scene
✓
✓
✗
✓
✗
✓
Face/head bounding boxes
✓
✗
✓
✗
✗
✓
Hand bounding boxes
✗
✓
✗
✗
✗
✓
Action/behavior labels
✗
✓
✗
✓
✓
✓
Person-level annotation
✓
✓
✓
✓
✓
✓
Frame-level annotation
✓
✓
✓
✓
✗
✓
TABLE I: Comparison of ChildGaze with existing child-centered gaze, behavior, and interaction datasets in terms of annotation type, spatial cues, multi-person support, behavioral labeling, annotation granularity, and evaluation methodology.
Label
Annotation Criteria
Collaborative
• Child listens to an adult and participates in the ongoing interaction. • Performs, attempts, or continues a shared task with an adult. • Plays with an object that is part of a shared activity. • Works or plays with another child on the same task or object. • Looks away but remains actively involved in the shared activity.
Non-Collaborative
• Child does not pay attention to the ongoing interaction and does not participate. • Plays or works independently without shared participation. • Attends to an unrelated person, object, or activity. • Is present in the scene but shows no observable participation in the shared interaction.
TABLE II: Collaboration Labeling Criteria Used for Child Annotations in ChildGaze
Fig. 2: Overview of the ChildGaze dataset construction and analysis pipeline. Source ChildPlay [ 1 ] videos are prepared and imported into Labelbox, where separate child and adult annotation classes are defined. Face, left-hand, and right-hand bounding boxes are annotated and propagated across frames, followed by assignment of collaborative or non-collaborative labels to individual children. The resulting annotations are organized at the row, person, and frame levels to preserve both individual and scene-level interaction information. The final annotated dataset supports validation and downstream evaluation of collaborative-behavior recognition using pretrained visual models [ 1 ] .
Property
Yes
No
Total
Dataset Overview
Annotated files
–
–
27
Total frames
–
–
10,641
Total annotation rows
51,155
22,113
73,268
Child Annotations
Child annotation rows
17,654
22,113
39,767
TABLE III: Detailed statistics of the ChildGaze dataset across frame-, person-, and body-part-level annotations.
Annotator B: Yes
Annotator B: No
Annotator A: Yes
644
59
Annotator A: No
38
677
TABLE IV: Inter-annotator agreement for child collaboration labels in the validation subset [ 18 ] .
TABLE VI: Baseline performance and inference efficiency of ViT and Swin Transformer for collaborative-behavior recognition at the frame and child-person levels.
Video recordings of child-caregiver interactions enable investigation of attentional dynamics during naturalistic behavior. Such multimodal recording also allows researchers to examine how attention interacts with action and language use in real time. However, manual annotation of such data is time-consuming. Here, we introduce GazeBehavior Annotation Toolkit, a deep-learning-based toolkit designed to facilitate three key processes in data preprocessing and feature extraction: post-hoc synchronization across multiple videos, semi-automatic annotation of gaze target categories, and categorization of participants' poses and hand actions. This toolkit improves the efficiency and scalability of feature extraction from human egocentric eye-tracking and video data. Such improvement is critical in supporting large-scale and longitudinal investigations of attentional dynamics and naturalistic behavior in human early development.
Iba Baig, Kevin Li, Yanbin Xu +5
Department of Psychology, University of Miami · Northeastern University · Ecole Normale Sup´erieure, PSL University, EHESS, CNRS +1
Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork. While much research has focused on modeling coordination and task execution, the cognitive processes that support such collaboration, particularly Theory of Mind (the ability to infer the mental states of others), remain difficult to study in natural settings. To address this gap, we introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios. The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects, social cues and interactions between agents, as well as agent-object interactions. We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning. By providing temporally aligned, richly annotated multimodal data, CoMind facilitates the development and evaluation of AI systems capable of modeling complex social interactions and reasoning about human behaviors in collaborative environments. Our dataset and benchmarks are made available at https://comind.ethz.ch/.
We introduce a new problem domain for human action recognition: the fine-grained analysis of children's gait behaviors from standard RGB video. We specifically target the ambulatory patterns of children aged 3-17 years. Such behaviors arise naturally in the diagnosis and treatment of several critical developmental and neuromuscular disorders, such as cerebral palsy and hemiplegia. Despite their clinical value, current 3D sensor-based gait analysis systems are expensive, intrusive, and often impractical for young subjects. To address this, we introduce a new dataset comprising over 1,100 high-frame-rate (60 FPS) video sequences from 110 subjects, accompanied by synchronized, anonymized pose sequences. In each session, the child performs a 5-second "walk-around" task, capturing the gait cycle from multiple viewpoints. Crucially, we demonstrate that current state-of-the-art approaches, including gait foundation models and Multimodal Large Language Models (MLLMs), fail to effectively resolve these clinical nuances. We identify the key technical challenges in analyzing these erratic and subtle motor patterns and describe a unified end-to-end framework for decoding fundamental components of pediatric gait. Through comprehensive experimental results, we demonstrate the potential of this dataset to drive novel research questions and establish a rigorous baseline for automated child gait assessment.
Yifan Shen, Boyi Li, Meihuan Huang +12
University of Illinois Urbana-Champaign · PediaMed AI · Shenzhen Children’s Hospital +2