Infant Care Video Dataset for Classification of Interventions Using Transformers

Igor Bogdanov; James Green

cs.CVAug 24, 2026

Infant Care Video Dataset for Classification of Interventions Using Transformers

Authors: Igor Bogdanov, James Green

Organizations: Department of Systems and Computer Engineering Carleton University Ottawa, Canada

Abstract

Healthcare documentation in the neonatal intensive care unit (NICU) presents significant challenges, with nurses spending approximately 25% of their time on record-keeping, while up to 60% of interventions remain undocumented. Motivated by the need to detect interventions from video automatically, we present the Infant Care Video Dataset (ICVD), a collection of 4,144 videos spanning 12 simulated intervention classes designed for developing automated documentation systems. Our manikin-based approach systematically varies conditions, such as camera angle and clinician skin tone, while ensuring privacy compliance. Using video transformer architectures (TimeSformer and MotionFormer), we establish strong baseline performance (93.97% and 93.17% top-1 accuracy) among the 12 infant care classes. Our ablation study comparing temporal models with a framewise approach (23.17% accuracy) demonstrates a 70.80% performance gap, validating the need for temporal modeling. The ICVD provides a foundation for developing automated documentation systems to reduce clinical burden in neonatal care environments and improve existing practices.

Figures & tables

Fig. 1: Camera configuration used for dataset recording. Eight cameras were positioned around the infant manikin in the crib. The cameras included several smartphone models and baby monitors to capture diverse perspectives and recording quality, representing real-world monitoring conditions.

Panel of Figure 2 — Fig. 2: Sample frames from a diaper-change intervention showing: (a) preparing, (b) removing, (c) starting diaper application, and (d) applying a fresh diaper.

Fig. 3: Systematic variability factors in the ICVD dataset. Left: Two background conditions. Center: Five hand covering types. Right: Two performers and eight camera perspectives.

Fig. 4: Sample frames showcase the visual diversity across intervention types, camera angles, hand coverings, and background conditions. Top row (left to right): blanket-cover with bare hands, putting baby to crib , feeding with green gloves. Middle row: diaper-remove with blue gloves on light sheet, pacifier-give with white gloves on dark sheet, diaper-change with dark blue gloves. Bottom row: wiping-face with blue gloves, blanket-uncover on white sheet, wiping-face with light blue gloves on dark sheet. Note the systematic variation in hand coverings (bare hands, white, green, dark blue, light blue gloves) and background conditions (dark and light sheets).

Split	Number of videos
Total	4144
Train	2892
Validation	624
Test	631

TABLE I: Dataset split for model training

Classes	Index	Light	Dark	Total
Giving Pacifier	7	172	162	334
Changing Diaper	10	160	145	305
Taking Pacifier	5	163	158	321
Feeding	6	163	152	315
Removing from crib	1	165	172	337
Replacing infant in crib	8	166	173	339

TABLE II: Number of videos by class and background type

Model	Top-1 (%)	Top-5 (%)	Precision (%)	Recall (%)	F1 (%)
TSF	93.97	100.00	94.11	93.29	92.92
MF	93.17	99.84	93.73	92.32	91.29
FW	23.17	65.87	23.08	23.00	19.45

TABLE III: Model Performance Metrics On The Test Set

Fig. 5: Per-class accuracy comparison between models. Temporal models achieve near-perfect accuracy on most classes, unlike the framewise model.

Fig. 6: Normalized confusion matrices comparing models. Left: TimeSformer shows near-perfect classification for most classes. Middle: MotionFormer exhibits similar patterns. Right: The framewise model shows substantial confusion across classes.

Fig. 7: Intervention classification performance metrics. Both temporal models substantially outperform the framewise model.

Healthcare

Healthcare AI develops and evaluates models and systems for clinical care, medical research, public health, and healthcare operations.

See all 621 papers

DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis
Sep 26, 2026Jiahui Li, Yutong Guo, Nan Yang +3Medical Diagnosis Medical VLMs
University of Georgia, Athens, USA · Beijing Luhe Hospital, Beijing, China
InfantFace: Detecting infant faces in neonatal clinical environments
Jun 18, 2026Abdullah Bin-Obaid, Maria M. Cobo, Rebeccah Slater +2Healthcare Object Detection
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK · Department of Paediatrics, University of Oxford, Oxford, UK · Universidad San Francisco de Quito USFQ, Colegio de Ciencias Biologicas y Ambientales, Quito, Ecuador
Respiratory Status Detection with Video Transformers
Mar 22, 2026Thomas Savage, Evan MadillVision Transformer Temporal Video Understanding
University of Pennsylvania · University of Washington

Sep 26, 2026J/K move · Enter open · S save

DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis

Jiahui Li, Yutong Guo, Nan Yang, Wenzhan Song+2

University of Georgia, Athens, USA · Beijing Luhe Hospital, Beijing, China

Medical Diagnosis Medical VLMs Clinical Decision-Making

Open paper AskPDF arXiv

Diagnosing a patient from video requires more than recognizing the sign: a vision-language model must turn what it sees into hypotheses, questions and tests. DynamicDx evaluates each step in 71 neurological consultations across 11 sign categories, linking authentic patient videos to confirmed diagnoses and fixed charts built from the same case reports, so that every model queries the same evidence. Across five such models, video improves accuracy by 9.9-22.5 percentage points over blind input, but neither recognition alone nor temporal order explains the gain: the cause is usually missing from the model's video-only differential diagnosis even when the sign is recognized, and shuffling the frames produces no reliable accuracy loss. Instead, a trajectory replay traces most of the gain to the investigation results the video prompts. Evidence acquisition is the bottleneck: supplying the decisive investigations raises accuracy to 73.2-93.0%. Two interventions act on it. A post-trained 4B video describer improves sign descriptions, especially from a short, densely sampled segment, and source-clean literature retrieval expands initial hypotheses; both bring the tests a model orders closer to those the treating clinicians documented and, through them, raise accuracy. For video-based diagnosis, seeing better helps when it leads to asking better.

Abstract

Healthcare

Explore similar work

DynamicDx: Evaluating Evidence Acquisition in Video-Based Diagnosis

InfantFace: Detecting infant faces in neonatal clinical environments

Respiratory Status Detection with Video Transformers