cs.CVOct 16, 2026

The Collection of a Real-World, Large-Scale Dataset for Automated Skill Evaluation in Central Line Dressing Change

Authors: Ummey Tanin, Shannan K. Hamlin, Hsin-Mei Chen, Peizhu, Qian

Organizations: Pam · University of Houston Houston, TX, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Central line-associated bloodstream infections (CLABSIs) remain a major source of preventable morbidity and mortality in healthcare settings. Proper execution of the central line dressing change (CLDC) procedure is essential for infection prevention, yet current approaches to competency assessment rely primarily on manual ob- servation by clinical educators, making evaluation time-intensive, difficult to scale, and limited in consistency and granularity. These challenges highlight the need for AI-assisted skill evaluation methods that can support more objective, reproducible, and scalable assessment of clinical performance. However, progress in this area has been constrained by the lack of broad-scale, clinically grounded datasets for developing and evaluating such methods. In this work, we present the collection process of a large-scale video dataset for CLDC skill evaluation, comprising 365 recordings of registered nurses collected during the 2026 annual competency exam at Houston Methodist Hospital. Each video is paired with subtask-level temporal annotations and clinical performance assessments spanning 17 subtasks of the procedure. We describe the data collection and curation pipeline, including pilot testing of record- ing configurations, annotation protocol design, and post-collection processing. We further establish the feasibility of video-based skill assessment through benchmark experiments with spatiotemporal deep learning models. This dataset provides a foundation for future research on automated clinical skill assessment, feedback generation, and AI-assisted training in nursing education.

Figures & tables

Figure 1. Overview of the proposed AI-assisted pipeline for central line dressing change evaluation. Recorded evaluation videos are segmented into task-specific clips, scored automatically at the subtask level, and used to generate performance feedback. describe the image.
Figure 2. Example of a central line dressing, illustrating the dressing, central lines, and securement device.
Paper / ModelDataModelTrain Accuracy
Funke et al. (2019)RGBI3D + TSN95.1–100%
Replicated Model 1RGBI3D + TSN95.45%
Replicated Model 2RGBC3D + TSN90.12%
Table 1. Benchmark replication results on JIGSAWS.
Activity IDActivity Description
A01_RemoveContentsRemove contents of the pocket
A02_HeadCoverMaskHead cover and masks
A03_PerformHandHygienePerform hand hygiene
A04_NonSterileGlovesNon-sterile gloves
A05_DressingRemovalDetachol to remove dressing
A06_RemoveStatlockRemove statlock
Table 2. Finalized subtasks for the CLDC dataset
CameraResolutionFOVFPS
RealSense D435i1280 ×\times 72087 ∘×58∘{}^{\circ}\times 58^{\circ}30
GoPro HERO134K156 ∘60
4K Camcorder4KUltra-wide30
Table 3. Cameras evaluated during pilot recording sessions.
Figure 3. Pilot recordings in the lab using angled top-down center view.
Figure 4. Other camera mounting positions tested during pilot recordings: (left) frontal-left top-down corner; (right) corner overhead diagonal.
Figure 5. Printed data annotation sheet used during live recording sessions at Houston Methodist.
Figure 6. Training session for the data collection team prior to clinical deployment.
Figure 7. Real-time data annotation and nursing educator feedback during data collection. Featured nursing educator: Mr. Jay Hooker.
Figure 8. Participant demographics.
Figure 9. Example frames from the dataset. Top: sponge application by Participant 309. Bottom: dressing removal by Participant 313.
More in this topic

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Explore similar work

CardsList
Aug 31, 2026

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.

Chaohui Dang, Zheheng Jiang, James Glasbey +3

Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1

Surgical Video UnderstandingMultimodal Model Evaluation
AskPDF
Aug 18, 2026

Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery

Persistent shortages in the surgical workforce and inherent limitations of traditional training methods highlight the necessity of automated, data-driven approaches in surgical education. This study addresses these challenges by introducing a novel, explainable AI-powered framework for automated skill assessment, specifically focusing on cataract surgery. We present the world's largest dataset of cataract surgery videos, comprising 2,000 recordings. Additionally, we propose an AI-powered analytical framework that employs advanced computer vision and signal-processing techniques to automatically evaluate surgical videos to derive objective, quantitative performance indicators that complement or potentially replace subjective scoring methods. A significant advantage of our framework over previous methods lies precisely in its explainability of outputs, elevating it beyond merely an opaque skill classification tool. Through experimental analysis of 83 cataract surgery videos, we demonstrate that the automatically computed metrics exhibit strong correlations with expert-based subjective evaluations, achieving up to 87% accuracy in surgical skill assessment. Each metric was individually examined, and expert surgeons provided subjective ratings using the newly introduced Capsulorhexis Skill Assessment System (CSAS). These subjective assessments were compared with ten objective motion-based metrics extracted through our framework. The results indicated a robust correlation between subjective ratings and automated indicators, underscoring the framework's capacity to accurately model surgical expertise.

Mohammad Javad Ahmadi, Hamid D. Taghirad

Applied Robotics and AI Solutions (ARAS), Faculties of Electrical and Computer Engineering, K.N. Toosi University of Technology, Tehran, Iran.

Automated EvaluationSurgical Video UnderstandingExplainable Artificial Intelligence
AskPDF
May 16, 2026

AI-Assisted Competency Assessment from Egocentric Video in Simulation-Based Nursing Education

Assessing learner competency in clinical simulation requires expert observation that is time-intensive, difficult to scale, and subject to inter-rater variability. Vision-language models have emerged as a promising tool for understanding complex visual behavior. In this work, we investigate whether visual observations can provide educationally meaningful signals for competency assessment through a three-stage framework that (1) extracts action timelines from egocentric nursing simulation video using frozen visual encoders and few-shot learning, (2) derives sequence-level features and per-session recognition metrics, and (3) relates these to instructor-rated competency. Across 22 densely annotated sessions (3.8 hours, 493 actions), a frozen DINOv2 backbone with HMM Viterbi decoding achieves 57.4% MOF in leave-one-out 1-shot recognition. Surprisingly, we observe a negative trend between recognition accuracy and competency (rho = -0.524, p = 0.012 for mIoU), robust to six confound controls: more competent students produce diverse, harder-to-classify workflows, while simple sequence features show no such relationship. Per-item analysis identifies patient safety protocols and team communication as the expected behaviors most reflected in this pattern, and process model comparisons reveal that higher-competency students exhibit more protocol-consistent action transitions. These findings suggest that recognition accuracy may complement predicted action timelines as a pedagogically informative signal in automated competency assessment.

Hanchen David Wang, Yilin Liu, Madison J. Lee +4

Vanderbilt University

Educational AssessmentEducational TechnologyEgocentric Video Understanding
AskPDF