Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations

Vincenzo Marco De Luca; Antonio Longa; Andrea Passerini

cs.LGAug 24, 2026

Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations

Authors: Vincenzo Marco De Luca, Antonio Longa, Andrea Passerini

Organizations: University of Trento Trento, Italy · UiT the Arctic University of Norway Tromsø, Norway

Abstract

Modeling team interactions in high-stakes environments such as operating rooms is critical for understanding how coordination, communication, and individual behaviors shape team performance and safety outcomes. Existing datasets in this domain are often fragmented across modalities, annotation schemes, and formats, limiting their ability to support integrated analyses of real-world collaborative processes. We address this limitation by introducing an extended multimodal dataset for surgical team interaction analysis, built from real operating room recordings. Starting from an existing corpus, we construct an analysis-ready version of the data by providing speaker diarization, transcripts, and multi-level annotations capturing team performance, interaction processes, and individual characteristics. Team performance is assessed using a standardized surgical teamwork evaluation protocol, while interaction quality and individual attributes are annotated through structured rating schemes covering collaboration, group dynamics, and non-technical skills. To further support the study of coordination breakdowns and performance variability, we introduce counterfactual annotations that describe plausible alternative team outcomes in the presence of observed interaction failures, enabling analysis of how specific behavioral patterns may relate to different trajectories of team performance. In addition, we provide structured temporal and relational representations designed to support computational modeling of teamwork processes and the design of AI-assisted collaborative systems. The dataset is designed to support the study of how individual actions, interaction patterns, and team-level processes jointly contribute to team outcomes in surgical settings, providing a unified resource for analyzing collaborative behavior in high-stakes domains.

Figures & tables

Property	Description
Procedures	Robotic knee replacement
Recordings	17 full-length + 22 short clips
Duration	$\sim$ 90 minutes per full procedure
Timepoints	92,983 (1 FPS synchronized)
Data Volume	$\sim$ 500 GB
Visual Modalities	5 RGB-D, 3 RGB, 1 low-exposure RGB

Table 1. Key characteristics of the MM-OR dataset.

Term	Definition
Communication	Quality and quantity of information exchanged among members of the team
Coordination	Management and timing of activities and tasks
Cooperation and Back-up behavior	Assistance provided among members of the team, supporting others, and correcting errors
Leadership	Provision of directions, assertiveness, and support among members of the team
Team monitoring and Situational awareness	Team observation and awareness of ongoing processes

Table 2. Definitions of team-level social construct introduced in OTAS

Figure 1. Violin plot of the distribution of samples across different phases, in particular we highlight the duration of each sample while

n

represents the number of samples per each class.

Team construct	Definition	# Items
Communication	Maintaining mutual understanding among team members and managing the flow of dialogue to ensure continuity.	2
Joint information processing	Sharing information to optimize group decision-making and ability to reach an agreement among team members.	2
Coordination	Dividing tasks effectively among team members, managing time efficiently during teamwork, and coordinating technical skills to complete the task.	3
Interpersonal coordination	Promoting healthy interaction between team members to foster good relationships.	1
Motivation	Individual team member orientation towards completing individual tasks.	1

Table 3. Definitions of interaction-level social constructs introduced in HMT

Team construct	Definition	# Items
Situation Awareness	Gathering information from the environment and the team, correctly interpreting the information to understand the situation and anticipating future states and potential problems	3
Decision Making	Considering options and consequences before deciding, making appropriate decisions and communicating them clearly, implementing decisions and monitoring their outcomes	3
Communication and Teamwork	Sharing information clearly with team members, ensuring team members understand tasks and plans, Organizing and coordinating tasks within the team	3
Leadership	Encouraging and supporting team members, maintaining performance under stress, and ensuring tasks progress efficiently and safely	3

Table 4. Definitions of individual social constructs introduced in NOTSS

Team construct	Definition	# Items
Adaptability (A)	The ability of a team to adjust roles, strategies, and resource allocation in response to environmental changes ( 46 ) .	5
Back-up behavior (BB)	The ability of team members to anticipate the needs of others and provide timely support when required ( 46 ) .	3
Mutual performance monitoring (MPM)	The ability to comprehend the team environment and continuously monitor team members’ performance to ensure effective coordination ( 46 ) .	5
Team leadership (TL)	The capacity to direct, coordinate, and guide the activities of team members toward shared objectives ( 46 ) .	2
Team orientation (TO)	The capacity of team members to prioritize collective goals and outcomes over individual interests ( 46 ) .	2
Closed-loop communication (CC)	The process by which team members explicitly direct, acknowledge, and confirm messages to ensure accurate receipt and shared understanding ( 46 ) .	5

Table 5. Core behavioral components of the Big Five in Teamwork framework.

Figure 2. Violin plot of the distribution of samples for the OTAS rating schema, considering the average score provided by annotators and normalizing these scores in the range 1-5.

Figure 3. Violin plot of the distribution of samples for the HMT rating schema, considering the average score provided by annotators.

Figure 4. Violin plot of the distribution of samples for the NOTSS rating schema, considering the average score provided by annotators.

Figure 5. Violin plot of the distribution of samples for the BFTQ rating schema, considering the average score provided by annotators.

Figure 6. Violin plot of the distribution of samples for the SYMLOG and GLIS rating schemas, considering the average score provided by annotators.

Figure 7. Violin plot of the distribution of the measures provided for the proposed counterfactual rating schemas.

Figure 8. Empirical cumulative distribution function of the duration of the proposed counterfactual

Figure 9. Average F1-macro over the different OTAS dimensions. For each modeling paradigm, the results reported correspond the best-performing model.

Paradigm	Model	Team		Individual				Interaction
Paradigm	Model	OTAS	Phase duration	NOTSS	BFTQ	SYMLOG	GLIS	HMT
SNN	MLP	59.8 $\pm$ 2.2	60.5 $\pm$ 1.2	51.5 $\pm$ 1.0	42.5 $\pm$ 1.1	48.5 $\pm$ 0.9	64.5 $\pm$ 0.8	41.5 $\pm$ 1.2
SNN	RF	60.2 $\pm$ 0.8	61.8 $\pm$ 1.8	52.1 $\pm$ 1.2	38.5 $\pm$ 0.7	42.1 $\pm$ 0.6	65.2 $\pm$ 0.7	44.1 $\pm$ 1.0
TNN	LSTM	62.3 $\pm$ 1.6	62.8 $\pm$ 0.9	56.2 $\pm$ 1.4	52.2 $\pm$ 1.3	55.2 $\pm$ 0.9	64.8 $\pm$ 1.1	48.2 $\pm$ 1.1
	GRU	61.8 $\pm$ 1.8	63.0 $\pm$ 0.8	58.1 $\pm$ 1.1	55.5 $\pm$ 1.2	57.3 $\pm$ 0.8	62.6 $\pm$ 1.3	51.2 $\pm$ 1.3
	MHA	63.8 $\pm$ 2.0	62.4 $\pm$ 1.3	60.4 $\pm$ 1.2	56.2 $\pm$ 1.0	61.3 $\pm$ 0.8	64.9 $\pm$ 1.0	53.0 $\pm$ 1.2

Table 6. Combined results measuring F1-macro according to a LOGO protocol, with experiments run over 10 different random seeds. Tasks are organized into three levels: Team (OTAS, Phase duration), Individual (NOTSS, BFTQ, SYMLOG, GLIS), and Interaction (HMT). Bold indicates best, underline indicates second best.

Figure 10. Test F1-macro computed according to the LOGO protocol for each team member, averaged across ten runs with different random seeds.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Aug 24, 2026

From Multimodal Observation to Interpretable Suggestions: Counterfactual Time-Expanded Relational Modeling of Surgical Teams

In surgery, patient safety is threatened not only by technical issues but also by poor teamwork. However, existing surgical AI-based solutions focus mainly on visual workflow and technical execution, neglecting the modeling of team interactions and missing opportunities to actively support clinicians in improving their teamwork skills. To address this gap, we propose a tempo-relational framework for modeling surgical team dynamics from multimodal observations. By leveraging Time-Expanded graphs, the approach captures both relational structure and temporal evolution, achieving strong expressivity while remaining robust in the low-data regime typical of surgical settings. Beyond prediction, such modeling enables the generation of efficient, interpretable, and actionable suggestions for clinicians. More specifically, we generate suggestions via a counterfactual procedure that identifies minimal yet structured changes in individual behaviors and interaction patterns associated with improvements in team performance. Experiments with simulated surgical procedures show that our approach improves predictive performance in diverse behavioral and interaction goals while offering meaningful insights into team dynamics. This work advances surgical AI beyond outcome-driven prediction towards a socially grounded, team-centric, and actionable paradigm to better understand and support the development of team skills in surgical settings.

Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni +1

University of Trento Trento, Italy · UiT the Arctic University of Norway Tromsø, Norway

Topic pending

Ask PDF

May 5, 2026

Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs

Surgical team performance arises from complex interactions between technical execution and non-technical skills, including communication and coordination dynamics. However, current surgical AI systems predominantly model visual workflow signals, lacking structured representations of intraoperative team interactions over time. We propose a real-time actionable approach for modeling surgical team dynamics using time-expanded interaction graphs, where team members are modeled as time-indexed nodes and communication exchanges define directed edges. This spatio-temporal expansion enables dynamic interaction modeling, while allowing efficient inference with a static graph neural network. The model predicts procedural efficiency as the deviation from the expected duration and supports real-time deployment. Beyond prediction, we perform a counterfactual analysis to identify minimal changes in communication structure and interpretable behavioral variables associated with improved predicted outcomes. Experiments on recorded surgical procedures show that structured modeling of team interactions improves early identification of prolonged interventions and provides coherent, actionable explanations. This work advances surgical AI toward real-time, team-aware, and actionable decision support in the operating room.

Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni +1

a University of Trento · b UiT the Arctic University of Norway

Clinical Outcome Prediction Dynamic Graph Learning Spatiotemporal GNNs

Ask PDF

Jul 27, 2026

Surgical Re-enactment for Operating Room Workflow Datasets

The introduction of new technologies, such as surgical robots, is driving the vision of a connected, smart operating room (OR). However, realizing this vision requires a deep understanding of surgical workflows, which relies on realistic datasets capturing the actions of all OR personnel from both full room and surgical field perspectives. Acquiring such data in real ORs is prohibitively challenging due to factors such as ethics committee approvals, limited space for camera installation, and sterility regulations preventing the use of tracking markers. We present a step-by-step methodology for re-enacting complete surgical procedures in a reconstructed OR. This approach enables the creation of repeatable and annotatable workflow datasets for training activity recognition models, generating scene graphs, and formalizing surgical process models. Developed for robot-assisted ophthalmic surgery, our methodology combines expert consultation, structured workflow formalization, OR reconstruction, role-based training, real OR observation, and iterative recording with post-take debriefing. We provide concrete recommendations to allow other research groups to seamlessly adopt this methodology for their own surgical domains.

Jana Nina Friedrich, Andrea Karin Maria Ross, Angelo Henriques +3

Medical Autonomy and Precision Surgery Lab, TUM University Hospital, Munich, Germany · TUM School of Medicine and Health, Department of Ophthalmology, TUM University Hospital, Munich, Germany · TUM School of Computation, Information, and Technology, TUM, Munich, Germany +1

Synthetic Data Generation Surgical Video Understanding

Ask PDF

Abstract

Surgical Video Understanding

Explore similar work

From Multimodal Observation to Interpretable Suggestions: Counterfactual Time-Expanded Relational Modeling of Surgical Teams

Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs

Surgical Re-enactment for Operating Room Workflow Datasets