From Multimodal Observation to Interpretable Suggestions: Counterfactual Time-Expanded Relational Modeling of Surgical Teams

Vincenzo Marco De Luca; Antonio Longa; Giovanna Varni; Andrea Passerini

cs.LGAug 24, 2026

From Multimodal Observation to Interpretable Suggestions: Counterfactual Time-Expanded Relational Modeling of Surgical Teams

Authors: Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, Andrea Passerini

Organizations: University of Trento Trento, Italy · UiT the Arctic University of Norway Tromsø, Norway

Abstract

In surgery, patient safety is threatened not only by technical issues but also by poor teamwork. However, existing surgical AI-based solutions focus mainly on visual workflow and technical execution, neglecting the modeling of team interactions and missing opportunities to actively support clinicians in improving their teamwork skills. To address this gap, we propose a tempo-relational framework for modeling surgical team dynamics from multimodal observations. By leveraging Time-Expanded graphs, the approach captures both relational structure and temporal evolution, achieving strong expressivity while remaining robust in the low-data regime typical of surgical settings. Beyond prediction, such modeling enables the generation of efficient, interpretable, and actionable suggestions for clinicians. More specifically, we generate suggestions via a counterfactual procedure that identifies minimal yet structured changes in individual behaviors and interaction patterns associated with improvements in team performance. Experiments with simulated surgical procedures show that our approach improves predictive performance in diverse behavioral and interaction goals while offering meaningful insights into team dynamics. This work advances surgical AI beyond outcome-driven prediction towards a socially grounded, team-centric, and actionable paradigm to better understand and support the development of team skills in surgical settings.

Figures & tables

Figure 1. Overview of the interaction modeling pipeline. Top row: multimodal time-series are segmented into fixed temporal windows (15 seconds). Middle row: for each window, a snapshot interaction graph

G_{t}

is built, in which nodes represent team members and edges encode broadcast verbal communication. Node features integrate interpretable paralinguistic (eGeMAPS), text, position, and human–tool interaction (relation with objects). Bottom row: snapshot graphs are connected through identity-preserving temporal edges into a Time-Expanded Graph

G^{TE}

, enabling joint modeling of team members’ coordination and longitudinal individual dynamics across time.

Activation	Control	Dominance	Behavioral Class
Low	Low	Low	Withdrawn / Disengaged
Low	Low	High	Restrained Conflict
Low	High	Low	Calm–Cooperative
Low	High	High	Quiet Authority
High	Low	Low	Agitated / Over-aroused
High	Low	High	Dominant–Aggressive

Table 1. Behavioral classes defined by combining three paralinguistic features extracted from the eGeMAPS set.

Paradigm	Model	Modeling Dimension			Avg. F1-macro (%)	F1-macro Measures (%)
		Feature	Time	Topology	Teamwork	Comm	Coord	Coop	Lead	Sit. Aw
SNN	MLP	✓	✗	✗	59.8 $\pm$ 2.2	60.1 $\pm$ 2.0	58.7 $\pm$ 2.3	60.3 $\pm$ 2.5	59.2 $\pm$ 1.8	60.7 $\pm$ 2.2
SNN	RF	✓	✗	✗	60.2 $\pm$ 0.8	61.0 $\pm$ 0.9	59.5 $\pm$ 0.5	60.0 $\pm$ 0.6	60.3 $\pm$ 1.0	60.3 $\pm$ 1.2
TNN	LSTM	✓	✓	✗	62.3 $\pm$ 1.6	63.0 $\pm$ 2.4	61.8 $\pm$ 1.4	62.5 $\pm$ 1.8	62.0 $\pm$ 1.1	62.3 $\pm$ 1.1
	GRU	✓	✓	✗	61.8 $\pm$ 1.8	62.2 $\pm$ 2.1	61.5 $\pm$ 1.7	61.7 $\pm$ 2.3	61.8 $\pm$ 1.7	61.8 $\pm$ 1.2
	MHA	✓	✓	✗	63.8 $\pm$ 2.0	64.0 $\pm$ 2.1	63.5 $\pm$ 2.5	63.9 $\pm$ 2.8	63.6 $\pm$ 1.4	63.9 $\pm$ 1.4

Table 2. Average F1-macro (%) and standard deviation (%) over ten seeds. Teamwork average F1-macro denotes the mean across five OTAS dimensions: Communication (Comm), Coordination (Coord), Cooperation (Coop), Leadership (Lead), and Situational Awareness (Sit. Aw). Bold indicates the best overall performance.

Figure 2. Average number of iterations required to find an appropriate counterfactual using Alg. 2 .

Figure 3. Average distance between the original and counterfactual paralinguistic features of the targeted team member, using Alg. 2 .

Figure 4. Examples of counterfactuals generated from the dataset. Panels 1) and 2) show topological counterfactuals: in 1), the leader’s distraction leads to poor communication, suggesting a more prompt response; in 2), irrelevant conversation reduces situational awareness, suggesting greater attention and fewer off-topic discussions. Panels 3) and 4) show paralinguistic counterfactuals based on the behavioral classes in Tab. 1 : in 3), the leader is disengaged from the nurse’s request, reducing team communication, and the counterfactual suggests a calm-cooperative behavior; in 4), the leader exhibits dominant-aggressive behavior toward the nurse, reducing cooperation, and the counterfactual recommends a calm behavioral style.

Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs
May 5, 2026Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni +1Clinical Outcome Prediction Dynamic Graph Learning
a University of Trento · b UiT the Arctic University of Norway
Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations
Aug 24, 2026Vincenzo Marco De Luca, Antonio Longa, Andrea PasseriniSurgical Video Understanding
University of Trento Trento, Italy · UiT the Arctic University of Norway Tromsø, Norway
Surg $Σ$ : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence
Mar 17, 2026Zhitao Zeng, Mengya Xu, Jian Jiang +13Multimodal Foundation Models Surgical Video Understanding
NUS · CUHK · SJTU +1

May 5, 2026J/K move · Enter open · S save

Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs

Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, Andrea Passerini

a University of Trento · b UiT the Arctic University of Norway

Clinical Outcome Prediction Dynamic Graph Learning Spatiotemporal GNNs

Open paper AskPDF arXiv

Surgical team performance arises from complex interactions between technical execution and non-technical skills, including communication and coordination dynamics. However, current surgical AI systems predominantly model visual workflow signals, lacking structured representations of intraoperative team interactions over time. We propose a real-time actionable approach for modeling surgical team dynamics using time-expanded interaction graphs, where team members are modeled as time-indexed nodes and communication exchanges define directed edges. This spatio-temporal expansion enables dynamic interaction modeling, while allowing efficient inference with a static graph neural network. The model predicts procedural efficiency as the deviation from the expected duration and supports real-time deployment. Beyond prediction, we perform a counterfactual analysis to identify minimal changes in communication structure and interpretable behavioral variables associated with improved predicted outcomes. Experiments on recorded surgical procedures show that structured modeling of team interactions improves early identification of prolonged interventions and provides coherent, actionable explanations. This work advances surgical AI toward real-time, team-aware, and actionable decision support in the operating room.

Abstract

Explore similar work

Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs

Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations

SurgΣΣΣ: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence

Surg $Σ$ : A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence