A 4D Representation for Training-Free Agentic Reasoning from Monocular Laparoscopic Video
Authors: Maximilian Fehrentz, Nicolas Stellwag, Robert Wiebe, Nicole Thorisch, Fabian Grob, Patrick Remerscheid, Ken-Joel Simmoteit, Benjamin D. Killeen, +2 more
Organizations: Computer Aided Medical Procedures, TU Munich, Munich, Germany · TUM.ai, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Hospital of the LMU Munich, Ludwig-Maximilians-Universität (LMU), Munich, Germany
Spatiotemporal reasoning is a fundamental capability for artificial intelligence (AI) in soft tissue surgery, paving the way for intelligent assistive systems and autonomous robotics. While 2D vision-language models show increasing promise at understanding surgical video, the spatial complexity of surgical scenes suggests that reasoning systems may benefit from explicit 4D representations. Here, we propose a framework for equipping surgical agents with spatiotemporal tools based on an explicit 4D representation, enabling AI systems to ground their natural language reasoning in both time and 3D space. Leveraging models for point tracking, depth, and segmentation, we develop a coherent 4D model with spatiotemporally consistent tool and tissue semantics. A Multimodal Large Language Model (MLLM) then acts as an agent on tools derived from the explicit 4D representation (e.g., trajectories) without any fine-tuning. We evaluate our method on a new dataset of 134 clinically relevant questions and find that the combination of a general purpose reasoning backbone and our 4D representation significantly improves spatiotemporal understanding and allows for 4D grounding. We demonstrate that spatiotemporal intelligence can be "assembled" from 2D MLLMs and 3D computer vision models without additional training. Code, data, and examples are available at https://tum-ai.github.io/surg4d/
Figures & tables
Figure 1: Given a monocular 2D video sequence, our method creates a tracked semantic 4D representation and provides spatiotemporal information as tools to a 2D MLLM Agent. Upon receiving a query, the agent iteratively requests and interprets spatiotemporal information from our 4D representation to inform its response.
Figure 2: Example result and partial reasoning trace on a spatial query. Despite not having access to a specific gallbladder semantic instance, the agent understands that the connective tissue class is the desired connection between gallbladder and liver and localizes the interaction correctly.
Spatial
Temp. PIT
Temp. Interval
Directional
Model
L2 (px) ↓
(timesteps) ↓
IoU ↑
L1 ↓
Baselines
MLLM
90.76±65.08
6.56±5.05
0.51±0.34
1.15±0.88
MLLM w/ SoM
142.77±73.62
5.12±3.39
0.51±0.31
0.83±0.85
Ours
Agent
46.54±53.40
3.62±3.79
0.39±0.31
0.18±0.54
Table 1: Spatial, temporal point-in-time, temporal interval, and directional grounding.
Figure 3: Temporal example showing 2D video, 4D from two perspectives, and the derived model decisions
National University of Singapore, Singapore · Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore · Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and the Department of Medical Physics and Biomedical Engineering, University College London, UK