SurgFormer: Scalable Learning of Organ Deformation with Resection Support and Real-Time Inference

Ashkan Shahbazi; Elaheh Akbari; Kyvia Pereira; Jon S. Heiselman; Annie C. Benson; Garrison L. H. Johnston; Jie Ying Wu; Nabil Simaan

cs.CVMar 6, 2026

SurgFormer: Scalable Learning of Organ Deformation with Resection Support and Real-Time Inference

Authors: Ashkan Shahbazi, Elaheh Akbari, Kyvia Pereira, Jon S. Heiselman, Annie C. Benson, Garrison L. H. Johnston, Jie Ying Wu, Nabil Simaan,

Organizations: Department of Computer Science, College of Connected Computing, Vanderbilt University · Vanderbilt Institute for Surgery and Engineering · Department of Biomedical Engineering, Vanderbilt University · Department of Mechanical Engineering, Vanderbilt University

Abstract

We introduce SurgFormer, a multiresolution gated transformer for data driven soft tissue simulation on volumetric meshes. High fidelity biomechanical solvers are often too costly for interactive use, so we train SurgFormer on solver generated data to predict nodewise displacement fields at near real time rates. SurgFormer builds a fixed mesh hierarchy and applies repeated multibranch blocks that combine local message passing, coarse global self attention, and pointwise feedforward updates, fused by learned per node, per channel gates to adaptively integrate local and long range information while remaining scalable on large meshes. For cut conditioned simulation, resection information is encoded as a learned cut embedding and provided as an additional input, enabling a unified model for both standard deformation prediction and topology altering cases. We also introduce two surgical simulation datasets generated under a unified protocol with XFEM based supervision: a cholecystectomy resection dataset and an appendectomy manipulation and resection dataset with cut and uncut cases. To our knowledge, this is the first learned volumetric surrogate setting to study XFEM supervised cut conditioned deformation within the same volumetric pipeline as standard deformation prediction. Across diverse baselines, SurgFormer achieves strong accuracy with favorable efficiency, making it a practical backbone for both tasks. {Code, data, and project page: available here}

Figures & tables

Figure 1: SurgFormer enables real-time, anatomically plausible soft-tissue simulation for minimally invasive surgery, jointly modeling tool-driven deformation and topology-changing resection. We demonstrate cholecystectomy (top) and appendectomy (bottom); each row shows the input, predicted deformation under tool interaction, and resulting resection state, highlighting geometric fidelity and visual realism.

Figure 2: XFEM based data generation pipeline for surgical tissue mechanics. CT volumes are segmented to obtain organ geometries, converted to solid models, and tetrahedralized into volumetric meshes. Procedure-specific boundary conditions, tool interaction flags, deleted-vertex indicators, and level-set fields are then defined to represent deformation and resection states. Quasistatic linear XFEM is solved on the active domain to produce full-field displacement targets.

Figure 3: Overview of the proposed multi resolution graph transformer for organ deformation. Simulator input signals are encoded by an Adapter and processed through $L$ hierarchical levels using farthest point sampling with pooling, where each level applies LayerNorm followed by parallel local graph attention, global multi head attention, and feed forward transformations fused by a learned gate. A symmetric decoder upsamples via unpooling and refines features at each level, using levelwise skip connections from the encoder, and a final regressor maps the finest level features to the output.

Method	RMSE $\downarrow$	Max Err $\downarrow$	DCM $\uparrow$	Time (ms) $\downarrow$	Params (M)
GAOT [ 22 ]	0.028	0.034	96.36 ±0.88	0.53 ±0.04	7.2
NIN [ 19 ]	0.033	0.045	93.61 ±0.61	1.57 ±0.15	6.8
MGN-T [ 7 ]	0.083	0.122	86.76 ±1.27	1.31 ±0.09	6.2
PointNet [ 15 ]	0.030	0.038	96.37 ±0.33	1.28 ±0.05	6.0
PVCNN [ 13 ]	0.039	0.053	92.58 ±1.34	1.69 ±0.23	6.0
SurgFormer	0.018	0.022	97.21 ±1.06	0.64 ±0.07	6.5

Table 1: Soft tissue deformation modeling results under the same evaluation protocol in [ 17 ] . SurgFormer achieves the best overall accuracy across baselines, attaining the lowest RMSE and Max Err and the highest deformation capture metric (DCM), while maintaining near real time inference and a comparable parameter budget.

Method	RMSE $\downarrow$	Max Err $\downarrow$	DCM $\uparrow$	Time (ms) $\downarrow$	Params (M)
GAOT [ 22 ]	$\text{0.158}\to\text{0.133}$	$\text{0.289}\to\text{0.221}$	$\text{63.35}\to\text{72.26}_{\pm 1.11}$	0.51 ±0.04	7.2
NIN [ 19 ]	$\text{0.216}\to\text{0.185}$	$\text{0.364}\to\text{0.284}$	$\text{61.11}\to\text{69.78}_{\pm 0.93}$	1.56 ±0.17	6.8
PointNet [ 15 ]	$\text{0.199}\to\text{0.157}$	$\text{0.334}\to\text{0.181}$	$\text{62.03}\to\text{70.12}_{\pm 0.26}$	1.33 ±0.06	6.0
PVCNN [ 13 ]	$\text{0.227}\to\text{0.144}$	$\text{0.382}\to\text{0.173}$	$\text{64.45}\to\text{72.05}_{\pm 1.62}$	1.59 ±0.18	6.0
SurgFormer	$\text{0.143}\to\text{{0.112}}$	$\text{0.191}\to\text{{0.164}}$	$\text{66.85}\to\text{{83.61}}_{\pm 0.44}$	0.72 ±0.09	6.5

Table 2: Cut conditioning results. Each entry reports performance without cut conditioning

\to

with cut conditioning under the same protocol.

Method	RMSE $\downarrow$	Max Err $\downarrow$	DCM $\uparrow$	Time (ms) $\downarrow$	Params (M)
GAOT [ 22 ]	0.164	0.272	79.44 ±1.17	0.49 ±0.08	7.2
NIN [ 19 ]	0.155	0.263	80.24 ±2.24	0.81 ±0.14	6.8
PointNet [ 15 ]	0.180	0.293	76.78 ±0.73	1.07 ±0.13	6.0
PVCNN [ 13 ]	0.119	0.257	88.74 ±1.80	1.23 ±0.19	6.0
SurgFormer	0.135	0.228	87.61 ±2.02	0.48 ±0.08	6.0

Table 3: Appendectomy results on mixed uncut and cut cases. Cut conditioning follows ( 20 ) with

c=\mathbf{0}

for uncut.

Signal	Model	RMSE $\downarrow$	Max Err $\downarrow$	DCM $\uparrow$	$\mathcal{M}_{Dr}$ $\downarrow$
Clean	Standard	0.018	0.022	97.21	0.08
Clean	Adv. FT	0.026	0.031	95.41	0.10
Adv. ( $\alpha=0.2$ )	Standard	—	—	—	0.49
Adv. ( $\alpha=0.2$ )	Adv. FT	—	—	—	0.11

Table 4: Adversarial smoothness stress test. Clean signals use full supervised metrics. For adversarial tool signals, no target displacement is available, so we report only the smoothness score

\mathcal{M}_{Dr}

Variant	RMSE $\downarrow$	Max Err $\downarrow$	DCM $\uparrow$	Time (ms) $\downarrow$	Params (M)
Full ( $\mathcal{B}_{\ell}=\{1,2,3\}$ )	$0.018$	$0.022$	$97.21$	$0.6$	$6.5$
No local ( $\mathcal{B}_{\ell}=\{2,3\}$ )	$0.026$	$0.034$	$93.84$	$0.5$	$5.2$
No global ( $\mathcal{B}_{\ell}=\{1,3\}$ )	$0.022$	$0.029$	$94.36$	$0.4$	$5.4$
No FF ( $\mathcal{B}_{\ell}=\{1,2\}$ )	$0.020$	$0.025$	$96.41$	$0.5$	$6.1$
Uniform mix ( $\mathcal{B}_{\ell}=\{1,2,3\}$ )	$0.021$	$0.027$	$96.02$	$0.6$	$5.9$

Table 5: Multibranch fusion ablation for the fusion operator in ( 15 ). We report design variants obtained by removing local, global, or feedforward branches from the active branch set

\mathcal{B}_{\ell}

. Branch indices are 1 local, 2 global, and 3 feedforward. Uniform mix uses

\Gamma_{i,b,c}=1/|\mathcal{B}_{\ell}|

for

b\in\mathcal{B}_{\ell}

Neural Surrogate Modeling

Neural surrogate modeling uses learned models to approximate costly simulations, enabling faster prediction, design, control, and inference.

See all 410 papers

Observation-Referenced Tissue Deformation Forecasting with Tool-Driven and Residual Dynamics
Oct 9, 2026Tom Olesch, Nan Xiao, Xiongye Xiao +2Deformable Object Simulation Surgical Robotics
Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville · Department of Surgery, University of Tennessee Medical Center, Knoxville
VoxelSage: Tool-Augmented 3D CT Analysis and Simulator-Shielded Sequential Resection Planning for Liver Tumors
Sep 29, 2026Binghong Qian, Xuanhe Liu, Yifan Xing +33D Medical Imaging LLM Tool Use
Zhejiang University
Generalizing Soft Tissue Deformation and Force Prediction Across Material Stiffness and Geometry
Aug 21, 2026Madina Kojanazarova, Sidaty El Hadramy, Philippe C. CattinDeformable Object Simulation Neural Surrogate Modeling
University of Basel, Department of Biomedical Engineering, Allschwil, Switzerland

Oct 9, 2026J/K move · Enter open · S save

Observation-Referenced Tissue Deformation Forecasting with Tool-Driven and Residual Dynamics

Tom Olesch, Nan Xiao, Xiongye Xiao, Jonathan C. DeLong+1

Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville · Department of Surgery, University of Tennessee Medical Center, Knoxville

Deformable Object Simulation Surgical Robotics Differentiable Physics

Open paper AskPDF arXiv

Forecasting tissue deformation from future instrument motion could support predictive planning and increasingly autonomous surgical systems. We study this problem using a scene-specific differentiable surface model fitted to an observed tool-tissue interaction window. Rather than forecasting absolute simulated positions, we introduce an observation-referenced formulation that anchors each tissue point to its last valid observation and adds only the deformation induced by the known, preplanned future tool trajectory. This reduces sensitivity to reconstruction, initialization, and persistent simulation offsets while preserving the predicted deformation response. The physical model uses differentiable, mesh-free position-based dynamics with scene-specific rigidity, attachment, region scale, and grasp coupling. We further introduce an optional observation-referenced residual model combining graph-spectral spatial modes with periodic harmonic dynamics for structured motion not explained by the tool input. We evaluate the framework retrospectively on 12 surgical manipulation clips from three datasets. Observation referencing provides the largest gain, with a median within-clip error ratio of

0.598

relative to the corresponding absolute simulation forecast. The optional periodic residual further improves forecasting when the remaining motion exhibits periodic structure. These results support observation-referenced deformation prediction as a practical building block for future surgical world models, digital twins, and generative simulators that must predict how tissue evolves under instrument actions.

Abstract

Neural Surrogate Modeling

Explore similar work

Observation-Referenced Tissue Deformation Forecasting with Tool-Driven and Residual Dynamics

VoxelSage: Tool-Augmented 3D CT Analysis and Simulator-Shielded Sequential Resection Planning for Liver Tumors

Generalizing Soft Tissue Deformation and Force Prediction Across Material Stiffness and Geometry