SurgFormer: Scalable Learning of Organ Deformation with Resection Support and Real-Time Inference
Authors: Ashkan Shahbazi, Elaheh Akbari, Kyvia Pereira, Jon S. Heiselman, Annie C. Benson, Garrison L. H. Johnston, Jie Ying Wu, Nabil Simaan, Michael I. Miga, Soheil Kolouri
Organizations: Department of Computer Science, College of Connected Computing, Vanderbilt University · Vanderbilt Institute for Surgery and Engineering · Department of Biomedical Engineering, Vanderbilt University · Department of Mechanical Engineering, Vanderbilt University
We introduce SurgFormer, a multiresolution gated transformer for data driven soft tissue simulation on volumetric meshes. High fidelity biomechanical solvers are often too costly for interactive use, so we train SurgFormer on solver generated data to predict nodewise displacement fields at near real time rates. SurgFormer builds a fixed mesh hierarchy and applies repeated multibranch blocks that combine local message passing, coarse global self attention, and pointwise feedforward updates, fused by learned per node, per channel gates to adaptively integrate local and long range information while remaining scalable on large meshes. For cut conditioned simulation, resection information is encoded as a learned cut embedding and provided as an additional input, enabling a unified model for both standard deformation prediction and topology altering cases. We also introduce two surgical simulation datasets generated under a unified protocol with XFEM based supervision: a cholecystectomy resection dataset and an appendectomy manipulation and resection dataset with cut and uncut cases. To our knowledge, this is the first learned volumetric surrogate setting to study XFEM supervised cut conditioned deformation within the same volumetric pipeline as standard deformation prediction. Across diverse baselines, SurgFormer achieves strong accuracy with favorable efficiency, making it a practical backbone for both tasks. {Code, data, and project page: available here}
Figures & tables
Figure 1: SurgFormer enables real-time, anatomically plausible soft-tissue simulation for minimally invasive surgery, jointly modeling tool-driven deformation and topology-changing resection. We demonstrate cholecystectomy (top) and appendectomy (bottom); each row shows the input, predicted deformation under tool interaction, and resulting resection state, highlighting geometric fidelity and visual realism.
Figure 2: XFEM based data generation pipeline for surgical tissue mechanics. CT volumes are segmented to obtain organ geometries, converted to solid models, and tetrahedralized into volumetric meshes. Procedure-specific boundary conditions, tool interaction flags, deleted-vertex indicators, and level-set fields are then defined to represent deformation and resection states. Quasistatic linear XFEM is solved on the active domain to produce full-field displacement targets.
Figure 3: Overview of the proposed multi resolution graph transformer for organ deformation. Simulator input signals are encoded by an Adapter and processed through L hierarchical levels using farthest point sampling with pooling, where each level applies LayerNorm followed by parallel local graph attention, global multi head attention, and feed forward transformations fused by a learned gate. A symmetric decoder upsamples via unpooling and refines features at each level, using levelwise skip connections from the encoder, and a final regressor maps the finest level features to the output.
Method
RMSE ↓
Max Err ↓
DCM ↑
Time (ms) ↓
Params (M)
GAOT [ 22 ]
0.028
0.034
96.36 ±0.88
0.53 ±0.04
7.2
NIN [ 19 ]
0.033
0.045
93.61 ±0.61
1.57 ±0.15
6.8
MGN-T [ 7 ]
0.083
0.122
86.76 ±1.27
1.31 ±0.09
6.2
PointNet [ 15 ]
0.030
0.038
96.37 ±0.33
1.28 ±0.05
6.0
PVCNN [ 13 ]
0.039
0.053
92.58 ±1.34
1.69 ±0.23
6.0
SurgFormer
0.018
0.022
97.21 ±1.06
0.64 ±0.07
6.5
Table 1: Soft tissue deformation modeling results under the same evaluation protocol in [ 17 ] . SurgFormer achieves the best overall accuracy across baselines, attaining the lowest RMSE and Max Err and the highest deformation capture metric (DCM), while maintaining near real time inference and a comparable parameter budget.
Method
RMSE ↓
Max Err ↓
DCM ↑
Time (ms) ↓
Params (M)
GAOT [ 22 ]
0.158→0.133
0.289→0.221
63.35→72.26±1.11
0.51 ±0.04
7.2
NIN [ 19 ]
0.216→0.185
0.364→0.284
61.11→69.78±0.93
1.56 ±0.17
6.8
PointNet [ 15 ]
0.199→0.157
0.334→0.181
62.03→70.12±0.26
1.33 ±0.06
6.0
PVCNN [ 13 ]
0.227→0.144
0.382→0.173
64.45→72.05±1.62
1.59 ±0.18
6.0
SurgFormer
0.143→0.112
0.191→0.164
66.85→83.61±0.44
0.72 ±0.09
6.5
Table 2: Cut conditioning results. Each entry reports performance without cut conditioning → with cut conditioning under the same protocol.
Method
RMSE ↓
Max Err ↓
DCM ↑
Time (ms) ↓
Params (M)
GAOT [ 22 ]
0.164
0.272
79.44 ±1.17
0.49 ±0.08
7.2
NIN [ 19 ]
0.155
0.263
80.24 ±2.24
0.81 ±0.14
6.8
PointNet [ 15 ]
0.180
0.293
76.78 ±0.73
1.07 ±0.13
6.0
PVCNN [ 13 ]
0.119
0.257
88.74 ±1.80
1.23 ±0.19
6.0
SurgFormer
0.135
0.228
87.61 ±2.02
0.48 ±0.08
6.0
Table 3: Appendectomy results on mixed uncut and cut cases. Cut conditioning follows ( 20 ) with c=0 for uncut.
Signal
Model
RMSE ↓
Max Err ↓
DCM ↑
MDr↓
Clean
Standard
0.018
0.022
97.21
0.08
Adv. FT
0.026
0.031
95.41
0.10
Adv. ( α=0.2 )
Standard
—
—
—
0.49
Adv. FT
—
—
—
0.11
Table 4: Adversarial smoothness stress test. Clean signals use full supervised metrics. For adversarial tool signals, no target displacement is available, so we report only the smoothness score MDr .
Variant
RMSE ↓
Max Err ↓
DCM ↑
Time (ms) ↓
Params (M)
Full ( Bℓ={1,2,3} )
0.018
0.022
97.21
0.6
6.5
No local ( Bℓ={2,3} )
0.026
0.034
93.84
0.5
5.2
No global ( Bℓ={1,3} )
0.022
0.029
94.36
0.4
5.4
No FF ( Bℓ={1,2} )
0.020
0.025
96.41
0.5
6.1
Uniform mix ( Bℓ={1,2,3} )
0.021
0.027
96.02
0.6
5.9
Table 5: Multibranch fusion ablation for the fusion operator in ( 15 ). We report design variants obtained by removing local, global, or feedforward branches from the active branch set Bℓ . Branch indices are 1 local, 2 global, and 3 feedforward. Uniform mix uses Γi,b,c=1/∣Bℓ∣ for b∈Bℓ .
Forecasting tissue deformation from future instrument motion could support predictive planning and increasingly autonomous surgical systems. We study this problem using a scene-specific differentiable surface model fitted to an observed tool-tissue interaction window. Rather than forecasting absolute simulated positions, we introduce an observation-referenced formulation that anchors each tissue point to its last valid observation and adds only the deformation induced by the known, preplanned future tool trajectory. This reduces sensitivity to reconstruction, initialization, and persistent simulation offsets while preserving the predicted deformation response. The physical model uses differentiable, mesh-free position-based dynamics with scene-specific rigidity, attachment, region scale, and grasp coupling. We further introduce an optional observation-referenced residual model combining graph-spectral spatial modes with periodic harmonic dynamics for structured motion not explained by the tool input. We evaluate the framework retrospectively on 12 surgical manipulation clips from three datasets. Observation referencing provides the largest gain, with a median within-clip error ratio of 0.598 relative to the corresponding absolute simulation forecast. The optional periodic residual further improves forecasting when the remaining motion exhibits periodic structure. These results support observation-referenced deformation prediction as a practical building block for future surgical world models, digital twins, and generative simulators that must predict how tissue evolves under instrument actions.
Tom Olesch, Nan Xiao, Xiongye Xiao +2
Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville · Department of Surgery, University of Tennessee Medical Center, Knoxville
Preoperative liver-tumor assessment requires segmentation, physical-space measurement, visual evidence, and resection planning from the same three-dimensional CT volume. Existing tools often handle these steps separately, while language models cannot reliably compute physical measurements from CT. To provide an integrated workflow, we present VoxelSage, a multi-modal system for two- and three-dimensional visualization, liver-tumor analysis, and preoperative resection planning. Its dual-port architecture separates language-model orchestration from image computation: Port A interprets requests and selects skills, while Port B applies them to CT volumes and segmentation masks and returns structured results. Keeping physical measurements in Port B prevents the LLM from computing them directly and reduces the risk of fabricated numerical results. Eight built-in skills support quantitative analysis, visual evidence generation, three-dimensional reconstruction, segmentation refinement, and sequential resection planning; user-defined skills can extend these functions. For sequence planning, a behavior-cloned neural ranker orders candidate resection targets, while a simulator-based shield checks them against predefined constraints. Across 256 unseen simulator scenes, this approach reduced mean simulated time from 34.274 to 33.388 min (0.886 min, 2.59%) and mean simulated blood loss from 300.847 to 183.852 mL (116.995 mL, 38.89%) relative to a deterministic baseline. These results demonstrate system integration and simulator-level performance, not clinical efficacy or safety. The public implementation is available at https://github.com/ZJUMAI/VoxelSage.
Accurate soft tissue simulation is essential for surgical training, pre-operative planning, and haptic feedback systems. While learning-based surrogate models trained on data using the finite element method (FEM) offer a promising path to real-time inference, their reliability depends on well-calibrated constitutive models. Existing approaches neither provide systematic guidance on model selection across stiffness levels, nor generalize across different tissue stiffnesses or geometries. We perform a comprehensive calibration of hyperelastic constitutive models in the SOFA Framework using gravity-loaded silicone beams with different stiffnesses. Using calibrated simulations as training data, we use a softness conditioned equivariant graph neural network, enabling deformation and force prediction across multiple tissue types and unseen geometries. Our model achieves sub-millimeter mean deformation accuracy at 0.010s inference time, while showing that force prediction quality is directly tied to upstream calibration consistency.
Madina Kojanazarova, Sidaty El Hadramy, Philippe C. Cattin
University of Basel, Department of Biomedical Engineering, Allschwil, Switzerland