cs.CVMar 26, 2026

arg-VU: Affordance Reasoning with Physics-Aware 3D Geometry for Visual Understanding in Robotic Surgery

Authors: Nan Xiao, Yunxin Fan, Farong Wang, Fei Liu

Organizations: University of Tennessee, Knoxville, TN, USA · Stanford University, Palo Alto, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Affordance reasoning provides a principled link between perception and action, yet remains underexplored in surgical robotics, where tissues are highly deformable, compliant, and dynamically coupled with tool motion. We present arg-VU, a physics-aware affordance reasoning framework that integrates temporally consistent geometry tracking with constraint-induced mechanical modeling for surgical visual understanding. Surgical scenes are reconstructed using 3D Gaussian Splatting (3DGS) and converted into a temporally tracked surface representation. Extended Position-Based Dynamics (XPBD) embeds local deformation constraints and produces representative geometry points (RGPs) whose constraint sensitivities define anisotropic stiffness metrics capturing the local constraint-manifold geometry. Robotic tool poses in SE(3) are incorporated to compute rigidly induced displacements at RGPs, from which we derive two complementary measures: a physics-aware compliance energy that evaluates mechanical feasibility with respect to local deformation constraints, and a positional agreement score that captures motion alignment (as kinematic motion baseline). Experiments on surgical video datasets show that arg-VU yields more stable, physically consistent, and interpretable affordance predictions than kinematic baselines. These results demonstrate that physics-aware geometric representations enable reliable affordance reasoning for deformable surgical environments and support embodied robotic interaction.

Figures & tables

Fig. 1 : Left: tracked surface geometry reconstructed from endoscopic video. Right: affordance score map (red indicates higher actionability). Our framework estimates physically plausible and actionable tool-tissue interaction regions rather than only reconstructing geometry.
Fig. 2 : Overall framework of arg⁡\arg -Vu . Th 3D point cloud from 3DGS provides a temporally tracked surface representation. XPBD enforces local constraints to produce representative geometry point (RGPs) with anisotropic stiffness structure. Tool poses in SE(3)\mathrm{SE}(3) yield induced displacements used for PACS (physical feasibility) and PAS (motion alignment, as kinematic motion baseline) for affordance reasoning.
Fig. 3 : Neighborhood definitions. Left : radius neighborhoods may cross anatomical boundaries. Right : segmentation-aware filtering constrains neighborhoods within anatomical regions, improving physical plausibility near organ boundaries.
Fig. 4 : Geometric interpretation of the constraint-induced stiffness metric. Local deformation constraints define a manifold M\mathcal{M} whose tangent space represents admissible deformation directions. The stiffness metric KiRGP\mathbf{K}_{i}^{\mathrm{RGP}} induces an ellipsoid that visualizes anisotropic resistance to motion. Long axes correspond to compliant tangent directions, whereas short axes correspond to stiff constraint-normal directions. A tool-induced displacement Δpi\Delta p_{i} at the representative point pip_{i} is evaluated through this metric.
Fig. 5 : Median cosine similarity between observed motion direction and the principal direction of predicted uncertainty (largest eigenvector). Our XPBD-derived covariance aligns better with future motion than the velocity-persistence baseline.
Fig. 6 : Left to right = increasing time. Each column is at the same timestep. First row: Raw RGB frames. Second row: Baseline PAS. Third row: Compliance score (PACS). Red regions have higher affordance values, and blue regions have lower affordance values. The affordance progressively concentrates near the tool-tissue interaction region as retraction begins. The PACS results exhibit more stable convergence over time.
Fig. 7 : Qualitative visualization of a sample frame. Top-left: Affordance heatmap in 2D. Top-right: 3D PACS score distribution heatmap on the tissue point cloud. Yellow: Higher score. Dark blue: Lower score. Lower-left: reconstructed noisy 3D tissue point cloud. Lower-right: original 2D RGB frame.

Explore similar work

CardsList

  1. SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

    Jul 5, 2026Lei Song, Yonghao Long, Mengya Xu +3Robotic PerceptionSurgical Robotics

    Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Thoracic Surgery, Peking University People’s Hospital, Beijing, China · Thoracic Oncology Institute, Peking University People’s Hospital, Beijing, China +3

  2. AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction

    Apr 1, 2026Aiza Maksutova, Lalithkumar Seenivasan, Hao Ding +5Surgical Robotics

    Johns Hopkins University, Baltimore MD, USA

  3. Learning Surgical Robotic Manipulation with 3D Spatial Priors

    Mar 4, 2026Yu Sheng, Lidian Wang, Xiaomeng Chu +6Robotic ManipulationSurgical Robotics

    University of Science and Technology of China · Tuodao Medical · Yale University