SurgiCLAW: An Interpretable Atomic-Skill Agent for Precision Robotic Micromanipulation

Angen Ye; junjie xie; Yujia Song; Junkai Huang; Dapeng Zhang

cs.ROOct 11, 2026

SurgiCLAW: An Interpretable Atomic-Skill Agent for Precision Robotic Micromanipulation

Authors: Angen Ye, junjie xie, Yujia Song, Junkai Huang, Dapeng Zhang

Organizations: Institute of Automation, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China

Abstract

Recent multimodal large language models (MLLMs) and vision--language--action models have provided robots with new capabilities for long-horizon task understanding and planning. In surgical robotics, however, end-to-end prediction of low-level actions remains limited by insufficient precision, opaque decisions, and scarce data. This paper presents SurgiCLAW, a hierarchical skill-based framework for precision surgical manipulation. SurgiCLAW confines an MLLM to intraoperative visual understanding, structured explanation, and atomic surgical-skill selection, while delegating physical execution to verifiable deterministic control programs. For brain--machine-interface implantation, a representative high-precision surgical task, we construct a state-to-skill invocation dataset and expand it through visual and state perturbations. The model first learns structured skill invocation through supervised fine-tuning and is then optimized using three rewards for action correctness, operation prerequisites, and format validity, improving action selection, operational safety, and output parseability. We evaluate subtask success, offline model comparison, real-robot performance, and ablations. On a fixed test set, SurgiCLAW achieves 94.6% skill-selection accuracy and 93.0% macro-averaged F1 (Macro-F1). Across 50 closed-loop real-robot implantation trials, it achieves an 88.0% task success rate.

Figures & tables

Function	Meaning	Primary prerequisite
ADJUST_EXPOSURE	Adjust microscopic exposure	Camera online; exposure outside the calibrated range
ADJUST_FOCUS	Adjust microscopic focus	Camera online; image out of focus
E_STOP	Trigger a safety stop	Singular configuration, system fault, or other high-risk anomaly
SPATIAL_REGISTRATION	Register image and robot coordinates	Usable image; system self-check passed
RESTART_CAMERA	Restart a failed or disconnected camera	Abnormal camera connection; robot held in a safe state
ALIGN_NEEDLE	Deterministic visual-servo needle alignment	Registration complete; image sharp; no singular configuration

TABLE I: Atomic Action Interface

Fig. 1: Multi-MLLM collaborative data-generation pipeline. Starting from raw image, robot-state, instruction, and skill-label samples, the pipeline expands the data through image/text augmentation and constructs structured rationales using three MLLMs. Candidate outputs are filtered by rule-based checks, leave-one-generator-out cross-validation, and human spot inspection before being retained in the final structured dataset.

Fig. 2: SFT and GDPO training pipeline. The MLLM first learns structured reasoning, answers, and function calls through SFT. GDPO then optimizes action correctness, operation prerequisites, and output format under a reference-model constraint.

Method	Accuracy (%) $\uparrow$	Macro-P (%) $\uparrow$	Macro-R (%) $\uparrow$	Macro-F1 (%) $\uparrow$	Safety violation (%) $\downarrow$
General VLMs (non-SFT)
MiniCPM-o-2.6 [ 23 ]	14.2	2.0	14.3	3.6	18.6
DeepSeek-VL2 [ 24 ]	20.6	22.4	20.9	14.8	14.2
Llama-4-Maverick-17B-128E-Instruct [ 25 ]	24.7	28.9	25.1	17.2	11.6
InternVL3.5-241B-A28B [ 26 ]	31.6	37.8	31.2	24.4	7.4
Qwen2.5-VL-3B-Instruct [ 27 ]	15.4	27.8	15.5	5.9	15.8

TABLE II: Atomic-Skill Selection on the Fixed Test Set

Fig. 3: Closed-loop decision example for flexible-electrode implantation. At each step, SurgiCLAW receives the current microscopic image, robot state, and operation query, then outputs a structured rationale, an operator-facing answer, and a single atomic-skill call. The example illustrates a typical decision sequence from spatial registration and image-quality correction to needle alignment and final insertion.

Method	Success (%) $\uparrow$	Stages $\downarrow$	Takeover (%) $\downarrow$	Unsafe proposal (%) $\downarrow$
Surgical-specific VLMs
SurgVLM-7B	46.0 (23/50)	$10.4\pm 2.6$	38.0 (19/50)	5.4 (11/204)
SurgVidLM	50.0 (25/50)	$9.8\pm 2.4$	34.0 (17/50)	4.9 (10/205)
General VLM + SFT
Qwen3-VL-4B-Instruct + SFT	66.0 (33/50)	$8.6\pm 1.9$	20.0 (10/50)	3.4 (7/207)
Qwen3-VL-32B-Instruct + SFT	76.0 (38/50)	$7.7\pm 1.6$	10.0 (5/50)	1.9 (4/206)

TABLE III: Closed-Loop Real-Robot Implantation Results

Setting	Accuracy (%) $\uparrow$	Macro-P (%) $\uparrow$	Macro-R (%) $\uparrow$	Macro-F1 (%) $\uparrow$	Safety violation (%) $\downarrow$
Full model	94.6	94.0	92.0	93.0	0.5
w/o structured rationale	88.2	87.4	85.9	86.4	2.3
w/o GDPO	83.0	81.4	81.0	81.1	3.2
w/o action-correctness reward	90.4	88.1	88.7	88.2	1.6
w/o prerequisite reward	91.2	90.8	88.9	89.7	3.2
w/o format reward	92.0	92.8	88.6	90.4	1.1

TABLE IV: Ablation of Model-Side Components

Subtask	Success criterion	Success (%) $\uparrow$	Mean error $\downarrow$	Time (s) $\downarrow$
Spatial registration	TRE $<0.50$ mm and coordinate transform completed	95.0 (38/40)	$0.347\pm 0.126$ mm (TRE)	$3.18\pm 0.37$
Target localization (internal)	Stable target detection available for alignment	97.5 (39/40)	$0.051\pm 0.028$ mm (position)	$0.82\pm 0.26$
Needle alignment	Orientation and image-plane position within thresholds	95.0 (38/40)	$0.91\pm 0.42^{\circ}$ / $0.0018\pm 0.0010$ mm	$19.4\pm 4.8$
Insertion execution	Direction and 3-D landing position within thresholds	90.0 (36/40)	$2.57\pm 1.16^{\circ}$ / $0.015\pm 0.006$ mm	$45.73\pm 13.49$
Image-quality adjustment	Image restored to a parseable quality range	97.5 (39/40)	–	$1.63\pm 0.61$
Safety stop	Timely refusal in a high-risk state	100.0 (40/40)	–	$0.12\pm 0.02$

TABLE V: Atomic-Subtask Success and Execution Accuracy

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,699 papers

MedVLA: A Hierarchical Vision-Language-Action Framework for Closed-Loop Precision Medical Robot Manipulation
Sep 22, 2026Junjie Xie, Angen Ye, Chuxuan He +2Robotic Control Robotic Manipulation
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Zhejiang Gongshang University, Hangzhou 310018, China
SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics
Jun 28, 2026Jiashuo Sun, Yue He, Wenxuan Liu +4Vision-Language-Action Models Robot Manipulation Benchmarks
the National Engineering Research Center of Robot Visual Perception and Control Technology, China · the School of Artificial Intelligence and Robotics, Hunan University, Hunan, China · the national graduate college for elite engineers, Hunan University, Hunan, China
SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition
Oct 18, 2026Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif +1Surgical Video Understanding Surgical Robotics
Electrical and Computer Engineering Department, North Carolina A&T State University, Greensboro, NC, USA · Swinburne University of Technology, Hawthorn, VIC, Australia · Harvard Paulson School of Engineering and Applied Sciences, Harvard University, Allston, MA, USA +3

Sep 22, 2026J/K move · Enter open · S save

MedVLA: A Hierarchical Vision-Language-Action Framework for Closed-Loop Precision Medical Robot Manipulation

Junjie Xie, Angen Ye, Chuxuan He, Yujia Song+1

Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Zhejiang Gongshang University, Hangzhou 310018, China

Robotic Control Robotic Manipulation Vision-Language-Action Models Healthcare Robotics

Open paper AskPDF arXiv

Precision medical robotics demands adaptive decision-making under strict safety, interpretability, and execution constraints. Although recent Vision-Language-Action (VLA) models show strong multimodal reasoning ability, their continuous action generation paradigm is not well suited for precision medical tasks, where reliable closed-loop operation may also depend on non-action system function calls. To address this gap, we propose MedVLA, a hierarchical framework that couples high-level multimodal reasoning with low-level function-constrained execution. We further introduce a scalable multi-agent pipeline to generate skill-oriented chain-of-thought(CoT) data for structured training. Built on different multimodal large-model backbones, MedVLA consistently improves performance after fine-tuning, demonstrating the effectiveness of the proposed framework across model variants. Under identical initial conditions, we perform 100 closed-loop flexible electrode implantation trials. The results show that MedVLA achieves a 95.0% task success rate, substantially outperforming representative VLA baselines, including OpenVLA (8%) and

π_0

(15%), in accuracy, stability, and safety. These results indicate that structured reasoning with constrained function-level execution is a practical route toward deployable precision medical robotics.

Abstract

Robotic Manipulation

Explore similar work

MedVLA: A Hierarchical Vision-Language-Action Framework for Closed-Loop Precision Medical Robot Manipulation

SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics

SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition