Robotic Optical Coherence Tomography-Guided Geometric Modeling of Soft Tissue: 3D Representations for Data-Driven Closed-Loop Shape Control

Chaoyu Zhang; Guangshen Ma; Mark Draelos

cs.ROOct 23, 2026

Robotic Optical Coherence Tomography-Guided Geometric Modeling of Soft Tissue: 3D Representations for Data-Driven Closed-Loop Shape Control

Authors: Chaoyu Zhang, Guangshen Ma, Mark Draelos

Organizations: University of Michigan, Ann Arbor

Abstract

In medical robotics, data-driven soft-tissue shape control requires deforming tissue toward desired morphologies through sequential physical interactions. This challenge is particularly important in microscale manipulation, where localized deformation is difficult to observe, represent, and control, while physics-based models can be computationally expensive and difficult to adapt to realistic environments. Optical coherence tomography (OCT) provides high-resolution volumetric perception of tissue deformation, but target-conditioned control requires effective representations of both observed and desired tissue states. In this work, we introduce an OCT-guided data-driven framework for tissue shape control, learned from user-guided demonstrations and executed with closed-loop OCT feedback. We study full-volume, tissue-masked sparse-volume, and voxelized point-cloud representations, with or without residual conditioning, using deterministic (Explicit) and generative diffusion (Diffusion) policies. Across 12 configurations and 576 phantom rollouts on 3-DoF depth and 6-DoF directional tasks, OCT-derived 3D representations support both Explicit and Diffusion policies, with performance depending on representation and residual conditioning. Residual conditioning reduces directional Shape RMSE from 0.748 to 0.555 mm for full-volume Diffusion, while voxelized point-cloud Explicit with residuals achieves 0.534 mm for depth and 0.443 mm for directional control, with comparable accuracy at intermediate unseen targets and consistent performance across deployments. We further evaluate the framework in 60 ex vivo porcine rollouts, extending to biological tissue with final geometric errors comparable to human demonstrations. These results establish volumetric OCT as current and goal representations for learned closed-loop tissue shape control across deterministic and generative action models.

Figures & tables

Figure 1: Closed-loop OCT-guided tissue shape control. The OCT captures volumetric tissue deformation induced by the robotic tooltip, while parameterized target generation constructs desired morphologies from predeformed OCT states. Current and target OCT volumes are encoded as full volumes, tissue-masked volumes, or voxelized point clouds, with optional current–target residual conditioning. These representations condition an Explicit or Diffusion policy to predict tooltip motions, with the first action executed before an OCT scan updates the state for closed-loop prediction.

Figure 2: Representation-specific residual conditioning. Full volumes use voxelwise intensity differences, sparse tissue volumes use intensity and occupancy differences over occupied coordinates, and voxelized point clouds use pooled feature differences. Values are schematic.

Figure 3: Phantom shape accuracy and reliability. Large symbols show mean final shape RMSE; three gray dots and their range show target-averaged deployment sweeps. Adjacent columns report mean final HD95 (details in Appendix B ) and per-rollout RMSE reduction. Purple dashed lines and bands show human endpoint means and reported

\pm 1

sample SD of final Shape RMSE across individual human demonstrations. Circles: Explicit; diamonds: Diffusion; filled: residual conditioning (

R

Figure 4: Target response and agreement with human demonstrations. Nearest-demonstration distances (“Nearest Demo”) show the mean across three deployment sweeps (various colors), with sweep means and ranges in gray. Columns report pooled

r_{d}

and

W

across all targets, while demonstration distances use seen targets only. Units apply to distances and

W

; symbols follow Fig. 3 .

Figure 5: Porcine tissue shape error before and after control. Open and filled markers show initial and final mean RMSE, respectively; arrows indicate the direction of error change after control. Symmetric denotes Shape RMSE combining both distance directions; obs. denotes observed tissue geometry. Human references include 30 demonstrations per task, showing policy and human errors on a similar scale and with similar trends across conditions, as indicated by arrow orientation.

Figure 6: Phantom deformation states and porcine endpoint geometry. (a,b) OCT-derived geometry at initial, intermediate, and final phantom states. (c) Porcine endpoints at unseen targets. Blue: observed tissue; brown: fixed target; insets: RGB observations. The porcine overlays show remaining differences in depression width and surrounding geometry.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Demonstrations	Train/validation	Actions	Observations
Phantom depth	40	32/8	405	445
Phantom directional	51	41/10	535	586
Porcine depth	30	24/6	302	332
Porcine directional	30	24/6	321	351
Total	151	121/30	1,563	1,714

Appendix

Table 1: Complete demonstration inventory and trajectory-level partitions.

Dataset	$A$ (mm)	$w$ (mm)	$\phi$	$\tau$
Phantom depth	$2,3,4,5$	6	–	–
Phantom directional	3.5	6	$0,45,90,135,180^{\circ}$	0.7
Porcine depth	$2,3,4$	3	–	–
Porcine directional	5.0	3	$0,45,90^{\circ}$	0.7

Appendix

Table 2: Target parameters used for user demonstration collection. The dash denotes a depth-only target. Direction angles follow the native OCT row–column convention.

Figure 7: Parameterized OCT target generation. A measured predeformation OCT volume is transformed into a target tissue morphology using a parameterized displacement field with deformation depth $A$ and width $w$ defined in Equation 7 . The resulting target volume preserves the measured OCT structure within the deformed geometry.

Input	Observation history	Fixed target
Full volume	$[B,3,256,256,256]$	$[B,256,256,256]$
Tissue-only	Three $\{[N_{t},3]_{\mathrm{coord}},[N_{t},1]_{\mathrm{intensity}}\}$ sets	One $\{[N_{g},3]_{\mathrm{coord}},[N_{g},1]_{\mathrm{intensity}}\}$ set
Voxelized Point cloud	$[B,3,1024,3]$	$[B,1024,3]$
Relative tool pose	$[B,3,6]$	–

Appendix

Table 3: Policy inputs after preprocessing.

B

denotes batch size;

N_{t}

and

N_{g}

denote the variable numbers of occupied sparse voxels in an observation and target, respectively. Sparse entries are shown per frame within one policy sample.

Input	Encoder	Tokens
Full volume	Four dense 3-D convolutional residual stages with channels $16/32/64/128$ , GroupNorm, SiLU, and adaptive average pooling to $4^{3}$ tokens per volume	259 / 451
Tissue-only	Sparse 3-D encoder with 27-neighbor blocks of channels $32/64/128/128$ and strides $1/2/2/2$ , followed by global mean pooling over active voxels	7 / 10
Voxelized point cloud	Pointwise MLP $3\to 64\to 128\to 128$ with ReLU, max pooling over 1,024 points, and pooled MLP $128\to 128\to 128$	7 / 10

Appendix

Table 4: Representation encoders and total condition-token counts without/with residual conditioning. Counts include three observations, one fixed target, and three relative tooltip poses.

Figure 8: OCT-guided control and action decoders. (a) OCT history, fixed targets, relative commanded poses, and optional residuals form condition tokens. The stack denotes three observations. Only the first of eight predicted actions is executed before rescanning; pose history is updated by command composition. (b) Four time-conditioned Transformer blocks predict clean actions for 20 DDIM-style updates. PE denotes learned action-position embeddings. (c) Explicit uses eight learned queries and one four-head cross-attention operation. Query skip addition, LayerNorm, and a SiLU MLP produce the action chunk. Representation encoders and token counts are specified in Table 4 .

Setting	Value	Setting	Value
Optimizer	AdamW	Learning rate	$10^{-4}$
Weight decay	$10^{-4}$	Batch size	1
Maximum/minimum epochs	100/10	Early-stop patience	10
Minimum improvement	$10^{-4}$	EMA decay	0.995
Gradient accumulation	None	Training seed	42

Appendix

Table 5: Shared physical-policy training settings. The learning rate is constant.

Representation / conditioning	Explicit	Diffusion
Full volume	$181.67\pm 0.08$	$272.85\pm 0.54$
Full volume + residual	$311.44\pm 0.17$	$402.91\pm 0.40$
Tissue-only	$35.06\pm 1.44$	$140.16\pm 14.73$
Tissue-only + residual	$517.86\pm 77.71$	$614.60\pm 39.28$
Point cloud	$2.16\pm 0.23$	$91.16\pm 0.57$
Point cloud + residual	$2.42\pm 0.11$	$94.26\pm 0.79$

Appendix

Table 6: Prepared-input model-only latency (mean

\pm

sample SD, ms) for the twelve phantom depth checkpoints on an RTX 4500 Ada GPU: one fixed input per checkpoint, batch size one, FP16 autocast, five warm-ups, and 20 CUDA-synchronized calls.

Phantom: all twelve configurations
Task	Targets	Rollouts	Actions	OCT states
Depth	7	252	3,024	3,276
Directional	9	324	3,888	4,212
Porcine: Full-volume Diffusion $+$ residual
Depth	5	15	180	195
Directional	5	15	180	195

Appendix

Table 7: Closed-loop evaluation inventory. Each phantom task evaluates all twelve policy configurations. Porcine experiments use task-specific checkpoints for the two configurations shown below. Each rollout contains twelve executed actions and thirteen OCT states, including the initial scan.

Material	Task	Target conditions	Fixed parameters
Phantom	Depth	Seen: $2.0,\,3.0,\,4.0,\,5.0$ mm Unseen: $2.5,\,3.5,\,4.5$ mm	Width: $6$ mm
Phantom	Directional	Seen: $0^{\circ},\,45^{\circ},\,90^{\circ},\,135^{\circ},\,180^{\circ}$ Unseen: $22.5^{\circ},\,67.5^{\circ},\,112.5^{\circ},\,157.5^{\circ}$	Depth: $3.5$ mm Width: $6$ mm Strength: $0.7$
Porcine	Depth	Seen: $2.0,\,3.0,\,4.0$ mm Unseen: $2.5,\,3.5$ mm	Width: $3$ mm
Porcine	Directional	Seen: $0^{\circ},\,45^{\circ},\,90^{\circ}$ Unseen: $22.5^{\circ},\,67.5^{\circ}$	Depth: $5$ mm Width: $3$ mm Strength: $0.7$

Appendix

Table 8: Phantom and porcine evaluation target conditions. Porcine experiments use task-specific checkpoints for Full-volume Diffusion

+

residual and Point-cloud Explicit

+

residual.

3-DoF depth
Configuration	$R$	$D_{12}$ all	$D_{12}$ seen	$D_{12}$ unseen	$H_{95}$	$\rho$	Improved
Full volume / Explicit	–	$0.593\pm 0.065$	0.602	0.581	1.076	21.3	17/21
Full volume / Explicit	$\checkmark$	$0.508\pm 0.008$	0.499	0.520	0.879	36.8	21/21
Full volume / Diffusion	–	$0.623\pm 0.006$	0.637	0.606	1.046	17.6	15/21
Full volume / Diffusion	$\checkmark$	$0.625\pm 0.100$	0.636	0.612	1.119	18.5	17/21
Tissue-only / Explicit	–	$0.703\pm 0.288$	0.710	0.693	1.425	7.4	16/21

Appendix

Table 9: Complete phantom endpoint results evaluation.

D_{12}

and

H_{95}

are in millimeters;

\rho

is the percentage reduction in Shape RMSE.

Material / task	Cohort	$n$	$D_{0}\to D_{\rm end}$	$H_{95,\rm end}$	Improved	Actions
Phantom / depth	Operators, all	40	$0.849\to 0.522$	0.831	40/40	10–12
Phantom / depth	Operators, val.	8	$0.849\to 0.552$	0.835	8/8	10–12
Phantom / depth	Point E+ $R$	12	$0.838\to 0.531$	0.896	11/12	12
Phantom / direction	Operators, all	51	$0.877\to 0.445$	0.964	51/51	9–15
Phantom / direction	Operators, val.	10	$0.876\to 0.421$	0.953	10/10	10–13
Phantom / direction	Point E+ $R$	15	$0.852\to 0.457$	1.004	15/15	12

Appendix

Table 10: Shape outcomes at demonstrated goal parameters.

D_{0}

and

D_{\rm end}

are initial and endpoint Shape RMSE;

H_{95,\rm end}

is endpoint HD95 (all in mm).

n

counts trajectories and “Improved” means

D_{\rm end}<D_{0}

. Operator trajectories end after 9–15 recorded actions; policy rollouts execute twelve. All and validation operator rows overlap; the latter are included in the former. Initial scans are distinct across cohorts.

Figure 9: Phantom depth error across observation steps. Step 0 is the initial observation; steps 1–12 follow executed actions. Rows show representations and columns show seen/unseen targets. Curves average targets and three sweeps; shading spans sweep means.

Figure 10: Phantom directional error at every observation step. Aggregation and shading follow Fig. 9 . Curves show deployments across nine target directions, including four unseen directions.

		3-DoF translation		6-DoF translation		6-DoF rotation
Configuration	$R$	$r_{d}$	$W$	$r_{d}$	$W$	$r_{d}$	$W$
Full volume / Explicit	–	0.577	0.368	0.421	2.174	0.461	5.940
Full volume / Explicit	$\checkmark$	0.994	1.027	0.671	2.899	0.487	6.659
Full volume / Diffusion	–	0.456	0.340	0.350	0.717	0.360	0.427
Full volume / Diffusion	$\checkmark$	0.955	0.920	0.944	5.203	0.895	11.030
Tissue-only / Explicit	–	0.774	0.385	0.810	3.328	0.894	8.999

Appendix

Table 11: Target association and across-target response spread for all phantom configurations.

r_{d}

measures the association between target differences and endpoint-response differences; larger is stronger.

W

measures the RMS spread of endpoint responses across targets and is reported in millimeters for translation and degrees for rotation.

		3-DoF translation		6-DoF translation		6-DoF rotation
Configuration	$R$	Demo avg.	Nearest demo	Demo avg.	Nearest demo	Demo avg.	Nearest demo
Full volume / Explicit	–	2.032	1.366	7.551	6.743	14.403	10.959
Full volume / Explicit	$\checkmark$	0.745	0.726	3.200	2.114	9.174	6.643
Full volume / Diffusion	–	2.297	1.867	6.330	5.426	11.056	6.931
Full volume / Diffusion	$\checkmark$	1.840	1.152	2.566	1.859	4.664	3.037
Tissue-only / Explicit	–	2.213	1.958	3.328	2.337	7.096	5.443

Appendix

Table 12: Cumulative-command agreement with demonstrations. “Demo avg.” is the distance to the average cumulative command of the ten demonstrations at the same target; “Nearest demo” is the distance to the closest individual demonstration.

Figure 11: Depth-task

y

translation commands across all targets. Columns show the seven depth targets, with asterisks marking unseen intermediate targets. Rows show all twelve policy configurations grouped by representation and action model.

Figure 12: Directional-task roll commands across all targets. Columns show the nine tested target directions, with asterisks marking unseen intermediate targets. Rows: twelve policy configurations.

Figure 13: Directional-task pitch commands across all targets.

Figure 14: Directional-task yaw commands across all targets.

Configuration	$D_{0}$	$D_{12}$	$H_{95,0}$	$H_{95,12}$	Improved
3-DoF depth
Full volume / Diffusion	$0.650\pm 0.044$	$1.102\pm 0.045$	2.003	4.390	0/15
Point cloud / Explicit	$0.592\pm 0.035$	$1.220\pm 0.088$	1.906	4.690	0/15
6-DoF directional
Full volume / Diffusion	$0.985\pm 0.021$	$0.961\pm 0.024$	3.376	2.892	8/15
Point cloud / Explicit	$1.022\pm 0.028$	$1.081\pm 0.038$	3.434	2.978	4/15

Appendix

Table 13: Ex vivo endpoint geometry (mm) for 60 rollouts, with 15 per row. Both configurations use residual conditioning.

D_{0}

and

D_{12}

are means

\pm

sample SD across three target-averaged sweeps (Eq. ( 13 )); HD95 columns report means.

3-DoF depth
	Target $\to$ observation		Observation $\to$ target
Configuration	RMSE	q95	RMSE	q95
Human reference	$0.760\to 0.395$	$1.988\to 0.759$	$0.486\to 1.544$	$1.216\to 4.344$
Full volume / Diffusion	$0.767\to 0.334$	$2.003\to 0.655$	$0.499\to 1.520$	$1.242\to 4.390$
Point cloud / Explicit	$0.732\to 0.530$	$1.906\to 0.951$	$0.394\to 1.639$	$0.972\to 4.690$
6-DoF directional

Appendix

Table 14: Directed components of Shape RMSE and HD95. Each entry shows the initial-to-final mean in millimeters. RMSE refers to the directed terms in Eq. ( 14 ); q95 refers to the two percentiles whose maximum defines HD95.

Figure 15: Directed ex vivo RMSE over execution. Rows show depth and directional tasks, and columns show the two policy configurations. Blue denotes target-to-observation RMSE and coral denotes observation-to-target RMSE.

Task	Configuration	Response	$r_{d}$	$W$
3-DoF depth
Depth	Full volume / Diffusion	Translation	0.854	1.221
Depth	Point cloud / Explicit	Translation	0.660	0.692
6-DoF directional
Directional	Full volume / Diffusion	Translation	0.926	0.984
Directional	Full volume / Diffusion	Rotation	0.974	2.051

Appendix

Table 15: Target-dependent ex vivo command responses. Statistics use the same definitions as in the phantom action-response analysis.

r_{d}

is the target-association score and

W

is the response spread.

	3-DoF translation		6-DoF translation		6-DoF rotation
Configuration	Demo avg.	Nearest demo	Demo avg.	Nearest demo	Demo avg.	Nearest demo
Full volume / Diffusion	1.148	0.697	0.790	0.852	0.911	0.478
Point cloud / Explicit	0.717	0.586	1.315	0.914	1.526	1.069

Appendix

Table 16: Ex vivo cumulative-command agreement with demonstrations. “Demo avg.” and “Nearest demo” use the same component-wise command sums and reference distances as Table 12 .

Figure 16: Representative ex vivo endpoint comparisons for Full-volume Diffusion + residual. Top row: depth targets $2.0$ – $4.0$ mm. Bottom row: directional targets $0^{\circ}$ – $90^{\circ}$ at depth $5.0$ mm and tilt strength $0.70$ . Blue shows the final observation and brown shows the target.

Figure 17: Representative ex vivo endpoint comparisons for Voxelized point-cloud Explicit with residual conditioning.

Figure 18: Demonstration collection and verification. Top: tooltip control with OCT and RGB feedback. Bottom: trajectory review with tissue–target overlays and geometric error.

Deformable Object Manipulation

Deformable Object Manipulation studies how robots perceive, model, and control flexible objects such as cloth, ropes, cables, and soft tissue.

See all 101 papers

Sep 6, 2026

OCTN: Neural OCT Representations for Robot-Guided Precision Intervention

Optical coherence tomography (OCT) offers compact, contactless, micron-scale imaging suitable for intraoperative guidance, but native OCT volumes are discretely sampled, anisotropic, and currently inefficient for downstream geometric reasoning and robot integration. We present OCTN (pronounced "octane"), an implicit neural representation framework that converts volumetric OCT scans into a continuous, differentiable, and spatially faithful tissue-intensity field. OCTN uses a two-stage hybrid training strategy that combines supervision from acquired voxels with inter-slice interpolations, preserving B-scan fidelity while improving continuity in sparsely sampled regions. For versatility, we first show that OCTN enables fast volumetric reasoning by storing the learned tissue representation natively on the GPU, supporting intensity-based spatial queries with up to 43x speedup over conventional CPU processing. We then demonstrate OCTN-enabled OCT-guided robotic laser surgery where the continuous tissue representation supports implicit surface discovery and surface-constrained path planning via multiple optimization strategies, including Newton- and SGD-based optimization. Next, OCTN enables reconstruction of dense volumetric structure from sparsely acquired B-scans, while reducing acquisition time by 4x and preserving clinically relevant structures. Across the newly generated Duke TissueOCT dataset and public OCT datasets, OCTN achieves robust, high-fidelity reconstruction with PSNR > 30 dB and training time < 10 s, while preserving surface consistency within 10

μ

m Chamfer distance relative to baseline reconstruction. The TissueOCT dataset and code are publicly available at raprakashvi.github.io/octn

Ravi Prakash, Ryan P. McNabb, Patrick J. Codd +1

Duke University, Durham, NC 27705 USA · Arizona State University, Phoenix, AZ 85287 USA

Optical Coherence Tomography Implicit Neural Representations Surgical Robotics

Ask PDF

Aug 30, 2026

Module Number Adaptive Visual Shape Control for Serial Modular Soft Robots

Image based shape control provides a simple means of controlling the whole body configuration of soft robots. However, existing data driven approaches are typically developed for fixed robot structures and require new control data when the number of modules changes. This paper presents a module number adaptive visual shape control method for serial modular soft pneumatic robots. A controller trained only on single module actuation shape data is reused for robots with one to five modules by decomposing whole body camera images into local module patches. A single common module segmenter localizes individual modules across all tested configurations, while the same local controller is applied to every extracted patch. Geometric data augmentation improves transferability to downstream modules, and a lightweight mask reconstruction network reconstructs a synthetically removed actuator mask channel. Experiments on physical robots demonstrate shape control across varying numbers of modules and under environmental changes and payload loading. The results show that single module control learning enables scalable whole body control without configuration specific control data collection.

Kyohei Akamine, Takato Horii, Yusuke Sakaue +1

Graduate School of Engineering Science, The University of Osaka, Osaka 560-0043, Japan · Faculty of Science and Technology, Sophia University, 7-1 Kioi-cho, Chiyoda-ku, Tokyo 102-8554, Japan

Soft Robotics Vision-Based Robot Control Modular Robotics

Ask PDF

May 18, 2026

Learning-Based Adaptive Control for Surgical Robotic Exposure Task on Deformable Tissues

In various surgical procedures, regions of interest (ROIs) such as organs or lesions are often occluded by overlying tissues, requiring surgeons to achieve adequate exposure for precise intervention. However, the irregular geometry, nonlinear biomechanical properties of overlying tissues, and limited intraoperative visibility of the ROI pose significant challenges to the autonomous execution of tissue retraction. To address this, we formulate a realistic model of the tissue retraction task and propose a learning-based adaptive control framework for achieving ROI exposure. The method optimizes control inputs online by monitoring changes in the visual boundary of the tissue, while leveraging a deep deformation estimation model trained on simulation data to identify the optimal grasping point and ensure the convergence and safety of the adaptive controller. Through simulations and real-world experiments on different deformable materials, it has been demonstrated that this framework exhibits zero-shot adaptation to similar tasks and can complete the autonomous retraction process, from initial grasp selection to full ROI exposure. Therefore, it has the potential to be applied in actual surgical assistance scenarios.

Jiayi Liu, Kaiqi Wei, Yiwei Wang +2

Huazhong University of Science and Technology, Wuhan, China

Deformable Object Manipulation Vision-Based Robot Control Surgical Robotics Adaptive Control

Ask PDF

Abstract

Deformable Object Manipulation

Explore similar work

OCTN: Neural OCT Representations for Robot-Guided Precision Intervention

Module Number Adaptive Visual Shape Control for Serial Modular Soft Robots

Learning-Based Adaptive Control for Surgical Robotic Exposure Task on Deformable Tissues