Task-Relevant Feature-Dynamics Fidelity Enables Zero-Shot Sim-to-Real Transfer for Robotic Ultrasound Scanning

Yizhao Qian; Jiayuan Luo; Wanyi Zhu; Yameng Zhang; Max Q. -H. Meng; Yixuan Yuan; Li Liu

cs.ROAug 30, 2026

Task-Relevant Feature-Dynamics Fidelity Enables Zero-Shot Sim-to-Real Transfer for Robotic Ultrasound Scanning

Authors: Yizhao Qian, Jiayuan Luo, Wanyi Zhu, Yameng Zhang, Max Q. -H. Meng, Yixuan Yuan, Li Liu

Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · School of Advanced Engineering, Great Bay University, and the Dongguan Great Bay Institute for Advanced Study, Dongguan, China · Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Shenzhen Key Laboratory of Robotics Perception and Intelligence and the Department of Electronic Engineering, Southern University of Science and Technology, Shenzhen, China

Abstract

Robotic ultrasound policies operating directly on B-mode images require extensive interaction data, whereas real-robot data collection is costly and safety-constrained. Simulation provides a scalable alternative, but zero-shot transfer depends not only on single-frame realism but also on whether simulated observations reproduce task-relevant feature changes induced by probe motion. We term this cross-domain consistency task-relevant feature-dynamics fidelity (TR-FDF). Under local regularity assumptions, our contraction analysis shows that greater sensitivity of TR-FDF mismatch to probe motion reduces the effective closed-loop contraction margin, whereas motion-independent errors primarily enlarge the residual error bound. Guided by this analysis, we develop a TR-FDF-oriented ultrasound simulator that combines a shared structural intermediate domain, trajectory-level fixed noise, and few-step conditional flow generation. In phantom experiments, a policy trained exclusively in simulation succeeded in 390 of 400 zero-shot deployments across four target planes. The simulator achieved an FID of 29.66 and generated observations at 67.1 Hz. Controlled interventions, ablations, and baseline comparisons showed that TR-FDF sensitivity complements single-frame realism in predicting zero-shot transfer performance.

Figures & tables

Fig. 1: Problem framework and the sim-to-real gap for raw B-mode observations. We identify TR-FDF sensitivity as a complementary metric beyond single-frame realism: the same probe motion should induce consistent task-relevant feature changes in simulation and on the real system. Motion-independent feature-transition mismatch and static residuals mainly enlarge the residual-error region around the target, whereas the local sensitivity of TR-FDF mismatch to probe motion reduces the effective contraction margin of the real closed loop.

Fig. 2: Two effects of raw B-mode observation mismatch on closed-loop transfer. Feature-transition mismatch under a stationary probe, $d_{0}$ , together with static task-feature bias and other residual terms, $B$ , enters the additive term $\beta=Cd_{0}+B$ and mainly enlarges the residual-error region around the target. Under probe motion, the local TR-FDF sensitivity $L_{\mathrm{TR}}$ enters the effective contraction factor and reduces the contraction margin.

Fig. 3: Observation-generator architecture. Probe poses are converted into structural masks, encoded into a shared structural latent space in Stage I, rendered as B-mode observations by the CFM generator in Stage II, and accelerated by reflow distillation in Stage III. Fixed- $X_{0}$ trajectory rendering suppresses motion-independent sampling variation.

Fig. 4: Experimental platform and abdominal phantom. (a) Real-robot autonomous ultrasound platform, including the UR5 manipulator, Mindray M9 ultrasound scanner, C5-1S convex probe, force sensor, and custom abdominal phantom. (b) Internal layout of the custom abdominal phantom.

Plane	Trials	FID	TR-FDF sensitivity	Speed	Success	98% CI	Pos. err. (mm)	Rot. err. (deg)	Duration (s)	Failure type
A1	100	29.66	31.61	67.1 Hz	98/100	91.5–99.6	$2.83\pm 0.95$	$3.88\pm 0.86$	$28.52\pm 11.14$	2 timeouts
A2					100/100	94.9–100.0	$2.77\pm 0.40$	$3.79\pm 0.99$	$23.79\pm 8.38$	No Failure
A3					100/100	94.9–100.0	$2.44\pm 0.80$	$4.13\pm 0.91$	$25.30\pm 7.90$	No Failure
A4					92/100	83.3–96.4	$2.61\pm 1.14$	$4.11\pm 1.23$	$28.62\pm 11.48$	8 timeouts
Overall	400	29.66	31.61	67.1 Hz	390/400	95.0–98.8	$2.66\pm 0.88$	$3.98\pm 1.02$	$26.56\pm 10.07$	10 timeouts

TABLE I: Real-robot zero-shot deployment results for four target planes and overall results.

Fig. 5: Real-robot zero-shot convergence trajectories for four target planes. Each panel shows top-down XY trajectories on A1–A4, with 100 episodes per plane and 400 deployments in total. Green denotes successful episodes, and red denotes timeout failures; the center marker indicates the target position, and the surrounding region indicates the $3\,\mathrm{mm}$ translational success threshold.

Fig. 6: Zero-shot deployment performance. (a) Success rate, terminal position error, and terminal rotation error for the four target planes, reported with means and standard deviations. (b) Changes in mean position and rotation errors.

Position error (mm)	FID	TR-FDF sensitivity	Sim. success (98% CI)	Real success (98% CI)	Train-real gap	Terminal err. (mm)
0 (clean)	0.815	10.99	100/100 [94.9, 100.0]	100/100 [94.9, 100.0]	0.0 pp	$1.86\pm 0.11$
2.5	1.100	11.59	100/100 [94.9, 100.0]	98/100 [91.5, 99.6]	+2.0 pp	$2.49\pm 1.13$
3.5	0.665	12.98	100/100 [94.9, 100.0]	89/100 [79.6, 94.4]	+11.0 pp	$2.27\pm 1.77$
5.0	0.665	18.09	100/100 [94.9, 100.0]	69/100 [57.5, 78.5]	+31.0 pp	$4.15\pm 3.78$
7.5	0.712	21.07	94/100 [85.9, 97.6]	35/100 [24.9, 46.6]	+59.0 pp	$7.92\pm 5.15$
10.0	0.732	22.38	78/100 [67.1, 86.1]	11/100 [5.6, 20.4]	+67.0 pp	$12.33\pm 5.04$

TABLE II: Results of the real-image lookup intervention.

Fig. 7: Relationship between TR-FDF sensitivity and sim-to-real success under real-image lookup intervention. (a) Simulator and real-robot success rates as a function of TR-FDF sensitivity. (b) Real-robot XY trajectories under different perturbation strengths; as the perturbation increases, the trajectories gradually shift from concentrated convergence to divergent failure.

Morphology error (mm)	FID	TR-FDF sensitivity	Sim. success (98% CI)	Real success (98% CI)	Train-real gap	Terminal err. (mm/deg)
0 (clean)	29.66	31.61	100/100 [94.9, 100.0]	98/100 [91.5, 99.6]	+2.0 pp	$2.83\pm 0.95$ / $3.88\pm 0.86^{\circ}$
$\approx 2.0$	31.29	59.94	100/100 [94.9, 100.0]	100/100 [94.9, 100.0]	0.0 pp	$2.63\pm 0.61$ / $3.76\pm 1.04^{\circ}$
$\approx 3.0$	31.23	123.49	100/100 [94.9, 100.0]	97/100 [90.0, 99.1]	+3.0 pp	$2.92\pm 1.11$ / $3.96\pm 0.97^{\circ}$
$\approx 4.5$	30.78	133.31	99/100 [93.1, 99.9]	100/100 [94.9, 100.0]	-1.0 pp	$2.78\pm 0.25$ / $3.77\pm 0.89^{\circ}$
$\approx 5.0$	31.15	183.75	100/100 [94.9, 100.0]	73/100 [61.7, 81.9]	+27.0 pp	$19.85\pm 31.95$ / $3.64\pm 2.14^{\circ}$
$\approx 5.5$	30.70	262.64	100/100 [94.9, 100.0]	2/100 [0.4, 8.5]	+98.0 pp	$48.13\pm 20.69$ / $5.17\pm 1.84^{\circ}$

TABLE III: Results of the learned-simulator structural intervention.

Fig. 8: Relationship between TR-FDF sensitivity and sim-to-real success under structural intervention in the learned simulator. (a) Simulator and real-robot success rates as a function of TR-FDF sensitivity. (b) Real-robot XY trajectories under different perturbation strengths; under large perturbations, convergent trajectories and a separate divergent cluster appear simultaneously. Green and red denote successful and failed episodes, respectively.

Experiment	Ablation setting	TR-FDF sensitivity	FID	Real success	Trans. err. (mm)	Rot. err. (deg)	Sim. success	Speed (Hz)
Baseline	No ablation	31.61	29.66	98/100	$2.83\pm 0.95$	$3.88\pm 0.86$	100%	67.1
Ablation 1	Random frame-level $X_{0}$	50.78	43.36	55/100	$3.29\pm 4.82$	$6.16\pm 2.04$	100%	68.4
Ablation 2	No Stage-I shaping	244.26	31.14	5/100	$20.44\pm 17.62$	$11.98\pm 5.80$	100%	76.4
Ablation 3	Weak Stage-I shaping	164.61	53.05	17/100	$6.01\pm 3.54$	$6.61\pm 1.64$	100%	64.9
Ablation 4	GAN-based generator	92.03	210.83	18/100	$5.95\pm 7.79$	$10.93\pm 4.73$	100%	130.7
Ablation 5	No reflow distillation	30.43	27.83	99/100	$2.54\pm 0.68$	$3.41\pm 0.81$	100%	3.07

TABLE IV: Component-level ablation results for the proposed simulator.

Fig. 9: Real-robot XY trajectories for the component-level ablations. Green and red denote successful and failed episodes, respectively; all panels use the same target and coordinate ranges.

TR-FDF-related latent dynamics
Ablation setting	TR-FDF sensitivity	FID
Full Stage I	31.61	29.66
No distance matching	35.01	29.98
No mask-latent smoothness	138.34	59.45
No appearance invariance	49.18	40.97
Non-collapse and shared structural representation

TABLE V: Ablation results for the key Stage-I losses.

Screening stage	Method	FPS	FID	TR-FDF sensitivity	Sim. success	Real success	Final error (mm/deg)
I2I-training failure	Recycle-GAN (2018) [ 47 ]	75.3	153.59	–	–	–	–
I2I-training failure	Unsup-Recycle-GAN (2023) [ 48 ]	71.3	312.23	–	–	–	–
SAC-training failure	UNIT-DDPM (2021) [ 49 ]	10.6	34.87	47.23	0/100	–	–
SAC-training failure	UNest (2024) [ 50 ]	179.8	51.90	126.78	0/100	–	–
SAC-training failure	STABLE (2025) [ 51 ]	105.5	45.31	93.71	0/100	–	–
Real-robot evaluation	CycleGAN (2017) [ 16 ]	146.8	56.09	89.45	100/100	1/100	$24.52\pm 16.05$ / $17.66\pm 4.34$

TABLE VI: Comparison with classical methods.

Contact force (N)	Real-robot success	Failure type
5	94/100	6 timeouts
15	98/100	2 timeouts

TABLE VII: Real-robot zero-shot deployment results under contact-force variation.

Fig. 11: Real-robot tracking results when the phantom moves along spiral and square trajectories. Blue denotes the target trajectory, and red denotes the actual probe trajectory.

Explore similar work

BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer
Jul 1, 2026Yunfu Deng, Josiah P. HannaZero-Shot Learning Robot Policy Learning
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA
Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound
Jul 24, 2026Siqi Fan, Mingcong Chen, Ran Liu +5World Model Learning Action-Conditioned World Models
Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Hong Kong SAR · Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong SAR · Department of Biomedical Engineering, City University of Hong Kong, Hong Kong SAR +3
Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning
Jul 2, 2026Satoshi Yamamori, Koji Ishihara, Kenjiro Minamikawa +4Robotic Control Sim-to-Real Transfer
Graduate School of Informatics, Kyoto University, Kyoto, Japan · Dept. of Brain Robot Interface, Computational Neuroscience Labs, ATR, Kyoto, Japan

Jul 1, 2026, cs.ROJ/K move · Enter open · S save

BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer

Yunfu Deng, Josiah P. Hanna

Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA

Zero-Shot Learning Robot Policy Learning Representation Learning Sim-to-Real Transfer Invariant Representation Learning

Open paper AskPDF arXiv

Sim2real transfer for robot policy learning suffers due to mismatch between simulation and reality. Existing methods typically address each gap in isolation through separate adaptation modules, which are composed or layered when both gaps coexist. Yet the basis for attempting sim2real in the first place is that there is shared structure between a task in simulation and reality, where equivalent actions from equivalent configurations produce equivalent long term outcomes regardless of domain specific differences in rendering or physics. In this paper, we study whether we can identify and exploit this shared structure from raw observations to train a policy that enables zero shot transfer. We introduce BIFROST, which learns a shared history encoder on paired cross-domain data via cross-domain bisimulation objective: observation-action sequences leading to equivalent long-term behavior are mapped to nearby latent states, regardless of domain. Policies trained on these latent states in simulation transfer zero-shot to reality. We provide empirical evidence on sim2sim visual navigation and sim2real contact rich manipulation task and visual servoing task that BIFROST achieves effective transfer where domain adaptation and co-training baselines fail under both visual and dynamics domain gaps.

Abstract

Explore similar work

BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer

Action-Conditioned World Model for Goal Plane Probe Guidance in Robotic Ultrasound

Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning