GeCo-SRT: Geometry-aware Continual Adaptation for Robotic Cross-Task Sim-to-Real Transfer

Wenbo Yu; Wenke Xia; Weitao Zhang; Di Hu

cs.ROFeb 24, 2026

GeCo-SRT: Geometry-aware Continual Adaptation for Robotic Cross-Task Sim-to-Real Transfer

Authors: Wenbo Yu, Wenke Xia, Weitao Zhang, Di Hu

Organizations: Beijing Forestry University, Beijing, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE

Abstract

Bridging the sim-to-real gap is important for applying low-cost simulation data to real-world robotic systems. However, previous methods are severely limited by treating each transfer as an isolated endeavor, demanding repeated, costly tuning and wasting prior transfer experience. To move beyond isolated sim-to-real, we build a continual cross-task sim-to-real transfer paradigm centered on knowledge accumulation across iterative transfers, thereby enabling effective and efficient adaptation to novel tasks. Thus, we propose GeCo-SRT, a geometry-aware continual adaptation method. It utilizes domain-invariant and task-invariant knowledge from local geometric features as a transferable foundation to accelerate adaptation during subsequent sim-to-real transfers. This method starts with a geometry-aware mixture-of-experts module, which dynamically activates experts to specialize in distinct geometric knowledge to bridge observation sim-to-real gap. Further, the geometry-expert-guided prioritized experience replay module preferentially samples from underutilized experts, refreshing specialized knowledge to combat forgetting and maintain robust cross-task performance. Leveraging knowledge accumulated during iterative transfer, GeCo-SRT method not only achieves 52% average performance improvement over the baseline, but also demonstrates significant data efficiency for new task adaptation with only 1/6 data. We hope this work inspires approaches for efficient, low-cost cross-task sim-to-real transfer.

Figures & tables

Figure 1: Our GeCo-SRT method accumulates transferable geometric knowledge from sequential sim-to-real tasks (Task $1...N$ ), which facilitates effective and efficient transfer to novel tasks.

Figure 2: Overview of our geometry-aware continual adaptation method. (a) demonstrates the sim-to-real transfer with human correction pipeline, which translates sim-to-real heuristics into learnable human correction trajectories. (b) represents the Geometry-aware Mixture-of-Experts module, which extracts local geometric features to dynamically activate experts, enabling them to specialize in distinct geometric knowledge for effective sim-to-real transfer. (c) illustrates the Geometry-expert-guided Prioritized Experience Replay, which leverages expert utilization to prioritize historical data, thereby mitigating catastrophic forgetting and promoting knowledge reuse.

Figure 3: We demonstrate the task setup and real-world setup. (a) illustrates 4 robotic manipulation task in simulation and real-world scenarios. (b) shows that the real-world setup, which consists of a XARM equipped with a robotiq-140 gripper.

Method	Pick Cube	Stack Cube	Pick Banana	Plug Insert	Avg. SR
Direct Deploy	5.7%	0%	6.7%	0%	3.1%
Action Residual	16.7%	3.3%	13.3%	0.0%	9.2%
Transic 1 1 1 We reproduced this method, adapting it to align with the distinct point cloud configuration utilized in our approach.	66.7%	30.0%	23.3%	33.3%	38.3%
Geo-MoE	80.0%	43.3%	40.0%	36.7%	50.0%

Table 1: Single task sim-to-real transfer performance. Our Geo-MoE significantly outperforms all baselines, demonstrating the advantage of specific activated geometric knowledge in bridging the sim-to-real gap.

Method	Pick Cube		Stack Cube		Pick Banana		Plug Insert		Avg.
Method	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$
Direct Deploy	$5.7\%$	/	$0.0\%$	/	$6.7\%$	/	$0.0\%$	/	$3.1\%$	/
Naive Fine-tuning	$16.7\%$	$100\%$	$3.3\%$	$100.0\%$	$13.3\%$	$100\%$	$3.3\%$	/	$9.2\%$	$75.0\%$
Transic + PER	$76.7\%$	$81.2\%$	$30.0\%$	$72.3\%$	$20.0\%$	$66.5\%$	$33.3\%$	/	$40.0\%$	$55.0\%$
Geo-MoE + PER	$83.3\%$	$34.6\%$	$50.0\%$	$76.8\%$	$46.7\%$	$7.1\%$	$43.3\%$	/	$55.7\%$	$29.6\%$
Geo-MoE + EWC	$80.0\%$	$70.8\%$	$36.7\%$	$77.3\%$	$20.0\%$	$50.0\%$	$16.7\%$	/	$38.3\%$	$49.5\%$

Table 2: Continual learning performance across four sequential sim-to-real tasks (learned in the order: Pick Cube

\to

Stack Cube

\to

Pick Banana

\to

Plug Insert). Our method (GeCo-SRT) achieves the best overall performance, with the highest average success rate and lowest average forgetting. This performance demonstrates Geo-MoE’s advantage in leveraging specific geometric knowledge to bridge the sim-to-real gap, combined with Geo-PER’s ability to mitigate catastrophic forgetting by continuously updating idle experts.

Observation Residual	MoE	Pick Cube		Stack Cube		Pick Banana		Plug Insert		Avg.
Observation Residual	MoE	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$	$SR\uparrow$	$N-NBT\downarrow$
$\times$	$\times$	$6.7\%\text{ }$	$100\%$	$3.3\%$	$100.0\%$	$3.3\%$	$100.0\%$	$0.0\%$	/	$3.3\%$	$75.0\%$
$\times$	$\checkmark$	$16.7\%$	$86.7\%$	$3.3\%$	$100.0\%$	$13.3\%$	$75.2\%$	$3.3\%$	/	$9.2\%$	$65.5\%$
$\checkmark$	$\times$	83.3%	$37.3\%$	$46.7\%$	$96.5\%\text{ }$	$23.3\%$	$14.2\%$	$30.0\%$	/	$45.8\%$	$37.0\%$
$\checkmark$	$\checkmark$	83.3%	34.6%	50.0%	76.8%	46.7%	7.1%	43.3%	/	55.8%	29.6%

Table 3: Ablation study on core components of Geo-MoE. For the Observation Residual component,

\checkmark

indicates using our point cloud encoder to align the observation sim-to-real gap, whereas

\times

denotes using the action residual module. For the MoE component,

\checkmark

indicates using our mixture-of-experts architecture to obtain distinct knowledge, while

\times

represents the baseline without it.

Figure 4: Data efficiency comparison of training “From Scratch” vs. “From Continued”. The performance gain (hatched area) highlights our method’s ability to utilize transferable geometric knowledge for cross-task sim-to-real transfer, which enhances data efficiency when adapting to new tasks, especially in low-data regimes.

Evaluate Task	Pre-trained Task	SR
StackCube	PickCube	40.0%
	PickBanana	33.3%
	PlugInsert	16.7%
	From scratch	26.7%
	From scratch (60 traj.)	43.3%
PlugInsert	PickCube	33.3%

Table 4: Impact of task similarity on knowledge transfer. All models use 10 trajectories unless otherwise specified.

Figure 5: Real-world workspace setup for human-in-the-loop data collection. The human operator provides online correction through a 3Dconnexion SpaceMouse (marked with a red box) while monitoring the robot’s execution.

Figure 6: Qualitative visualization of action adaptation on the “Stack Cube” and “Plug Insert” task. (Top) In the initial phase (Task 1), human correction trajectories are used to rectify the failure modes of direct deployment. (Bottom) In the continual learning phase (Task N), our Geo-MoE model utilizes the Re-Use mechanism to achieve success with significantly less new correction data compared to the baseline, demonstrating high data efficiency.

Number of Experts ( $N$ )	Avg. SR (%) $\uparrow$	NBT (%) $\downarrow$
2	60.0	40.0
3	66.7	33.3
8	65.0	30.0

Table 5: Sensitivity Analysis of Expert Numbers (

N

). We evaluate the impact of

N

on Success Rate (SR) and Net Benefit of Transfer (NBT). Performance remains robust across configurations, with

N=3

providing an optimal balance between efficiency and specialization.

Figure 7: Qualitative Visualization of Expert Specialization. Different colors indicate the routing preference of each expert. Our MoE layer naturally learns to partition the input space based on geometric primitives, specializing in edges, corners, and planar surfaces.

Method	Faucet (%) $\uparrow$	Tidying (%) $\uparrow$
Direct Deploy	10.0	0.0
Geo-MoE (zero-shot)	53.3	30.0
Geo-MoE (scratch)	76.6	43.3
Geo-MoE (continual)	83.3	56.7

Table 6: Quantitative Results on New Tasks. Success Rate (SR) for Faucet and Tidying tasks. Our Geo-MoE (continual) significantly outperforms zero-shot and scratch-trained baselines.

Figure 8: Visualization of Newly Integrated Tasks. Experimental setups for Faucet and Tidying scenarios. Our method effectively handles these novel geometries through efficient knowledge transfer.

Figure 9: Qualitative Results of Depth Completion. Our method effectively restores missing depth information in geometrically complex regions, providing a dense, noise-resilient point cloud for downstream manipulation.

Method	$T1^{\dagger}$ (%) $\uparrow$	$T2^{\dagger}$ (%) $\uparrow$
Direct Deploy	3.3	0.0
Domain Randomization	10.0	6.7
Ours	40.0	20/33 ∗

Table 7: Success Rate (SR) with RGB Input. Comparison under RGB-only observations.

T1^{\dagger}

and

T2^{\dagger}

denote test scenarios with increased geometric complexity. ∗ indicates results under partial observation.

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 316 papers

An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer
Mar 13, 2025Yuxuan Xu, Shiyu Wang, Jinhao Huang +6Robotic Data Collection Sim-to-Real Transfer
Institute for AI Industry Research (AIR), Tsinghua University · Beijing Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou)
DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation
May 3, 2026Zijian Zeng, Fei Ding, Huiming Yang +2Domain Randomization Dexterous Manipulation
Tsinghua University · Alibaba Group · Bengbu University +1
RoboBridge: A Self-Evolving Embodied Agent Framework for Sim-to-Real Transfer
Oct 2, 2026Chenxi Li, Zhangrui Zhao, Rui Li +8Sim-to-Real Transfer Robot Policy Adaptation
Shanghai Artificial Intelligence Laboratory · Zhejiang University · Beihang University +3

Mar 13, 2025J/K move · Enter open · S save

An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer

Yuxuan Xu, Shiyu Wang, Jinhao Huang, Wenhao Zhao+5

Institute for AI Industry Research (AIR), Tsinghua University · Beijing Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou)

Robotic Data Collection Sim-to-Real Transfer RL Exploration Robot Policy Generalization

Open paper AskPDF arXiv

The sim-to-real gap remains a critical challenge in robotics, hindering the deployment of algorithms trained in simulation to real-world systems. We propose a flexible Real-to-Sim-to-Real (RSR) framework whose central contribution is an information-theoretic cost function that explicitly accounts for sim-to-real discrepancies. This cost balances two objectives, completing the task and steering the policy to collect real-world samples that are maximally informative for improving transfer. It can be integrated seamlessly into existing reinforcement learning algorithms (e.g., PPO, SAC) and ensures a balanced exploration of critical regions in the real domain. The framework treats differentiable simulation as optional: when a differentiable simulator is available, the collected informative data can also be used to tune simulator parameters. We implement the framework with the MuJoCo MJX platform and demonstrate its generality by evaluating on both manipulation tasks with a 6-DOF robotic arm and locomotion tasks on a legged robot. Empirical results show that our RSR loop yields more efficient data acquisition and substantially improves task performance in real-world that achieves a smoother sim-to-real transfer.

Abstract

Sim-to-Real Transfer

Explore similar work

An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

RoboBridge: A Self-Evolving Embodied Agent Framework for Sim-to-Real Transfer