SARI: Phase-Split Sim-Real Co-Training for Contact-Rich Manipulation

Xingxin He; Yuxuan Jiang; Haonan Zhang; Chuhan Cui; Kaile Li; Zhongxing Zheng; Caihao Xu; Ziqi Wang

cs.ROOct 2, 2026

SARI: Phase-Split Sim-Real Co-Training for Contact-Rich Manipulation

Authors: Xingxin He, Yuxuan Jiang, Haonan Zhang, Chuhan Cui, Kaile Li, Zhongxing Zheng, Caihao Xu, Ziqi Wang

Organizations: The Hong Kong University of Science and Technology · BYD Company Limited

Abstract

Vision-language-action (VLA) models often require costly real-world demonstrations to adapt to contact-rich manipulation tasks, particularly when generalization across object placements is needed. We propose SARI (Simulated Approach, Real Interaction), a phase-split sim-and-real co-training framework built on a simple insight: spatial coverage and contact physics should be acquired from the domains best suited to them. Specifically, free-space approaches require spatial diversity but tolerate modest simulation gaps, making them ideal for synthetic generation; conversely, contact interactions demand accurate physics but vary little across object placements, allowing a few real demonstrations to generalize across the workspace. SARI generates diverse simulated approaches in a photorealistic digital twin while collecting real contact interactions at only a few placements. Post-trained on these phase-segmented demonstrations, a single policy seamlessly stitches simulated approaches with real contact interactions using visual appearance alignment and a shared camera-relative action representation--without explicit phase labels or hand-coded switches. Across five real-world contact-rich manipulation tasks, SARI reduces real-data collection time by 34.3% and achieves 27.5% success at unseen placements, where all full-task sim-real baselines fail completely (0%).

Figures & tables

Fig. 2: Overview of SARI . We reconstruct a digital twin of the real workspace, generate spatially diverse simulated approach segments, and collect real contact segments. Appearance-aligned observations and a shared camera-relative action representation allow a single policy to post-train on their union and execute the full task in the real world.

Method	Metric	Seen		Unseen				Mean
		P1	P3	P2	P4	P5	P6	Seen	Unseen
Full Sim	Approach	100	100	100	100	100	100	100.0	100.0
Full Sim	Completion	0	0	0	0	0	0	0.0	0.0
Full Real	Approach	100	100	100	0	0	0	100.0	25.0
Full Real	Completion	50	50	0	0	0	0	50.0	0.0
Full-task Co-training [ 9 ]	Approach	100	100	100	0	0	0	100.0	25.0

TABLE I: Lamp-task performance (%). Each object placement is evaluated in 10 trials. Approach denotes approaching the pre-contact region, and Completion denotes successful completion of the lamp-pressing task.

Method	Sim. demos	Real demos	Real time (min)
Full Sim	180 full-task	–	0
Full Real	–	60 full-task	28.03
Full-task Co-training [ 9 ]	180 full-task	60 full-task	28.03
SARI (Ours)	180 approaches	60 contacts	18.42

TABLE II: Data allocation and real collection cost for the lamp task. SARI uses simulated approaches and real contacts; all methods use matched initialization, post-training compute, and evaluation.

Fig. 3: Lamp-task placement protocol. The six candidate lamp placements form a $2\times 3$ grid with 25 cm lateral and 15 cm depth spacing. Simulated approach data cover all placements (pink markers), while real contact data are collected only at P1 and P3 (blue markers). The remaining four placements are unseen for real contact demonstrations and are used for evaluation.

Fig. 4: Representative real-world task trajectories. From top to bottom: press button, open lamp, open drawer, open book, and pull tissue. Each row progresses from the home configuration to task completion.

Task	Seen	Unseen
Book	87.5	81.25
Button	87.5	75.0
Tissue	62.5	43.75
Lamp	50.0	27.5
Drawer	50.0	25.0

TABLE III: Five–task success rates (%). The lamp task is evaluated in ten trials per placement, while all other tasks are evaluated in four trials per placement. Seen: P1/P3; Unseen: P2/P4–P6. Unseen placements have no real contact demonstrations.

Fig. 5: Appearance configurations evaluated in Table IV . From left to right: low-fidelity rendering; low-fidelity rendering with color correction; high-fidelity rendering with color correction and corrupted GS; and high-fidelity rendering with color correction and correct GS.

Configuration			Success (%)
Rendering	Color corr.	GS corr.	$\pi_{0.5}$ [ 1 ]	GR00T N1.7 [ 2 ]
Low	–	$\checkmark$	8.3	3.3
Low	$\checkmark$	$\checkmark$	13.3	11.7
High	$\checkmark$	–	18.3	13.3
High (Ours)	$\checkmark$	$\checkmark$	35.0	25.0

TABLE IV: Appearance-ablation success rates on the lamp task (%). We compare four visual configurations under

\pi_{0.5}

and GR00T N1.7 pretraining. Checkmarks denote the listed condition. Each policy and configuration is evaluated in 60 trials (10 trials per placement across six object placements).

Configuration	Seen	Unseen
Joint velocity	0.00	0.00
Joint position	0.00	0.00
EEF-relative	35.00	20.00
Camera-relative	50.00	27.50

TABLE V: Action-representation success rates on the lamp task (%). EEF-relative uses GR00T N1.7 [ 2 ] ; all other configurations use

\pi_{0.5}

[ 1 ] , and camera-relative is ours. Seen: P1/P3; Unseen: P2/P4–P6.

Explore similar work

TacCoRL: Integrating Tactile Feedback into VLA via Simulation
Jun 10, 2026Siyu Ma, Yuqi Liang, Chang Yu +5RL for Robotics Robot Skill Learning
University of California, Los Angeles · University of California, San Diego · 4Peking University +1
SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation
Jun 13, 2026Wenkang Hu, Haoran Wang, Yitong Li +10Zero-Shot Robotic Manipulation Robot Skill Learning
1Shanghai Jiao Tong University · 2Horizon Robotics · 3Style3D Research
ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation
Sep 21, 2026Bowei Li, Yuner Zhang, Changliu LiuRobotics Simulation Sim-to-Real Transfer
Carnegie Mellon University, Pittsburgh, PA 15213, USA

Jun 10, 2026, cs.ROJ/K move · Enter open · S save

TacCoRL: Integrating Tactile Feedback into VLA via Simulation

Siyu Ma, Yuqi Liang, Chang Yu, Yunuo Chen+4

University of California, Los Angeles · University of California, San Diego · 4Peking University+1

RL for Robotics Robot Skill Learning Contact-Rich Robotic Manipulation Sim-to-Real Transfer Vision-Language-Action Models Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

Vision-language-action (VLA) models provide strong visual, language, and action priors for robot manipulation, but visual observations alone often miss the local contact state required for contact-rich tasks. We present TacCoRL, a scalable framework that injects Tactile feedback into VLA policies and improves them through sim-real Co-training and simulation-based reinforcement learning (RL), without requiring large-scale tactile pretraining or extensive real-world contact exploration. The key idea is not only adding touch as an input, but learning how contact readings should modulate action responses in near-failure states that are rare in demonstrations and risky to collect on hardware. We use a real-aligned simulator as a closed-loop training environment for contact interaction. Mixed simulated and real trajectories first warm-start tactile-conditioned actions in the pretrained policy. Reinforcement learning with verifiable task rewards then optimizes the policy using simulated contact rollouts. It reinforces tactile-conditioned actions that lead to task completion, while a supervised objective on real trajectories keeps the refined policy anchored to deployment visual, tactile, and action distributions. The resulting policy transfers directly to the real robot without privileged simulation state or online real-world RL. Across four bimanual contact-rich tasks, the final visuo-tactile policy achieves an average success rate of 72.5%, compared to baseline of 50.0%. Result videos and more details are available at https://tac-corl.github.io/

Abstract

Explore similar work

TacCoRL: Integrating Tactile Feedback into VLA via Simulation

SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation