Vision-language-action (VLA) models often require costly real-world demonstrations to adapt to contact-rich manipulation tasks, particularly when generalization across object placements is needed. We propose SARI (Simulated Approach, Real Interaction), a phase-split sim-and-real co-training framework built on a simple insight: spatial coverage and contact physics should be acquired from the domains best suited to them. Specifically, free-space approaches require spatial diversity but tolerate modest simulation gaps, making them ideal for synthetic generation; conversely, contact interactions demand accurate physics but vary little across object placements, allowing a few real demonstrations to generalize across the workspace. SARI generates diverse simulated approaches in a photorealistic digital twin while collecting real contact interactions at only a few placements. Post-trained on these phase-segmented demonstrations, a single policy seamlessly stitches simulated approaches with real contact interactions using visual appearance alignment and a shared camera-relative action representation--without explicit phase labels or hand-coded switches. Across five real-world contact-rich manipulation tasks, SARI reduces real-data collection time by 34.3% and achieves 27.5% success at unseen placements, where all full-task sim-real baselines fail completely (0%).
Figures & tables
Fig. 2: Overview of SARI . We reconstruct a digital twin of the real workspace, generate spatially diverse simulated approach segments, and collect real contact segments. Appearance-aligned observations and a shared camera-relative action representation allow a single policy to post-train on their union and execute the full task in the real world.
Method
Metric
Seen
Unseen
Mean
P1
P3
P2
P4
P5
P6
Seen
Unseen
Full Sim
Approach
100
100
100
100
100
100
100.0
100.0
Completion
0
0
0
0
0
0
0.0
0.0
Full Real
Approach
100
100
100
0
0
0
100.0
25.0
Completion
50
50
0
0
0
0
50.0
0.0
Full-task Co-training [ 9 ]
Approach
100
100
100
0
0
0
100.0
25.0
TABLE I: Lamp-task performance (%). Each object placement is evaluated in 10 trials. Approach denotes approaching the pre-contact region, and Completion denotes successful completion of the lamp-pressing task.
Method
Sim. demos
Real demos
Real time (min)
Full Sim
180 full-task
–
0
Full Real
–
60 full-task
28.03
Full-task Co-training [ 9 ]
180 full-task
60 full-task
28.03
SARI (Ours)
180 approaches
60 contacts
18.42
TABLE II: Data allocation and real collection cost for the lamp task. SARI uses simulated approaches and real contacts; all methods use matched initialization, post-training compute, and evaluation.
Fig. 3: Lamp-task placement protocol. The six candidate lamp placements form a 2×3 grid with 25 cm lateral and 15 cm depth spacing. Simulated approach data cover all placements (pink markers), while real contact data are collected only at P1 and P3 (blue markers). The remaining four placements are unseen for real contact demonstrations and are used for evaluation.
Fig. 4: Representative real-world task trajectories. From top to bottom: press button, open lamp, open drawer, open book, and pull tissue. Each row progresses from the home configuration to task completion.
Task
Seen
Unseen
Book
87.5
81.25
Button
87.5
75.0
Tissue
62.5
43.75
Lamp
50.0
27.5
Drawer
50.0
25.0
TABLE III: Five–task success rates (%). The lamp task is evaluated in ten trials per placement, while all other tasks are evaluated in four trials per placement. Seen: P1/P3; Unseen: P2/P4–P6. Unseen placements have no real contact demonstrations.
Fig. 5: Appearance configurations evaluated in Table IV . From left to right: low-fidelity rendering; low-fidelity rendering with color correction; high-fidelity rendering with color correction and corrupted GS; and high-fidelity rendering with color correction and correct GS.
Configuration
Success (%)
Rendering
Color corr.
GS corr.
π0.5 [ 1 ]
GR00T N1.7 [ 2 ]
Low
–
✓
8.3
3.3
Low
✓
✓
13.3
11.7
High
✓
–
18.3
13.3
High (Ours)
✓
✓
35.0
25.0
TABLE IV: Appearance-ablation success rates on the lamp task (%). We compare four visual configurations under π0.5 and GR00T N1.7 pretraining. Checkmarks denote the listed condition. Each policy and configuration is evaluated in 60 trials (10 trials per placement across six object placements).
Configuration
Seen
Unseen
Joint velocity
0.00
0.00
Joint position
0.00
0.00
EEF-relative
35.00
20.00
Camera-relative
50.00
27.50
TABLE V: Action-representation success rates on the lamp task (%). EEF-relative uses GR00T N1.7 [ 2 ] ; all other configurations use π0.5 [ 1 ] , and camera-relative is ours. Seen: P1/P3; Unseen: P2/P4–P6.
Vision-language-action (VLA) models provide strong visual, language, and action priors for robot manipulation, but visual observations alone often miss the local contact state required for contact-rich tasks. We present TacCoRL, a scalable framework that injects Tactile feedback into VLA policies and improves them through sim-real Co-training and simulation-based reinforcement learning (RL), without requiring large-scale tactile pretraining or extensive real-world contact exploration. The key idea is not only adding touch as an input, but learning how contact readings should modulate action responses in near-failure states that are rare in demonstrations and risky to collect on hardware. We use a real-aligned simulator as a closed-loop training environment for contact interaction. Mixed simulated and real trajectories first warm-start tactile-conditioned actions in the pretrained policy. Reinforcement learning with verifiable task rewards then optimizes the policy using simulated contact rollouts. It reinforces tactile-conditioned actions that lead to task completion, while a supervised objective on real trajectories keeps the refined policy anchored to deployment visual, tactile, and action distributions. The resulting policy transfers directly to the real robot without privileged simulation state or online real-world RL. Across four bimanual contact-rich tasks, the final visuo-tactile policy achieves an average success rate of 72.5%, compared to baseline of 50.0%. Result videos and more details are available at https://tac-corl.github.io/
Siyu Ma, Yuqi Liang, Chang Yu +5
University of California, Los Angeles · University of California, San Diego · 4Peking University +1
RGB sim-to-real for deformable manipulation has remained largely unsolved without real-world fine-tuning. We present SimWeaver, which trains zero-shot RGB VLA policies on 200 simulated demonstrations per task, reaching above 80% per-task and 91% average real-world success across 5 diverse deformable tasks including plastic-bag manipulation, without teleoperation or per-task calibration. SimWeaver combines a reliable measurement-backed simulator (SimWeaver-Sim) with an extensible asset framework supporting single-image generation(SimWeaver-Asset), a deterministic topology-aware trajectory synthesizer (SimWeaver-Syn), and a sim-to-real protocol with ISP-aware photometric augmentation (SimWeaver-Real). On silk grasping, the sim-trained policy reaches 100% under visual distribution shifts where real-data baselines drop to 9-70%, at two orders of magnitude lower per-trajectory cost. We will release SimWeaver and a representative asset subset. Project page: https://simweaver.github.io/
Wenkang Hu, Haoran Wang, Yitong Li +10
1Shanghai Jiao Tong University · 2Horizon Robotics · 3Style3D Research
Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.
Bowei Li, Yuner Zhang, Changliu Liu
Carnegie Mellon University, Pittsburgh, PA 15213, USA