Reinforcement learning in massively parallel physics simulations has driven major progress in sim-to-real robot learning. However, current approaches remain brittle and task-specific, relying on extensive per-task engineering to design rewards, curricula, and demonstrations. Even with this engineering, they often fail on long-horizon, contact-rich manipulation tasks and do not meaningfully scale with compute, as performance quickly saturates when training revisits the same narrow regions of state space. We introduce OmniReset, a simple and scalable framework that enables on-policy reinforcement learning to robustly solve a broad class of dexterous manipulation tasks using a single reward function, fixed algorithm hyperparameters, no curricula, and no human demonstrations. Our key insight is that long-horizon exploration can be dramatically simplified by using simulator resets to systematically expose the RL algorithm to the diverse set of robot-object interactions which underlie dexterous manipulation. OmniReset programmatically generates such resets with minimal human input, converting additional compute directly into broader behavioral coverage and continued performance gains. We show that OmniReset gracefully scales to long-horizon dexterous manipulation tasks beyond the capabilities of existing approaches and is able to learn robust policies over significantly wider ranges of initial conditions than baselines. Finally, we distill OmniReset into visuomotor policies which display robust retrying behavior and substantially higher success rates than baselines when transferred to the real world zero-shot. Project webpage: https://weirdlabuw.github.io/omnireset/
Figures & tables
Figure 1: OmniReset is a scalable framework for dexterous manipulation which uses diverse simulator resets and large scale reinforcement learning to solve contact-rich, long horizon tasks beyond the capabilities of existing approaches. These policies are then distilled to RGB camera observations and robustly transferred to the real world zero-shot, where they are able to consistently solve challenging tasks over much wider ranges of initial conditions than baselines.
Figure 2: Sim-to-Real Pipeline with OmniReset (1) After generating partial assemblies and grasps from the simulator, (2) we collect reset states: reaching, near object, grasped, and near goal. (3) We then train a state-based RL policy initialized from these reset states, which is used to (4) train student-teacher distillation to get a RGB policy. (5) We finally deploy the RGB-based policy in the real world zero-shot.
(a) Grasp Sampling. We display the grasp poses ( SG⊂S ) sampled for the table leg. From left-to-right, the grasp sampling ranges are broad, moderate, and narrow. Our method uses the broad range.
Figure 4: Additional tasks. Visualizations of the new manipulation tasks solved with OmniReset. From left to right: Four-Leg Table Assembly, Cube Stacking, Cupcake on Plate, Block Reorientation on Wall
Figure 5: Success rates during RL training. We plot success rates over learning process for tasks described in Sec. 4 . We see that OmniReset scales to task where baselines struggle to make meaningful progress, especially with the wide range of initial conditions specified in the Hard variants of tasks.
Figure 6: Success rate over perturbations. We plot the decline in success rate (measured by ratio of success rate between no perturbations and current level of perturbations). We find that OmniReset is robust to perturbations while performance of baselines drops significantly.
Figure 7: Successful initial states of RL policy. For the Lew Twisting , we plot the xy configurations from which RL polices trained with Demo Curriculum and OmniReset succeed. We find that OmniReset succeeds from a much broader range of initial conditions.
Figure 8: Ablation on number of environments. We plot the success rates over course of RL using different number of environments. We find that the number of environments significantly impacts training performance.
Figure 9: Ablation on grasp sampling range. For this ablation on the Leg Twisting task, we find that training RL on narrower grasp sampling ranges leads to worse sample efficiency and lower success rate.
Figure 10: Sim2Real Tasks. Simulation setup for pretraining (top) and real-world deployment (bottom) of peg insertion (left), leg twisting (middle), and drawer assembly (right).
Figure 11: Visual randomizations. Examples of domain randomization applied during training. We vary lighting conditions, backgrounds, object and robot appearance, and workspace textures, along with camera pose and field-of-view jitter, to improve robustness to real-world visual variation.
Table 1: Sim-to-real performance across tasks: We report state-based RL performance in simulation, distilled image policy performance in simulation and real-world deployment, and a behavior cloning baseline trained on only 100 real-world demos.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Success Rates on Different Stages of Task. We plot the success rates for the Leg Twisting task when starting from states that are in the Near-Goal region and also the Reaching Region of the state space. When evaluating these success rates, we sample resets from the demonstrations used by the baseline algorithms to ensure the resulting policies start from in-distribution states. We see that the baselines can achieve moderate success rates when starting close to the goal (Near Goal), but struggle to make meaningful progress on the full long-horizon task (captured by the reaching resets).
Figure 13: Effect of system identification. Comparison of chirp trajectory fit with and without system identification.
Category
Parameter
Range / Distribution
Dynamics
Robot friction (static)
U(0.3,1.2)
Robot friction (dynamic)
U(0.2,1.0)
Insertive object friction
U(1.0,2.0) (static), U(0.9,1.9) (dynamic)
Table friction
U(0.3,0.6) (static), U(0.2,0.5) (dynamic)
Robot mass scale
×U(0.7,1.3)
Insertive object mass
U(0.02,0.2) kg
Appendix
Table 2: Domain randomization and state initialization used during training. Parameters are either sampled directly from distributions or applied as multiplicative scaling to nominal values (denoted by × ).
(a) Behavior which doesn’t transfer well. We show an example of a behavior which doesn’t transfer well, where the policy reorient the peg by dropping it above the hole and re-grasping it.
Reinforcement learning (RL) in simulation can train dexterous manipulation policies without robot demonstrations, but training a single generalist policy with task-agnostic rewards faces a severe exploration problem: approaching, grasping, and reorienting diverse objects with many degrees of freedom is difficult to discover from scratch. Prior works make exploration tractable with high-quality robot demonstrations, per-task reward shaping, or by restricting policies to narrow modes of behavior. We propose X-Reset, a framework that instead resolves exploration with human hand-object demonstrations. Rather than imitating or tracking retargeted human motion, X-Reset kinematically retargets hand-object states to noisy robot states, filters out states that are unstable in simulation, and samples the remainder as resets during RL training with general-purpose object-centric rewards. The resulting policy depends only on object state and goal, with demonstrations entering training through the reset distribution. We show that X-Reset trains generalist policies on 20 objects across three embodiments---a 22-DoF hand on two different arms and a parallel-jaw gripper---and resolves the exploration challenges of RL from scratch. X-Reset scales with the number of training objects, generalizes to unseen objects, can learn from imperfect hand-pose estimates, and transfers behaviors zero-shot from sim-to-real.
Prithwish Dan, Chenyang Ma, Wei Zhan
Applied Intuition · University of North Carolina at Chapel Hill · University of California, Berkeley
Imitation learning enables robots to acquire manipulation skills from demonstrations, but the resulting policies can fail outside the training data, while collecting more demonstrations requires substantial human effort. Human-in-the-loop reinforcement learning uses corrective feedback during online training, but typically learns the complete task policy rather than refining a pretrained imitation policy. We introduce Res-HIL, a human-in-the-loop residual reinforcement learning framework that learns corrective actions on top of a frozen imitation policy. Each human intervention provides two complementary learning signals: direct supervision of the residual policy and reward shaping of preceding autonomous behavior. Res-HIL combines these signals with zero initialization of the residual policy to stabilize and accelerate online learning. We evaluate Res-HIL on five contact-rich manipulation tasks spanning high-precision and long-horizon behaviors. With only 20 initial demonstrations, Res-HIL outperforms state-of-the-art full-policy human-in-the-loop reinforcement learning and residual fine-tuning without human guidance on every task after ten minutes of online training. Res-HIL improves its pretrained base policies and outperforms imitation policies trained with five times more demonstrations. An ablation study shows that direct residual supervision is critical to performance, while intervention-aware reward shaping substantially improves training efficiency.
Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht +1
Siemens AG, Research and Predevelopment, Germany. · Technical University of Munich, Germany.
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
Jusuk Lee, Sungha Kim, Yeonsoo Park +8
Seoul National University · University of Maryland, College Park · All Purpose AI +1