Contact-Guided Exploration for Non-Prehensile Locomanipulation with Multi-Critic RL
Authors: Simone Tolomei, Mayank Mittal, Franco Angelini, Manolo Garabini, Paolo Salaris, Marco Hutter
Organizations: Centro di Ricerca E. Piaggio, Dipartimento di Ingegneria dell’Informazione, Universit`a di Pisa, Italy · ETH Zürich, Switzerland · NVIDIA
Non-prehensile manipulation offers versatile skills for moving and rearranging heavy or bulky objects, particularly when combined with a mobile manipulation platform. However, both model-based and model-free approaches struggle with the complex hybrid dynamics and the sparsity of the contact in these tasks. To address these challenges, we propose a contact-guided exploration strategy implemented within a Multi-Critic Reinforcement Learning (RL) framework. A dedicated exploration critic is trained with a dense contact-seeking reward that guides the end-effector toward meaningful contact points; its influence is progressively decayed to recover a task-optimal policy. We obtain candidate interaction points from a general-purpose grasping algorithm, enabling the exploration mechanism to generalise across various object geometries. We evaluate the approach on multiple tasks, including box pushing, chair transportation, and a dishwasher opening task. Finally, we validate the chair transportation policy through extensive experiments on a quadrupedal mobile manipulator, demonstrating deployable non-prehensile manipulation in the real world.
Figures & tables
Fig. 1: Learning Non-Prehensile Manipulation via Contact-guided exploration. We guide reinforcement learning exploration using object-centric contact priors. The colored dashed lines illustrate the learned policy’s ability to execute contact-rich behaviours (e.g., pushing a chair to a target) by focusing on high-probability interaction regions.
Fig. 2: Overview of the System Architecture. (Top Left) Contact candidates are generated from object meshes using surface sampling or grasping algorithm to guide exploration. (Center) The Multi-Critic module employs a shared backbone with three specialized heads for task ( rtask ), exploration ( rexp ), and regularization ( rreg ) rewards. These are combined using a schedule-weighted advantage ∑Aiwi , where the influence of exploration ( wexp ) and regularization ( wreg ) is decayed according to a predefined schedule. (Bottom Left) The Actor utilizes an LSTM-based policy to output target arm joint positions qa∗ , base height, and base velocity. The base commands are realized by a frozen, pre-trained locomotion policy that generates low-level leg joint targets ql∗ . (Right) The versatility of the architecture is demonstrated across three non-prehensile tasks: chair transportation, box pushing, and dishwasher opening.
Fig. 3: Contact Candidates on Complex Geometries. Red markers indicate candidate contact points computed via a grasping algorithm. During training, a single candidate point is sampled to serve as a target for the end-effector position. The dataset includes both procedurally randomized chair models and a curated subset of CAD models from the IKEA furniture dataset.
Reward Term
Weight
Task Group (Goal Achievement: rtask )
Object Vel. to Goal
10.0
Object Pos. Tracking
15.0
Group Weight wtask
0.75
Exploration Group (Contact Guidance: rexp )
EE Tracking Contact Point
3.0
TABLE I: Reward Structure and Simulation Parameters
Fig. 4: Manipulation Strategy for Target-Directed Transportation. The learned policy exhibits an orientation-aware strategy to maximize control authority over the chair. The agent is initialized relative to the object with various target locations (colored discs). The overlapping silhouettes illustrate the continuous transition from initial approach to stabilized pushing, demonstrating the policy’s ability to balance base locomotion with precise arm coordination.
Fig. 5: Quantitative performance and failure mode analysis. Comparison of success rates and termination causes for (a) Box Pushing and (b) Chair Transportation. All methods use the contact-guided exploration reward during training. While vanilla PPO and PPO with Weight Schedule (WS) suffer from significant rates of Timeouts and Missed Contacts, the addition of the Multi-Critic architecture substantially improves efficiency. The full proposed method (Multi-Critic PPO + WS) achieves the highest success rates (over 90%), effectively mitigating Tipover events.
Fig. 6: Experimental validation and zero-shot real-world generalization. We deploy the policy on a quadrupedal manipulator and evaluate it on diverse physical geometries. Panels (a)–(c) show successful transportation of multiple chair models, while panel (d) shows robustness on a LOVBACKEN three-legged table. The policy generalizes to the table’s backrest-free topology, reorienting the base to maintain an optimal contact normal and transferring contact-guided exploration from simulation to the real world without additional fine-tuning.
Object
Successes
Total Runs
Success Rate
Primary Benchmark
ADDE
27
37
72.90%
SANDSBERG
8
14
57.14%
Case Studies
VIHALS
3
3
100.00%
LOVBACKEN
2
4
50.00%
TABLE II: Hardware Experiments on IKEA Objects
Fig. 7: Reactive recovery behavior under contact failure. Following an initial slip or failed contact attempt, the agent does not enter an unstable state but instead executes a recovery sequence: first re-positioning the base, then increasing the arm’s swing amplitude and re-establishing a stable contact point to resume the transportation task.
Fig. 8: Robustness to physical perturbations and payload changes. (a) The agent transports a chair with an additional 5 kg payload, showing robustness to large, unseen variations in inertia and friction. (b) The system rejects real-time disturbances when a human abruptly shifts the object, with the policy immediately re-planning the contact trajectory.
Fig. 9: Emergent contact sequencing for articulated object manipulation. In the dishwasher opening task, the contact-guided exploration provides candidate points (red markers) on both the handle and the door panel. The learned policy demonstrates a multi-stage strategy: (Top) the agent first uses the handle to initiate the opening motion. (Bottom) As the door kinematics change, the agent switch contact to the panel, utilizing a pushing motion to lower the door.
Training non-prehensile manipulation policies in contact-rich settings is a core challenge in robotics. While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and discover complex manipulation strategies. To address this, we combine two basic ideas: First, designing appropriate reset strategies (the start state distribution of episodes) has shown promise in improving RL exploration and effectiveness. Second, while model-based approaches to finding trajectories through manipulation are hard, recent work showed that model-based approaches to sampling states on constrained manifolds can be highly efficient. Based on these observations, we propose a novel state sampler that boosts the performance of goal-conditioned RL in complex contact-rich manipulation tasks. Our sampler explicitly takes into account the structure of contact in order to provide a rich covering of diverse contact modes. By combining constrained sampling resets with projected interpolation and curriculum learning, our novel approach outperforms RL without constrained sampling and alternative reset methods, and effectively trains universal, non-prehensile, and dynamic manipulation policies in contact-rich settings. See https://www.user.tu-berlin.de/mtoussai/26-CSRL/ for supplementary material.
Marc Toussaint, Cornelius V. Braun, Armand Jordana +5
Technische Universität Berlin · Robotics Institute Germany · LAAS-CNRS +1
A key challenge in contact-rich dexterous manipulation is the need to jointly reason over global geometry and nonsmooth contact dynamics. End-to-end policies bypass this complexity, but often require large amounts of data and transfer poorly from simulation to reality. We address the limitations with a simple insight: dexterous manipulation is inherently hierarchical--at a high level, a robot decides where to touch (geometry); at a low level it determines how to move the object through contact dynamics. Building on this insight, we propose a hierarchical RL--MPC framework in which a high-level reinforcement learning (RL) policy predicts a contact intention, a novel object-centric interface that specifies (i) an object-surface contact location and (ii) a post-contact object subgoal pose. Conditioned on the contact intention, a low-level contact-implicit model predictive control (MPC) optimizes local contact modes and real-time (re)plans through contact dynamics to generate robot actions that robustly move the object toward each subgoal. We evaluate the framework on non-prehensile tasks, including geometry-generalized pushing across diverse object shapes, pivoting/flipping-based object reorientation, and environment-assisted object repositioning. It achieves high success rate with substantially reduced data (10 times less than end-to-end baselines), highly robust performance, and zero-shot sim-to-real transfer.
Zhixian Xie, Yu Xiang, Michael Posa +1
Arizona State University · University of Texas at Dallas · University of Pennsylvania
Reinforcement learning explores effectively in domains such as Atari games, navigation, and locomotion, where novelty over states or dynamics is a sufficient signal. In contrast, dexterous manipulation requires rich physical hand--object interactions, but existing methods often suffer from unstable contact-based novelty signals, inefficient distance novelty signals, or reliance on task-specific priors. We propose ContactExplorer, a general exploration method for dexterous manipulation tasks. ContactExplorer represents contact as the intersection between object surface points and hand keypoints, encouraging dexterous hands to discover diverse and novel contact patterns, namely which fingers contact which object regions. It maintains a contact counter conditioned on discretized object states obtained via learned hash codes. This counter is leveraged in two complementary ways: (1) a count-based contact coverage reward that promotes exploration of novel contact patterns, and (2) an energy-based reaching reward that guides the agent toward under-explored contact regions. We evaluate ContactExplorer on seven contact-rich manipulation tasks and five dexterous hand embodiments. Experimental results show that ContactExplorer substantially improves sample efficiency and success rates over existing exploration methods, that it reduces the need for task-specific priors, and that it remains effective across hand embodiments and transfers to the real world. Project page is https://contact-explorer.github.io.
Zixuan Liu, Ruoyi Qiao, Chenrui Tie +5
School of Computing, National University of Singapore · RoboScience