Bimanual manipulation in cluttered, contact-rich environments remains challenging because it requires coordinated motion generation, interaction-aware planning, and reliable execution under tight kinematic constraints. We present PRISM, a projection-integrated sampling-based Model Predictive Control (MPC) framework that uses a GPU-accelerated physics simulator as an online world model for complex dual-arm manipulation. The main algorithmic contribution is a QP-guided control sampling strategy that decouples trajectory exploration from kinematic feasibility. At each MPC step, sampled joint-velocity trajectories are projected onto the set of motions satisfying joint position, velocity, acceleration, and jerk bounds, together with an initial-velocity boundary condition, before rollout evaluation. This enables broad yet feasible exploration of coordinated bimanual behaviors. To support efficient online execution, we derive a custom ADMM/Bregman-splitting QP solver that exploits joint-wise separability and reusable matrix factorizations. We further use Bayesian optimization to tune task-cost weights offline, reducing manual parameter selection. We evaluate PRISM on challenging variants of PerAct2 tasks, including obstacle-constrained ball transport, tray transport, cube handover, and box lifting. Experiments show improved robustness and task success relative to representative sampling-based baselines, while maintaining real-time or near-real-time execution. We also demonstrate successful sim-to-real transfer on dual UR5e manipulators, highlighting the practical potential of physics-based online planning for contact-rich bimanual manipulation. Project details, including code and supplementary videos, are available at https://sites.google.com/view/prismbimanual.
Fig. 2: System architecture of our proposed framework. The outer Model Predictive Control (MPC) loop handles real-time state feedback ( θ,θ˙,x ) and warm-starts the distribution parameters. The inner sampling-based optimizer (Alg. 1) enforces constraints via QP projection and leverages highly parallelized MJX rollouts to evaluate task-specific costs.
Fig. 3: Comparison of sampling behaviors for a single joint. The QP-projected samples (right) share a common initial velocity θ˙1=θ˙init and fan out to cover the full feasible envelope defined by velocity, acceleration, and jerk bounds. The baseline (left) and LPF (middle) are restricted to conservative covariances, yielding negligible coverage of the reachable set.
Fig. 4: The bimanual setup and global reference frame. The coordinate system ( x,y,z ) is explicitly modeled in the task-specific cost functions to enforce constraints, such as planar alignment in the yz -plane during the ball transport and handover tasks.
Fig. 5: Experimental Setup wherein real-world and simulation are tightly coupled with OptiTrack motion capture.
Fig. 6: Execution of the ball lifting and transport task. The manipulators must apply continuous, opposing contact forces in the yz -plane to maintain the unattached ball while dynamically adjusting their joint configurations to clear the central obstacle.
Fig. 7: Execution of the handover task. The sequence demonstrates the transition from the pick phase to the pass phase, requiring precise mid-air synchronization and collision avoidance amidst a dense cluster of static spherical obstacles.
Fig. 8: Execution of the box lifting task. Unlike the spherical payload, the asymmetric flat-sided box requires the optimizer to enforce strict orientation costs during the move phase to prevent rotational slipping while sustaining frictional contact.
Fig. 9: Statistics of all tasks. (a) Success rate, (b) computation time, and (c) task time with respect to different batch sizes used in Alg. 1 . Note that success rates generally plateau after a batch size of 1250, while the computation time for the asymmetric box lifting task scales significantly higher than other tasks.
Fig. 10: Comparison across all tasks between manually tuned and Bayesian-optimized cost weights for (a) Success rate, (b) computation time, and (c) task completion time. Bayesian optimization systematically improves success rates across all tasks—most notably a 45% increase in the tray moving task—without introducing additional computational overhead during online execution.
Fig. 11: Comparison across all tasks with existing approaches for (a) Success rate, (b) computation time, and (c) task completion time. Baseline-CEM refers to the standard unfiltered formulation of CEM [ 7 ] . The proposed QP-projection method consistently outperforms vanilla CEM, LPF, and SGF baselines in success rate, particularly in multi-phase tasks requiring sustained bimanual coordination
Fig. 12: Illustrative comparison of our approach (top) with [ 2 ] (bottom) on a ball-lifting task with an added obstacle. In the obstacle-free setting, both approaches succeed. When an obstacle is introduced—a configuration outside the training distribution of [ 2 ] —the learned policy struggles to adapt its lifting strategy, while our online planner discovers a feasible solution by reasoning over the current scene.
We present a primitive-informed sampling-based model predictive control (MPC) framework for multi-fingered dexterous manipulation. Sampling-based MPC evaluates candidate control trajectories through forward simulation without requiring gradients through complex contact dynamics. However, directly sampling these trajectories in the high-dimensional joint space of a dexterous hand is inefficient and makes performance strongly dependent on the sampling distribution. Our framework biases sampling using low-dimensional manipulation primitives that encode coordinated finger motions, while simultaneously optimizing joint-level residuals to adapt these motions to the current hand-object configuration. Task-related rollout constraints reject infeasible trajectories during forward simulation, improving the effective use of the sampling budget. We evaluate the approach on a 16 DoF Allegro hand using a synchronized MuJoCo digital twin. Ablations show that both the primitive and residual are necessary for reliable continuous in-hand rotation, that increasing the sampling budget alone does not recover this coordination, and that rollout constraints substantially improve success rate. A primitive extracted for one object size transfers to other sizes and remains effective under model mismatch. The framework further supports grasping, object reorientation, and coordinated arm-hand manipulation, using primitives extracted from both a simulation-trained policy and human hand-motion data.
Emek Barış Küçüktabak, Karankumar Patel, Jinda Cui +3
Honda Research Institute USA, San Jose, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA (Work done during an internship at HRI)
Sampling-based Model Predictive Control (SMPC) is a promising strategy for contact-rich robotic manipulation, combining gradient-free optimization with massively parallel GPU simulation. Yet, most prior work relies on simplified dynamics or remains confined to simulation. We present an MPC framework that leverages JAX for large-scale parallelization and efficient computation, coupled with the high-fidelity MuJoCo MJX simulator, and deploy it on a Franka Research 3 executing the Push-T manipulation task through a complete real-to-sim-to-real pipeline. The MTP variant with structured global sampling outperforms unimodal baselines such as CEM, MPPI, and PS across tasks that require mode switching, both in simulation and on hardware. Furthermore, we evaluate online domain randomization within the MPC sample budget, showing that contact-initiation parameters yield interpretable adaptation signals, whereas global physics parameters provide feedback that is too weak for reliable exploitation at typical replanning frequencies. These findings highlight key challenges for sampling-based MPC in contact-rich manipulation-contact sensitivity, tight compute budgets, and the difficulty of obtaining informative domain-randomization signals in real time.
Magnus Dierking, Joao Carvalho, An Thai Le +2
Intelligent Autonomous Systems Lab, TU Darmstadt, Germany · Interactive Robot Perception & Learning Lab, TU Darmstadt, Germany · German Research Center for AI (DFKI) +2
Bimanual mobile manipulation requires a seamless integration between high-level semantic reasoning and safe, compliant physical interaction - a challenge that end-to-end models approach opaquely and classical controllers lack the context to address. This paper presents GenerativeMPC, a hierarchical cyber-physical framework that explicitly bridges semantic scene understanding with physical control parameters for bimanual mobile manipulators. The system utilizes a Vision-Language Model with Retrieval-Augmented Generation (VLM-RAG) to translate visual and linguistic context into grounded control constraints, specifically outputting dynamic velocity limits and safety margins for a Whole-Body Model Predictive Controller (MPC). Simultaneously, the VLM-RAG module modulates virtual stiffness and damping gains for a unified impedance-admittance controller, enabling context-aware compliance during human-robot interaction. Our framework leverages an experience-driven vector database to ensure consistent parameter grounding without retraining. Experimental results in MuJoCo, IsaacSim, and on a physical bimanual platform confirm a 60% speed reduction near humans and safe, socially-aware navigation and manipulation through semantic-to-physical parameter grounding. This work advances the field of human-centric cybernetics by grounding large-scale cognitive models into predictable, high-frequency physical control loops.
Marcelino Julio Fernando, Miguel Altamirano Cabrera, Jeffrin Sam +3
Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology, Moscow, Russia