PRISM: Projection-Integrated Sampling-Based MPC with Bayesian Cost Tuning for Bimanual Manipulation

Alinjar Dan; Iryna Hurova; Karl Kruusamäe; Arun Kumar Singh

cs.ROAug 26, 2026

PRISM: Projection-Integrated Sampling-Based MPC with Bayesian Cost Tuning for Bimanual Manipulation

Authors: Alinjar Dan, Iryna Hurova, Karl Kruusamäe, Arun Kumar Singh

Organizations: University of Tartu, Narva mnt 18, 51009, Tartu, Estonia

Abstract

Bimanual manipulation in cluttered, contact-rich environments remains challenging because it requires coordinated motion generation, interaction-aware planning, and reliable execution under tight kinematic constraints. We present PRISM, a projection-integrated sampling-based Model Predictive Control (MPC) framework that uses a GPU-accelerated physics simulator as an online world model for complex dual-arm manipulation. The main algorithmic contribution is a QP-guided control sampling strategy that decouples trajectory exploration from kinematic feasibility. At each MPC step, sampled joint-velocity trajectories are projected onto the set of motions satisfying joint position, velocity, acceleration, and jerk bounds, together with an initial-velocity boundary condition, before rollout evaluation. This enables broad yet feasible exploration of coordinated bimanual behaviors. To support efficient online execution, we derive a custom ADMM/Bregman-splitting QP solver that exploits joint-wise separability and reusable matrix factorizations. We further use Bayesian optimization to tune task-cost weights offline, reducing manual parameter selection. We evaluate PRISM on challenging variants of PerAct $^{2}$ tasks, including obstacle-constrained ball transport, tray transport, cube handover, and box lifting. Experiments show improved robustness and task success relative to representative sampling-based baselines, while maintaining real-time or near-real-time execution. We also demonstrate successful sim-to-real transfer on dual UR5e manipulators, highlighting the practical potential of physics-based online planning for contact-rich bimanual manipulation. Project details, including code and supplementary videos, are available at https://sites.google.com/view/prismbimanual.

Figures & tables

\left(\boldsymbol{\theta}_{k+1,j},\mathbf{x}_{k+1,j}\right)=\mathbf{f}\left(\boldsymbol{\theta}_{k,j},\dot{\boldsymbol{\theta}}^{*}_{k,j},\mathbf{x}_{k,j}\right).

Algorithm 1 Sampling-Based Optimizer for Solving ( 1a )–( 1c )

Fig. 2: System architecture of our proposed framework. The outer Model Predictive Control (MPC) loop handles real-time state feedback (

\boldsymbol{\theta},\dot{\boldsymbol{\theta}},\mathbf{x}

) and warm-starts the distribution parameters. The inner sampling-based optimizer (Alg. 1) enforces constraints via QP projection and leverages highly parallelized MJX rollouts to evaluate task-specific costs.

Fig. 3: Comparison of sampling behaviors for a single joint. The QP-projected samples (right) share a common initial velocity $\dot{\theta}_{1}=\dot{\theta}_{\mathrm{init}}$ and fan out to cover the full feasible envelope defined by velocity, acceleration, and jerk bounds. The baseline (left) and LPF (middle) are restricted to conservative covariances, yielding negligible coverage of the reachable set.

Fig. 4: The bimanual setup and global reference frame. The coordinate system ( $x,y,z$ ) is explicitly modeled in the task-specific cost functions to enforce constraints, such as planar alignment in the $yz$ -plane during the ball transport and handover tasks.

Fig. 5: Experimental Setup wherein real-world and simulation are tightly coupled with OptiTrack motion capture.

Fig. 6: Execution of the ball lifting and transport task. The manipulators must apply continuous, opposing contact forces in the $yz$ -plane to maintain the unattached ball while dynamically adjusting their joint configurations to clear the central obstacle.

Fig. 7: Execution of the handover task. The sequence demonstrates the transition from the pick phase to the pass phase, requiring precise mid-air synchronization and collision avoidance amidst a dense cluster of static spherical obstacles.

Fig. 8: Execution of the box lifting task. Unlike the spherical payload, the asymmetric flat-sided box requires the optimizer to enforce strict orientation costs during the move phase to prevent rotational slipping while sustaining frictional contact.

Fig. 9: Statistics of all tasks. (a) Success rate, (b) computation time, and (c) task time with respect to different batch sizes used in Alg. 1 . Note that success rates generally plateau after a batch size of 1250, while the computation time for the asymmetric box lifting task scales significantly higher than other tasks.

Fig. 10: Comparison across all tasks between manually tuned and Bayesian-optimized cost weights for (a) Success rate, (b) computation time, and (c) task completion time. Bayesian optimization systematically improves success rates across all tasks—most notably a $45\%$ increase in the tray moving task—without introducing additional computational overhead during online execution.

Fig. 11: Comparison across all tasks with existing approaches for (a) Success rate, (b) computation time, and (c) task completion time. Baseline-CEM refers to the standard unfiltered formulation of CEM [ 7 ] . The proposed QP-projection method consistently outperforms vanilla CEM, LPF, and SGF baselines in success rate, particularly in multi-phase tasks requiring sustained bimanual coordination

Fig. 12: Illustrative comparison of our approach (top) with [ 2 ] (bottom) on a ball-lifting task with an added obstacle. In the obstacle-free setting, both approaches succeed. When an obstacle is introduced—a configuration outside the training distribution of [ 2 ] —the learned policy struggles to adapt its lifting strategy, while our online planner discovers a feasible solution by reasoning over the current scene.

Bimanual Robotic Manipulation

Bimanual robotic manipulation coordinates two robot arms to handle objects, tools, and tasks that require collaboration.

See all 192 papers

Primitive-Informed Sampling-Based MPC for Multi-Fingered Dexterous Manipulation
Sep 14, 2026Emek Barış Küçüktabak, Karankumar Patel, Jinda Cui +3Robotic Grasping Dexterous Manipulation
Honda Research Institute USA, San Jose, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA (Work done during an internship at HRI)
Real-World Deployment of Massively Parallel Sampling-Based MPC for Contact-Rich Manipulation
Jun 16, 2026Magnus Dierking, Joao Carvalho, An Thai Le +2Contact-Rich Robotic Manipulation Model Predictive Control
Intelligent Autonomous Systems Lab, TU Darmstadt, Germany · Interactive Robot Perception & Learning Lab, TU Darmstadt, Germany · German Research Center for AI (DFKI) +2
GenerativeMPC: VLM-RAG-guided Whole-Body MPC with Virtual Impedance for Bimanual Mobile Manipulation
Apr 21, 2026Marcelino Julio Fernando, Miguel Altamirano Cabrera, Jeffrin Sam +3Mobile Manipulation Bimanual Robotic Manipulation
Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology, Moscow, Russia

Sep 14, 2026J/K move · Enter open · S save

Primitive-Informed Sampling-Based MPC for Multi-Fingered Dexterous Manipulation

Emek Barış Küçüktabak, Karankumar Patel, Jinda Cui, Zhaodong Yang+2

Honda Research Institute USA, San Jose, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA (Work done during an internship at HRI)

Robotic Grasping Dexterous Manipulation In-Hand Manipulation Model Predictive Control Robotic Manipulation

Open paper AskPDF arXiv

We present a primitive-informed sampling-based model predictive control (MPC) framework for multi-fingered dexterous manipulation. Sampling-based MPC evaluates candidate control trajectories through forward simulation without requiring gradients through complex contact dynamics. However, directly sampling these trajectories in the high-dimensional joint space of a dexterous hand is inefficient and makes performance strongly dependent on the sampling distribution. Our framework biases sampling using low-dimensional manipulation primitives that encode coordinated finger motions, while simultaneously optimizing joint-level residuals to adapt these motions to the current hand-object configuration. Task-related rollout constraints reject infeasible trajectories during forward simulation, improving the effective use of the sampling budget. We evaluate the approach on a 16 DoF Allegro hand using a synchronized MuJoCo digital twin. Ablations show that both the primitive and residual are necessary for reliable continuous in-hand rotation, that increasing the sampling budget alone does not recover this coordination, and that rollout constraints substantially improve success rate. A primitive extracted for one object size transfers to other sizes and remains effective under model mismatch. The framework further supports grasping, object reorientation, and coordinated arm-hand manipulation, using primitives extracted from both a simulation-trained policy and human hand-motion data.

Abstract

Bimanual Robotic Manipulation

Explore similar work

Primitive-Informed Sampling-Based MPC for Multi-Fingered Dexterous Manipulation

Real-World Deployment of Massively Parallel Sampling-Based MPC for Contact-Rich Manipulation

GenerativeMPC: VLM-RAG-guided Whole-Body MPC with Virtual Impedance for Bimanual Mobile Manipulation