Dynamics models, whether simulators or learned world models, have long been central to robotic manipulation, but most focus on minimizing prediction error rather than confronting a more fundamental challenge: real-world manipulation is inherently uncertain. We argue that robust manipulation under uncertainty is fundamentally an integration problem: uncertainties must be represented, propagated, and constrained within the planning loop, not merely suppressed during training. We present and open-source ManiDreams, a modular framework for uncertainty-aware manipulation planning over intuitive physics models. It realizes this integration through composable abstractions for distributional state representation, backend-agnostic dynamics prediction, and declarative constraint specification for action optimization. The framework explicitly addresses three sources of uncertainty: perceptual, parametric, and structural. It wraps any base policy with a sample-predict-constrain loop that evaluates candidate actions against distributional outcomes, adding robustness without retraining. Experiments on ManiSkill tasks show that ManiDreams maintains robust performance under various perturbations where the RL baseline degrades significantly. Runnable examples on pushing, picking, catching, and real-world deployment demonstrate flexibility across different policies, optimizers, physics backends, and executors. The framework is publicly available at https://github.com/Rice-RobotPI-Lab/ManiDreams
Figures & tables
Figure 1: ManiDreams unifies perceptual, parametric, and structural uncertainties through Task-specific Intuitive Physics (TSIP) and Domain-randomized Instance Set (DRIS). Left : Simulation-based TSIP across standard tasks ( PushCube , PickCube , PushT ), where the overlaid objects are DRIS instances. Right : A card picking task and a pushing task driven by simulation-based and learning-based TSIPs.
Figure 2: Three-layer architecture. Layer 1 defines abstract interfaces for the four core abstractions, analogous to OMPL’s StateSpace , StatePropagator , StateValidityChecker , and Planner . Layer 2 provides concrete implementations unified behind a single environment class. Layer 3 is user-facing: adding a new task requires only populating this layer. Dashed arrows indicate inheritance.
Figure 3: Runtime pipeline of ManiDreams. The planning loop (yellow): The solver generates candidate actions, predicts their DRIS outcomes via TSIP, and selects the best action satisfying the caging constraint; the loop may iterate for refinement. The execution loop (green): The selected action is appended to an action chunk; once complete, the chunk is dispatched to the real robot or simulator, which observes the outcome and produces an updated DRIS for the next planning cycle.
Figure 4: Minimal working example. Three steps configure and launch the full sample-predict-constrain pipeline. Setting num_samples=1 and omitting the cage reduces the pipeline to direct policy execution, serving as the baseline in all experiments.
Figure 5: Object pushing with circular trajectory following using different TSIPs. Random target objects are selected from the YCB dataset Calli et al. (2017) . Top : Simulation-based TSIP. Bottom : Learning-based TSIP based on a pre-trained diffusion model.
Figure 6: Card picking example with a simulation-based TSIP and a Geometric3DCage (transparent green sphere).
Figure 7: Dynamic ball catching example with a simulation-based TSIP and a pretrained PPO sampler.
Figure 8: Real-robot deployment. Left : Cluttered push-picking (corn, avocado, chicken leg, croissant), where the robot pushes obstructing objects aside before grasping the target. Right : Scooping via wall corner for flat objects (steak, pancake, thin box, pizza). In both tasks, ManiDreams plans with learning-based TSIP and pixel-based DRIS (red region behind the object) for consecutive action chunks.
Figure 9: Success rate under increasing perturbations across three ManiSkill tasks (top to bottom: PushCube , PickCube , PushT ). Left : Observation noise ( σ in meters). Center : Observation delay (in simulation steps). Right : Physics perturbation severity. Shaded regions indicate one standard deviation over 10 runs.
DRIS instances m
1
4
8
16
32
Success rate(%)
58
72
82
85
86
Solver samples N
1
4
8
16
32
Success rate(%)
52
71
82
87
88
Distribution width
Narrow
Medium
Wide
Success rate(%)
74
82
76
Table 1: Ablation study on PushCube . Success rate (%) averaged over 100 episodes under combined perceptual and parametric perturbations. Default configuration: m=8 , N=8 , Medium.
Figure 10: Wall-clock time per control step on PushCube for increasing N×m configurations and a diffusion-based TSIP.
Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.
Manipulating fast and dynamically moving targets in unstructured 3D environments remains challenging for embodied AI. Existing visual-language-action models and world models struggle with accurate 3D geometry and physically meaningful forecasting. We propose PhysMani, a framework that couples a physics-principled 3D Gaussian world model with a future-aware action policy model. The world model learns a divergence-free Gaussian velocity field via online optimization for fast and physically grounded future dynamics prediction. The policy model integrates the predicted 3D scene future dynamics through a learnable token based cross-attention module. We introduce PhysMani-Bench, a dynamic manipulation benchmark with 16 tasks, and demonstrate a superior success rate over strong baselines in both simulation and real-world robot experiments.
Peng Yun, Shouwang Huang, Hao Li +3
vLAR Group, The Hong Kong Polytechnic University, Hong Kong SAR, China · Astribot, China
World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar robot-target configuration. The resulting plan is connected to the robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data. Project page: https://methoder00.github.io/DPP/