cs.ROOct 19, 2026
SaveLong-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions
Organizations: Robotics Institute, Carnegie Mellon University · Stanford University
Abstract
Despite recent advances in policy learning, long-horizon manipulation remains difficult because learned policies must avoid compounding errors while preserving future feasibility. While search-based planning can explicitly reason over such future consequences, it quickly becomes expensive in high-dimensional continuous action spaces. This raises a central question: how can long-horizon reasoning from search be transferred into policies in continuous action spaces? To this end, we propose DISCO: Distilled Tree Search over Continuous Actions. DISCO uses progressive-widening MCTS in physics simulation to generate long-horizon plans. During search, a learned policy proposes continuous actions, while a learned value function scores actions by their expected progress toward task completion. Using GPU-parallelized simulation, DISCO periodically distills MCTS-generated plans into the policy-value model, improving the action proposals and value estimates used in subsequent searches. Finally, the learned policy-value model can be directly deployed without the need for simulator-based search at test time. Across simulated and real-world rearrangement tasks, DISCO outperforms behavior cloning, reinforcement learning, and search-based baselines.
Figures & tables
Figure 1: DISCO: Di stilled Tree S earch over Co ntinuous Actions. DISCO addresses long-horizon multi-object rearrangement by using progressive-widening Monte Carlo Tree Search over continuous actions. Using an initial behavior-cloned policy to propose short-horizon actions, DISCO expands search trees in parallel GPU simulation to discover long-horizon plans. The search trees are then distilled into a policy and a value function, enabling direct rollout on held-out tasks, such as real-world shelf packing. MCTS and simulation are used only during training. At test time, the distilled model acts directly from point clouds.
Figure 2: MCTS with DISCO . We embed a policy and a value function into the MCTS loop. The policy proposes continuous actions for tree expansion, while the value function guides rollout action selection toward terminal states. Periodically, both models are updated from newly generated search trees, enabling direct policy-value deployment at test time.
Figure 3: Task Setup : Representative states from the tasks used in our evaluation. In Shelf Packing and Object Stacking, the robot must fit objects into a shelf and construct stable stacks, respectively. Constrained Packing requires objects to be placed inside a bounded region without collisions. We also evaluate on a real-world version of Shelf Packing along with a Table Bussing task where dinnerware must be placed upright in a tray.
Figure 4: Simulation Results. DISCO improves reward and success with increasing simulation budget on shelf packing and object stacking, and remains more robust as task complexity increases. The simulation budget counts simulator calls made by search during offline training, not at test time.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Prior data | Single-step demonstrations used by DISCO |
| Discount factor | |
| Entropy temperature | |
| Batch size | |
| Gradient updates per environment iteration | |
| Critic learning rate |
Table 3: RL-PD Hyperparameters. We report the actor-critic hyperparameters used for the RL-PD baseline. RL-PD uses the same progress reward, action representation, and task horizon as DISCO, but replaces search-generated supervision with off-policy reinforcement learning from prior and online data.
| Hyperparameter | Value |
|---|---|
| Simulation budget | 270,000 (shelf packing), 27,000 (object stacking) |
| Progressive-widening coefficient | 2.0 |
| Progressive-widening exponent | 0.3 |
| Exploration constant | 1.5 |
| Expansion batch size | 32 |
| Maximum task horizon | number of objects (3–7 per task) |
Table 4: MCTS and distillation hyperparameters.
Figure 9: Policy and Value Function Architectures. Top : the policy encodes the target-object point cloud with EdgeConv layers and denoises a pose token with a flow-matching transformer to predict a grasp or place pose. Bottom : the value function embeds the full-scene point cloud with a per-point MLP, and an action-conditioned query attends to the scene tokens to predict the success probability .
Figure 10: Value Function Targets. We visualize the distribution of value-function targets and the effect of evidence weighting. Left : histogram of targets over 11,015 eligible nodes from 50 training trees, showing that the distribution is dominated by saturated targets at 0 and 1. Right : raw target-bucket mass compared to the effective training mass under the weighted BCE loss with , which increases the contribution of intermediate targets backed by more rollout evidence.
| Model | Metric | Training Steps | Final |
|---|---|---|---|
| Grasp policy | ( ) | 82.8k | 0.0457 |
| Grasp policy | ( ) | 82.8k | 0.714 |
| Place policy | ( ) | 58.9k | 0.0530 |
| Place policy | ( ) | 58.9k | 0.384 |
| Q function | ( ) | 30k | 0.00681 |
| Q function | ( ) | 30k | 0.0160 |
Table 5: Policy and Value Training Metrics. We report final training losses for the grasp policy, place policy, and Q function. The grasp policy is trained for 82.8k steps and the place policy is trained for 58.9k steps using the flow-matching policy objective, where denotes translation error and denotes rotation error. The Q function is trained for 30k steps, where and denote mean squared error and mean absolute error on subtree-success targets.
Figure 11: Shelf Packing Environments . Visualization of all 50 train environments and 15 test environments for the shelf packing task setup.
Figure 12: Object Stacking Environments . Visualization of all 50 train environments and 15 test environments for the object stacking task setup.
| Policy trained on | Success (%) ( ) | Dataset size |
|---|---|---|
| Only successful plans | 1,868 | |
| All feasible edges (Ours) | 8,655 |
Table 6: Distillation Ablations. Left : training the policy on all feasible edges gives higher success in subsequent search than training on only successful plans. Right : pre-training on fractions of the sub-task dataset . Reward after 25 search iterations saturates beyond 1,332 demonstrations.
Figure 13: Convergence of values. We visualize the convergence of root-action value estimates during MCTS in a representative training environment. Left: trajectories for representative depth-1 actions, where markers indicate updates on iterations when the corresponding action is selected. Right: mean absolute gap to the final estimate, , averaged over all visited root actions. Root-action values fan out from noisy early estimates and converge steadily as search budget increases.
Figure 14: Visit count compared to -values. We plot the MCTS value estimate of each visited root action against its visit count across the training trees. Gray points show individual depth-1 nodes, and the blue curve shows the binned mean with standard error. Spearman measures rank correlation between visit count and without assuming a linear relationship. The positive but modest correlation ( ) reflects an early rise followed by a plateau. Mean rises sharply over the first few visits and then saturates, indicating that MCTS quickly abandons poor actions and allocates additional search budget primarily to actions already confirmed to be promising.
Figure 15: Simulation Results. DISCO improves reward and success with increasing simulation budget on shelf packing and object stacking, and remains more robust as task complexity increases.
| Shelf Packing (Sim) Metrics | Object Stacking Metrics | |||||
|---|---|---|---|---|---|---|
| Algorithm | Reward ( ) | Success (%) ( ) | Train Plans ( ) | Reward ( ) | Success (%) ( ) | Train Plans ( ) |
| Beam Search | ||||||
| MPPI | ||||||
| RL-PD | ||||||
| DISCO w/o Policy | ||||||
| DISCO w/o PW | ||||||
Table 7: Simulation Metrics . We report average reward and success rate for DISCO against baselines on a held-out test set for shelf packing and object stacking. We also report number of plans found by each method during training. Results are averaged over 32 environment seeds and show a 95% confidence interval.
Figure 16: Decision Making . The graph above shows an object stacking task vertex with several children. In this graph we see that the higher Q-values of 0.944 and 0.917 are given most often to stable placements of the light blue object, by placing it on flat surfaces, rather than the unstable placement in node id=29 with a Q-value of 0.25
Figure 17: Mind Change . The object stacking graph above shows how the DISCO algorithm changes its mind after progressive widening adds a new action and a new vertex. Here, vertex id=140 was the preferred vertex with more objects getting stacked. Later during search, vertex id=1010 is added, offering a better placement for the yellow object, that allows more objects to be placed on top. Thus, progressive widening leads DISCO to find better actions later during search.
Figure 18: Multi-Modal Proposals . The graph above shows multi-modal action proposals of object stacking originating from a single node.
Figure 19: Multi-Modal Plans . The graph of above visualizes multi-modal plans found by DISCO. Here, we show a graph of the object stacking environment where DISCO finds many different ways of creating the stack of objects.
More in this topic
Long-Horizon Robotic Manipulation
Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.
See all 435 papers