Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions

Kallol Saha; Yining Hong; Amber Li; Yejin Choi; Maxim Likhachev; David Held

cs.ROOct 19, 2026

Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions

Authors: Kallol Saha, Yining Hong, Amber Li, Yejin Choi, Maxim Likhachev, David Held

Organizations: Robotics Institute, Carnegie Mellon University · Stanford University

Abstract

Despite recent advances in policy learning, long-horizon manipulation remains difficult because learned policies must avoid compounding errors while preserving future feasibility. While search-based planning can explicitly reason over such future consequences, it quickly becomes expensive in high-dimensional continuous action spaces. This raises a central question: how can long-horizon reasoning from search be transferred into policies in continuous action spaces? To this end, we propose DISCO: Distilled Tree Search over Continuous Actions. DISCO uses progressive-widening MCTS in physics simulation to generate long-horizon plans. During search, a learned policy proposes continuous actions, while a learned value function scores actions by their expected progress toward task completion. Using GPU-parallelized simulation, DISCO periodically distills MCTS-generated plans into the policy-value model, improving the action proposals and value estimates used in subsequent searches. Finally, the learned policy-value model can be directly deployed without the need for simulator-based search at test time. Across simulated and real-world rearrangement tasks, DISCO outperforms behavior cloning, reinforcement learning, and search-based baselines.

Figures & tables

Figure 1: DISCO: Di stilled Tree S earch over Co ntinuous Actions. DISCO addresses long-horizon multi-object rearrangement by using progressive-widening Monte Carlo Tree Search over continuous actions. Using an initial behavior-cloned policy to propose short-horizon actions, DISCO expands search trees in parallel GPU simulation to discover long-horizon plans. The search trees are then distilled into a policy and a value function, enabling direct rollout on held-out tasks, such as real-world shelf packing. MCTS and simulation are used only during training. At test time, the distilled model acts directly from point clouds.

Figure 2: MCTS with DISCO . We embed a policy $\pi_{\theta}$ and a value function $Q_{\phi}$ into the MCTS loop. The policy proposes continuous actions for tree expansion, while the value function guides rollout action selection toward terminal states. Periodically, both models are updated from newly generated search trees, enabling direct policy-value deployment at test time.

Figure 3: Task Setup : Representative states from the tasks used in our evaluation. In Shelf Packing and Object Stacking, the robot must fit objects into a shelf and construct stable stacks, respectively. Constrained Packing requires objects to be placed inside a bounded region without collisions. We also evaluate on a real-world version of Shelf Packing along with a Table Bussing task where dinnerware must be placed upright in a tray.

Figure 4: Simulation Results. DISCO improves reward and success with increasing simulation budget on shelf packing and object stacking, and remains more robust as task complexity increases. The simulation budget counts simulator calls made by search during offline training, not at test time.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Prior data	Single-step demonstrations used by DISCO
Discount factor $\gamma$	$0.99$
Entropy temperature $\alpha$	$0.1$
Batch size	$64$
Gradient updates per environment iteration	$10$
Critic learning rate	$0.1$

Appendix

Table 3: RL-PD Hyperparameters. We report the actor-critic hyperparameters used for the RL-PD baseline. RL-PD uses the same progress reward, action representation, and task horizon as DISCO, but replaces search-generated supervision with off-policy reinforcement learning from prior and online data.

Figure 8

Figure 9

Hyperparameter	Value
Simulation budget $\beta$	270,000 (shelf packing), 27,000 (object stacking)
Progressive-widening coefficient $k$	2.0
Progressive-widening exponent $\alpha$	0.3
Exploration constant $C$	1.5
Expansion batch size $B$	32
Maximum task horizon $K$	number of objects (3–7 per task)

Appendix

Table 4: MCTS and distillation hyperparameters.

Figure 9: Policy and Value Function Architectures. Top : the policy

\pi_{\theta}

encodes the target-object point cloud with EdgeConv layers and denoises a pose token with a flow-matching transformer to predict a grasp or place pose. Bottom : the value function

Q_{\phi}

embeds the full-scene point cloud with a per-point MLP, and an action-conditioned query attends to the scene tokens to predict the success probability

Q_{\phi}(\mathbf{o},\mathbf{a})\in[0,1]

Figure 10: Value Function Targets. We visualize the distribution of value-function targets and the effect of evidence weighting. Left : histogram of targets $v=n_{\mathrm{succ}}/n_{\mathrm{leaves}}$ over 11,015 eligible nodes from 50 training trees, showing that the distribution is dominated by saturated targets at 0 and 1. Right : raw target-bucket mass compared to the effective training mass under the weighted BCE loss with $w=\log(1+n_{\mathrm{leaves}})$ , which increases the contribution of intermediate targets backed by more rollout evidence.

Model	Metric	Training Steps	Final
Grasp policy	$\mathcal{L}_{\mathrm{pos}}$ ( $\downarrow$ )	82.8k	0.0457
Grasp policy	$\mathcal{L}_{\mathrm{rot}}$ ( $\downarrow$ )	82.8k	0.714
Place policy	$\mathcal{L}_{\mathrm{pos}}$ ( $\downarrow$ )	58.9k	0.0530
Place policy	$\mathcal{L}_{\mathrm{rot}}$ ( $\downarrow$ )	58.9k	0.384
Q function	$\mathcal{L}_{\mathrm{MSE}}^{Q}$ ( $\downarrow$ )	30k	0.00681
Q function	$\mathcal{L}_{\mathrm{MAE}}^{Q}$ ( $\downarrow$ )	30k	0.0160

Appendix

Table 5: Policy and Value Training Metrics. We report final training losses for the grasp policy, place policy, and Q function. The grasp policy is trained for 82.8k steps and the place policy is trained for 58.9k steps using the flow-matching policy objective, where

\mathcal{L}_{\mathrm{pos}}

denotes translation error and

\mathcal{L}_{\mathrm{rot}}

denotes rotation error. The Q function is trained for 30k steps, where

\mathcal{L}_{\mathrm{MSE}}^{Q}

and

\mathcal{L}_{\mathrm{MAE}}^{Q}

denote mean squared error and mean absolute error on subtree-success targets.

Figure 11: Shelf Packing Environments . Visualization of all 50 train environments and 15 test environments for the shelf packing task setup.

Figure 12: Object Stacking Environments . Visualization of all 50 train environments and 15 test environments for the object stacking task setup.

Policy trained on	Success (%) ( $\uparrow$ )	Dataset size
Only successful plans	$32.3\scriptstyle{\pm 24.9}$	1,868
All feasible edges (Ours)	$\mathbf{52.8\scriptstyle{\pm 38.9}}$	8,655

Appendix

Table 6: Distillation Ablations. Left : training the policy on all feasible edges gives higher success in subsequent search than training on only successful plans. Right : pre-training on fractions of the sub-task dataset

\mathcal{D}

. Reward after 25 search iterations saturates beyond 1,332 demonstrations.

Figure 13: Convergence of $Q$ values. We visualize the convergence of root-action value estimates during MCTS in a representative training environment. Left: $Q$ trajectories for representative depth-1 actions, where markers indicate updates on iterations when the corresponding action is selected. Right: mean absolute gap to the final estimate, $\lvert Q(t)-Q_{\mathrm{final}}\rvert$ , averaged over all visited root actions. Root-action values fan out from noisy early estimates and converge steadily as search budget increases.

Figure 14: Visit count compared to $Q$ -values. We plot the MCTS value estimate of each visited root action against its visit count across the training trees. Gray points show individual depth-1 nodes, and the blue curve shows the binned mean with standard error. Spearman $\rho$ measures rank correlation between visit count and $Q$ without assuming a linear relationship. The positive but modest correlation ( $\rho=+0.15$ ) reflects an early rise followed by a plateau. Mean $Q$ rises sharply over the first few visits and then saturates, indicating that MCTS quickly abandons poor actions and allocates additional search budget primarily to actions already confirmed to be promising.

Figure 15: Simulation Results. DISCO improves reward and success with increasing simulation budget on shelf packing and object stacking, and remains more robust as task complexity increases.

	Shelf Packing (Sim) Metrics			Object Stacking Metrics
Algorithm	Reward ( $\uparrow$ )	Success (%) ( $\uparrow$ )	Train Plans ( $\uparrow$ )	Reward ( $\uparrow$ )	Success (%) ( $\uparrow$ )	Train Plans ( $\uparrow$ )
Beam Search	$24.3\scriptstyle{\pm 8.0}$	$6.7\scriptstyle{\pm 5.1}$	$4.26\scriptstyle{\pm 1.14}$	$79.9\scriptstyle{\pm 4.2}$	$33.3\scriptstyle{\pm 13.6}$	$5.84\scriptstyle{\pm 1.00}$
MPPI	$36.7\scriptstyle{\pm 7.8}$	$13.3\scriptstyle{\pm 7.2}$	$12.20\scriptstyle{\pm 3.25}$	$60.7\scriptstyle{\pm 6.2}$	$13.3\scriptstyle{\pm 12.2}$	$3.32\scriptstyle{\pm 0.50}$
RL-PD	$71.0\scriptstyle{\pm 4.8}$	$20.0\scriptstyle{\pm 13.7}$	$138.52\scriptstyle{\pm 40.81}$	$64.5\scriptstyle{\pm 5.7}$	$20.0\scriptstyle{\pm 8.5}$	$\mathbf{46.46\scriptstyle{\pm 10.13}}$
DISCO w/o Policy	$65.9\scriptstyle{\pm 6.2}$	$6.7\scriptstyle{\pm 2.3}$	$7.56\scriptstyle{\pm 1.59}$	$75.2\scriptstyle{\pm 5.0}$	$33.3\scriptstyle{\pm 8.9}$	$3.02\scriptstyle{\pm 0.56}$
DISCO w/o PW	$39.6\scriptstyle{\pm 7.2}$	$0.0\scriptstyle{\pm 0.0}$	$133.14\scriptstyle{\pm 31.88}$	$64.8\scriptstyle{\pm 7.5}$	$26.7\scriptstyle{\pm 10.8}$	$22.98\scriptstyle{\pm 3.79}$

Appendix

Table 7: Simulation Metrics . We report average reward and success rate for DISCO against baselines on a held-out test set for shelf packing and object stacking. We also report number of plans found by each method during training. Results are averaged over 32 environment seeds and show a 95% confidence interval.

Figure 16: Decision Making . The graph above shows an object stacking task vertex with several children. In this graph we see that the higher Q-values of 0.944 and 0.917 are given most often to stable placements of the light blue object, by placing it on flat surfaces, rather than the unstable placement in node id=29 with a Q-value of 0.25

Figure 17: Mind Change . The object stacking graph above shows how the DISCO algorithm changes its mind after progressive widening adds a new action and a new vertex. Here, vertex id=140 was the preferred vertex with more objects getting stacked. Later during search, vertex id=1010 is added, offering a better placement for the yellow object, that allows more objects to be placed on top. Thus, progressive widening leads DISCO to find better actions later during search.

Figure 18: Multi-Modal Proposals . The graph above shows multi-modal action proposals of object stacking originating from a single node.

Figure 19: Multi-Modal Plans . The graph of above visualizes multi-modal plans found by DISCO. Here, we show a graph of the object stacking environment where DISCO finds many different ways of creating the stack of objects.

Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.

See all 435 papers

Sep 28, 2026

Don't Throw Away the Tail: Action Upcycling for Policy Acceleration

Modern robot policies predict a chunk of future actions from a single observation, execute only a prefix, and discard the rest before replanning. Choosing the length of this prefix, the execution horizon, poses a trade-off between reactivity and efficiency. A short horizon keeps the policy reactive to the environment, but requires frequent policy calls. Recent test-time methods adaptively select the horizon for each chunk, but they either read model internals, where the signal must be chosen for each architecture, or draw extra samples, which adds cost. We propose Action Upcycling, a training-free algorithm that reuses actions the policy would otherwise discard, without accessing model internals or drawing extra samples. We find that discarded actions stay close to their replanned versions as long as the action velocity remains smooth. Action Upcycling therefore extends the execution horizon up to the point where the velocity begins to fluctuate. Extensive experiments on simulated and real-world manipulation tasks show that Action Upcycling reduces policy calls by 1.2-1.7x with no loss in success rate, across multiple Vision-Language-Action Models (VLAs) and even a World Action Model (WAM). It applies to any chunked policy at negligible cost and is orthogonal to other policy acceleration methods such as few-step sampling and streaming action decoding, opening a new axis for policy acceleration.

Taesung Kwon, Jangho Park, Sunwoo Park +5

KAIST · Sungkyunkwan University

Robot Policy Learning Efficient VLA Model Inference Action Chunking

Ask PDF

Apr 23, 2026

Long-Horizon Manipulation via Trace-Conditioned VLA Planning

Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip

Isabella Liu, An-Chieh Cheng, Rui Yan +7

University of California, San Diego · NVIDIA

Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation Long-Horizon Planning Vision-Language-Action Models

Ask PDF

Oct 9, 2026

PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Visual Subgoals and Policy Priors

Latent world-model planning has shown promising results in goal-conditioned robotic manipulation, but remains challenging for long-horizon planning due to limited temporal guidance and inefficient action search. A distant visual goal provides little guidance for preparatory actions, while uninformed action sampling makes local optimization inefficient. We propose PILOT (Policy-Initialized Local Optimization with Temporal subgoals), which uses a video model to generate intermediate visual subgoals and a learned policy to initialize the cross-entropy method (CEM). CEM refines candidate action sequences toward each subgoal using an action-conditioned world model in a receding-horizon control loop. In our primary real-robot evaluation across four UR5e manipulation tasks, PILOT achieves 68.8% success, compared with 55.0% without policy initialization, 13.8% for policy-only execution, and 0.0% for direct final-goal planning. Simulation results show that these gains persist over longer horizons: PILOT exceeds direct execution of the same policy by 30 percentage points on long Push-T tasks and by 22 percentage points on LIBERO-10. Policy initialization also accelerates local optimization: on Push-T, CEM matches the cost reached by generic initialization after 10 iterations in only 4 iterations and reduces the final planning cost by 24.1% under the same sampling budget. These results show that combining temporal subgoals with policy-guided action search improves both long-horizon task execution and local planning efficiency. Videos are available on the webpage: https://pilot-planner.github.io/.

Yang Liu, Hanchen Cui, Karthik Desingh

The authors are with the Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN 55455, USA.

Long-Horizon Robotic Manipulation Robot Task Planning World Model-Based Planning

Ask PDF

Abstract

Long-Horizon Robotic Manipulation

Explore similar work

Don't Throw Away the Tail: Action Upcycling for Policy Acceleration

Long-Horizon Manipulation via Trace-Conditioned VLA Planning

PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Visual Subgoals and Policy Priors