Learning to Move from and for Manipulation Policies: Spatializing Latent Action Representations for Goal-Pose-Free Mobile Manipulation
Authors: Siyuan Zheng, Zhong Cao, Henry X. Liu
Organizations: Department of Electrical and Computer Engineering, University of Michigan · Department of Civil and Environmental Engineering, University of Michigan · University of Michigan Transportation Research Institute
The success of manipulation policies is closely tied to the spatial relationship between the robot and the target object, as well as the task being performed, making the ability to adjust the robot's pose critical for successful execution. When the robot starts away from the task, such adjustments require coordinating navigation with local manipulation. Jointly training these capabilities incurs substantial computational and data costs, while navigating toward a predefined goal pose limits the flexibility needed to adapt to task-specific manipulation requirements. To address these challenges, we propose Learning to Move from and for Manipulation Policies (M4M), a framework that learns manipulation-aware mobility directly from pretrained manipulation policies. By leveraging the hidden representations of the policy's action-generating network as dense supervision, together with a limited number of manipulation execution outcomes, we learn a spatially conditioned embedding that captures task-dependent manipulation success across candidate robot poses. From the robot's current stereo observations, this representation estimates manipulation success at unvisited poses without rolling out the manipulation policy at each location. It is further integrated into navigation policy training, while a learned handoff policy decides when to transition from navigation to manipulation, enabling the robot to move and manipulate in a unified, task-aware manner. Across eight articulated-object tasks in unseen RoboCasa kitchens, M4M achieves 47.0% mean task success, compared with 26.5% for the strongest baseline and 19.2% for direct manipulation execution, and refrigerator-closing experiments show consistent improvements across four frozen manipulation policies.
Figures & tables
Figure 2: Success-guided approach and manipulation. At training time, the frozen policy observes candidate stance b , supplying compressed action-expert features hb and the probe score g(hb) . The student predicts z(a,b) from stereo and language at the current pose a , conditioned on the candidate. Feature matching supervises z and score matching supervises g(z) . The frozen probe and supervised SuccessHead form the success field S^a(b) . A compact controller reads the success, target, and traversability fields to choose collision-checked moves and compare stopping with continuing before handoff. The success-field inset illustrates the predicted field used for movement and handoff.
Figure 3: Manipulation stance preferences depend on the policy. Four frozen policies— π0.5 [ 2 ] , GR00T N1.5 [ 26 ] , π0 [ 3 ] , and Diffusion Policy [ 7 ] —are evaluated at the same 100 poses around a refrigerator. Circles mark sampled stances; the star marks the fixture. Color denotes empirical success from 0 to 1 in the shared coordinate frame. This diagnostic motivates policy-aware stance selection; its means are separate from the held-out multi-task benchmark.
Figure 4: One current view predicts success across candidate stances. Six candidate views in the training scene provide teacher features hb , frozen-probe scores g(hb) , and measured outcomes kb/6 . From the view at a , the student predicts z(a,b) and the fused score S^a(b) for each of the same stances. The feature strips are schematic; the counts and scores are from the example.
Figure 5: Eight-task completion on unseen kitchens. Each task uses two held-out kitchens and 180 episodes per method; error bars show SD. N2M and M4M (one-shot) predict once from the initial view, followed by placement and manipulation. M4M (closed-loop) moves and re-observes before handoff. Dashed lines mark separately measured favorable-stance references.
Condition
Success (%)
Empirical standoff (1.3 m)
51.7±6.0
Distilled probe only
69.4±5.1
Outcome-only head
75.6±7.9
Dijkstra
72.8±6.0
Full model
82.2±5.4
Table 1: Repositioning analysis on two unseen fridge kitchens. (a) Stance-scoring and navigation ablations: three groups of 60 episodes, 90 per layout, at 0.5–2.5 m. (b) Box obstacle: three groups of 24 episodes, 36 per layout, at 2.0/2.5 m with obstacle geometry available only during training. Values are success rates (%, mean ± SD across groups).
Labels
ResNet-18
SigLIP
Prefix
H
Action
5%
0.274±0.076
0.240±0.091
0.211±0.045
0.174±0.015
0.275±0.051
20%
0.293±0.034
0.248±0.032
0.177±0.014
0.148±0.018
0.258±0.045
50%
0.262±0.007
0.228±0.023
0.176±0.019
0.151±0.015
0.216±0.009
100%
0.264±0.003
0.210±0.005
0.176±0.011
0.158±0.003
0.214±0.010
Table 2: Cross-layout selection regret (lower is better). Probes train on one kitchen and evaluate on three unseen layouts. Values are mean ± SD over five label-subset repeats; bold marks the lowest mean. Supplementary Table S6 also reports within-kitchen results.
Setting
H
g
Regret ↓
Brier ↓
A
–
–
0.414±0.030
0.379±0.019
B
✓
–
0.412±0.038
0.364±0.026
C
–
✓
0.372±0.041
0.373±0.025
D
✓
✓
0.312±0.057
0.303±0.026
Table 3: Offline loss ablation (1,000-rollout budget, unseen kitchens; mean ± SD over five seeds). H and g denote feature and score matching; all rows include outcome supervision and use SuccessHead. Lower is better.
Figure 6: Matched total budgets of 200–1,500 rollouts. Lines show mean completion; shading is one SD across three evaluation blocks. Budgets include both stance-label collection and controller training (100 controller rollouts at budget 200; 200 at each larger budget).
Policy
M4M
N2M
Direct
π0.5
82.2±5.4
51.7±3.3
47.2±3.8
π0
55.0±6.0
14.4±1.9
5.0±1.7
GR00T N1.5
55.6±9.8
16.1±3.8
4.4±2.5
Diffusion Policy
23.3±5.0
12.2±4.2
3.3±1.7
Table 4: Refrigerator closing across frozen policies. Success (%, mean ± SD) uses the same held-out kitchens and 0.5–2.5 m starts as the π0.5 test. Direct executes from the initial pose; bold marks the best mean.