EgoPush: Egocentric Multi-Object Rearrangement for Mobile Robots via Constrained Teacher Observability

Boyuan An; Zhexiong Wang; Yipeng Wang; Jiaqi Li; Sihang Li; Jing Zhang; Chen Feng

cs.ROFeb 20, 2026

EgoPush: Egocentric Multi-Object Rearrangement for Mobile Robots via Constrained Teacher Observability

Authors: Boyuan An, Zhexiong Wang, Yipeng Wang, Jiaqi Li, Sihang Li, Jing Zhang, Chen Feng

Organizations: New York University

Abstract

Humans rearrange objects in cluttered environments using egocentric perception, actively moving to keep task-relevant spatial cues in view. Mobile robots have not matched this: rearrangement is usually built on a global pose estimate or a map, which is exactly what a robot carrying one camera lacks, while pushing keeps changing the scene it would have to be built from. We present EgoPush, which pushes objects into anchor-relative formations from onboard RGB-D alone, with no global localization, external tracking, or map at deployment, and transfers zero-shot to a TurtleBot in controlled and visually cluttered scenes. What makes this learnable turns out to be a property of the teacher rather than of the student: three privileged teachers trained with identical rewards, architecture, and hyperparameters all exceed $98\%$ success, yet their distilled egocentric students reach $0\%$ , $54.8\%$ , and $87.3\%$ , the only variable being the teacher's observation function. EgoPush therefore trains the teacher under egocentric observability constraints, restricting it to visibility-limited cues and revealing target references only when the anchor is centrally visible, so that its supervision is recoverable by a depth-based student distilled online. Making the teacher trainable in the first place needs two further pieces: a role-grouped object-centric interface shared by teacher and student, and stage-wise temporally decayed rewards for long-horizon credit assignment. Videos, the playable task, and code are available at https://ai4ce.github.io/EgoPush/.

Figures & tables

Method	Onboard	No global	Multi	Horizon	Real
ReloPush [ 1 ]	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
SAMaps [ 20 ] / SIM [ 2 ]	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
DyWA [ 14 ]	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$
SPIN [ 13 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$
EgoPush	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE I: Deployment Assumptions of Related Systems Onboard : the policy consumes only robot-mounted sensing. No global : no global pose, map, or external tracking at deployment. Multi : more than one object rearranged. Horizon : several sequential placements.

Fig. 2: EgoPush Overview . A privileged teacher trains from visibility-limited sparse keypoints; the egocentric student consumes role-grouped depth (RGB used only for instance masks) and is distilled online via action plus relational losses, enabling zero-shot sim-to-real transfer.

Fig. 3: Relational Distillation. The pairwise cosine-similarity matrix over the shared act/anc/obs groups is matched between teacher and student (Eq. 5 ).

Two-object setup (stage one of the cross task)
Method	Input	Reach $\uparrow$	SR $\uparrow$	Traj. $\downarrow$
SIM [ 2 ]	RGB-D+map+odom	30.94	19.26	9.65
End-to-end RL
RGBD-Sem	RGB-D+seg	39.12	0.10	N/A
E2E-Curr.	RGB-D	32.75	0.78	N/A
RGBD-RNN	RGB-D	15.43	0.10	N/A

TABLE II: Baseline Comparison Stages : mean completed placements (of

4

). N/A: fewer than

1\%

successes, too few for a trajectory statistic. E2E-RGB and E2E-RGBD are omitted for space (Reach

5.57

6.54

, SR

0.00

0.29

). Wilson

95\%

half-widths at

n{=}1024

{\pm}0.4

near

0

{\pm}2.4

18\%

{\pm}2.0

87\%

;

0/1024

bounds SR below

0.37\%

Fig. 4: Behavior cloning struggles in EgoPush. Action-chunk PCA collapses on human teleop demos (left) and is faithful but insufficient on RL demos (right) .

Fig. 5: Imitation-gap baselines on the full four-object cross. Left: boxes placed (

4

= success) and SR for each teacher (Tea) and its egocentric student (Stu). Right: student distillation loss under one pipeline (SITT with

\lambda_{1}\in\{0.01,0.02\}

Panel of Table 7 — TABLE III: Teacher Observation Ablation Top: behaviors; bottom: teacher/student metrics, including the deployed student’s active gain $g$ (Sec. IV-D ). Student-SR Wilson $95\%$ intervals ( $n{=}1024$ ) are disjoint: $[85.1,89.2]$ , $[51.7,57.8]$ , $[0,0.37]$ .

Method	SR $\uparrow$	Time $\downarrow$	Traj. $\downarrow$
Base	16.02	1076.42	13.78
SWR	87.50	505.13	14.24
SWR+TD	97.95	455.46	13.77
Ours	98.63	443.26	12.83

TABLE IV: Credit-Assignment Ablation on Cross Final performance: Ours after

{\sim}45

k vectorized steps, the ablations after

{\sim}90

k. Time and Traj. are computed over successful episodes only.

Fig. 6: Credit-assignment ablation on Cross. First 50k vectorized steps (

8{,}192

environments each). Left: per-variant instantaneous reward (scales differ); right: episode length.

Fig. 7: Why ACT saturates. Left: teleop demos give $0\%$ , RL demos saturate at $18.2\%$ . Right: active perception over $256$ two-object episodes; $V$ in-view rate, $g$ active gain against a position-matched random-heading null, $\tau$ re-acquisition steps.

Segmenter	#Tr.	SR	Pos. (cm)	Ori. (°)	Img. Proc. (ms)	Net Infer. (ms)	Comm. (ms)
HSV (controlled)	30	21/30	4.0	5.8	26.65 $\pm$ 2.07	5.97 $\pm$ 0.88	4.46 $\pm$ 0.97
MobileSAM (cluttered)	30	20/30	4.1	5.8	52.95 $\pm$ 8.33	6.19 $\pm$ 1.13	5.77 $\pm$ 0.53

TABLE V: Real-World Evaluation SR is trial-level: the cross completed with no collision, every centroid within

7.5\,

cm of its target and every orientation within

15^{\circ}

. Pos./Ori.: mean position and orientation error over every object of every trial; latency: per-frame mean

\pm

std.

Fig. 8: Real-world deployment. (a) MobileSAM-refined segmentation in cluttered scenes (RGB only forms masks). (b) Pose-criterion evaluation rollouts (real vs. target layout).

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

Oct 9, 2026

EgoReveal: Learning Active Vision and Object Manipulation under Partial Occlusions from Egocentric Demonstrations

Learning from Demonstrations (LfD) enables robots to learn object manipulation directly from human demonstrations. In contrast to resource-intensive and labor-intensive teleoperation, egocentric demonstrations are easy to collect, enabling scalable data collection. Manipulated objects are often partially occluded in everyday environments, yet prior egocentric demonstration-based methods that require no robots or specialized teaching devices do not explicitly address occlusions. To fill this gap, we propose EgoReveal, a framework that learns viewpoint-repositioning together with manipulation actions from egocentric human demonstrations to reposition the viewpoint and obtain target visibility under partial occlusions. EgoReveal instantiates this as two imitation policies: one predicts the future camera-arm pose sequence from the current view and camera-arm pose, and the other predicts the future control commands from object keypoints and a history of control commands. EgoReveal switches between the two based on target visibility. In experiments where the manipulated object is partially occluded initially, EgoReveal repositions the camera to obtain visibility and completes the task more reliably than fixed-camera baselines.

Hiromu Kurihara, Kazutoshi Tanaka, Masashi Hatano +3

Keio University, Yokohama, Kanagawa 223-8522, Japan · OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan

Active Vision for Robot Manipulation Robotic Manipulation Learning from Demonstration Robot Imitation Learning

Ask PDF

Aug 27, 2026

Embodied Scene Rearrangement Planning

This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.

Canzhi Chen, Zan Wang, Siqi Zhu +3

School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China

Robot Task Planning Long-Horizon Robotic Manipulation Robot Manipulation Benchmarks 3D Scene Understanding

Ask PDF

Jun 12, 2026

EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning

Robot learning from real-world demonstrations is currently constrained by data scaling. Universal Manipulation Interface (UMI) provides an efficient robot-free data collection interface, yet current UMI-style pipelines often collect redundant demonstrations and lack global scene context. To improve data efficiency, we present EgoGuide, a collection interface that records synchronized wrist and head/egocentric observations and couples them with online visual-geometric data quality guidance. We also introduce a Gated Egocentric Residual Policy for robust learning from a viewpoint-varying egocentric camera, allowing head/egocentric context to correct ambiguous local observations while preserving stable wrist-view control. Real-world experiments show that EgoGuide reduces the required number of data episodes and improves data efficiency. The residual policy further improves robustness under visual occlusion. Project Page: https://silicx.github.io/EgoGuide

Yue Xu, Mingtao Nie, Tianle Li +4

Shanghai Jiao Tong University · Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Innovation Institute

Robot Learning Robotic Data Collection Visuomotor Policy Learning Robotic Manipulation Data-Efficient Robot Learning Learning from Demonstration+3

Ask PDF

Abstract

Robotic Manipulation

Explore similar work

EgoReveal: Learning Active Vision and Object Manipulation under Partial Occlusions from Egocentric Demonstrations

Embodied Scene Rearrangement Planning

EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning