EgoPush: Egocentric Multi-Object Rearrangement for Mobile Robots via Constrained Teacher Observability

Boyuan An; Zhexiong Wang; Yipeng Wang; Jiaqi Li; Sihang Li; Jing Zhang; Chen Feng

cs.ROFeb 20, 2026

EgoPush: Egocentric Multi-Object Rearrangement for Mobile Robots via Constrained Teacher Observability

Authors: Boyuan An, Zhexiong Wang, Yipeng Wang, Jiaqi Li, Sihang Li, Jing Zhang, Chen Feng

Organizations: New York University

Abstract

Humans rearrange objects in cluttered environments using egocentric perception, actively moving to keep task-relevant spatial cues in view. Mobile robots have not matched this: rearrangement is usually built on a global pose estimate or a map, which is exactly what a robot carrying one camera lacks, while pushing keeps changing the scene it would have to be built from. We present EgoPush, which pushes objects into anchor-relative formations from onboard RGB-D alone, with no global localization, external tracking, or map at deployment, and transfers zero-shot to a TurtleBot in controlled and visually cluttered scenes. What makes this learnable turns out to be a property of the teacher rather than of the student: three privileged teachers trained with identical rewards, architecture, and hyperparameters all exceed $98\%$ success, yet their distilled egocentric students reach $0\%$ , $54.8\%$ , and $87.3\%$ , the only variable being the teacher's observation function. EgoPush therefore trains the teacher under egocentric observability constraints, restricting it to visibility-limited cues and revealing target references only when the anchor is centrally visible, so that its supervision is recoverable by a depth-based student distilled online. Making the teacher trainable in the first place needs two further pieces: a role-grouped object-centric interface shared by teacher and student, and stage-wise temporally decayed rewards for long-horizon credit assignment. Videos, the playable task, and code are available at https://ai4ce.github.io/EgoPush/.

Figures & tables

Method	Onboard	No global	Multi	Horizon	Real
ReloPush [ 1 ]	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
SAMaps [ 20 ] / SIM [ 2 ]	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
DyWA [ 14 ]	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$
SPIN [ 13 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$
EgoPush	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE I: Deployment Assumptions of Related Systems Onboard : the policy consumes only robot-mounted sensing. No global : no global pose, map, or external tracking at deployment. Multi : more than one object rearranged. Horizon : several sequential placements.

Fig. 2: EgoPush Overview . A privileged teacher trains from visibility-limited sparse keypoints; the egocentric student consumes role-grouped depth (RGB used only for instance masks) and is distilled online via action plus relational losses, enabling zero-shot sim-to-real transfer.

Fig. 3: Relational Distillation. The pairwise cosine-similarity matrix over the shared act/anc/obs groups is matched between teacher and student (Eq. 5 ).

Two-object setup (stage one of the cross task)
Method	Input	Reach $\uparrow$	SR $\uparrow$	Traj. $\downarrow$
SIM [ 2 ]	RGB-D+map+odom	30.94	19.26	9.65
End-to-end RL
RGBD-Sem	RGB-D+seg	39.12	0.10	N/A
E2E-Curr.	RGB-D	32.75	0.78	N/A
RGBD-RNN	RGB-D	15.43	0.10	N/A

TABLE II: Baseline Comparison Stages : mean completed placements (of

4

). N/A: fewer than

1\%

successes, too few for a trajectory statistic. E2E-RGB and E2E-RGBD are omitted for space (Reach

5.57

6.54

, SR

0.00

0.29

). Wilson

95\%

half-widths at

n{=}1024

{\pm}0.4

near

0

{\pm}2.4

18\%

{\pm}2.0

87\%

;

0/1024

bounds SR below

0.37\%

Fig. 4: Behavior cloning struggles in EgoPush. Action-chunk PCA collapses on human teleop demos (left) and is faithful but insufficient on RL demos (right) .

Fig. 5: Imitation-gap baselines on the full four-object cross. Left: boxes placed (

4

= success) and SR for each teacher (Tea) and its egocentric student (Stu). Right: student distillation loss under one pipeline (SITT with

\lambda_{1}\in\{0.01,0.02\}

Panel of Table 7 — TABLE III: Teacher Observation Ablation Top: behaviors; bottom: teacher/student metrics, including the deployed student’s active gain $g$ (Sec. IV-D ). Student-SR Wilson $95\%$ intervals ( $n{=}1024$ ) are disjoint: $[85.1,89.2]$ , $[51.7,57.8]$ , $[0,0.37]$ .

Method	SR $\uparrow$	Time $\downarrow$	Traj. $\downarrow$
Base	16.02	1076.42	13.78
SWR	87.50	505.13	14.24
SWR+TD	97.95	455.46	13.77
Ours	98.63	443.26	12.83

TABLE IV: Credit-Assignment Ablation on Cross Final performance: Ours after

{\sim}45

k vectorized steps, the ablations after

{\sim}90

k. Time and Traj. are computed over successful episodes only.

Fig. 6: Credit-assignment ablation on Cross. First 50k vectorized steps (

8{,}192

environments each). Left: per-variant instantaneous reward (scales differ); right: episode length.

Fig. 7: Why ACT saturates. Left: teleop demos give $0\%$ , RL demos saturate at $18.2\%$ . Right: active perception over $256$ two-object episodes; $V$ in-view rate, $g$ active gain against a position-matched random-heading null, $\tau$ re-acquisition steps.

Segmenter	#Tr.	SR	Pos. (cm)	Ori. (°)	Img. Proc. (ms)	Net Infer. (ms)	Comm. (ms)
HSV (controlled)	30	21/30	4.0	5.8	26.65 $\pm$ 2.07	5.97 $\pm$ 0.88	4.46 $\pm$ 0.97
MobileSAM (cluttered)	30	20/30	4.1	5.8	52.95 $\pm$ 8.33	6.19 $\pm$ 1.13	5.77 $\pm$ 0.53

TABLE V: Real-World Evaluation SR is trial-level: the cross completed with no collision, every centroid within

7.5\,

cm of its target and every orientation within

15^{\circ}

. Pos./Ori.: mean position and orientation error over every object of every trial; latency: per-frame mean

\pm

std.

Fig. 8: Real-world deployment. (a) MobileSAM-refined segmentation in cluttered scenes (RGB only forms masks). (b) Pose-criterion evaluation rollouts (real vs. target layout).

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

EgoReveal: Learning Active Vision and Object Manipulation under Partial Occlusions from Egocentric Demonstrations
Oct 9, 2026Hiromu Kurihara, Kazutoshi Tanaka, Masashi Hatano +3Active Vision for Robot Manipulation Robotic Manipulation
Keio University, Yokohama, Kanagawa 223-8522, Japan · OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan
Embodied Scene Rearrangement Planning
Aug 27, 2026Canzhi Chen, Zan Wang, Siqi Zhu +3Robot Task Planning Long-Horizon Robotic Manipulation
School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China
EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning
Jun 12, 2026Yue Xu, Mingtao Nie, Tianle Li +4Robot Learning Robotic Data Collection
Shanghai Jiao Tong University · Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Innovation Institute

Oct 9, 2026J/K move · Enter open · S save

EgoReveal: Learning Active Vision and Object Manipulation under Partial Occlusions from Egocentric Demonstrations

Hiromu Kurihara, Kazutoshi Tanaka, Masashi Hatano, Tadashi Kozuno+2

Keio University, Yokohama, Kanagawa 223-8522, Japan · OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan

Active Vision for Robot Manipulation Robotic Manipulation Learning from Demonstration Robot Imitation Learning

Open paper AskPDF arXiv

Learning from Demonstrations (LfD) enables robots to learn object manipulation directly from human demonstrations. In contrast to resource-intensive and labor-intensive teleoperation, egocentric demonstrations are easy to collect, enabling scalable data collection. Manipulated objects are often partially occluded in everyday environments, yet prior egocentric demonstration-based methods that require no robots or specialized teaching devices do not explicitly address occlusions. To fill this gap, we propose EgoReveal, a framework that learns viewpoint-repositioning together with manipulation actions from egocentric human demonstrations to reposition the viewpoint and obtain target visibility under partial occlusions. EgoReveal instantiates this as two imitation policies: one predicts the future camera-arm pose sequence from the current view and camera-arm pose, and the other predicts the future control commands from object keypoints and a history of control commands. EgoReveal switches between the two based on target visibility. In experiments where the manipulated object is partially occluded initially, EgoReveal repositions the camera to obtain visibility and completes the task more reliably than fixed-camera baselines.

Abstract

Robotic Manipulation

Explore similar work

EgoReveal: Learning Active Vision and Object Manipulation under Partial Occlusions from Egocentric Demonstrations

Embodied Scene Rearrangement Planning

EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning