Pixel: A Co-Designed Full-Stack Robot System for Learning General-Purpose Mobile Manipulation

Tony Tao; Jason Jingzhou Liu; Ruslan Salakhutdinov; Deepak Pathak

cs.RONov 2, 2026

Pixel: A Co-Designed Full-Stack Robot System for Learning General-Purpose Mobile Manipulation

Authors: Tony Tao, Jason Jingzhou Liu, Ruslan Salakhutdinov, Deepak Pathak

Organizations: Carnegie Mellon University

Abstract

General-purpose mobile manipulation requires a combination of capable robot hardware, scalable demonstration collection, expressive whole-body control, and effective policy learning. Yet these components are often developed independently, creating mismatches that limit the capabilities of the complete system. We present Pixel, a co-designed full-stack system for learning general-purpose mobile manipulation. Pixel combines a low-cost wheeled humanoid with human-aligned kinematics and compliant grippers, multiple teleoperation and robot-free human data-collection interfaces, a unified whole-body controller, and a common policy learning pipeline across human and robot demonstrations. A hierarchical learning stack further enables language-conditioned visuomotor control and long-horizon task execution. We evaluate the system across hardware, data collection, and policy learning, and demonstrate mobile-manipulation behaviors requiring dexterity, active perception, navigation, and whole-body coordination. We hope that Pixel provides an integrated and extensible foundation for studying how robot hardware, data, control, and learning can be designed together for robot learning. More results at https://pixel-robot.com/pixel

Figures & tables

Fig. 2: Hardware co-design. Our robot hardware and sensorized human data collection system are co-designed at all levels, with shared camera perspectives, aligned human–robot kinematics, paired end effectors designed from the ground up, and a functionally aligned lower-body design.

Fig. 3: Pixel hardware overview. Our low-cost robot platform is designed for high performance, reliability, and reproducibility. The system integrates a holonomic mobile base, telescoping vertical lift, articulated torso, active-perception head, and custom high-force, compliant end effectors.

Fig. 4: Data collection interfaces. Pixel supports three main forms of data collection. Left: A kinematically matched leader-arm station for precise bimanual teleoperation. Center: VR for untethered whole-body teleoperation. Right: Sensorized human data collection system for scaling robot-free demonstrations. All interfaces produce a common task-space representation used by the shared whole-body controller.

Fig. 5: Policy overview. Left: The high-level policy processes voice commands, prior interaction context, and selected keyframes to determine the next subtask and generate a verbal response to the user. Right: The low-level policy conditions on the subtask description and visual observations to generate robot actions using a DiT-based action expert.

Fig. 6: Evaluation tasks. We evaluate our robot hardware, data-collection interfaces, and policy-design choices across everyday mobile-manipulation tasks requiring precision, long-horizon execution, and whole-body coordination.

Fig. 7: Top: Test objects for the gripper test. Bottom: Experimental setup showing a grasping test with the AgileX Piper gripper.

Fig. 8: Relative-base action space. We express predicted head and bimanual end-effector trajectories in a shared pseudo-base frame constructed at the start of each action chunk. This representation preserves long-horizon precision and provides a common action space for sensorized human and robot demonstrations. Relative-pure and Relative-head reference frames are shown for comparison.

Fig. 9: Workspace comparison. Pixel achieves the strongest human–robot workspace alignment among the evaluated platforms. Mobile ALOHA, YOR, and AGIBOT G1 cover a smaller fraction of the human workspace, while Dexmate Vega and RB-Y1M achieve broader reach through articulated lower bodies but substantially exceed human workspace volume.

Robot	Overall Match $\uparrow$	Arm-Spec. Match $\uparrow$	Human Coverage $\uparrow$	Relative Volume $\rightarrow 1$
Pixel (Ours)	90.99%	81.04%	89.89%	1.11
YOR	89.80%	84.27%	79.65%	0.89
AGIBOT-G1	75.22%	74.44%	80.71%	1.30
Mobile ALOHA	66.50%	57.47%	46.00%	0.52
RB-Y1M	54.65%	52.29%	100.00%	3.02
Dexmate	41.28%	46.79%	100.00%	4.37

TABLE I: Human–robot workspace comparison. Reach-match scores use a

50\,\mathrm{mm}

tolerance; coverage and relative volume use the original

25\,\mathrm{mm}

voxel occupancy.

Fig. 10: (a) We find that torso, lift, and head motion jointly support consistent task success across the evaluated tasks. (b) Pixel’s gripper outperforms commercial alternatives such as the AgileX Piper gripper in grasping force and stability.

Fig. 11: (a) User preferences obtained through a survey on different data collection methods. (b) Mean completion times for two user-study tasks using different data collection interfaces. (c) Our shared interface and controller design enables co-training to improve performance.

Fig. 12: (a) Comparison of observation- and action-space design choices for policy training with sensorized human data . (b) Comparison of observation- and action-space design choices for policy training with robot teleoperation data .

Task	$m$	$k$	$d$	$r_{d}$
Each hand pose	3	60	50	0.2
Head pose	4	50	100	0.05
Upper-body posture	0.2	50	100	0.2
Torso posture	0.3	150	100	0.2
Lift posture	1	50	5000	1.0
Base yaw attractor	1	50	100	0.2

TABLE II: Attractor parameters used in Equation 11 . All rows use

\alpha=40

and

s_{d}=10

. The head row applies separately to its forcing and geometric terms; damping applies only to forcing terms.

Fig. 13: Offline and online temporal synchronization. Dots indicate source samples; orange dots show the latest samples at or before the highlighted grid time. (a) Offline synchronization resamples the shared recording interval (shaded), with one grid time illustrated. (b) Online synchronization selects from currently buffered samples. Rings mark the latest sample in each stream; the earliest of these determines the reference time after rounding down to the grid. Timing is schematic.

Parameter	Setting
Temperature	0.1
Maximum output tokens	4096
Image resolution / API tier	Low / priority
Response format	Schema-constrained JSON
Thinking level	Planning: low; monitoring: minimal
Request timeout	Planning: 8 s; monitoring: 4 s

TABLE III: High-level VLM configuration.

Skill	Low-level policy instruction	Visual completion criteria
Tidy toys	Pick up the toys from the table and place them neatly in the toy basket.	No toys remain on the table, and the toys are visibly in the basket.
Tidy pillows	Navigate to the couch and tidy the pillows.	Both pillows are fully visible, upright, and neatly arranged on the couch.
Hang clothing	Grab the clothing item from the person and hang it on the clothes rack.	The clothing item is visibly hanging on the rack (a wooden pole).
Deliver jacket	Grab the jacket from the clothes rack and give it to the person.	The requesting person visibly receives the jacket.
Clean table	Pick up the handheld vacuum and clean the dining table.	The table is visibly clean after vacuuming, and the vacuum is retracted from the surface.
Return vacuum	Put the vacuum back.	The handheld table vacuum is visibly returned to storage.

TABLE IV: Configured high-level skill library.

Hyperparameter	Value
Vision backbone	DINOv3 ViT-B/16
Vision parameters per view	Approximately 86M
Language encoder	Frozen DistilBERT
Language embedding dimension	768
Visual context length	591 tokens
Token / hidden dimension	768

TABLE V: Low-level policy training hyperparameters.

Hyperparameter	Value
Random resized crop area	0.9–1.0
Random rotation	$\pm 5^{\circ}$
Color jitter: brightness	0.3
Color jitter: contrast	0.4
Color jitter: saturation	0.5
Color jitter: hue	0.08

TABLE VI: Image augmentation settings.

Fig. 14: Representative robot task sequences. Each row shows four key stages, ordered from left to right. Rows (a–c) correspond to the sensorized human data tasks, and rows (d–f) to the robot teleoperation data tasks.

Task	Score	Condition
Toy Cleanup	$0.000$	No task-relevant progress.
	$0.500$	The toy is grasped and lifted from the table.
	$1.000$	The toy is released inside the basket.
Tidy Pillow	$0.000$	No task-relevant progress.
	$0.143$	The robot navigates to the couch.
	$0.429$	The first pillow is picked up.

TABLE VII: Progress-based task scoring rubrics. Each score is cumulative.

Configuration	Toy Cleanup	Tidy Pillow	Trash Disposal	Average
Complete system	$\boldsymbol{100.0\pm 0.0}$	$\boldsymbol{100.0\pm 0.0}$	$\boldsymbol{100.0\pm 0.0}$	$\boldsymbol{100.0\pm 0.0}$
Torso disabled	$60.0\pm 54.8$	$14.3\pm 0.0$	$68.0\pm 46.0$	$47.4\pm 29.0$
Lift disabled	$60.0\pm 54.8$	$85.7\pm 24.7$	$68.0\pm 46.0$	$71.2\pm 13.2$
Head control disabled	$\boldsymbol{100.0\pm 0.0}$	$20.0\pm 12.8$	$36.0\pm 41.0$	$52.0\pm 42.3$

TABLE VIII: Hardware degree of freedom ablations.

Configuration	Toy Cleanup	Tidy Pillow	Trash Disposal	Average
All cameras, relative-base	$75.0\pm 44.4$	$\boldsymbol{95.0\pm 10.6}$	$\boldsymbol{76.0\pm 37.6}$	$\boldsymbol{82.0\pm 11.3}$
Head camera only	$5.0\pm 15.4$	$80.7\pm 25.9$	$34.0\pm 42.6$	$39.9\pm 38.2$
Wrist cameras only	$82.5\pm 37.3$	$72.9\pm 29.3$	$44.0\pm 41.3$	$66.5\pm 20.0$
Relative-pure actions	$\boldsymbol{85.0\pm 36.6}$	$47.9\pm 22.4$	$69.0\pm 25.5$	$67.3\pm 18.6$
Relative-head actions	$20.0\pm 41.0$	$25.0\pm 27.4$	$32.0\pm 37.5$	$25.7\pm 6.0$

TABLE IX: Policy ablations with sensorized human data.

Configuration	Vacuum Cleaning	Snack Retrieval	Hang Jacket	Average
All cameras, relative-base	$65.0\pm 45.9$	$88.3\pm 27.3$	$82.9\pm 22.1$	$78.7\pm 12.2$
Head camera only	$0.0\pm 0.0$	$\boldsymbol{100.0\pm 0.0}$	$78.6\pm 22.6$	$59.5\pm 52.7$
Wrist cameras only	$70.0\pm 38.7$	$93.3\pm 14.1$	$58.6\pm 22.8$	$74.0\pm 17.7$
Absolute joint-space actions	$\boldsymbol{77.5\pm 41.6}$	$90.0\pm 31.6$	$\boldsymbol{91.4\pm 18.1}$	$\boldsymbol{86.3\pm 7.7}$

TABLE X: Policy ablations with robot teleoperation data.

Training data	Toy Cleanup	Tidy Pillow	Average
Sensorized human only	$75.0\pm 44.4$	$\boldsymbol{95.0\pm 10.6}$	$85.0\pm 14.1$
Human + robot (balanced sampling)	$\boldsymbol{95.0\pm 22.4}$	$93.6\pm 21.0$	$\boldsymbol{94.3\pm 1.0}$

TABLE XI: Human–robot co-training results.

Gripper	Horizontal pull-out	Vertical pull-out	Closing force
Pixel	$\boldsymbol{82.7\pm 2.6}$	$\boldsymbol{71.0\pm 3.0}$	$\boldsymbol{92.9\pm 2.4}$
AgileX Piper	$46.4\pm 3.4$	$43.4\pm 6.1$	$63.3\pm 1.5$

TABLE XII: Gripper force measurements.

Interface	Tidy Pillow	Wine-Glass Pouring
Sensorized Human	$\boldsymbol{8.4\pm 1.7}$	$\boldsymbol{6.7\pm 1.9}$
VR Joystick	$39.0\pm 14.9$	$29.1\pm 9.9$
Leader Arm	$35.3\pm 8.5$	$36.9\pm 20.8$
VR Body Tracking	$52.1\pm 18.7$	$34.0\pm 11.7$

TABLE XIII: User-study completion times.

Mobile Manipulation

Mobile manipulation studies robots that navigate through environments while using arms or other effectors to interact with objects.

See all 95 papers

Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations
Nov 2, 2026Jason Jingzhou Liu, Tony Tao, Deepak Pathak +1Whole-Body Robotic Manipulation Robot Imitation Learning
Carnegie Mellon University
HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations
Mar 3, 2026Xiaomeng Xu, Jisang Park, Han Zhang +6Mobile Manipulation Whole-Body Robotic Manipulation
Stanford University · Toyota Research Institute
HumanoidUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation
Jun 25, 2026Hongwu Wang, Chenhao Yu, Youhao Hu +3Robot Skill Learning Robotic Data Collection
Beijing Academy of Artificial Intelligence

Nov 2, 2026J/K move · Enter open · S save

Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations

Jason Jingzhou Liu, Tony Tao, Deepak Pathak, Ruslan Salakhutdinov

Carnegie Mellon University

Whole-Body Robotic Manipulation Robot Imitation Learning Cross-Embodiment Robot Learning Human-to-Robot Motion Imitation Motion Retargeting

Open paper AskPDF arXiv

Sensorized human demonstrations offer scalable data for mobile manipulation, but embodiment differences complicate policy learning. We present Pixel-UMI, a pipeline for learning environment-aware whole-body policies from human data. We first introduce Environment-Aware Retargeting (EAR), which combines reconstructed scene geometry with demonstrated head and gripper trajectories to determine robot posture and generate collision-aware joint and base targets. Delay-calibrated action assignment converts these trajectories into actions. However, kinematic differences can leave residual pose errors, creating viewpoint mismatches between human demonstrations and robot observations. We therefore introduce Retargeting-Error-Aware Masking (REAM), which uses these errors during training to reduce reliance on mismatched views at deployment. Finally, because estimated actions cannot fully capture contact-rich interactions, we collect on-policy human corrections through DAgger to improve performance. We evaluate Pixel-UMI on five challenging tasks spanning articulated-object manipulation, deformable-object handling, constrained-space reaching, whole-body collision avoidance, and contact-rich bimanual coordination, achieving 87.0% average task progress. Video results are available at https://pixel-robot.com/pixel-umi

Abstract

Mobile Manipulation

Explore similar work

Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations

HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations

HumanoidUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation