RoboCap: A New Platform for Egocentric Robot Learning

Grounded Superintelligence; BitRobot

cs.ROOct 5, 2026

RoboCap: A New Platform for Egocentric Robot Learning

Authors: Grounded Superintelligence, BitRobot

Organizations: Grounded Superintelligence · BitRobot

Abstract

Despite its promise for scaling robot learning, egocentric manipulation data is still scarce today. Collection at scale requires vertically integrating ergonomic hardware with centimeter-precise 3D algorithms, at a precision that has not been publicly demonstrated. To address this gap, we introduce RoboCap, a 250,g six-camera dual-IMU hat designed for in-the-wild egocentric data capture, and the Grounded API, a suite of device-agnostic 3D algorithms tuned for RoboCap. In this report, we demonstrate how hardware, calibration, and 3D algorithms interact to achieve state-of-the-art performance on the public benchmarks: our SLAM across diverse settings and rigs, our depth estimation on egocentric settings, and our hand tracking when adapted to third-party devices.

Figures & tables

Figure 1 : RoboCap camera coverage. Left: azimuth coverage of the front stereo and lateral cameras. Center: azimuth coverage of the eye stereo cameras, visualized forward for legibility. Right: elevation coverage of the front and eye pairs. FoV-overlap in degrees at infinity.

Cameras (6, hardware-synchronized)
Sensor	Global-shutter RGB, 10-bit, 3.0 µm px
Frame resolution	$1920{\times}1080$
Frame rate	30 Hz
Lens	Kannala-Brandt 4
Focal length	$f=590$ – $630$ px
Exposure	Per-camera AE, bounded $\leq 10$ ms

Table 1: RoboCap sensor suite. The two stereo pairs use fisheye optics, the lateral cameras a wide lens. Stereo overlap is the binocular field at infinity from the calibrated extrinsics. IMU noise parameters are fleet-level estimates from per-device continuous-time calibration [ 16 ] .

Panel of Figure 2 — Figure 2 : Epipolar residuals before (top) and after (bottom) self-calibration. Identical features are marked in the left and right cameras with a black dot, and epipolar lines are drawn across each pair. Self-calibration eliminates almost all epipolar drift and improves 3D estimation.

Panel of Figure 3 — Figure 3 : Histograms of the epipolar drift in the eye pair (left), front pair (middle), and front-eye pairs (right), over a subset of 150 deployed devices. We find that rotational correction (Section 4.1 ) is sufficient to restore epipolar drift, which follows a Gaussian centered around 0 ∘ .

Figure 4 : Scene reconstruction over 4 RoboCap sessions. For each session, we generate point clouds (Section 4.3 ) and project them into the SLAM world frame, leveled with the estimated gravity direction (Section 4.2 ). Then, we solve multi-session frame alignment with RANSAC on 3D-projected SIFT features.

Method	Inputs	Short	Medium	Long	Low light	Moving	Avg.
Aria’s SLAM (r) ‡	bino, IMU	90.7	78.5	70.9	84.2	55.0	75.9
ORB-SLAM3 (r) [ 7 ]	mono, IMU	23.0	10.9	11.2	3.1	2.0	10.0
OKVIS2 (r) [ 30 ]	bino, IMU	20.0	11.6	2.6	14.5	4.7	10.7
OpenVINS+Maplab (r)	bino, IMU	27.7	23.4	12.8	19.8	13.9	19.5
Best of other submissions	mono, IMU	34.2	29.3	16.8	25.4	15.5	24.2
Ours	bino, IMU	80.2	61.6	59.9	67.7	41.9	62.3

Table 2: Official LaMAria test-set results (control-point score, higher is better). (r): reproduced by the benchmark authors. “Best of other submissions” is the highest score per category over all other public entries. ‡ Shown in gray as a reference and excluded from the ranking: Aria’s SLAM is the device manufacturer’s own system, developed together with the glasses, and is the solution used to initialize the benchmark’s ground-truth pipeline (its pose recall is therefore not reported). Bold marks the best result among the remaining methods.

Panel of Figure 6 — Figure 6 : Examples from our hand tracking model, which fuses multi-view images into a single hand mesh. Our model’s pretraining makes it robust to blur, visual ambiguity, complexity, gloves, occlusion, and lighting. See Figures 10 and 11 for more difficult examples.

	Scratch	Pretrained
HOT3D [ 3 ]	17.41	14.47
UmeTrack [ 24 ]	19.00	11.49
SHOW3D [ 45 ]	19.34	13.68

Table 5 : Best validation MPJPE (mm) on held-out subjects across datasets, varying only initialization. More details in Appendix A .

	POEMv2 [ 64 ]	Ours
DexYCB-Mv	6.69	5.22
OakInk-Mv	8.34	7.25
HO3D-Mv	7.72	13.01

Table 6 : MPJPE (mm) on third-person multi-view benchmarks.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10 : Robustness to degraded imaging and extreme viewpoints. Each panel shows the four camera views of a single frame with the predicted mesh overlaid. The model remains multi-view consistent when the hand is blurred by fast motion, when the lens is fogged, when the hand is close enough to the camera to fill the frame, and when the hand is seen through a transparent glove.

Panel of Figure 11 — Figure 11 : Robustness to hand-object interaction and unusual configurations. Each panel shows the four camera views of a single frame with the predicted mesh overlaid. The model recovers plausible poses while grasping small objects, while holding a tool with the fingers largely occluded by it, during fine manipulation such as drawing, and when the hands appear alongside other body parts.

Panel of Figure 12 — Figure 12 : Comparison of random vs pretrained weights on egocentric benchmarks. Within each panel the columns are the left and right camera views of a single frame, the top row is the model trained from scratch and the bottom row is the model initialized from our pretrained weights; both are trained until they achieve their best multi-dataset validation metrics. The pretrained model aligns more closely with the observed hand, particularly at the finger joints, where the scratch model tends to mispredict articulation while keeping the global hand position roughly correct.

	Held-out (ours)		Official test
	subjects	share	subjects	share
UmeTrack	4	10.1%	4	—
SHOW3D	6	20.0%	6	21.0%
HOT3D	2	16.0%	2	—

Appendix

Table 7: Composition of the surrogate held-out sets. UmeTrack: 103 sequences, 12.6% hand-hand interaction against the official split’s 11.6%; real sequences only. SHOW3D: 337 scenes covering 23 of 28 object classes against the official split’s 23; novel-activity rate 12.7% against 12.2%. HOT3D: two held-out subjects per device, matching the official pose track.

	views	train	test	split
DexYCB-Mv [ 10 ]	8	25,387	4,951	S0
OakInk-Mv [ 60 ]	4	58,692	19,909	SP2
HO3D-Mv [ 22 ]	5	9,087	2,706	7 seq.

Appendix

Table 8: Multi-view benchmark splits, in multi-view frames, following [ 64 ] . HO3D-Mv comprises the seven sequences: ABF1, BB1, GSF1, MDF1 and SiBF1 for training, GPMF1 and SB1 for evaluation.

Explore similar work

Jun 12, 2026cs.RO

EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning

Robot learning from real-world demonstrations is currently constrained by data scaling. Universal Manipulation Interface (UMI) provides an efficient robot-free data collection interface, yet current UMI-style pipelines often collect redundant demonstrations and lack global scene context. To improve data efficiency, we present EgoGuide, a collection interface that records synchronized wrist and head/egocentric observations and couples them with online visual-geometric data quality guidance. We also introduce a Gated Egocentric Residual Policy for robust learning from a viewpoint-varying egocentric camera, allowing head/egocentric context to correct ambiguous local observations while preserving stable wrist-view control. Real-world experiments show that EgoGuide reduces the required number of data episodes and improves data efficiency. The residual policy further improves robustness under visual occlusion. Project Page: https://silicx.github.io/EgoGuide

Yue Xu, Mingtao Nie, Tianle Li +4

Shanghai Jiao Tong University · Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Innovation Institute

Robot-Free Demonstrations Robotic Data Acquisition Egocentric Vision Wrist Residual Policy Redundancy

Ask PDF

May 16, 2026cs.CV

EgoKit: Towards Unified Low-Cost Egocentric Data Collection with Heterogeneous Devices

Egocentric video is increasingly used as a data source for robot learning, activity understanding, and embodied AI research, but collecting it at scale remains fragmented in practice: each candidate host device, such as an Android phone, iPhone, iPad, smart glasses, or extended reality (XR) headset, exposes a different SDK, a different policy on raw camera access, and different limitations on external USB cameras and on-device tracking. Synchronized ego-view and wrist-view capture is therefore typically obtained by either committing to a single proprietary platform or building one-off rigs that do not transfer across devices. To address this gap, we present EgoKit, a toolkit that exposes the same egocentric recording workflow across six heterogeneous host devices. Across all supported devices, EgoKit presents the same recording interaction and produces locally stored video with a uniform log format; on XR headsets, it additionally logs head pose and OpenXR-standard 26-joint hand tracking aligned to the video streams. The companion accessories, including two wrist cameras with mounts, a head strap, and a USB-C hub, add wrist-view capture to any supported host without custom hardware fabrication. EgoKit is available at https://egokit.chuange.org/.

Liuchuan Yu, Erdem Murat, Beichen Wang +10

George Mason University, Computer Science, Fairfax, Virginia 22030, USA.

Egocentric Dataset Egocentric Video Robotic Data Acquisition 3D Hand Wrist Device

Ask PDF

Jun 6, 2026cs.RO

Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Data

Robotics faces a fundamental challenge of data scarcity. Unlike language or vision research, there is no internet-scale dataset for robotic manipulation. A promising path forward is to leverage egocentric human data, which can be collected more easily, with greater breadth, and at a larger scale. Towards this end, we investigate key design choices for learning across human and humanoid embodiments equipped with dexterous five-finger hands, using the

π_{0.5}

model as a foundation. Our results show that human data enables robots to learn new task semantics and compose existing skills into novel behaviors without corresponding robot data. The paper website is here: https://egopipaper.github.io/

Ji Woong Kim, Ke Wang, Zipeng Fu +4

1Stanford University · 2Meta

Egocentric Dataset Humanoid

Ask PDF