HubEgo-100K: In-the-Wild Egocentric Video of Skilled Work for Robot Learning
Organizations: Hub Labs, San Francisco
Abstract
Robot policies trained by imitation generalize poorly outside the environments and objects seen in training. Published data-scaling studies find that generalization grows with the diversity of training environments and objects rather than with more demonstrations per environment. Robot data is costly to collect across many environments, whereas egocentric human video can be recorded in the wild at scale; existing egocentric video, however, mostly shows daily-life activity and contains little skilled work in real workplaces. We present HubEgo-100K, a dataset built for diversity. This version of the paper covers the first release, v1.0: 10,124 hours of 1080p egocentric video with synchronized 100 Hz inertial (IMU) data from one head-mounted phone camera. In v1.0, 12,227 contributors record their own unscripted work in 15,548 environments (each one contributor's own setting of one type, such as their kitchen or workshop) across 11 setting types, 30 work categories, 56 task types and 35 action families, with 83% of hours in workplace settings. The release comprises 72,754 episodes with machine-generated action and language annotations, in LeRobot and MCAP formats under CC BY 4.0. We measure diversity along each axis, show that inside v1.0, with episodes in random order, new contributors and words keep appearing, and relate the measurements to published evidence on six data properties tied to policy performance, including hand visibility and inertial sensing. We also document collection, curation, privacy measures and data format, and provide a datasheet. Later versions of this paper will add benchmarks and cover further releases toward 100,000 hours.
Figures & tables
| Dataset | Hours | Viewpoint | Setting | Modalities | People | License |
| Ego4D ( Grauman et al., 2022 ) | 3,670 | head-mounted (7 camera types) | daily life; 74 locations, 9 countries | RGB; audio, IMU (836 h), gaze, 3D scans, stereo on subsets | 931 d | Ego4D license agreement |
| Ego-Exo4D ( Grauman et al., 2024 ) | 1,286 (221 ego) a | Aria glasses + 4 to 5 static cameras | skilled activities; 123 scenes, 13 cities | ego and exo RGB, audio, IMU, gaze, 3D, poses, language | 740 | Ego-Exo4D license agreement |
| EPIC-KITCHENS-100 ( Damen et al., 2022 ) | 100 | head-mounted | participants’ own kitchens (45) | RGB, audio, narrations | 37 | CC BY-NC 4.0 |
| EGTEA Gaze+ ( Li et al., 2018 ) | 28 | eye-tracking glasses | one kitchen, 7 recipes | RGB, audio, gaze, hand masks | 32 | n/a |
| Assembly101 ( Sener et al., 2022 ) | 513 (167 ego) a | 8 static + 4 headset (monochrome) | lab desk, toy assembly | RGB, monochrome ego, 3D hand pose | 53 | CC BY-NC 4.0 |
| HOI4D ( Liu et al., 2022 ) | n/a (2.4M frames) | helmet-mounted RGB-D | 610 indoor rooms | RGB-D, 3D hand and object pose, segmentation | 9 | n/a |
| Quantity | Value |
| Hours | 10,124 |
| Episodes | 72,754 (median 4.9 min, at most 30 min) |
| Recordings | 41,688 |
| Frames | 1,093,194,072 |
| Contributors | 12,227 |
| Environments | 15,548, across 11 setting types |
| Property | Published evidence (selected) | HubEgo-100K measurement |
| Environment and object diversity | Generalization follows a power law in the number of environments and objects; demonstrations per environment saturate; 32 environments with 50 demonstrations each reach about 90% success on new environments and objects ( Lin et al., 2025 ) . At matched size, more diverse scenes improve out-of-distribution success ( Khazatsky et al., 2024 ) ; more training locations improve success in unseen homes ( Black et al., 2025a ) . | 15,548 environments across 11 setting types; median 268 environments per task type (47 of 56 task types have at least 100); median 489 per work category; effective number of setting types 7.7 of 11. |
| Egocentric human video | Ego4D pretraining beats ImageNet for manipulation (34% to 42%) ( Nair et al., 2022 ) ; human data improves robot policies by 34 to 228% relative score ( Kareer et al., 2025 ) , raises out-of-distribution success from 34.7% to 59.4% ( Qiu et al., 2025 ) and nearly doubles success on settings seen only in human data after diverse pretraining ( Kareer et al., 2026 ) . | 10,124 hours, 1.09 billion frames, head-mounted ultra-wide view (median 114 degrees diagonal), 72.8% of hours in episodes longer than 10 minutes. |
| Hand visibility and curation | On Ego4D, keeping stable clips with a visible hand (975 of 3,976 hours) turns a negative co-training effect into a positive one ( Long et al., 2026 ) ; hand and contact labels mined from egocentric video raise real-robot success ( Srirama et al., 2024 ) . | Model-estimated shares of sampled frames: at least one hand in 94.7%, both hands in 76.3%; model-estimated on-task share 96.9%; 83.0% of hours in episodes with a hand in at least 90% of sampled frames. |
| Task breadth and long tail | Removing 25% of tasks hurts generalization as much as removing 49% of data ( Brohan et al., 2023 ) ; training on more than 100 tasks gives 44% success on 24 unseen tasks ( Jang et al., 2021 ) . | 56 task types, 30 work categories, 35 action families; 1,429 verb and 3,709 noun lemmas in the machine-generated descriptions; inside v1.0, with episodes in random order, the last 10% of hours adds about 39 verb and 84 noun lemmas; 34.5% of hours labeled hard. |
| Contributor diversity | Diversity of people and places is a design goal of Ego4D ( Grauman et al., 2022 ) ; mixed-skill multi-operator data is harder to imitate at small scale ( Mandlekar et al., 2021 ; Belkhale et al., 2023 ) . | 12,227 contributors; no contributor above 0.22% of hours; 2,824 contributors recorded one hour or more; effective number of contributors 5,666 (exponential of entropy). |
| Inertial sensing | IMU is a low-cost egocentric modality (video takes 60 times more storage and 43 times more compute) ( Gong et al., 2023 ) ; IMU pretraining aligned with text and video nearly doubles activity-recognition F1 on Ego4D ( Moon et al., 2023 ) ; a camera plus a low-cost IMU is the minimum suite for metric 6-DoF state estimation ( Qin et al., 2018 ) . | Accelerometer and gyroscope aligned to every frame; native accelerometer rate median 99.98 Hz; in MCAP, device motion for every episode, magnetometer and heading for 99.1% and barometer for 93.6% of episodes. |
| Axis | ||||
| Contributors | 12,227 | 5,666 | 0.92 | 3,385 |
| Environments | 15,548 | 7,244 | 0.92 | 4,197 |
| Setting types | 11 | 7.7 | 0.85 | 6.2 |
| Work categories | 30 | 18.9 | 0.86 | 14.4 |
| Task types | 56 | 31.0 | 0.85 | 20.9 |
| Action families | 35 | 20.8 | 0.85 | 16.4 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Task type | Work category | Hours | Episodes | Contrib. | Env. |
| Building site & renovation crew | Construction & renovation | 1,227 | 8,317 | 1,247 | 1,276 |
| Garment & shoe workshop | Textile & apparel production | 1,094 | 4,752 | 868 | 1,035 |
| Building systems trades | Building trades (electrical, plumbing, HVAC) | 691 | 3,293 | 690 | 865 |
| Salon, barber & nail services | Personal services | 415 | 5,218 | 927 | 930 |
| Phone & electronics repair bench | Repair & craft workshops | 323 | 1,538 | 418 | 513 |
| Auto shop & fuel station work | Vehicle repair & services | 323 | 4,622 | 474 | 560 |
Robotic Data Collection
Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.
See all 196 papers