cs.ROOct 13, 2026
SaveEgoExplorer: A Vision-Motion-Touch Observatory for Human Loco-Manipulation In Situ
Organizations: Shanghai AI Laboratory · Zhejiang University · Shanghai Jiao Tong University
Abstract
Learning humanoid loco-manipulation from human demonstrations rests on observing how locomotion, posture, and object interaction evolve together as tasks unfold. Scaling collection across diverse environments, in turn, favors lightweight, portable capture systems. Yet, existing datasets provide only partial views of this coordination. Thus, we introduce EgoExplorer, a 100-hour vision-motion-touch observatory of human loco-manipulation built with a wearable system. With an emphasis on professional activities, its continuous recordings align egocentric video with reconstructed whole-body motion, fine-grained finger articulation, and tactile observations, accompanied by step-level descriptions. To achieve this coverage with sparse body sensing, we develop the Calibrate-Complete-Constrain pipeline. Concretely, reusable calibration fixes participant geometry and sensor attachments before motion inference to disentangle mounting offsets from articulation. A temporal prior then completes unobserved joint motion, while layered constraints promote measurement agreement, temporal continuity, and geometric consistency, properties that motion plausibility alone does not guarantee. Against tracker-fitted references, the method reduces body-point error by 71.9% relative to the strongest baseline; matched evaluations also show gains in hand stability and contact estimation. Preliminary studies explore multimodal generation and conditioning alongside humanoid retargeting with whole-body control in simulation.
Figures & tables
| Dataset | Duration (h) | Body state | Hand state | Touch | Mobile capture | Loco- manip. | Step labels | |||
| metric full body | non-visual anchors | feet observed | articulated | non-visual anchors | ||||||
| Ego4D ( Grauman et al., 2022 ) | 3,670 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |
| Ego-Exo4D ( Grauman et al., 2024 ) | 221 | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ||
| Nymeria ( Ma et al., 2024 ) | 300 | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| EgoDex ( Hoque et al., 2026 ) | 829 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ||
| EgoVerse ( Punamiya et al., 2026 ) | 1,362 | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | |
Table 1: Comparison of representative egocentric datasets. In the body and hand groups, ✓ is sensor-grounded label, a state inferred from images or covering a subset, ✗ absence; elsewhere ✓/ are full/partial support. Anchors : non-visual measurements (wrists, feet, and fingers) unaffected by occlusion. Mobile capture : the wearer’s root travels rather than staying at one workstation. Loco-manip. : both are measurable in the motion data. Step labels : sub-task annotations, ✓ if written or checked by experts, if left as model output. † : judged from the released sample.
Figure 2: Overview of EgoExplorer . The rig records egocentric video, poses from five trackers, finger articulation, and touch ( Sec. 3.1 ). ( Calibrate-Complete-Constrain ) estimates a reusable body-sensor profile through protocol-guided calibration. A temporal prior generates a whole-body motion proposal from anchors; joint refinement then promotes skeletal measurement agreement, participant-specific surface clearance, and whole-mesh ground non-penetration ( Sec. 3.2 ). Hand and touch are integrated with the reconstructed body, followed by expert-guided step annotation and review ( Sec. 3.3 ). The resulting episodes align multimodal data on a shared timeline ( Sec. 4 ).
Figure 3: Data statistics . (a) Session durations; (b) regular and recovery recordings across 10 tasks.
Figure 4: Synchronized body motion and touch across ten tasks. Each panel shows a matched instant, with tactile readings at lower right; gray-to-orange shading indicates baseline-relative increases.
Figure 5: Step annotations. Three color-coded subtasks with timestamps and descriptions.
| Method | Anchors | Body points | Non-tracked | PA-MPJPE | Root | Velocity |
| AvatarPoser ( Jiang et al., 2022 ) | 3 | 128.5 | 122.5 | 86.3 | 99.5 | 106.1 |
| AGRoL ( Du et al., 2023 ) | 3 | 119.6 | 114.0 | 81.0 | 89.6 | 113.0 |
| EgoPoser ( Jiang et al., 2024b ) | 3 | 155.5 | 143.2 | 89.3 | 121.2 | 113.2 |
| AvatarJLM ( Zheng et al., 2023 ) | 3 | 117.9 | 113.3 | 78.6 | 95.0 | 105.9 |
| SAGE ( Feng et al., 2024 ) | 3 | 114.7 | 109.5 | 81.4 | 83.4 | 110.3 |
| SparsePoser ( Ponton et al., 2023 ) † | 6 † | 219.2 | 189.9 | 134.4 | 84.3 | 119.1 |
Table 2: Body reconstruction on the Evaluation Subset (mm; velocity: mm/s). Non-tracked: 17 unobserved joints. † Reconstructed pelvis; ‡ raw tracker poses; ∗ locally trained variants.
| Annotation | Fact error | Unsupported | Missed steps | Extra steps | Boundary error | Unjudgeable |
| Model-generated | 11.09 | 2.46 | 8.16 | 2.05 | 3.08 | 0.82 |
Table 5: Expert review of model-generated text annotations (%).
Table 7: Ablations on the Evaluation Subset and Natural Recordings. Pose errors: mm; velocity/slide: mm/s; jerk: m/s 3 ; floor: %. Blue: primary metric (Avg.); gray: the baseline.
Figure 7: Geometric refinement reduces self-intersection and floor penetration.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Stream | Hardware | Acquisition / representation |
| Ego RGB | GoPro HERO13 Black with Media Mod | 30 Hz; pixels; intrinsics below |
| Body tracking | Five VIVE Ultimate Trackers; head, wrists, ankles | 60 Hz; room-fixed frame with its origin at a designated reference location |
| Finger articulation | MANUS Metagloves Pro (no haptic feedback) or Wuji gloves | Five 6-DoF fingertip EMF sensors per glove; 120 Hz nominal rate |
| Regional touch | Wuji tactile configuration | matrix; 744 taxels per hand; 526 retained by the evaluation health mask; 120 Hz nominal rate |
Table 8: Capture hardware and acquisition settings. Glove rates are manufacturer specifications; per-recording timing follows native timestamps.
| Field group | Per-row shape / type | Contents and conventions |
| Index and time | Scalars | Episode, frame, global row, task index, and timestamp in seconds |
| Ego RGB | Linked video frame | observation.images.ego ; frame matched to the row timestamp |
| Tracker poses | ; | Position in meters; quaternion order ; per-device validity |
| Body motion | World pelvis position (m), root orientation, and 21 local body-joint rotations (axis-angle, rad) | |
| Hand motion | Left/right SMPL-X finger rotations in local axis-angle form | |
| Tactile readings | Normalized taxel readings with invalid cells masked; tactile subset only |
Table 9: Contents associated with each release row. Dataset-specific features extend LeRobot’s standard indexing fields.
Figure 8: Protocol-guided body–sensor calibration sequence. Model-generated reference motions across 17 stages (144 s): (A) static holds and body articulation; (B) finger actions; (C) turning and stepping. These illustrate the protocol, not measured participant motion.
| Interface | Meaning |
| L0: raw signal | Native taxel readings and validity, with range and any sentinels defined by the device schema. Wearing-dependent offsets and gain remain. |
| L1: signal change | Changes relative to a no-contact baseline; still dependent on sensor response and configuration. |
| L2: regional contact | Binary regional contact with validity/health recorded separately. Regional mapping and threshold calibration are versioned. |
Table 10: Tactile interfaces used by the release and evaluation. Binary contact and channel validity are distinct; neither normalized amplitude nor a health indicator is an absolute pressure measurement.
| Experimental stage | Information provided to the annotation model |
| Material preparation | Retrieve reagents, catalysts, containers, and consumables; bring them to the preparation bench and organize them by use. Preparation may include rinsing pipettes, taring a balance, weighing solids, and transferring material. |
| Reaction setup | Add materials to a reaction flask in the specified sequence, adjust heating and stirring, and monitor the apparatus. The document distinguishes reagent addition from device adjustment. |
| Reaction monitoring | For TLC, take a sample, spot a plate, and inspect it under UV illumination. For HPLC preparation, transfer and dilute a sample, mix it, filter it into a vial, and label the vial. |
| Quenching and workup | Move the reaction vessel to the workup area, handle receiving vessels, pour liquids, and inspect temperature or pH readouts as described in the procedure. |
| Extraction | Fill a separatory funnel, allow layers to settle, open and close the stopcock, collect layers in separate vessels, and repeat mixing and separation when required. |
| Rotary evaporation | Attach the flask, set the apparatus, operate vacuum/heating/rotation controls, monitor concentration, stop the apparatus, release vacuum, detach the flask, and weigh it. |
Table 11: Example procedural knowledge supplied by laboratory Skills documents.
Figure 9: Collection scale and coverage: (a) recording and duration shares by environment; (b) glove configurations; (c) sensing-stream coverage. Panels (b,c) use regular recordings.
Figure 10: Movement statistics: (a) 3D head paths; (b) locomotion and manipulation bout durations; (c) five-tracker dynamics, with units indicated on the axes.
Figure 11: A reaction-setup episode in two consecutive intervals. Rows align ego frames, tracker speed, behavior, and posture; dashed lines mark detected movement-to-manipulation transitions.
Figure 12: Hand activity and contact: (a) unilateral/bimanual occupancy; (b) event frequency; (c) locomotion and stationary hand activity. Panels (a,b) retain healthy tactile streams.
Figure 13: Contact-frame occupancy by region: (a) thumb, index, and middle fingers; (b) ring and little fingers, and palm. L/R denotes the hand; n/a indicates insufficient live-taxel support.
Figure 14: Reconstructed capture environments: laboratory, industrial workspace, living room, food-preparation area, and study. Each panel shows one registered view.
| Statistic | Value |
| Temporal coverage (video-duration weighted) | 99.9998% |
| Segment duration p50 / p90 (s) | 7.486 / 24.000 |
| Mean caption length (characters) | 98.87 |
| Caption length p50 / p90 (characters) | 87 / 163 |
| Exact duplicate rate | 0.0402% |
Table 12: Caption coverage, duration, length, and exact duplication.
Figure 15: Step-annotation vocabulary. Word size indicates term frequency; frequent terms describe laboratory objects, equipment, and manipulation.
| Modality | IDs | Action coverage / controls |
| Hands | H01–H03 | Spread, hook, fist; articulation and self-occlusion |
| Hands | H04–H07 | Thumb opposition to each of the other fingers |
| Hands | H08–H10 | Tripod pinch, lateral pinch, spherical grasp |
| Body, static | B01–B06 | Overhead, left/right cross-body reach, bend, left/right lunge |
| Body, dynamic | B07–B10 | Sit/stand, rectangular step sequence, separate left/right forward kicks |
| Contact | T01–T05 | Thumb–index press, bar press, spherical press, five-finger press, palm press; ON/OFF controls |
Table 13: Action groups in the Evaluation Subset. Guides specify intended actions; references and recording counts are defined separately for each modality.
| Comparison | Inputs | Controls / reference |
| Public body baselines | Head and wrists | Calibrated anchor conversion; raw-pose variants omit it |
| AvatarPoser ∗ / HMD-Poser ∗ | Five trackers | Local training; distinct training data and adapters ( Sec. F.2 ) |
| SparsePoser † | Five trackers + borrowed pelvis | Assisted input; pelvis from our reconstruction |
| EgoExplorer body reconstruction | Five trackers | Fixed profile; gloves are fused only after body fitting |
| HaWoR / HandFlow / glove | Video or glove skeleton | Matched held configurations; shared retargeter ( Sec. F.3 ) |
| HACO / pose rule / glove | RGB, hand pose, or touch | Same scored frames and regions; exclusions disclosed ( Sec. F.3 ) |
Table 14: Inputs and reference controls for the main-text comparisons.
Figure 16: Full-body action guides for the Evaluation Subset (B01–B10): static holds followed by multi-phase actions.
Figure 17: Hand action guides for the Evaluation Subset (H01–H10, in order). Generated MANO poses illustrate the instructions and local hand configurations.
| Metric | Without Skills | With Skills |
| Windows flagged for factual errors | 76.07 | 69.55 |
| Windows flagged for unsupported claims | 71.78 | 66.26 |
| Missed model-reference events | 48.97 | 41.92 |
Table 15: Professional-skill document ablation with the same annotation model. Automated judge results on unrevised drafts (%; lower is better).
Figure 19: Humanoid retargeting of a recorded episode. Green: reconstructed human motion; blue: G1 kinematic retargeting; gold: physics-based whole-body control in simulation. The trajectories are offset for visualization.
Figure 20: Qualitative reconstruction comparison on the Evaluation Subset. Each row shows the same instant for the fitted reference (grey), EgoExplorer (orange), and baselines (blue), rendered from a shared viewpoint.
Figure 21: Body and touch over time, ordered left to right. Rows: reagent weighing, oven retrieval, reaction setup, extraction, rotary evaporation, column chromatography, bookshelf organization, sandwich preparation, clothes folding, and cardboard-box folding. Insets show both hands’ synchronized touch; gray-to-orange denotes baseline-relative increments.
| Approach / example | Sensing arrangement | Deployment requirement |
| Multi-view RGB | External RGB cameras | Overlapping views, synchronization, camera calibration |
| HOT3D | Optical markers on hands and objects | Calibrated infrared-camera volume |
| Nymeria | Inertial body suit and Aria devices | Wearable sensors; no external camera array |
| ParaHome | IMU suit, gloves, and 70 RGB cameras | Instrumented camera volume |
| EgoExplorer | Five inside-out trackers, ego camera, gloves | Body–sensor calibration; no external array or base stations |
Table 16: Capture arrangements and deployment requirements.
| Hardware / configuration | Quantity | Cost (USD) |
| RGB array: Logitech C920s | 4–6 cameras | 280–420 |
| EgoExplorer body tracking: Ultimate Trackers + dongle | 5 + 1 | 1,184 |
| Rokoko Smartsuit Pro II | 1 suit | 2,295 |
| OptiTrack Flex 13 | 8 cameras | 8,792 |
| Ego camera: HERO13 Black + Media Mod | 1 set | 480 |
| MANUS Metagloves Pro, non-haptic | 1 pair | 6,315 |
Table 17: Reference hardware costs in USD. Camera counts are illustrative; glove options are alternatives.