Learning from human data offers a scalable path toward embodied intelligence, yet it requires models to accurately understand fine-grained human actions from egocentric observations. However, existing benchmarks remain limited in action atomicity, temporal precision, and hand-action association. We introduce EgoActionBench, a benchmark that represents continuous human manipulation as temporally localized atomic actions associated with the left hand, right hand, or both hands. EgoActionBench contains 1,407 episodes with 41,132 human-refined action annotations. To evaluate such dense predictions, we propose an evaluation protocol that retrieves temporally eligible candidates, selects a prediction by content agreement, and scores semantic fidelity and temporal overlap alongside action precision and recall, jointly capturing what happens, when it happens, and which hand performs it. We benchmark 20 baseline models and systems on EgoActionBench and further adapt two backbones, Qwen3-VL-4B and LLaVA-OneVision-2-8B, through supervised fine-tuning followed by reinforcement learning on automatically annotated training episodes. EgoAction improves EgoActionScore by 15.9 and 23.2 points on the two backbones, and both adapted models outperform all open-source baselines. Our experiments reveal substantial limitations of current models in action atomicity, temporal precision, and hand-action association, highlighting the gap between general video understanding and the action understanding required for embodied intelligence.
Figures & tables
Figure 1: Overview of EgoAction. Continuous egocentric video is represented as temporally localized atomic actions on left-hand, right-hand, and both-hands tracks (top). The bottom-left example highlights the action verb, manipulated object, and tool. Adjacent frames illustrate the start and end boundaries of a joint action (bottom right).
Dataset / Benchmark
Scene
Target unit
Ego.
Temp. loc.
Caption gen.
Hand-specific captions
Independent hand timelines
Structured caption scoring
ActivityNet Captions ( Krishna et al., 2017 )
Real-world
Events
✗
✓
✓
✗
✗
✗
EPIC-KITCHENS-100 ( Damen et al., 2022 )
Kitchen
Actions
✓
✓
✗
✗
✗
✗
Ego4D ( Grauman et al., 2022 )
Real-world
Narrated events
✓
✗
✗
✗
✗
✗
Ego-Exo4D ( Grauman et al., 2024 )
Real-world
Atomic descriptions
✓
✗
✗
✓
✗
✗
EgoYC2 ( Ohkawa et al., 2025 )
Kitchen
Procedural steps
✓
✓
✓
✗
✗
✗
HD-EPIC ( Perrett et al., 2025 )
Kitchen
Actions
✓
✗
✗
✓
✗
✗
Table 1: Comparison with related datasets and benchmarks. Actions may group several operations under one label; atomic actions are delimited by meaningful changes in the operation, manipulated object, contact relation, or dominant motion. Temporal localization and caption generation denote evaluated tasks. Hand-specific captions identify acting hands; independent hand timelines assign separate action intervals to each hand. Structured scoring evaluates semantic fields or facts. Checks denote documented protocol features. Rows use Ego4D narrations and Ego-Exo4D atomic descriptions. Percentages denote human egocentric episodes.
Source
Clips
Hours
Actions
Left
Right
Both
Ego4D
451
3.130
16,774
6,825
8,349
1,600
EgoLife
485
2.676
13,306
5,895
6,109
1,302
EPIC-KITCHENS-100
471
2.176
11,052
4,599
5,300
1,153
Total
1,407
7.982
41,132
17,319
19,758
4,055
Table 2: Dataset Statistics of EgoActionBench. Video clips, total video duration, and human-refined action annotations by source. Left, Right, and Both report annotation counts for the corresponding hand tracks.
Figure 2: Dataset Composition and Action Statistics. (a) Activity taxonomy with six groups and 17 scenes. (b) Source composition within each activity group. (c) Episode counts by scene. (d) Action-duration distribution on a logarithmic time axis, with a median of 0.60 s. (e) Caption vocabulary. (f) The 30 most frequent primary verbs, with counts on a logarithmic scale. Statistics cover 1,407 episodes and 41,132 action captions; sector angles do not encode counts.
Figure 3: Human-in-the-Loop Construction of EgoActionBench. Hand-aware preprocessing and sliding-window captioning generate draft per-hand annotations. Human annotation, agentic quality control, and final human verification refine these drafts against the original video. Episodes needing correction return for human annotation and QC.
Figure 4: EgoAction evaluation pipeline. For each ground-truth action, temporal overlap and hand-track compatibility determine the eligible candidates. Positive field-level semantic agreement selects one prediction, whose original caption, interval, and hand label are used to compute Semantic and temporal IoU. These scores are averaged separately over ground-truth actions with completed evaluations.
Figure 5: Why F1 score matters. We complement tIoU, which measures interval overlap, with one-to-one F1 to explicitly penalize errors in event coverage and segmentation granularity. The four schematic examples illustrate (1) action insertion, (2) action omission, (3) under-segmentation, and (4) over-segmentation.
Figure 6: Metric sensitivity to candidate overlap thresholds and frame sampling. Panels (a) to (d) show EgoActionScore, Semantic, Temporal, and Hand as the candidate overlap threshold varies, using predictions generated at 2 fps. The shaded bands mark the default threshold of τsem=0.5 . Panels (e) and (f) show EgoActionScore under fixed-rate and fixed-count sampling, with τsem=0.5 . Panel (f) uses a logarithmic frame-count axis. All scores range from 0 to 100; vertical axis ranges vary across panels.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
SFT
GRPO
Optimizer
AdamW
AdamW
Learning rate
1×10−5
5×10−7
Learning rate scheduler
Cosine
Cosine
Learning rate warmup
5% of training updates
10 updates
Training batch size
32
24
Gradient norm clipping
1.0
1.0
Appendix
Table 7: Key optimization and sampling hyperparameters for SFT and GRPO. Values apply to both backbones. Batch size counts supervised examples for SFT and generated responses for GRPO. Each GRPO batch contains six videos with G=4 responses per video. A dash indicates that the parameter is not applicable.
Figure 7: Agreement with human reviewers on the 400-pair calibration set. Percentage agreement (left) and Cohen’s κ (right) on 1,377 semantic judgments, based on whether each requirement is satisfied. Qwen denotes Qwen3-32B; R1 and R2 denote the two reviewers.
Figure 8: Model scores under two semantic judges. Blue circles and purple diamonds show Qwen3-32B and GPT-5.6 Sol judgments of fixed predictions on the same 99 clips and 2,506 ground-truth actions. Both panels share the same axis scale.
Source
Episodes
Frame rate
Video resolution
Ego4D
451
30 fps
480×360 , 640×360 , 960×540 , 960×720 , or 960×1280
EgoLife
485
20 fps
960×960
EPIC-KITCHENS-100
471
50 fps
1920×1080
Appendix
Table 8: Video specifications by source. Frame rates and resolutions of the benchmark episodes.
Activity group
Episodes
Action captions
Kitchen and dining
771
20,205
Household and daily life
230
6,783
Construction and repair
185
6,832
Gardening and outdoor
28
894
Crafts and laboratory
103
3,812
Desk, devices, and leisure
90
2,606
Appendix
Table 9: Activity coverage of EgoActionBench. Episode and action-caption counts across the six activity groups.
Figure 10: Inference prompt for dense per-hand action captioning. The placeholder {duration_sec} is replaced with the duration of each clip.
Figure 11: Template for sliding-window caption generation. The inputs include a video summary, recent captions, and the current output window. The model returns captions for the Left, Right, and Both tracks.
Figure 12: Instructions for temporal merging. Adjacent segments are reviewed within each hand track using visual evidence of hand-object contact, hand-motion phases, and object-state changes.
Figure 13: Qualitative Comparison of Evaluation Metrics. We compare SODA-c ( Fujita et al., 2020 ) , SodaM ( Yao et al., 2026 ) , and our evaluation protocol. The examples illustrate how our protocol preserves action-level content matches while evaluating temporal overlap on the original predictions.
Figure 14: Ego4D example. Selected frames and complete ground-truth annotations for handling a writing board.
Figure 15: EgoLife example. Selected frames and complete ground-truth annotations for handling a vase.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.