Humans consistently outperform state-of-the-art AI models in action recognition, particularly in challenging real-world conditions involving low resolution, occlusion, and visual clutter. Understanding the sources of this performance gap is essential for developing more robust and human-aligned models. In this paper, we present a large-scale human-AI comparative study of egocentric action recognition using Minimal Identifiable Recognition Crops (MIRCs), defined as the smallest spatial or spatiotemporal regions sufficient for reliable human recognition. We used our previously introduced, Epic ReduAct, a systematically spatially reduced and temporally scrambled dataset derived from 36 EPIC KITCHENS videos, spanning multiple spatial reduction levels and temporal conditions. Recognition performance is evaluated using over 3,000 human participants and the Side4Video model. Our analysis combines quantitative metrics, Average Reduction Rate and Recognition Gap, with qualitative analyses of spatial (high-, mid-, and low-level visual features) and spatiotemporal factors, including a categorisation of actions into Low Temporal Actions (LTA) and High Temporal Actions (HTA). Results show that human performance exhibits sharp declines when transitioning from MIRCs to subMIRCs, reflecting a strong reliance on sparse, semantically critical cues such as hand-object interactions. In contrast, the model degrades more gradually and often relies on contextual and mid- to low-level features, sometimes even exhibiting increased confidence under spatial reduction. Temporally, humans remain robust to scrambling when key spatial cues are preserved, whereas the model often shows insensitivity to temporal disruption, revealing class-dependent temporal sensitivities.
Figures & tables
Figure 1: Research pipeline highlighting different processes, including data preparation and classification by human and AI classifiers and the final analysis stage.
Table 1: Summary of the Epic-ReduAct dataset. Values in parentheses indicate spatiotemporal unrecognisable quadrants.
Figure 2: Illustration of the reduction process. The ground-truth action label for the video is close . Panel A undergoes three successive levels of spatial reduction, resulting in Quadrant B, which is correctly recognised by 65% of participants. Quadrant B serves as the parent quadrant for four spatially reduced child quadrants, Upper-Left, Bottom-Left, Upper-Right, and Bottom-Right (C–F), each recognised by fewer than 50% of participants, thereby classifying them as sub-MIRCs and Quadrant B as the corresponding MIRC. In the spatiotemporal branch, the MIRC video is temporally scrambled, producing an additional child video (G), a spatiotemporal sub-MIRC, with a recognition rate of 40%.
Figure 3: The AI classifier (see Figure 1 ) is described in greater detail, consisting of a video feature encoder augmented by a lightweight spatiotemporal side network integrated with a frozen, pre-trained vision backbone.
Figure 4: The human classifier (see Figure 1 ) is presented in more detail, comprising two stages: classification and response cleaning.
Figure 5: Example segmentations illustrating the Active Hand ( Figure 5(a) ), Active Object ( Figure 5(b) ), and Contextual Objects ( Figure 5(c) ) in a video depicting the action labelled as “ hang gloves ”.
Figure 6: Computation of Object Surface Area Features ( Figure 6(a) ) and Mid-level Feature Maps ( Figure 6(c) )
Figure 7: Class-wise recognition gap for the AI model relative to a human-calibrated threshold. The recognition gap is computed for each MIRC–sub-MIRC pair using class-specific acceptance thresholds derived from human recognition performance. The top row shows results for spatial reductions, while the bottom row correspond to spatiotemporal scramblings, denoted with S and ST, respectively. In the spatial setting, ( Figures 7(a) , 7(b) and 7(c) ) present three representative classes and sample 31 of the put class is highlighted by the red box and analysed in detail in Section 4 (see Figure 8 ). For the spatiotemporal setting, ( Figures 7(d) , 7(e) and 7(f) ) present the same classes as in the spatial case and sample 16 of the wash class is highlighted by the red box for further analysis in Figure 16 .
Classes
Classifier
Metric
close
cut
hang
insert
open
peel
pour
put
remove
serve
take
turn-off
turn-on
wash
Human
RG
+42.92
+36.59
+40.52
+40.00
+38.07
+38.20
+32.21
+35.49
+31.36
+42.88
+39.03
+38.75
+31.11
+31.56
Std.
15.25
14.46
16.54
15.61
14.35
16.70
14.78
15.92
13.13
18.24
14.75
19.53
15.96
17.33
AI
RG
-2.30
+1.21
+0.34
+0.36
+5.40
+0.04
+0.06
-1.05
+5.71
0.00
+15.11
+5.36
+19.43
+3.12
Std.
10.15
2.92
1.17
1.58
10.09
0.04
0.76
8.46
4.67
0.01
12.89
6.96
11.78
6.98
Table 2: Spatial recognition gaps (RG) and standard deviations (Std.) for human and AI classifiers, reported per action class. Values are percentages. Positive RG values indicate a reduction in recognition performance under scrambling, whereas negative values indicate an improvement.
Feature
Definition
DKLcolour
Chromatic contrast in the Derrington‑Krauskopf‑Lennie colour space
Table 3: GBVS feature channels and their definitions.
Figure 8: Sample illustration of spatial feature loss across the original frame, MIRC, and sub-MIRC levels for a video from the put action class (corresponding to highlighted sample in Figure 7(a) ). The red bounding boxes highlight the spatially reduced child video at the next reduction level. As spatial resolution decreases, background distractors are progressively cropped out, allowing the AI model to recognise the action correctly and thereby reducing the recognition gap between levels.
Figure 9: Frequency distribution of recognition gaps between AI model and humans.
Figure 10: Average reduction rate comparison between the AI model and humans. Figure 10(a) shows the distribution of parent–child pairs across all reduction levels, binned at intervals of 0.1 in reduction rate. Figure 10(b) shows the average reduction rate as a function of reduction level for all parent–child quadrant pairs. Figure 10(c) shows the same comparison, restricted to MIRC/sub-MIRC pairs, with the number of pairs visualised for each reduction level.
Figure 11: Feature drop analysis. Quantifying the mid- to high-level features that lead to recognition failure and recovery. Figure 11(a) and Figure 11(b) show the mean feature change ( Δ ) in terms of feature visibility for Failure (Correct → Incorrect) and Recovery (Incorrect → Correct) transitions in human observers and the AI model. Note that human observers did not show any recovery transitions.
Figure 12: Correlation matrix for AI failure cases, illustrating the strength of feature correlations across transitions in failure cases. (Correct → Incorrect).
Figure 13: Example of a failure case under spatial reduction in a video in which the model produces an incorrect prediction after the first level of spatial reduction due to the complete occlusion of the Contextual Objects , while correctly recognising the action when the Contextual Objects and mid-level features remains visible.
Figure 14: Correlation matrix for AI recovery cases, illustrating the strength of feature correlations across transitions in recovery cases (Incorrect → Correct).
Figure 15: Performance recovery via spatial reduction for a video with substantial Active Hand and Active Object suppression at Level 2 ( Δ=−96% compared to the original video). Preserving Contextual Objects and other mid-level features enables correct verb (action) recognition ( put ) after the initial misclassification.
Figure 16: An example of temporal scrambling applied to a spatial MIRC. Although the same frames are retained, their temporal ordering differs between Figure 15(b) and Figure 15(c) . As no spatial reduction is applied between the two levels, the observed differences arise solely from changes in temporal structure.
Manipulation Type
Classifier
Min
Max
Std.
Mean
Spatial
AI
− 0.425
+0.315
+0.112
− 0.037
Human
− 0.050
+0.800
+0.159
+0.373
Spatiotemporal
AI
− 0.607
+0.320
+0.128
− 0.038
Human
− 0.250
+0.750
+0.179
+0.247
Table 4: Recognition gap statistics (percentages) for human and AI classifiers under spatial and spatiotemporal reductions. Positive values indicate reduced recognition performance, while negative values indicate improvement.
Classes
Classifier
Metric
close
cut
hang
insert
open
peel
pour
put
remove
serve
take
turn-off
turn-on
wash
Human
RG
+32.35
+15.00
+35.63
+24.44
+23.56
+31.25
+19.05
+24.17
+20.00
+16.88
+26.69
+35.82
+15.00
+14.41
Std.
16.40
12.91
19.25
8.08
16.37
16.94
15.27
17.00
15.53
11.67
16.26
21.51
18.81
16.91
AI
RG
+3.58
+0.03
-1.52
-0.02
+14.44
+0.08
+0.81
+10.10
-2.51
0.00
+15.51
+5.66
+18.76
-7.52
Std.
10.02
0.88
2.61
2.19
14.53
0.03
0.36
7.23
9.26
0.01
13.90
8.08
19.49
10.17
Table 5: Spatiotemporal Recognition Gaps (RG) and standard deviations (Std) for both classifiers, split by class labels for spatiotemporal MIRC-subMIRC videos. Values are percentages. Positive RG values indicate a reduction in recognition performance under scrambling, whereas negative values indicate an improvement.
Figure 17: Frequency distribution of the recognition gap between humans and the AI model under spatiotemporal reductions.
Figure 18: Average reduction rate as a function of reduction level for MIRCs and spatiotemporal sub-MIRC pairs, along with the number of pairs in each level of reduction.
Table 6: Action categories grouped into High Temporal Actions (HTA) and Low Temporal Actions (LTA). The table reports the total number of pairs and the proportion of samples in which recognition improved after temporal scrambling for both human observers and the AI model.
Figure 19: Mean difference of Humans and AI Model prediction confidence between each MIRC and its spatiotemporal sub-MIRC.
Fine-grained action recognition in egocentric video is challenging for Vision-Language Models (VLMs): actions often differ only in small visual cues, and a single model tends to be biased toward a subset of these cues. We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework in which (i) a VLM orchestrator chunks the video and proposes a top-k candidate label list per segment, (ii) an ensemble of heterogeneous VLM specialists, drawn from different open model families, engages in a structured deliberation that includes a peer-consultation round of questions, and (iii) agent rankings are aggregated with a Borda count and the orchestrator re-ranks its own prediction in light of the specialists' evidence. The entire pipeline runs locally with no fine-tuning. Experiments show that our method positively improves zero-shot action recognition performance over the baseline, highlighting the influence of a heterogeneous deliberation step, showing that the gain stems from decorrelated model priors rather than from additional compute.
Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when first-person body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent's perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection.
Jihyeok Jung, Jeewu Lee, Sanghyeop Kim +2
KAIST AI · Sogang University · Ministry of Science and ICT
The EPIC-KITCHENS-100 Action Detection challenge evaluates whether a model can localize the start and end of each action in long untrimmed egocentric videos and assign the corresponding verb--noun action label. In this report, we formulate our submission as EgoAction (Egocentric Action Composition with Reliability-Aware Temporal Fusion), a unified decoupled detection and fusion pipeline. The pipeline uses EPIC-finetuned VideoMAE-L features, trains separate noun and verb temporal detectors with causal temporal modeling, composes action hypotheses from top noun--verb pairs, and introduces a confidence-adaptive boundary fusion rule at post-processing time. The key observation is that verb and noun streams often fail differently: verb scores are sensitive to motion transitions, whereas noun scores are sensitive to hand-object visibility and object clutter. A fixed arithmetic mean of their predicted boundaries can therefore amplify localization errors when one stream degenerates. We replace this hard-coded mean with Dynamic Weighted Fusion (DWF), which normalizes the maximum noun and verb classification confidences into proposal-wise boundary weights and linearly combines the two intervals. This lightweight tensor-only operator shifts boundary authority toward the more reliable stream while preserving the decoupled action scoring mechanism. Together with sliding-window inference, top-K noun--verb action composition, and class-wise Soft-NMS, EgoAction provides a compact and reproducible system for egocentric temporal action detection.
Zhiheng Fu, Zixu Li, Zhiwei Chen +4
1Shandong University · 2Harbin Institute of Technology (Shenzhen)