cs.CVMar 9, 2026
SaveHuman-AI Divergence in Ego-centric Action Recognition under Spatial and Spatiotemporal Manipulations
Organizations: University of Surrey, Guildford, UK · Newcastle University, Newcastle upon Tyne, UK
Abstract
Humans consistently outperform state-of-the-art AI models in action recognition, particularly in challenging real-world conditions involving low resolution, occlusion, and visual clutter. Understanding the sources of this performance gap is essential for developing more robust and human-aligned models. In this paper, we present a large-scale human-AI comparative study of egocentric action recognition using Minimal Identifiable Recognition Crops (MIRCs), defined as the smallest spatial or spatiotemporal regions sufficient for reliable human recognition. We used our previously introduced, Epic ReduAct, a systematically spatially reduced and temporally scrambled dataset derived from 36 EPIC KITCHENS videos, spanning multiple spatial reduction levels and temporal conditions. Recognition performance is evaluated using over 3,000 human participants and the Side4Video model. Our analysis combines quantitative metrics, Average Reduction Rate and Recognition Gap, with qualitative analyses of spatial (high-, mid-, and low-level visual features) and spatiotemporal factors, including a categorisation of actions into Low Temporal Actions (LTA) and High Temporal Actions (HTA). Results show that human performance exhibits sharp declines when transitioning from MIRCs to subMIRCs, reflecting a strong reliance on sparse, semantically critical cues such as hand-object interactions. In contrast, the model degrades more gradually and often relies on contextual and mid- to low-level features, sometimes even exhibiting increased confidence under spatial reduction. Temporally, humans remain robust to scrambling when key spatial cues are preserved, whereas the model often shows insensitivity to temporal disruption, revealing class-dependent temporal sensitivities.
Figures & tables
Figure 1: Research pipeline highlighting different processes, including data preparation and classification by human and AI classifiers and the final analysis stage.
| Category | Videos | Samples | MIRCs | Spatial sub-MIRCs | Spatiotemporal quadrants | Verb Classes |
|---|---|---|---|---|---|---|
| Easy | 18 | 4,503 | 273 | 1,092 | 273 (200) | close, cut, hang, open, pour, put, remove, take, turn-off, turn-on, wash |
| Hard | 18 | 3,173 | 402 | 804 | 201 (145) | close, hang, insert, open, peel, pour, put, remove, serve, take, turn-off, wash |
Table 1: Summary of the Epic-ReduAct dataset. Values in parentheses indicate spatiotemporal unrecognisable quadrants.
Figure 2: Illustration of the reduction process. The ground-truth action label for the video is close . Panel A undergoes three successive levels of spatial reduction, resulting in Quadrant B, which is correctly recognised by 65% of participants. Quadrant B serves as the parent quadrant for four spatially reduced child quadrants, Upper-Left, Bottom-Left, Upper-Right, and Bottom-Right (C–F), each recognised by fewer than 50% of participants, thereby classifying them as sub-MIRCs and Quadrant B as the corresponding MIRC. In the spatiotemporal branch, the MIRC video is temporally scrambled, producing an additional child video (G), a spatiotemporal sub-MIRC, with a recognition rate of 40%.
Figure 3: The AI classifier (see Figure 1 ) is described in greater detail, consisting of a video feature encoder augmented by a lightweight spatiotemporal side network integrated with a frozen, pre-trained vision backbone.
Figure 4: The human classifier (see Figure 1 ) is presented in more detail, comprising two stages: classification and response cleaning.
Figure 5: Example segmentations illustrating the Active Hand ( Figure 5(a) ), Active Object ( Figure 5(b) ), and Contextual Objects ( Figure 5(c) ) in a video depicting the action labelled as “ hang gloves ”.
Figure 6: Computation of Object Surface Area Features ( Figure 6(a) ) and Mid-level Feature Maps ( Figure 6(c) )
Figure 7: Class-wise recognition gap for the AI model relative to a human-calibrated threshold. The recognition gap is computed for each MIRC–sub-MIRC pair using class-specific acceptance thresholds derived from human recognition performance. The top row shows results for spatial reductions, while the bottom row correspond to spatiotemporal scramblings, denoted with S and ST, respectively. In the spatial setting, ( Figures 7(a) , 7(b) and 7(c) ) present three representative classes and sample 31 of the put class is highlighted by the red box and analysed in detail in Section 4 (see Figure 8 ). For the spatiotemporal setting, ( Figures 7(d) , 7(e) and 7(f) ) present the same classes as in the spatial case and sample 16 of the wash class is highlighted by the red box for further analysis in Figure 16 .
| Classes | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classifier | Metric | close | cut | hang | insert | open | peel | pour | put | remove | serve | take | turn-off | turn-on | wash |
| Human | RG | +42.92 | +36.59 | +40.52 | +40.00 | +38.07 | +38.20 | +32.21 | +35.49 | +31.36 | +42.88 | +39.03 | +38.75 | +31.11 | +31.56 |
| Std. | 15.25 | 14.46 | 16.54 | 15.61 | 14.35 | 16.70 | 14.78 | 15.92 | 13.13 | 18.24 | 14.75 | 19.53 | 15.96 | 17.33 | |
| AI | RG | -2.30 | +1.21 | +0.34 | +0.36 | +5.40 | +0.04 | +0.06 | -1.05 | +5.71 | 0.00 | +15.11 | +5.36 | +19.43 | +3.12 |
| Std. | 10.15 | 2.92 | 1.17 | 1.58 | 10.09 | 0.04 | 0.76 | 8.46 | 4.67 | 0.01 | 12.89 | 6.96 | 11.78 | 6.98 | |
Table 2: Spatial recognition gaps (RG) and standard deviations (Std.) for human and AI classifiers, reported per action class. Values are percentages. Positive RG values indicate a reduction in recognition performance under scrambling, whereas negative values indicate an improvement.
| Feature | Definition |
|---|---|
| DKLcolour | Chromatic contrast in the Derrington‑Krauskopf‑Lennie colour space |
| Intensity | Luminance (brightness) variations |
| Orientation | Local edge orientation energy |
| Colour | Double‑opponent Red–Green / Blue–Yellow contrasts |
| Flicker | Frame-to-frame luminance change (temporal contrast) |
| Contrast | Local brightness variability |
Table 3: GBVS feature channels and their definitions.
Figure 8: Sample illustration of spatial feature loss across the original frame, MIRC, and sub-MIRC levels for a video from the put action class (corresponding to highlighted sample in Figure 7(a) ). The red bounding boxes highlight the spatially reduced child video at the next reduction level. As spatial resolution decreases, background distractors are progressively cropped out, allowing the AI model to recognise the action correctly and thereby reducing the recognition gap between levels.
Figure 9: Frequency distribution of recognition gaps between AI model and humans.
Figure 10: Average reduction rate comparison between the AI model and humans. Figure 10(a) shows the distribution of parent–child pairs across all reduction levels, binned at intervals of 0.1 in reduction rate. Figure 10(b) shows the average reduction rate as a function of reduction level for all parent–child quadrant pairs. Figure 10(c) shows the same comparison, restricted to MIRC/sub-MIRC pairs, with the number of pairs visualised for each reduction level.
Figure 11: Feature drop analysis. Quantifying the mid- to high-level features that lead to recognition failure and recovery. Figure 11(a) and Figure 11(b) show the mean feature change ( ) in terms of feature visibility for Failure (Correct Incorrect) and Recovery (Incorrect Correct) transitions in human observers and the AI model. Note that human observers did not show any recovery transitions.
Figure 12: Correlation matrix for AI failure cases, illustrating the strength of feature correlations across transitions in failure cases. (Correct Incorrect).
Figure 13: Example of a failure case under spatial reduction in a video in which the model produces an incorrect prediction after the first level of spatial reduction due to the complete occlusion of the Contextual Objects , while correctly recognising the action when the Contextual Objects and mid-level features remains visible.
Figure 14: Correlation matrix for AI recovery cases, illustrating the strength of feature correlations across transitions in recovery cases (Incorrect Correct).
Figure 15: Performance recovery via spatial reduction for a video with substantial Active Hand and Active Object suppression at Level 2 ( compared to the original video). Preserving Contextual Objects and other mid-level features enables correct verb (action) recognition ( put ) after the initial misclassification.
Figure 16: An example of temporal scrambling applied to a spatial MIRC. Although the same frames are retained, their temporal ordering differs between Figure 15(b) and Figure 15(c) . As no spatial reduction is applied between the two levels, the observed differences arise solely from changes in temporal structure.
| Manipulation Type | Classifier | Min | Max | Std. | Mean |
|---|---|---|---|---|---|
| Spatial | AI | 0.425 | +0.315 | +0.112 | 0.037 |
| Human | 0.050 | +0.800 | +0.159 | +0.373 | |
| Spatiotemporal | AI | 0.607 | +0.320 | +0.128 | 0.038 |
| Human | 0.250 | +0.750 | +0.179 | +0.247 |
Table 4: Recognition gap statistics (percentages) for human and AI classifiers under spatial and spatiotemporal reductions. Positive values indicate reduced recognition performance, while negative values indicate improvement.
| Classes | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classifier | Metric | close | cut | hang | insert | open | peel | pour | put | remove | serve | take | turn-off | turn-on | wash |
| Human | RG | +32.35 | +15.00 | +35.63 | +24.44 | +23.56 | +31.25 | +19.05 | +24.17 | +20.00 | +16.88 | +26.69 | +35.82 | +15.00 | +14.41 |
| Std. | 16.40 | 12.91 | 19.25 | 8.08 | 16.37 | 16.94 | 15.27 | 17.00 | 15.53 | 11.67 | 16.26 | 21.51 | 18.81 | 16.91 | |
| AI | RG | +3.58 | +0.03 | -1.52 | -0.02 | +14.44 | +0.08 | +0.81 | +10.10 | -2.51 | 0.00 | +15.51 | +5.66 | +18.76 | -7.52 |
| Std. | 10.02 | 0.88 | 2.61 | 2.19 | 14.53 | 0.03 | 0.36 | 7.23 | 9.26 | 0.01 | 13.90 | 8.08 | 19.49 | 10.17 | |
Table 5: Spatiotemporal Recognition Gaps (RG) and standard deviations (Std) for both classifiers, split by class labels for spatiotemporal MIRC-subMIRC videos. Values are percentages. Positive RG values indicate a reduction in recognition performance under scrambling, whereas negative values indicate an improvement.
Figure 17: Frequency distribution of the recognition gap between humans and the AI model under spatiotemporal reductions.
Figure 18: Average reduction rate as a function of reduction level for MIRCs and spatiotemporal sub-MIRC pairs, along with the number of pairs in each level of reduction.
| Verb Category | Verbs | Num. Pairs | Human Improved ( \text{Conf. \Delta }<0 ) | AI Improved ( \text{Conf. \Delta}<0 ) | ||
|---|---|---|---|---|---|---|
| Count | Percentage | Count | Percentage | |||
| HTA | opening, closing, turning, putting, taking, pouring, filling/serving, hanging, removing | 406 | 21 | 5.17% | 106 | 26.11% |
| LTA | washing, cutting, peeling | 68 | 8 | 11.76% | 41 | 60.29% |
Table 6: Action categories grouped into High Temporal Actions (HTA) and Low Temporal Actions (LTA). The table reports the total number of pairs and the proportion of samples in which recognition improved after temporal scrambling for both human observers and the AI model.
Figure 19: Mean difference of Humans and AI Model prediction confidence between each MIRC and its spatiotemporal sub-MIRC.