Wearable action anticipation systems must remain reliable despite missing frames, masking, and sensor noise, yet existing egocentric anticipation methods largely assume clean observations. We identify two complementary failure modes under temporal corruption: unreliable temporal evidence during encoding and implausible, low-support verb-noun compositions during decoding. We address them with a lightweight framework combining Temporal Reliability Suppression (TRS) and Robust Verb-Noun Graph (RVG) decoding. TRS predicts a per-frame suppression score from the projected input embedding and uses it as a learned key-side attention penalty at every encoder block and to derive reliability-weighted temporal pooling. RVG re-ranks verb-noun pairs using a PMI-based compatibility graph constructed from training labels. Under corruption-augmented training, TRS+CA+RVG reaches 29.1% average corrupted accuracy and 88.2% relative robustness across six corruptions, including three mechanisms absent during training, while reducing rare verb-noun predictions from 15.4% to 1.1%. Multi-seed and diagnostic experiments show that TRS responds to synthetic masking; shuffled-graph and frequency-only controls further indicate that RVG gains depend on genuine pairwise compatibility rather than marginal-frequency effects alone.
Figures & tables
Figure 1: Overview and complementarity of TRS+RVG. Temporal corruption affects egocentric anticipation at two levels: TRS suppresses unreliable temporal evidence during encoding and pooling, whereas RVG corrects low-support verb–noun compositions at decision time.
Method
Cl
Mask
Noise
Comb
Drop
Jitter
Freeze
AvgC
RR
Clean baseline
33.6
9.4
23.4
6.8
24.4
27.0
32.0
20.5
61.0
FrameDrop
32.7
9.6
23.4
7.6
23.7
26.1
30.5
20.1
61.6
FrameDrop + RVG
36.5
12.7
30.6
11.3
28.5
29.4
35.2
24.6
67.5
Corruption-Augmented (CA)
28.4
22.1
19.7
20.7
22.0
25.5
25.5
22.6
79.6
CA + RVG
31.3
25.2
25.1
25.5
24.4
28.1
29.7
26.3
84.0
TRS + CA
29.5
23.6
22.9
24.0
24.3
26.9
27.5
24.9
84.4
Table 1: EGTEA top-1 action accuracy (%), high severity. Drop, Jitter, and Freeze use mechanisms absent from training. AvgC averages the six corruptions; RR =100× AvgC/Cl. FrameDrop is a temporal-augmentation baseline [ 26 ] . Control rows were run on Mask, Noise, and Combined only. Bold : best per column among the full methods.
Figure 2: EGTEA robustness profile (top-1 action accuracy %; red: best per column). TRS+CA+RVG leads on Mask, Combined, and Jitter; FrameDrop+RVG leads on Noise, Drop, and Freeze.
Method
Cl
Mask
Noise
Comb
Avg
Clean baseline
8.9
0.5
5.8
0.5
2.3
Corruption-Augmented (CA)
8.2
6.9
6.2
6.9
6.7
CA + RVG
9.4
9.0
8.3
8.7
8.7
TRS + CA
9.4
8.4
6.3
6.7
7.1
TRS + CA + RVG
10.2
8.1
8.5
8.0
8.2
TRS + CA + Shuffled
9.4
7.7
5.9
6.7
6.8
Table 2: EK100 top-1 action accuracy (%). Avg averages Mask, Noise, and Combined. λ=2 is transferred from EGTEA without re-tuning. Bold : best per column.
Figure 3: TRS suppression scores on an EGTEA clip. Scores rise on masked frames t3 – t5 and remain low elsewhere.
Figure 4: Qualitative RVG correction: graph re-scoring replaces the low-support open knife prediction with cut knife .
Per-stream
By GT rarity
Method
Verb
Noun
Action
All
Seen-GT
Rare-GT
TRS + CA
36.6
37.8
22.5
22.2
22.8
0 6.4
TRS + CA + RVG
37.9
37.1
27.4
28.3
28.9
15.4
Table 4: Left: verb, noun, and joint accuracy under EGTEA Combined corruption. Right: accuracy on seen versus rare ground-truth pairs (rare = training count <30 , n=78 of 2022).
Action-chunked visuomotor policies predict overlapping trajectories, so every executed action is covered by several predictions. Temporal ensembling smooths execution by combining these predictions with an exponentially weighted mean. One corrupted prediction can move the aggregate without bound: its breakdown point is 0. We use adversarial corruption to stress this deployed aggregator and to compare two kinds of guarantee. A metric guarantee bounds the response to a perturbation of a given size. A combinatorial guarantee instead bounds the damage when at most q of the M candidates covering a timestep are corrupted, whatever their size. Encoder adversarial fine-tuning recovers 44% of the loss under the published patch attack, but only 7.3% after the attacker's step size is increased. By contrast, the coordinate-wise median of the same candidate set keeps its recovered fraction flat as attack optimisation increases. Median temporal ensembling costs one line and requires no retraining. Across 25 (configuration, corruption-level) combinations it is never worse than the mean and is significantly better in 15. It also transfers to a second policy class, and it recovers performance under a failure with no attacker in the loop at all: camera frames that arrive blank. Its effect on clean data is configuration-dependent, from -0.04 to +0.07. We also give the boundary: corruption that shifts every covering prediction by the same amount is invisible to this whole family of statistics, and no equivariant aggregator can remove it.
We propose JFAA, a JEPA-based Future Action Anticipation method for the EPIC-KITCHENS-100 (EK-100) Action Anticipation task. Inspired by the representation learning and future prediction ability of V-JEPA 2.1, JFAA uses a frozen encoder and predictor to extract observed context features and near-future latent tokens. A lightweight attentive probe is then trained to predict verb, noun, and action logits with separate task queries. To improve robustness, we further build a field-aware ensemble over selected epoch-level predictions, allowing each output field to benefit from its most reliable candidates. Experimental results on the official challenge server show that JFAA achieves first place in the EgoVis 2026 EK-100 Action Anticipation Challenge. Our code will be released at https://github.com/CorrineQiu/JFAA.
Qiaohui Chu, Haoyu Zhang, Yisen Feng +4
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory · Shandong Jianzhu University
This report presents TAP-JEPA, our runner-up submission to the EPIC-KITCHENS-100 (EK-100) Action Anticipation Challenge at EgoVis 2026. The task is to anticipate the next verb, noun, and verb-noun action from an egocentric clip that ends before the target action begins. Instead of fine-tuning a large video backbone, TAP-JEPA builds a compact anticipation model on frozen V-JEPA 2.1 features: a ViT-G/384 encoder extracts visible pre-action tokens, the pre-trained latent predictor estimates near-future tokens from the observed context, and both token groups are fused by attentive probes with task-specific queries for verbs, nouns, and action pairs. For the final submission, we expand supervised training with the official training split and most of the validation split, reserving a small subset for sanity checks and qualitative inspection, and adopt a two-stage score fusion that first averages eight independently initialized probe replicas within each epoch and then merges candidates from epochs 12-20 with field-dependent weights. On the official open-testing leaderboard, our sunshinesky entry achieves 27.91 percent overall action Mean Top-5 Recall (MT5R), ranking second and only 0.04 percentage points behind the top score.