EgoExo proficiency estimation aims to assess action quality by integrating fine-grained motion cues from egocentric (1st-person) views with spatial context from multiple exocentric (3rd-person) views. Simply adding more exocentric views degrades EgoExo performance, as redundant or noisy perspectives dilute useful motion cues. Our analysis identifies two key causes: (1) Multiview redundancy - From the data perspective, certain views provide limited or noisy information, diluting discriminative cues; (2) Overfitting - From the feature perspective, conventional fusion increases representational complexity, causing the model to memorise view-specific patterns rather than learn generalisable representations. To address these issues, we propose two complementary modules: AdaMVS, which adaptively identifies and fuses the most informative view tokens under weak supervision from the data perspective, and VIB-GB, which combines Gradient Blending and Variational Information Bottleneck regularisation from the feature perspective to compress redundant signals and suppress overfitting during training. Experiments on EgoExo-4D and EgoExo-Fitness demonstrate that our method learns both which view to look at and how to fuse them, achieving new state-of-the-art results. Our source code is available at https://github.com/dx199771/AdaMVS
Figures & tables
Figure 1: Multi-Exo-View Fusion Challenges. 1(a) demonstrates that redundant views (e.g., Views 2 and 3 account for just 17.2% of the total attention weight) contribute only marginal information; 1(b) shows that simply adding more views can degrade overall performance. In contrast, our method adaptively identifies and fuses the most informative views to enhance robustness and accuracy.
Figure 2: Overview of our proposed network. (Top) The egocentric branch extracts and decodes features through a dedicated pathway. (Bottom) The exocentric branch uses the Score Predictor and Exo-AdaMVS to compute scores, selecting key views and removing redundancy. The Ego–Exo AdaMVS fuses streams, while the VIB-GB module regularises the latent space to reduce overfitting and enhance generalisation. Solid arrows denote gradient flow, and dash-dotted arrows indicate mixed paths involving gradient and non-gradient operations used by VIB-GB.
Figure 3: Overview of the proposed VIB-GB regulariser. Ego and exo features are regularised through separate VIB branches, and Gradient Blending Regularisation (GB) updates gradient weights online using both training and validation losses to balance learning and suppress overfitting. The KL term constrains the latent representation, while Δego/exo guides proficiency learning.
Method
Pretrain
Ego-Exo4D [ 13 ]
EgoExo-Fitness [ 24 ]
Ego
Exos
Ego+Exos
Ego
Exos
Ego+Exos
Random
-
26.4
26.4
26.4
26.2
26.2
26.2
Majority-class
-
32.3
32.3
32.3
33.3
33.3
33.3
TimeSformer [ 13 ]
-
40.6
39.0
39.9
32.1
32.1
32.1
TimeSformer
EgoVLPv2 [ 39 ]
46.7
37.0
37.1
39.3
35.7
41.7
TimeSformer
EgoVLP [ 25 ]
44.7
40.5
39.4
41.7
36.9
39.3
Table 1: Proficiency estimation accuracy (%) comparison on the Ego-Exo4D and EgoExo-Fitness datasets in three settings. Bold numbers denote the best performance, and underlined numbers denote the second best.
Method
Ego
Exos
Ego+Exos
GFLOPs
Params (M)
Summation
47.2
44.4
45.2
1.11
23.42
Concat (early)
47.2
45.5
47.2
16.55
344.55
Concat (late)
47.2
47.7
49.0
11.74
23.42
CrossFusion
47.2
47.5
46.8
1.67
4.20
MTV [ 54 ]
45.2
43.7
43.5
1.15
7.37
MVCNN [ 41 ]
48.3
45.2
46.6
1.51
9.97
Table 2: Performance (%) and model complexity comparison on the Ego-Exo4D datasets with different multiview and multi-modal fusion methods. Bold indicates the best performance, and underlined indicates the second best.
Method (K400)
EgoExo4D
EgoExo-Fitness
Baseline
44.1
44.0
+ Feature-Mixup
45.2
40.5
+ Weight Decay (1e-2)
44.6
44.0
+ Weight Decay (1e-3)
44.1
44.0
+ Modality Dropout ( p=0.3 )
46.8
38.1
+ Modality Dropout ( p=0.5 )
47.2
41.7
Table 3: Quantitative comparison against standard overfitting mitigation baselines with varying regularisation configurations.
Table 7Table 8
Figure 4: Analysis of overfitting and generalisation on the EgoExo-4D dataset. 4(a) – 4(b) : Training and validation curves of baseline without and with the proposed VIB-GB module, respectively. 4(c) – 4(d) : Corresponding Overfitting-to-Generalisation Ratio (OGR) curves. Without VIB-GB, the model exhibits strong overfitting ( 4(a) – 4(c) ), while our VIB-GB effectively regularises training dynamics and improves generalisation ( 4(b) – 4(d) ).
Figure 5: Visualisation of the view selection results from the AdaMVS module. Top : success cases where the model selects the top- K most informative exocentric views ( K=2 ) to avoid multiview redundancy — green boxes denote selected views and red dotted boxes denote pruned ones. Bottom : failure cases where all views receive similar scores, leading to suboptimal selection.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Visualisation of the view selection results from the AdaMVS module. Top : success cases where the model selects the top- K most informative exocentric views ( K=2 ) to avoid multiview redundancy — green boxes denote selected views and red dotted boxes denote pruned ones. Bottom : failure cases where all views receive similar scores, leading to suboptimal selection.
Fusion Method
1 Exo View
2 Exo Views (6 Permutations)
3 Exo Views (4 Permutations)
4 Exo Views
Ego + Exo
1
2
3
4
Avg.
SD
12
13
14
23
24
34
Avg.
SD
123
234
134
124
Avg.
SD
1234
Avg.
SD
Crossfusion
44.12
43.68
45.68
46.56
45.01
1.34
46.12
46.12
44.12
47.01
46.34
44.79
45.75
1.08
47.01
46.34
46.78
48.12
47.06
0.76
47.45
46.78
1.34
Early Cat
44.12
43.68
45.68
46.56
45.01
1.34
46.12
47.23
46.12
45.90
45.90
48.12
46.57
0.91
49.67
47.89
47.01
46.12
47.67
1.52
45.45
45.23
1.34
Late Cat
44.12
43.68
45.68
46.56
45.01
1.34
45.45
46.12
46.34
43.46
48.12
44.57
45.68
1.60
46.56
46.12
47.01
47.45
46.79
0.57
47.67
49.00
1.34
Early Sum
44.12
43.68
45.68
46.56
45.01
1.34
43.02
44.79
46.56
44.79
48.78
44.79
45.46
1.98
45.23
45.90
43.02
45.01
44.79
1.24
44.35
45.23
1.34
Ours
43.02
43.68
45.90
46.56
44.79
1.71
43.46
48.12
48.78
45.23
44.79
46.34
46.12
2.04
46.78
45.68
48.34
45.45
46.56
1.32
49.00
50.60
1.71
Appendix
Table 1: Detailed action recognition accuracy (%) across all view configurations. Results are reported for each fusion method under varying numbers of exocentric views (1–4) and ego–exo combinations. ‘Avg.‘ denotes the mean accuracy and ‘SD‘ represents the standard deviation across corresponding view configurations.
Method
Exo
Same Exo × 4
All Exos
Summation
45.7
46.6 ( ↑ )
44.4 ( ↓ )
Early Cat
45.7
43.0( ↓ )
45.5( ↓ )
Late Cat
45.7
43.7( ↓ )
47.7( ↑ )
Cross Fusion
45.7
43.7( ↓ )
47.5( ↑ )
AdaMVS (Ours)
45.9
46.6 ( ↑ )
49.2 ( ↑↑ )
Appendix
Table 2: Ablation and controlled analysis on view redundancy and model overfitting.
Method
view1
+view2
+view3
+view4
ego+exos
Summation
43.9
47.2 1st
44.4 2nd
47.2 1st
45.2
Concat (early)
45.7 2nd
45.9 1st
44.1
44.6
49.0
Concat (late)
43.9 2nd
44.6 1st
43.7
43.7
47.2
Crossfusion
46.1 2nd
43.5
43.9
47.7 1st
46.8
Ours
46.1 2nd
45.9
46.8 1st
45.9
50.6
Appendix
Table 3: Quantitative evaluation across different combinations of the egocentric view paired with individual exocentric views.
Figure 2: Illustration of the Overfitting-to-Generalisation Ratio (OGR) concepts during training.
Figure 3: This module is a Transformer Decoder structure which refines Learnable Queries via Self-Attention before fusing them with Feature Extractor Embeddings via Multi-Head Attention.