Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency

Zhaofeng Shi; Heqian Qiu; Lanxiao Wang; Qingbo Wu; Fanman Meng; Lili Pan; Hongliang Li

cs.CVMar 10, 2026

Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency

Authors: Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Qingbo Wu, Fanman Meng, Lili Pan, Hongliang Li

Organizations: University of Electronic Science and Technology of China, Chengdu, China

Abstract

Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE $^{2}$ A $^{3}$ ) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at https://github.com/ZhaofengSHI/DCPGN.

Figures & tables

Figure 1: Schematic of the newly proposed TE 2 A 3 task.

Figure 2: Overview of our DCPGN. The model is trained using the source-view data. During testing, the ML-PGM assigns each representation to multiple labels, then performs confidence-based reweighting under an entropy priority queue strategy to balance multiple positive classes. Moreover, to complement the visual clue containing various objects, the DCCM incorporates a lightweight narrator to generate descriptions for action progressions with nouns (green) and verbs (purple). We integrate cross-modality clues by constraining their logits for temporally and spatially bridging the view gap. Finally, the prototype, visual, and textual logits are added for target-view anticipation.

Methods	EgoMe-anti				EgoExoLearn
	Exo2Ego		Ego2Exo		Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
Ours without Adaptation	71.94	32.46	64.24	30.07	31.91	34.36	35.28	33.03
Tent [ 63 ]	74.14	35.63	68.50	32.32	34.71	36.22	38.61	35.04
TPT [ 57 ]	74.88	35.97	68.65	32.55	35.29	36.48	38.82	35.14
VITTA [ 34 ]	75.65	35.94	68.68	33.84	35.06	36.44	39.17	35.65

Table 1: Quantitative results on the EgoMe-anti and EgoExoLearn benchmarks under the Exo2Ego and Ego2Exo settings.

$L_{C}$	VC	TC	Conf	ML	EgoMe-anti				EgoExoLearn
					Exo2Ego		Ego2Exo		Exo2Ego		Ego2Exo
					Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
✓	✓	✓	✓	✓	79.03	43.84	72.01	40.10	46.26	42.98	48.48	46.51
	✓	✓	✓	✓	78.67	42.72	71.29	39.20	44.80	42.73	47.97	45.84
		✓	✓	✓	76.92	42.37	69.65	37.99	41.32	42.25	45.73	45.23
	✓		✓	✓	77.56	42.15	70.47	37.66	41.94	41.93	45.88	43.72

Table 2: Ablation results under the Exo2Ego and Ego2Exo benchmarks on the EgoMe-anti and EgoExoLearn benchmarks.

L_{C}

is the dual-clue consistency loss. “VC” and “TC” denote the visual clues and textual clues, respectively. “Conf” represents the confidence-based reweighting, and “ML” denotes the multi-label assignment. “S” means only assign a single class as the pseudo label for the representation.

EgoMe-anti
$K$	Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb
$K$ =1	76.12	39.89	69.55	36.49
$K$ =2	77.89	42.65	70.67	38.97
$K$ =3	79.03	43.84	72.01	40.10
$K$ =4	78.95	43.65	71.43	40.42

Table 3: Analysis of the number of assigned pseudo labels

K

	Baseline	ML-PGM	NG	TCFE
FLOPs (G)	367.55	0.00	0.03	4.06
$\#$ Params (M)	251.18	8.54	2.38	54.04

Table 4: The inference FLOPs and the number of parameters of the baseline model and key components in the proposed method. “NG” denotes the narration generation, and “TCFE” represents the textual clue feature extraction.

Figure 3: Visualization of representations in memory banks and the corresponding prototypes of several classes. Dots denote saved representations and stars denote prototypes (best viewed in color).

Figure 4: Visualization of noun and verb anticipation candidates with or without DCCM under the Exo2Ego setting.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	EgoMe-anti				EgoExoLearn
	Exo		Ego		Exo		Ego
	Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
Train set	8336	8336	8939	8939	11762	17516	78055	33496
Validation set	1817	1817	1970	1970	8235	1960	84726	91454
Test set	3584	3584	3889	3889	12395	5371	15231	50471

Appendix

Table A1 : The number of samples under each view of the train / val / test set of the EgoMe-anti and EgoExoLearn benchmarks.

Methods	EgoMe-anti				EgoExoLearn
	Exo2Ego		Ego2Exo		Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
Ours without Adaptation	27.30	6.24	22.72	6.45	6.07	5.43	6.90	5.35
Tent [ 63 ]	31.72	8.09	26.92	8.98	7.72	6.57	7.98	5.40
TPT [ 57 ]	31.97	8.05	27.14	8.89	8.11	6.35	8.13	5.43
VITTA [ 34 ]	32.21	8.75	26.87	9.14	7.49	6.13	8.27	5.56

Appendix

Table A2 : Quantitative results on the EgoMe-anti and EgoExoLearn under the Exo2Ego and Ego2Exo settings in Top-1 recall evaluation.

$N$	EgoMe-anti				EgoExoLearn				$\#$ Params (M)
	Exo2Ego		Ego2Exo		Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
100	78.99	43.64	71.97	39.62	45.07	42.60	47.48	45.58	1.71
200	79.24	42.47	71.97	39.71	45.36	42.72	47.87	45.61	3.42
300	78.95	42.95	71.81	39.82	45.50	43.00	48.18	45.99	5.13
400	78.86	43.17	71.57	38.95	45.78	42.74	48.06	46.10	6.84

Appendix

Table A3 : Analysis of the maximum capacity

N

of the memory bank. Note that the saved data in the memory banks is also included in the statistics of the number of parameters.

$\alpha$	EgoMe-anti				EgoExoLearn
	Exo2Ego		Ego2Exo		Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
0.25	78.61	42.64	71.85	39.21	44.44	42.89	47.70	46.22
0.50	79.03	43.84	72.01	40.10	46.26	42.98	48.48	46.51
0.75	79.19	41.62	71.09	37.81	45.39	42.62	48.43	46.31
1.00	78.17	40.89	71.37	38.31	45.37	41.98	48.13	46.11

Appendix

Table A4 : Analysis of the hyperparameter

\alpha

for balancing distinct logits inferred by ML-PGM and DCCM.

$\mu_{1}$	EgoMe-anti				EgoExoLearn
	Exo2Ego		Ego2Exo		Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
0.25	77.52	43.16	71.41	38.90	45.36	42.82	48.10	46.26
0.50	78.57	42.62	71.67	39.62	45.53	42.97	48.08	45.89
0.75	78.47	42.37	71.82	39.40	45.59	42.93	48.38	45.66
1.00	79.03	43.84	72.01	40.10	46.26	42.98	48.48	46.51

Appendix

Table A5 : Analysis of the hyperparameter

\mu_{1}

for scaling the visual logits in the DCCM.

$\mu_{2}$	EgoMe-anti				EgoExoLearn
	Exo2Ego		Ego2Exo		Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb	Noun	Verb	Noun	Verb
0.25	78.48	42.98	71.90	39.37	43.99	42.62	47.90	46.14
0.50	79.03	43.84	72.01	40.10	46.26	42.98	48.48	46.51
0.75	78.86	42.76	71.87	39.05	46.27	42.78	48.34	46.14
1.00	77.99	42.39	70.60	39.21	44.33	43.01	48.39	45.68

Appendix

Table A6 : Analysis of the hyperparameter

\mu_{2}

for scaling the textual logits in the DCCM.

$N_{test}$	Exo2Ego		Ego2Exo
	Noun	Verb	Noun	Verb
Full	79.03	43.84	72.01	40.10
50 $\%$	78.59	42.76	71.38	39.28
25 $\%$	78.41	42.38	71.01	38.76
10 $\%$	75.42	38.32	64.84	35.75
5 $\%$	72.84	36.08	63.40	33.51

Appendix

Table A7 : Ablation on the number of test samples on EgoMe-anti benchmark (with 3889 Ego and 3584 Exo test samples in total).

Figure A1 : t-SNE visualization of representations in the memory banks and the corresponding prototypes of five dominant classes under Exo2Ego and Ego2Exo settings. The left panel shows the results on the EgoExoLearn benchmark and the right panel shows the results on the EgoMe-anti benchmark. Dots denote class-wise representations and stars denote prototypes (best viewed in color).

Figure A2 : Visualization of the Top-5 candidates for noun anticipation with or without the proposed DCCM under the Exo2Ego and Ego2Exo setting. The left panel shows the results on the EgoExoLearn benchmark, while the right panel shows the results on EgoMe-anti .

Figure A3 : Visualization of the Top-5 candidates for verb anticipation with or without the proposed DCCM under the Exo2Ego and Ego2Exo setting. The left panel shows the results on the EgoExoLearn benchmark, while the right panel shows the results on EgoMe-anti .

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 322 papers

Aug 26, 2026

Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation

EgoExo proficiency estimation aims to assess action quality by integrating fine-grained motion cues from egocentric (1st-person) views with spatial context from multiple exocentric (3rd-person) views. Simply adding more exocentric views degrades EgoExo performance, as redundant or noisy perspectives dilute useful motion cues. Our analysis identifies two key causes: (1) Multiview redundancy - From the data perspective, certain views provide limited or noisy information, diluting discriminative cues; (2) Overfitting - From the feature perspective, conventional fusion increases representational complexity, causing the model to memorise view-specific patterns rather than learn generalisable representations. To address these issues, we propose two complementary modules: AdaMVS, which adaptively identifies and fuses the most informative view tokens under weak supervision from the data perspective, and VIB-GB, which combines Gradient Blending and Variational Information Bottleneck regularisation from the feature perspective to compress redundant signals and suppress overfitting during training. Experiments on EgoExo-4D and EgoExo-Fitness demonstrate that our method learns both which view to look at and how to fuse them, achieving new state-of-the-art results. Our source code is available at https://github.com/dx199771/AdaMVS

Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye +1

University of Surrey, Guildford, United Kingdom · University of Wollongong, Wollongong, Australia

Action Quality Assessment Multi-View Feature Fusion Multi-View Learning Egocentric Video Understanding Active View Selection

Ask PDF

Sep 20, 2026

CE $^4$ L: Continual Ego, Exo, and Ego-Exo Learning

Perception for embodied agents is video-based, often multi-view (ego, exo, or both), and inherently continual, with simultaneous task and viewpoint shifts. Yet continual learning (CL) remains dominated by exo-only recognition tasks, obscuring behavior under these real-world coupled shifts. We introduce Continual Ego, E}xo, and Ego-Exo Learning (CE

^4

L), a unified multi-view CL benchmark spanning four representative tasks: cross-view referenced skill assessment, temporal action segmentation, cross-view association, and action anticipation & planning. CE

^4

L highlights challenges largely absent in prior CL benchmarks, including cross-view correspondence, view-dependent asynchrony, and heterogeneous semantic objectives. To this end, we propose Video Incremental Subspace-routed Task Adapters (VISTA), a parameter-efficient baseline method that stores task-specific updates in lightweight adapters and performs training-free routing via residual distance to task-specific whitened subspaces estimated from second-order statistics. Extensive experiments demonstrate the significantly varied efficacy of representative CL methods across CE

^4

L settings, while VISTA is consistently competitive and achieves state-of-the-art overall performance. Our source code for benchmarks and methods is available at https://github.com/AnAppleCore/CE4L .

Hongwei Yan, Kanglei Zhou, Yuchen Liu +3

School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Tsinghua-Peking Center for Life Sciences · Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China +2

Temporal Action Segmentation Next-Action Prediction Storage-Efficient Continual Learning Continual Learning Multi-View Learning Egocentric Video Understanding

Ask PDF

Jan 10, 2025

From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

Vision Language Models (VLMs) have achieved strong performance across a wide range of video understanding tasks. However, their viewpoint-invariant training limits their ability to infer egocentric properties, such as human-object interactions, from exocentric video observations. This limitation is particularly critical for applications such as Activities of Daily Living (ADL) monitoring, where understanding egocentric properties is essential but deploying wearable egocentric cameras is often impractical. We propose Ego2ExoVLM, a framework that enables VLMs to infer egocentric properties directly from exocentric videos by leveraging time-synchronized ego-exo video pairs during training. Our key insight is to treat the egocentric viewpoint as privileged supervision, providing rich interaction signals that are available only during training. Ego2ExoVLM consists of two complementary components: Ego2Exo Sequence Distillation, which transfers egocentric reasoning through a language-level sequence distillation objective, and Ego Adaptive Visual Tokens, which encourage the model to surface interaction-relevant cues within exocentric visual representations. To evaluate this capability, we introduce Ego-in-Exo Perception, a benchmark for assessing the understanding of egocentric properties from exocentric videos. We evaluate Ego2ExoVLM on 10 tasks spanning Ego-in-Exo Perception and existing ADL benchmarks, achieving state-of-the-art performance on the ADL-X benchmark suite and consistently outperforming strong baselines on our proposed benchmark. All code, models, and data will be released at https://github.com/dominickrei/EgoExo4ADL.

Dominick Reilly, Manish Kumar Govind, Le Xue +1

University of North Carolina at Charlotte · Elorian AI

Video Understanding VLM Adaptation Egocentric Video Understanding Learning Using Privileged Information

Ask PDF

Abstract

Test-Time Adaptation

Explore similar work

Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation

CE4^44L: Continual Ego, Exo, and Ego-Exo Learning

From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

CE $^4$ L: Continual Ego, Exo, and Ego-Exo Learning