EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities

Xinyuan Qian; Xinjia Zhu; Alessio Brutti; Dong Liang

cs.MMMar 18, 2026

EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities

Authors: Xinyuan Qian, Xinjia Zhu, Alessio Brutti, Dong Liang

Organizations: School of Computer and Communication Enngineering, University of Science and Technology Beijing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China · Fondazione Bruno Kessler, Italy · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Shenzhen Research Institute, Nanjing University of Aeronautics and Astronautics, China

Abstract

TTM (Talking to Me) task is a pivotal component in understanding human social interactions, aiming to determine who is engaged in conversation with the camera-wearer. Traditional models often face challenges in real-world scenarios due to missing visual data, neglecting the role of head orientation, and background noise. This study addresses these limitations by introducing EgoAdapt, an adaptive framework designed for robust egocentric "Talking to Me" speaker detection under missing modalities. Specifically, EgoAdapt incorporates three key modules: (1) a Visual Speaker Target Recognition (VSTR) module that captures head orientation as a non-verbal cue and lip movement as a verbal cue, allowing a comprehensive interpretation of both verbal and non-verbal signals to address TTM, setting it apart from tasks focused solely on detecting speaking status; (2) a Parallel Shared-weight Audio (PSA) encoder for enhanced audio feature extraction in noisy environments; and (3) a Visual Modality Missing Awareness (VMMA) module that estimates the presence or absence of each modality at each frame to adjust the system response dynamically.Comprehensive evaluations on the TTM benchmark of the Ego4D dataset demonstrate that EgoAdapt achieves a mean Average Precision (mAP) of 67.39% and an Accuracy (Acc) of 62.01%, significantly outperforming the state-of-the-art method by 4.96% in Accuracy and 1.56% in mAP.

Figures & tables

Panel of Figure 1 — (a) The TTM Task Setting

Figure 2. Overview of the EgoAdapt framework for egocentric interactive speaker detection. The VMMA ( VMMA ) module assesses modality availability and outputs $p_{vmm}$ to guide adaptation. The VSTR ( VSTR ) module extract non-verbal and verbal cues ( $z_{h}$ and $z_{l}$ ) from head crops $D_{h}$ and lip crops $D_{l}$ , respectively, while the PSA ( PSA ) encoder processes clean and noisy audio ( $D_{a}$ and $D_{m}$ ) to produce $z_{a}$ . These cues are fused in the Visual-Audio Fusion module for the final TTM Prediction of "Talking to Me" or "Non Talking to Me".

Figure 3. VSTR ( VSTR ) module. It includes two components: the Head Pose Feature Extraction Module, which captures head orientation (non-verbal cues), and the Lip Feature Extraction Module, which extracts lip movement features (verbal cues).

Figure 4. PSA ( PSA ) encoder. It processes clean speech $D_{a}$ and noise-mixed speech $D_{m}$ (ratio $\gamma$ ) into Mel spectrograms, encodes them with shared weights to obtain embeddings $z_{a}$ and $z_{m}$ , and uses MSE loss to enforce noise-robust audio features.

Figure 5. VMMA ( VMMA ) module dynamically handles missing visual inputs by processing head crops $D_{h}$ and the initialized prompt $p_{\text{vmm}}$ . It extracts fine-grained features by tracking frame presence and coarse-grained features by quantifying the missing frame ratio $\Delta_{c}$ .

Figure 6. Oveview of Visual-Audio Fusion module. It fuses head pose $z_{h}$ , lip motion $z_{l}$ , and audio $z_{a}$ via Head-Lip, Lip-Audio, and Audio-Head cross-attention blocks. Aggregated features with VMMA prompt $p_{\mathrm{vmm}}$ are refined by self-attention for final “Talking to Me” prediction.

Parameter	Head Pose Feature Extraction
Backbone Model	6DRepNet
Pretrained	Yes (frozen weights)
Deploy Mode	True
Input Feature	224x224 pixels image
Pooling Layer	Adaptive Average Pooling
Output Dimension	6 (Ortho6D Representation)

Table 1. Head pose feature extraction parameters.

Parameter	Lip Feature Extraction	Audio Feature Extraction
Backbone Model	CLIP ViT-B/32	Whisper-small
Pretrained	Yes (frozen weights)	Yes (frozen weights)
Input Feature	224x224 pixels image	Log-Mel Spectrogram
Embedding Dimension	512	768
Temporal Module Dimension	128	128
Number of Temporal Layers	1	1

Table 2. Lip and audio feature extraction parameters.

Method	Val		Test
	Acc (%)	mAP (%)	Acc (%)	mAP (%)
Random Guess ( Grauman et al., 2022 )	33.44	53.82	47.41	50.16
ResNet-18 Bi-LSTM ( Grauman et al., 2022 )	64.31	56.50	49.75	55.06
EgoT2 ( Xue et al., 2023 )	66.07	60.80	56.17	57.82
QuAVF ( Lin et al., 2023 )	71.80	71.20	57.05	65.83
Ours	81.14	84.83	62.01	67.39

Table 3. Comparison of our proposed EgoAdapt with existing methods on TTM benchmark of Ego4D dataset.

Modality	Visual Backbone	Audio Backbone	Acc (%)	mAP (%)
V	ViT ( Radford et al., 2021 )	✗	67.25	62.51
A	✗	Whisper ( Radford et al., 2021 )	75.64	77.55
V, A	ViT	WavLM ( Chen et al., 2022 )	72.69	72.96
V, A	TalkNet ( Tao et al., 2021 )	WavLM	72.52	72.59
V, A	TalkNet	Whisper	73.65	77.51
V, A	ViT	Whisper	75.33	77.21

Table 4. Comparison of the performance for different modality and backbone combinations.

Table 11

Figure 7. Comparison of the performance with or w/o VSTR.

Panel of Figure 8 — Figure 8. Comparison of model performance with PSA training and without it (i.e., using the pre-trained Whisper backbone) under different SNR levels.

Figure 9. Comparison of our proposed method with QuAVF ( Lin et al., 2023 ) (currently the SOTA method) in a multi-speaker environment where multiple individuals are speaking, and the model must correctly identify who is speaking to the camera wearer.

Figure 10. Comparison of our proposed method with QuAVF ( Lin et al., 2023 ) in noisy background scenario, where the model must distinguish between relevant and irrelevant audio signals.

Figure 11. Comparison of our proposed method with QuAVF ( Lin et al., 2023 ) in the scenario where the visual modality is partially missing, and the model relies on audio and contextual information to make predictions.

Multimodal Robustness

Multimodal robustness studies how systems combining different data types remain reliable when inputs are missing, noisy, shifted, or misleading.

See all 279 papers

Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos
Sep 12, 2026Feiyu Du, Xi He, Jia Li +2Egocentric Video Understanding Audio-Visual Understanding
The University of Texas at Dallas Richardson, United States
ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion
Sep 22, 2026Pu Wang, Hugo Van hammeAudio-Visual Understanding Multimodal Robustness
KU Leuven, Department of Electrical Engineering, Leuven, Belgium
Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration
Oct 23, 2026Pu Wang, Hugo Van hammeAudio-Visual Understanding
KU Leuven, Department of Electrical Engineering, Leuven, Belgium

Sep 12, 2026J/K move · Enter open · S save

Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

Feiyu Du, Xi He, Jia Li, Yapeng Tian+1

The University of Texas at Dallas Richardson, United States

Egocentric Video Understanding Audio-Visual Understanding

Open paper AskPDF arXiv

Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-level recognition, which is poorly aligned with online interaction and overlooks the diverse non-TTM speaking states that naturally arise in egocentric videos. In this paper, we revisit this problem by reformulating it as an online, frame-level prediction task. Instead of treating TTM as a binary problem against a single negative class, we model it in the presence of diverse and previously underexplored non-TTM states, such as talking-to-others, self-talking, and background conditions. To support this new formulation, we construct an Online TTM Dataset consisting of 406 egocentric video clips with approximately 900K annotated frames, each labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking), by extending the Ego4D social interaction benchmark. In this benchmark, we evaluate five adapted baselines and develop a new model that integrates social cues across modalities. Experimental results show that our multimodal model, which jointly leverages audio, visual, and speech-semantic cues, achieves 75.5% frame-level F1 on TTM, outperforming strong baselines and enabling a systematic analysis of how different speaking states affect TTM recognition.

Abstract

Multimodal Robustness

Explore similar work

Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration