Organizations: School of Computer and Communication Enngineering, University of Science and Technology Beijing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China · Fondazione Bruno Kessler, Italy · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Shenzhen Research Institute, Nanjing University of Aeronautics and Astronautics, China
TTM (Talking to Me) task is a pivotal component in understanding human social interactions, aiming to determine who is engaged in conversation with the camera-wearer. Traditional models often face challenges in real-world scenarios due to missing visual data, neglecting the role of head orientation, and background noise. This study addresses these limitations by introducing EgoAdapt, an adaptive framework designed for robust egocentric "Talking to Me" speaker detection under missing modalities. Specifically, EgoAdapt incorporates three key modules: (1) a Visual Speaker Target Recognition (VSTR) module that captures head orientation as a non-verbal cue and lip movement as a verbal cue, allowing a comprehensive interpretation of both verbal and non-verbal signals to address TTM, setting it apart from tasks focused solely on detecting speaking status; (2) a Parallel Shared-weight Audio (PSA) encoder for enhanced audio feature extraction in noisy environments; and (3) a Visual Modality Missing Awareness (VMMA) module that estimates the presence or absence of each modality at each frame to adjust the system response dynamically.Comprehensive evaluations on the TTM benchmark of the Ego4D dataset demonstrate that EgoAdapt achieves a mean Average Precision (mAP) of 67.39% and an Accuracy (Acc) of 62.01%, significantly outperforming the state-of-the-art method by 4.96% in Accuracy and 1.56% in mAP.
Figures & tables
(a) The TTM Task Setting
Figure 2. Overview of the EgoAdapt framework for egocentric interactive speaker detection. The VMMA ( VMMA ) module assesses modality availability and outputs pvmm to guide adaptation. The VSTR ( VSTR ) module extract non-verbal and verbal cues ( zh and zl ) from head crops Dh and lip crops Dl , respectively, while the PSA ( PSA ) encoder processes clean and noisy audio ( Da and Dm ) to produce za . These cues are fused in the Visual-Audio Fusion module for the final TTM Prediction of "Talking to Me" or "Non Talking to Me".
Figure 3. VSTR ( VSTR ) module. It includes two components: the Head Pose Feature Extraction Module, which captures head orientation (non-verbal cues), and the Lip Feature Extraction Module, which extracts lip movement features (verbal cues).
Figure 4. PSA ( PSA ) encoder. It processes clean speech Da and noise-mixed speech Dm (ratio γ ) into Mel spectrograms, encodes them with shared weights to obtain embeddings za and zm , and uses MSE loss to enforce noise-robust audio features.
Figure 5. VMMA ( VMMA ) module dynamically handles missing visual inputs by processing head crops Dh and the initialized prompt pvmm . It extracts fine-grained features by tracking frame presence and coarse-grained features by quantifying the missing frame ratio Δc .
Figure 6. Oveview of Visual-Audio Fusion module. It fuses head pose zh , lip motion zl , and audio za via Head-Lip, Lip-Audio, and Audio-Head cross-attention blocks. Aggregated features with VMMA prompt pvmm are refined by self-attention for final “Talking to Me” prediction.
Parameter
Head Pose Feature Extraction
Backbone Model
6DRepNet
Pretrained
Yes (frozen weights)
Deploy Mode
True
Input Feature
224x224 pixels image
Pooling Layer
Adaptive Average Pooling
Output Dimension
6 (Ortho6D Representation)
Table 1. Head pose feature extraction parameters.
Parameter
Lip Feature Extraction
Audio Feature Extraction
Backbone Model
CLIP ViT-B/32
Whisper-small
Pretrained
Yes (frozen weights)
Yes (frozen weights)
Input Feature
224x224 pixels image
Log-Mel Spectrogram
Embedding Dimension
512
768
Temporal Module Dimension
128
128
Number of Temporal Layers
1
1
Table 2. Lip and audio feature extraction parameters.
Method
Val
Test
Acc (%)
mAP (%)
Acc (%)
mAP (%)
Random Guess ( Grauman et al., 2022 )
33.44
53.82
47.41
50.16
ResNet-18 Bi-LSTM ( Grauman et al., 2022 )
64.31
56.50
49.75
55.06
EgoT2 ( Xue et al., 2023 )
66.07
60.80
56.17
57.82
QuAVF ( Lin et al., 2023 )
71.80
71.20
57.05
65.83
Ours
81.14
84.83
62.01
67.39
Table 3. Comparison of our proposed EgoAdapt with existing methods on TTM benchmark of Ego4D dataset.
Modality
Visual Backbone
Audio Backbone
Acc (%)
mAP (%)
V
ViT ( Radford et al., 2021 )
✗
67.25
62.51
A
✗
Whisper ( Radford et al., 2021 )
75.64
77.55
V, A
ViT
WavLM ( Chen et al., 2022 )
72.69
72.96
V, A
TalkNet ( Tao et al., 2021 )
WavLM
72.52
72.59
V, A
TalkNet
Whisper
73.65
77.51
V, A
ViT
Whisper
75.33
77.21
Table 4. Comparison of the performance for different modality and backbone combinations.
Table 11
Figure 7. Comparison of the performance with or w/o VSTR.
Figure 8. Comparison of model performance with PSA training and without it (i.e., using the pre-trained Whisper backbone) under different SNR levels.
Figure 9. Comparison of our proposed method with QuAVF ( Lin et al., 2023 ) (currently the SOTA method) in a multi-speaker environment where multiple individuals are speaking, and the model must correctly identify who is speaking to the camera wearer.
Figure 10. Comparison of our proposed method with QuAVF ( Lin et al., 2023 ) in noisy background scenario, where the model must distinguish between relevant and irrelevant audio signals.
Figure 11. Comparison of our proposed method with QuAVF ( Lin et al., 2023 ) in the scenario where the visual modality is partially missing, and the model relies on audio and contextual information to make predictions.
Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-level recognition, which is poorly aligned with online interaction and overlooks the diverse non-TTM speaking states that naturally arise in egocentric videos. In this paper, we revisit this problem by reformulating it as an online, frame-level prediction task. Instead of treating TTM as a binary problem against a single negative class, we model it in the presence of diverse and previously underexplored non-TTM states, such as talking-to-others, self-talking, and background conditions. To support this new formulation, we construct an Online TTM Dataset consisting of 406 egocentric video clips with approximately 900K annotated frames, each labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking), by extending the Ego4D social interaction benchmark. In this benchmark, we evaluate five adapted baselines and develop a new model that integrates social cues across modalities. Experimental results show that our multimodal model, which jointly leverages audio, visual, and speech-semantic cues, achieves 75.5% frame-level F1 on TTM, outperforming strong baselines and enabling a systematic analysis of how different speaking states affect TTM recognition.
Feiyu Du, Xi He, Jia Li +2
The University of Texas at Dallas Richardson, United States
Active speaker detection (ASD) requires reliable association between visible faces and acoustic speech, yet existing systems often degrade under challenging domains or incomplete observations. We introduce ROAM-ASD, a robust audiovisual framework that jointly models audio, full-face, and fine-grained mouth representations. A unified joint self-attention mechanism processes all input streams together with modality-agnostic query tokens, enabling direct interaction among available modality inputs. Modality dropout further improves robustness when input streams are unavailable. ROAM-ASD achieves state-of-the-art performance across five ASD benchmarks: 98.8% mAP on WASD, 87.9% on UniTalk, 96.5% on AVA, 99.3% on ASW, and 98.2% on Talkies, improving over previous best systems by 5.1, 4.7, 0.9, 1.0, and 2.1 mAP points, respectively. ROAM-ASD also substantially improves zero-shot cross-dataset generalization and remains robust to missing observations.
Pu Wang, Hugo Van hamme
KU Leuven, Department of Electrical Engineering, Leuven, Belgium
Active speaker detection (ASD) relies on audiovisual correspondence between speech and visible faces, yet this correspondence becomes unreliable in crowded scenes, or when visual evidence is unavailable. We introduce ROAM-ASD-C and ROAM-ASD-CR to extend ROAM-ASD with contextual and identity-aware reasoning. ROAM-ASD-C jointly reasons across simultaneously visible candidates, allowing other faces to provide complementary evidence when the target observation is ambiguous or incomplete. ROAM-ASD-CR incorporates registered-speaker representations derived from enrollment speech, allowing detection of whether a speaker is active even when that speaker is not visually observed. ROAM-ASD-C achieves state-of-the-art performance across five ASD benchmarks: 96.8% mAP on AVA, 98.9% on WASD, 88.8% on UniTalk, 98.7% on Talkies, and 99.5% on ASW. It also improves ROAM-ASD by up to 13.3 mAP under 75% target-track visual removal.
Pu Wang, Hugo Van hamme
KU Leuven, Department of Electrical Engineering, Leuven, Belgium