cs.CVSep 12, 2026

Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

Authors: Feiyu Du, Xi He, Jia Li, Yapeng Tian, Weili Wu

Organizations: The University of Texas at Dallas Richardson, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-level recognition, which is poorly aligned with online interaction and overlooks the diverse non-TTM speaking states that naturally arise in egocentric videos. In this paper, we revisit this problem by reformulating it as an online, frame-level prediction task. Instead of treating TTM as a binary problem against a single negative class, we model it in the presence of diverse and previously underexplored non-TTM states, such as talking-to-others, self-talking, and background conditions. To support this new formulation, we construct an Online TTM Dataset consisting of 406 egocentric video clips with approximately 900K annotated frames, each labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking), by extending the Ego4D social interaction benchmark. In this benchmark, we evaluate five adapted baselines and develop a new model that integrates social cues across modalities. Experimental results show that our multimodal model, which jointly leverages audio, visual, and speech-semantic cues, achieves 75.5% frame-level F1 on TTM, outperforming strong baselines and enabling a systematic analysis of how different speaking states affect TTM recognition.

Figures & tables

Figure 1. We introduce the Online TTM Dataset, a multimodal benchmark for online who-is-talking-to-me perception in realistic first-person scenarios. It temporally grounds social speaking states in continuous egocentric video streams while integrating visual, audio, and speech information. Within such a continuous first-person stream, the current social speaking state may shift over short temporal windows from background to talking-to-others, talking-to-me, or self-talking. An overview figure showing a continuous egocentric video stream annotated with four social speaking states over time. The visual layout highlights how the state of interaction changes within a first-person scene, including background, talking-to-others, talking-to-me, and self-talking. The figure emphasizes temporal transitions between these states and illustrates that wearer-directed speech appears as only one part of a richer stream of multimodal social activity.
Figure 2. Qualitative illustration of the original Ego4D social annotations from two different clips. (a) One clip is dominated by long background intervals, while TTM segments are sparse and fragmented. (b) Another clip contains frequent transitions among speaking states, with blank intervals indicating frames without valid social-state annotations.
DatasetPrediction unitGranularityVideosTotal framesTTM ratioOnline eval.Benchmark purpose
Ego4D SocialSegment labelClip-level4393,951,00011.49%×\timesOffline clip recognition
Online TTM (ours)State labelFrame-level406885,86641.49%✓Online frame prediction
Table 1. Comparison between the Ego4D social annotations and our Online TTM Dataset.
Figure 3. Overview of the proposed online multimodal model for egocentric social speaking recognition. From a continuous first-person video stream, we construct a person-centered visual stream and an aligned audio-text stream. The visual encoder extracts complementary cues from facial appearance, head-related representation, mouth motion, and wearer-centered geometry, while the audio-text branch captures frame-synchronous acoustic evidence and Whisper-derived speech semantics. These features are fused by cross-attention and then processed by a causal temporal Transformer with a causal mask to produce frame-level four-state predictions over BG, SELF, TTO, and TTM. A pipeline diagram of the proposed online multimodal model. A continuous egocentric video stream is split into a person-centered visual branch and an aligned audio-text branch. The visual side contains modules for facial appearance, head-related features, mouth motion, and wearer-centered geometry. The audio-text side provides acoustic evidence and text-semantic features. These representations are fused through cross-attention and then sent to a causal temporal Transformer, which outputs frame-level predictions for four classes: background, self-talking, talking-to-others, and talking-to-me.
 Frame-level F1 (%)Segment-level (%)Efficiency
MethodBGSELFTTOTTMTTM mAPParams (M)Model latency (ms/frame)
Ego4D baseline ( Ego4D, )49.959.22.361.362.618.0668.2
Streaming ASD ( Kundu et al., 2025 )66.569.425.366.372.83.4514.0
CMeRT ( Pang et al., 2025 )64.971.025.065.971.038.08812.0
LoCoNet ( Wang et al., 2024 )64.063.422.763.370.617.3047.4
ProTAS ( Shen and Elhamifar, 2024 )62.360.720.360.168.520.6386.4
Table 2. Comparison with baselines on the Online TTM Dataset.
Figure 4. Results comparison of our method and Streaming ASD on challenging scenarios from the Online TTM test set. Green boxes denote TTM targets, orange boxes denote TTO targets, and red circles indicate false predictions. The first row shows the ground truth, the second row shows our predictions, and the third row shows the predictions of Streaming ASD. A qualitative comparison figure showing several challenging examples from the Online TTM test set. Each example is arranged in three rows: the first row shows the ground truth, the second row shows predictions from the proposed method, and the third row shows predictions from Streaming ASD. Faces identified as talking-to-me or talking-to-others are marked with colored boxes, and incorrect predictions are marked with circles. The figure shows that the proposed method more often places the correct target annotation on socially confusable faces than the baseline.
ComponentsFrame-level F1 (%)Segment-level (%)
ViTHeadMouthGeomTextBGSELFTTOTTMTTM mAP
✓    56.3314.280.0562.2767.00
✓✓   64.8063.210.1563.2371.07
✓✓✓  66.6870.4519.4264.3972.13
✓✓✓✓ 70.6770.8826.0765.0072.21
✓✓✓✓✓99.4272.6040.1975.5479.78
Table 3. Ablation study of the main components.
 TTM False NegativesTTM False Positives
VariantBGSELFTTOBGSELFTTO
No Text17,5679,5114915,5831,30525,694
Full Model1115,10614,18705,29721,493
Table 4. TTM-centered confusion attribution with and without text semantics. All values denote the number of frames.
More in this topic

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

Explore similar work

CardsList
Mar 18, 2026

EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities

TTM (Talking to Me) task is a pivotal component in understanding human social interactions, aiming to determine who is engaged in conversation with the camera-wearer. Traditional models often face challenges in real-world scenarios due to missing visual data, neglecting the role of head orientation, and background noise. This study addresses these limitations by introducing EgoAdapt, an adaptive framework designed for robust egocentric "Talking to Me" speaker detection under missing modalities. Specifically, EgoAdapt incorporates three key modules: (1) a Visual Speaker Target Recognition (VSTR) module that captures head orientation as a non-verbal cue and lip movement as a verbal cue, allowing a comprehensive interpretation of both verbal and non-verbal signals to address TTM, setting it apart from tasks focused solely on detecting speaking status; (2) a Parallel Shared-weight Audio (PSA) encoder for enhanced audio feature extraction in noisy environments; and (3) a Visual Modality Missing Awareness (VMMA) module that estimates the presence or absence of each modality at each frame to adjust the system response dynamically.Comprehensive evaluations on the TTM benchmark of the Ego4D dataset demonstrate that EgoAdapt achieves a mean Average Precision (mAP) of 67.39% and an Accuracy (Acc) of 62.01%, significantly outperforming the state-of-the-art method by 4.96% in Accuracy and 1.56% in mAP.

Xinyuan Qian, Xinjia Zhu, Alessio Brutti +1

School of Computer and Communication Enngineering, University of Science and Technology Beijing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China · Fondazione Bruno Kessler, Italy +1

Multimodal RobustnessEgocentric Video UnderstandingMissing-Modality Learning
AskPDF
Mar 9, 2026

Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time

In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me that''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing stroke. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the audio--visual alignment required by deictic interaction. To bridge this gap, we introduce Egocentric Co-Speech Grounding (EcoG), where grounding is executable only if an agent jointly predicts What, Where, and When. To operationalize this, we present EcoG-Bench, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of 811 egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a Progressive Cognitive Evaluation protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (96.9% strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: 17.0%). Moreover, in a diagnostic ablation, replacing the native video--audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model (17.0%→\to42.9%). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech--gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.

Weijie Zhou, Xuantang Xiong, Zhenlin Hu +6

Beijing Jiaotong University · Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA) · Tencent Robotics X +2

Multimodal GroundingAudio-Visual AlignmentEgocentric Video Understanding
AskPDF
Sep 25, 2026

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

In face-to-face, real-time communication, a talking agent must track the target speaker through natural pauses, turn-taking, and backchannels, often amid background cross-talk. Most target speaker extraction (TSE) studies, however, rely on simulated mixtures with full or sparse overlap and ignore the turn-taking of real conversations. We therefore introduce, to our knowledge, the first benchmark for online audio-visual TSE (AV-TSE), built from intact dyadic interactions with independent third-party interference. Observing that anticipating upcoming activity from semantic, acoustic, and facial cues benefits online AV-TSE, we propose an LLM-based target-speaker voice activity projection (TS-VAP) module. Unlike conventional VAP with separated speaker channels, it forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-latency separator. We further combine this predictive context with historical and synchronous speaker context. Experiments show that TS-VAP consistently improves streaming extraction across multiple AV-TSE backbones, and that further combining historical, synchronous, and predictive context yields nearly 1 dB gain on real AV conversations. Project page: https://jjjjiaozi.github.io/TS-VAP/.

Shuhan Zhang, Wenxuan Wu, Haizhou Li

Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China

Target Speaker ExtractionSpeech Processing
AskPDF