Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

Feiyu Du; Xi He; Jia Li; Yapeng Tian; Weili Wu

cs.CVSep 12, 2026

Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

Authors: Feiyu Du, Xi He, Jia Li, Yapeng Tian, Weili Wu

Organizations: The University of Texas at Dallas Richardson, United States

Abstract

Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-level recognition, which is poorly aligned with online interaction and overlooks the diverse non-TTM speaking states that naturally arise in egocentric videos. In this paper, we revisit this problem by reformulating it as an online, frame-level prediction task. Instead of treating TTM as a binary problem against a single negative class, we model it in the presence of diverse and previously underexplored non-TTM states, such as talking-to-others, self-talking, and background conditions. To support this new formulation, we construct an Online TTM Dataset consisting of 406 egocentric video clips with approximately 900K annotated frames, each labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking), by extending the Ego4D social interaction benchmark. In this benchmark, we evaluate five adapted baselines and develop a new model that integrates social cues across modalities. Experimental results show that our multimodal model, which jointly leverages audio, visual, and speech-semantic cues, achieves 75.5% frame-level F1 on TTM, outperforming strong baselines and enabling a systematic analysis of how different speaking states affect TTM recognition.

Figures & tables

Figure 1. We introduce the Online TTM Dataset, a multimodal benchmark for online who-is-talking-to-me perception in realistic first-person scenarios. It temporally grounds social speaking states in continuous egocentric video streams while integrating visual, audio, and speech information. Within such a continuous first-person stream, the current social speaking state may shift over short temporal windows from background to talking-to-others, talking-to-me, or self-talking. An overview figure showing a continuous egocentric video stream annotated with four social speaking states over time. The visual layout highlights how the state of interaction changes within a first-person scene, including background, talking-to-others, talking-to-me, and self-talking. The figure emphasizes temporal transitions between these states and illustrates that wearer-directed speech appears as only one part of a richer stream of multimodal social activity.

Figure 2. Qualitative illustration of the original Ego4D social annotations from two different clips. (a) One clip is dominated by long background intervals, while TTM segments are sparse and fragmented. (b) Another clip contains frequent transitions among speaking states, with blank intervals indicating frames without valid social-state annotations.

Dataset	Prediction unit	Granularity	Videos	Total frames	TTM ratio	Online eval.	Benchmark purpose
Ego4D Social	Segment label	Clip-level	439	3,951,000	11.49%	$\times$	Offline clip recognition
Online TTM (ours)	State label	Frame-level	406	885,866	41.49%	✓	Online frame prediction

Table 1. Comparison between the Ego4D social annotations and our Online TTM Dataset.

Figure 3. Overview of the proposed online multimodal model for egocentric social speaking recognition. From a continuous first-person video stream, we construct a person-centered visual stream and an aligned audio-text stream. The visual encoder extracts complementary cues from facial appearance, head-related representation, mouth motion, and wearer-centered geometry, while the audio-text branch captures frame-synchronous acoustic evidence and Whisper-derived speech semantics. These features are fused by cross-attention and then processed by a causal temporal Transformer with a causal mask to produce frame-level four-state predictions over BG, SELF, TTO, and TTM. A pipeline diagram of the proposed online multimodal model. A continuous egocentric video stream is split into a person-centered visual branch and an aligned audio-text branch. The visual side contains modules for facial appearance, head-related features, mouth motion, and wearer-centered geometry. The audio-text side provides acoustic evidence and text-semantic features. These representations are fused through cross-attention and then sent to a causal temporal Transformer, which outputs frame-level predictions for four classes: background, self-talking, talking-to-others, and talking-to-me.

	Frame-level F1 (%)				Segment-level (%)	Efficiency
Method	BG	SELF	TTO	TTM	TTM mAP	Params (M)	Model latency (ms/frame)
Ego4D baseline ( Ego4D, )	49.9	59.2	2.3	61.3	62.6	18.066	8.2
Streaming ASD ( Kundu et al., 2025 )	66.5	69.4	25.3	66.3	72.8	3.451	4.0
CMeRT ( Pang et al., 2025 )	64.9	71.0	25.0	65.9	71.0	38.088	12.0
LoCoNet ( Wang et al., 2024 )	64.0	63.4	22.7	63.3	70.6	17.304	7.4
ProTAS ( Shen and Elhamifar, 2024 )	62.3	60.7	20.3	60.1	68.5	20.638	6.4

Table 2. Comparison with baselines on the Online TTM Dataset.

Figure 4. Results comparison of our method and Streaming ASD on challenging scenarios from the Online TTM test set. Green boxes denote TTM targets, orange boxes denote TTO targets, and red circles indicate false predictions. The first row shows the ground truth, the second row shows our predictions, and the third row shows the predictions of Streaming ASD. A qualitative comparison figure showing several challenging examples from the Online TTM test set. Each example is arranged in three rows: the first row shows the ground truth, the second row shows predictions from the proposed method, and the third row shows predictions from Streaming ASD. Faces identified as talking-to-me or talking-to-others are marked with colored boxes, and incorrect predictions are marked with circles. The figure shows that the proposed method more often places the correct target annotation on socially confusable faces than the baseline.

Components					Frame-level F1 (%)				Segment-level (%)
ViT	Head	Mouth	Geom	Text	BG	SELF	TTO	TTM	TTM mAP
✓					56.33	14.28	0.05	62.27	67.00
✓	✓				64.80	63.21	0.15	63.23	71.07
✓	✓	✓			66.68	70.45	19.42	64.39	72.13
✓	✓	✓	✓		70.67	70.88	26.07	65.00	72.21
✓	✓	✓	✓	✓	99.42	72.60	40.19	75.54	79.78

Table 3. Ablation study of the main components.

	TTM False Negatives			TTM False Positives
Variant	BG	SELF	TTO	BG	SELF	TTO
No Text	17,567	9,511	49	15,583	1,305	25,694
Full Model	111	5,106	14,187	0	5,297	21,493

Table 4. TTM-centered confusion attribution with and without text semantics. All values denote the number of frames.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities
Mar 18, 2026Xinyuan Qian, Xinjia Zhu, Alessio Brutti +1Multimodal Robustness Egocentric Video Understanding
School of Computer and Communication Enngineering, University of Science and Technology Beijing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China · Fondazione Bruno Kessler, Italy +1
Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
Mar 9, 2026Weijie Zhou, Xuantang Xiong, Zhenlin Hu +6Multimodal Grounding Audio-Visual Alignment
Beijing Jiaotong University · Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA) · Tencent Robotics X +2
Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information
Sep 25, 2026Shuhan Zhang, Wenxuan Wu, Haizhou LiTarget Speaker Extraction Speech Processing
Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China

Mar 18, 2026J/K move · Enter open · S save

EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities

Xinyuan Qian, Xinjia Zhu, Alessio Brutti, Dong Liang

School of Computer and Communication Enngineering, University of Science and Technology Beijing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China · Fondazione Bruno Kessler, Italy+1

Multimodal Robustness Egocentric Video Understanding Missing-Modality Learning

Open paper AskPDF arXiv

TTM (Talking to Me) task is a pivotal component in understanding human social interactions, aiming to determine who is engaged in conversation with the camera-wearer. Traditional models often face challenges in real-world scenarios due to missing visual data, neglecting the role of head orientation, and background noise. This study addresses these limitations by introducing EgoAdapt, an adaptive framework designed for robust egocentric "Talking to Me" speaker detection under missing modalities. Specifically, EgoAdapt incorporates three key modules: (1) a Visual Speaker Target Recognition (VSTR) module that captures head orientation as a non-verbal cue and lip movement as a verbal cue, allowing a comprehensive interpretation of both verbal and non-verbal signals to address TTM, setting it apart from tasks focused solely on detecting speaking status; (2) a Parallel Shared-weight Audio (PSA) encoder for enhanced audio feature extraction in noisy environments; and (3) a Visual Modality Missing Awareness (VMMA) module that estimates the presence or absence of each modality at each frame to adjust the system response dynamically.Comprehensive evaluations on the TTM benchmark of the Ego4D dataset demonstrate that EgoAdapt achieves a mean Average Precision (mAP) of 67.39% and an Accuracy (Acc) of 62.01%, significantly outperforming the state-of-the-art method by 4.96% in Accuracy and 1.56% in mAP.

Abstract

Egocentric Video Understanding

Explore similar work

EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities

Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information