Identifying whom a person looks at is central to understanding multi-party conversations, yet gaze geometry alone becomes ambiguous as more people appear. Conversations offer cues that geometry lacks, such as who is speaking and what is said, but no gaze dataset provides them for many multi-party scenes. We introduce SORBET, a benchmark of 3,340 samples from 1,578 clips with synchronized speech, transcripts, and attended person labels, covering more multi-party scenes than existing gaze datasets. Prior gaze models struggle to use these cues, which come in different forms, must be linked to the right person, and can conflict with gaze geometry. We propose Nunchi, a framework that constructs a text prompt from the transcript and the outputs of gaze and speaker models, links every cue to the candidates with shared labels, and lets a vision-language model (VLM) weigh gaze geometry against conversational cues to identify the attended person. Nunchi outperforms the strongest existing method by 4.8 percentage points. On SORBET, conversational cues help whereas additional visual cues do not, and the gains hold across three VLMs, with automatic transcripts, and on two existing datasets.
Figures & tables
Figure 1: Attended person identification in a multi-party conversation. As more people appear in a scene, several candidates lie in a similar direction from the observer, so gaze geometry alone cannot determine whom the observer looks at. Gaze-LLE [ 35 ] and MTGS [ 15 ] each select an incorrect candidate. Nunchi combines gaze geometry with conversational cues, the speaker and the transcript, and identifies the attended person C2 .
K=2
K=3 – 4
K≥5
Dataset
G
M
G
M
G
M
VAT †
83.9
85.1
80.2
77.0
64.3
63.8
ChildPlay †
87.7
86.9
81.3
79.2
57.1
67.3
UCO-LAEO †
100.0
97.4
96.2
100.0
–
–
VGS
79.3
75.9
55.9
58.8
–
–
Table 1: Effect of the number of people on Gaze-LLE (G) and MTGS (M) , in identity accuracy (%). Both models use their released checkpoints. K is the number of annotated people other than the observer. † marks datasets used to train MTGS.
K≥3 (test)
Dataset
Video
Audio
Transcript
Clips
Samples
GazeFollow [ 33 ]
✗
✗
✗
–
580
VAT [ 8 ]
✓
✗
✗
32
383
ChildPlay [ 37 ]
✓
✗
✗
9
97
UCO-LAEO [ 24 ]
✓
✓
✗
5
26
AVA-LAEO [ 24 ]
✓
✗
✗
10 ∗
110 ∗
Table 2: Comparison with existing gaze following and social gaze datasets. Audio and Transcript mark synchronized audio and transcripts released with the dataset. K≥3 counts test clips and samples with three or more candidates. As adjacent video frames are nearly identical, we sample video datasets every 25 frames, about once per second, and SORBET at keyframes. ∗ No test split, so we count the validation movies.
Figure 2: Construction of SORBET. From two multi-party conversation datasets, we keep frames with at least three identified people and a visible speaker, and take the middle frame of each segment of such frames as a keyframe. Each identified person serves in turn as the observer, and the annotator labels what the observer looks at. SORBET keeps samples whose gaze target is a candidate.
Series
Split
Episode
Sample
K=2
K=3
K=4
K≥5
Spk. (%)
Friends
train
108
1,118
328
540
197
53
49.6
val
42
485
143
278
60
4
48.5
test
27
480
108
220
74
78
50.8
BBT
train
16
219
134
65
20
0
64.8
val
2
48
35
13
0
0
58.3
test
5
139
94
40
5
0
72.7
Table 3: SORBET splits . Splits are disjoint at the episode level. K is the number of candidates per sample. Spk. is the percentage of samples whose target is the current speaker.
Figure 3: Overview of Nunchi. Cue extraction obtains gaze and speaker information with frozen experts, along with the transcript. Prompt construction writes them as text, and every cue refers to each candidate by the label Ck drawn in the keyframe. Candidate scoring reads the two images and the prompt with a single VLM and scores each candidate by the logit of its label. In this example, the gaze prior and the speaker cue point to different candidates, and Nunchi selects the attended person C2 . Colors mark the gaze prior, speaker, and transcript paths, and ✓/✗ mark agreement with the ground truth, which the model does not see.
Model
Training
Output
Cue
Params (trainable)
All
Friends
BBT
Superstore
K≥5
Gaze following
Gaze-LLE [ 35 ]
zero-shot
Mass
–
307.3M (0)
79.6
77.5
86.3
79.9
64.0
decoder
Mass
–
307.3M (2.9M)
81.9
81.7
86.3
79.5
66.0
LoRA + dec.
Mass
–
309.6M (5.3M)
81.6
80.2
85.6
82.1
67.0
full
Mass
–
307.3M (307.3M)
79.8
78.1
87.1
79.0
64.0
Sharingan [ 38 ]
zero-shot
Mass
–
106.1M (0)
76.3
75.2
80.6
75.9
58.0
decoder
Mass
–
106.1M (7.9M)
76.4
75.0
80.6
76.8
57.0
Table 4: Comparison with state-of-the-art methods on the SORBET test set , in identity accuracy (%). The Training column lists the parts fine-tuned on SORBET, all starting from the released checkpoints except MMGaze, which follows its original recipe. For each baseline, we report the best of several configurations. All configurations, including the frozen parts in each setting, are in the supplementary material. S and T denote the speaker and transcript cues, and Output follows the prediction rules in Sec. 5.1 . Frozen Gaze-LLE and LR-ASD are not counted in our parameters.
Figure 4: Qualitative comparison on the SORBET test set. Each column shows the candidate predicted by a method, with an arrow from the observer O (green) to the prediction (yellow). Green and red marks indicate correct and incorrect predictions. We show examples where the gaze following and social gaze models fail and our model succeeds.
Cue
All
Friends
BBT
Superstore
K≥5
base input
83.3
79.8
88.5
87.5
67.0
Conversational
+ speaker
85.2 ( +1.9 )
81.9
89.9
89.3
74.0
+ transcript
83.2 ( − 0.1 )
79.0
89.9
88.0
64.0
+ speaker + transcript
86.7 ( +3.4 )
82.9
92.1
91.5
76.0
Visual
Table 5: Cue ablation on the SORBET test set , in identity accuracy (%). Each row adds cues to the base input, the two images and the gaze prior. Numbers in parentheses are changes from the base input in points. Speaker + transcript is our full model.
Gaze-LLE
S+T
Qwen3-VL
InternVL3.5
Qwen2.5-Omni
✗
✗
76.2
72.7
70.3
✗
✓
81.7 (+5.5)
79.4 (+6.7)
75.3 (+5.0)
✓
✗
83.3 (+7.1)
81.6 (+8.9)
80.6 (+10.3)
✓
✓
86.7 (+10.5)
86.2 (+13.5)
84.6 (+14.3)
Table 6: Effect of the gaze prior and the conversational cues on the SORBET test set , in identity accuracy (%). The Gaze-LLE column marks the gaze prior in the input, and S+T marks the speaker and transcript cues. Numbers in parentheses are gains over the first row, which uses neither, in points. All models are fine-tuned with LoRA under the same setting.
Figure 5: Qualitative cue ablation on the SORBET test set. All columns use Qwen3-VL-8B with the gaze prior (base), and each adds one cue. Green and red marks indicate correct and incorrect predictions. We show examples where the base input fails and our model succeeds.
Method
Train
SORBET
VGS
AVA-LAEO
Chance
–
37.3
44.0
44.3
Gaze-LLE [ 35 ] (Mass)
GazeFollow
79.6
51.5
89.1
MTGS [ 15 ] (LAH)
VSGaze
75.2
40.6
86.7
Ours
SORBET
83.3
48.9
89.4
Ours + S
SORBET
85.2
51.5
90.1
Ours + S + T ASR
SORBET
86.4 (86.7)
52.6
91.0
Table 7: Results on the SORBET, VGS, and AVA-LAEO test sets , in identity accuracy (%). The Train column gives the training data, and the baselines use their released checkpoints. S denotes the speaker cue, and T ASR denotes the Whisper transcript. Numbers in parentheses use human transcripts instead. The test sets have 843, 266, and 322 samples.
While Omni-modal Large Language Models have made strides in joint sensory processing, they fundamentally struggle with a cornerstone of human interaction: deciphering complex, multi-person conversational dynamics to accurately answer Who said what and when.'' Current models suffer from an illusion of competence'' -- they exploit visual biases in conventional benchmarks to bypass genuine cross-modal alignment, while relying on sparse, low-frame-rate visual sampling that destroys crucial high-frequency dynamics like lip movements. To shatter this illusion, we introduce Visual-Registered Speaker Diarization and Recognition (VR-SDR) and the HumanOmni-Speaker Benchmark. By strictly eliminating visual shortcuts, this rigorous paradigm demands true end-to-end spatio-temporal identity binding using only natural language queries. To overcome the underlying architectural perception gap, we propose HumanOmni-Speaker, powered by a Visual Delta Encoder. By sampling raw video at 25 fps and explicitly compressing inter-frame motion residuals into just 6 tokens per frame, it captures fine-grained visemes and speaker trajectories without triggering a catastrophic token explosion. Ultimately, HumanOmni-Speaker demonstrates strong multimodal synergy, natively enabling end-to-end lip-reading and high-precision spatial localization without intrusive cropping, and achieving superior performance across a wide spectrum of speaker-centric tasks.
Detao Bai, Zhiheng Ma, Xihan Wei
Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology · Guangdong Provincial Key Laboratory of Computility Microelectronics +1
Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.
Seunghyun Oh, Hirotaka Hiraki, Shuyue Stella Li +2
Paul G. Allen School of Computer Science and Engineering, University of Washington, WA, USA · Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Bunkyo, Tokyo, Japan
Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.
Haotian Qi, Gabriel Skantze
Department of Speech Music and Hearing, KTH Stockholm, Sweden