Identifying whom a person looks at is central to understanding multi-party conversations, yet gaze geometry alone becomes ambiguous as more people appear. Conversations offer cues that geometry lacks, such as who is speaking and what is said, but no gaze dataset provides them for many multi-party scenes. We introduce SORBET, a benchmark of 3,340 samples from 1,578 clips with synchronized speech, transcripts, and attended person labels, covering more multi-party scenes than existing gaze datasets. Prior gaze models struggle to use these cues, which come in different forms, must be linked to the right person, and can conflict with gaze geometry. We propose Nunchi, a framework that constructs a text prompt from the transcript and the outputs of gaze and speaker models, links every cue to the candidates with shared labels, and lets a vision-language model (VLM) weigh gaze geometry against conversational cues to identify the attended person. Nunchi outperforms the strongest existing method by 4.8 percentage points. On SORBET, conversational cues help whereas additional visual cues do not, and the gains hold across three VLMs, with automatic transcripts, and on two existing datasets.
Figures & tables
Figure 1: Attended person identification in a multi-party conversation. As more people appear in a scene, several candidates lie in a similar direction from the observer, so gaze geometry alone cannot determine whom the observer looks at. Gaze-LLE [ 35 ] and MTGS [ 15 ] each select an incorrect candidate. Nunchi combines gaze geometry with conversational cues, the speaker and the transcript, and identifies the attended person C2 .
K=2
K=3 – 4
K≥5
Dataset
G
M
G
M
G
M
VAT †
83.9
85.1
80.2
77.0
64.3
63.8
ChildPlay †
87.7
86.9
81.3
79.2
57.1
67.3
UCO-LAEO †
100.0
97.4
96.2
100.0
–
–
VGS
79.3
75.9
55.9
58.8
–
–
Table 1: Effect of the number of people on Gaze-LLE (G) and MTGS (M) , in identity accuracy (%). Both models use their released checkpoints. K is the number of annotated people other than the observer. † marks datasets used to train MTGS.
K≥3 (test)
Dataset
Video
Audio
Transcript
Clips
Samples
GazeFollow [ 33 ]
✗
✗
✗
–
580
VAT [ 8 ]
✓
✗
✗
32
383
ChildPlay [ 37 ]
✓
✗
✗
9
97
UCO-LAEO [ 24 ]
✓
✓
✗
5
26
AVA-LAEO [ 24 ]
✓
✗
✗
10 ∗
110 ∗
Table 2: Comparison with existing gaze following and social gaze datasets. Audio and Transcript mark synchronized audio and transcripts released with the dataset. K≥3 counts test clips and samples with three or more candidates. As adjacent video frames are nearly identical, we sample video datasets every 25 frames, about once per second, and SORBET at keyframes. ∗ No test split, so we count the validation movies.
Figure 2: Construction of SORBET. From two multi-party conversation datasets, we keep frames with at least three identified people and a visible speaker, and take the middle frame of each segment of such frames as a keyframe. Each identified person serves in turn as the observer, and the annotator labels what the observer looks at. SORBET keeps samples whose gaze target is a candidate.
Series
Split
Episode
Sample
K=2
K=3
K=4
K≥5
Spk. (%)
Friends
train
108
1,118
328
540
197
53
49.6
val
42
485
143
278
60
4
48.5
test
27
480
108
220
74
78
50.8
BBT
train
16
219
134
65
20
0
64.8
val
2
48
35
13
0
0
58.3
test
5
139
94
40
5
0
72.7
Table 3: SORBET splits . Splits are disjoint at the episode level. K is the number of candidates per sample. Spk. is the percentage of samples whose target is the current speaker.
Figure 3: Overview of Nunchi. Cue extraction obtains gaze and speaker information with frozen experts, along with the transcript. Prompt construction writes them as text, and every cue refers to each candidate by the label Ck drawn in the keyframe. Candidate scoring reads the two images and the prompt with a single VLM and scores each candidate by the logit of its label. In this example, the gaze prior and the speaker cue point to different candidates, and Nunchi selects the attended person C2 . Colors mark the gaze prior, speaker, and transcript paths, and ✓/✗ mark agreement with the ground truth, which the model does not see.
Model
Training
Output
Cue
Params (trainable)
All
Friends
BBT
Superstore
K≥5
Gaze following
Gaze-LLE [ 35 ]
zero-shot
Mass
–
307.3M (0)
79.6
77.5
86.3
79.9
64.0
decoder
Mass
–
307.3M (2.9M)
81.9
81.7
86.3
79.5
66.0
LoRA + dec.
Mass
–
309.6M (5.3M)
81.6
80.2
85.6
82.1
67.0
full
Mass
–
307.3M (307.3M)
79.8
78.1
87.1
79.0
64.0
Sharingan [ 38 ]
zero-shot
Mass
–
106.1M (0)
76.3
75.2
80.6
75.9
58.0
decoder
Mass
–
106.1M (7.9M)
76.4
75.0
80.6
76.8
57.0
Table 4: Comparison with state-of-the-art methods on the SORBET test set , in identity accuracy (%). The Training column lists the parts fine-tuned on SORBET, all starting from the released checkpoints except MMGaze, which follows its original recipe. For each baseline, we report the best of several configurations. All configurations, including the frozen parts in each setting, are in the supplementary material. S and T denote the speaker and transcript cues, and Output follows the prediction rules in Sec. 5.1 . Frozen Gaze-LLE and LR-ASD are not counted in our parameters.
Figure 4: Qualitative comparison on the SORBET test set. Each column shows the candidate predicted by a method, with an arrow from the observer O (green) to the prediction (yellow). Green and red marks indicate correct and incorrect predictions. We show examples where the gaze following and social gaze models fail and our model succeeds.
Cue
All
Friends
BBT
Superstore
K≥5
base input
83.3
79.8
88.5
87.5
67.0
Conversational
+ speaker
85.2 ( +1.9 )
81.9
89.9
89.3
74.0
+ transcript
83.2 ( − 0.1 )
79.0
89.9
88.0
64.0
+ speaker + transcript
86.7 ( +3.4 )
82.9
92.1
91.5
76.0
Visual
Table 5: Cue ablation on the SORBET test set , in identity accuracy (%). Each row adds cues to the base input, the two images and the gaze prior. Numbers in parentheses are changes from the base input in points. Speaker + transcript is our full model.
Gaze-LLE
S+T
Qwen3-VL
InternVL3.5
Qwen2.5-Omni
✗
✗
76.2
72.7
70.3
✗
✓
81.7 (+5.5)
79.4 (+6.7)
75.3 (+5.0)
✓
✗
83.3 (+7.1)
81.6 (+8.9)
80.6 (+10.3)
✓
✓
86.7 (+10.5)
86.2 (+13.5)
84.6 (+14.3)
Table 6: Effect of the gaze prior and the conversational cues on the SORBET test set , in identity accuracy (%). The Gaze-LLE column marks the gaze prior in the input, and S+T marks the speaker and transcript cues. Numbers in parentheses are gains over the first row, which uses neither, in points. All models are fine-tuned with LoRA under the same setting.
Figure 5: Qualitative cue ablation on the SORBET test set. All columns use Qwen3-VL-8B with the gaze prior (base), and each adds one cue. Green and red marks indicate correct and incorrect predictions. We show examples where the base input fails and our model succeeds.
Method
Train
SORBET
VGS
AVA-LAEO
Chance
–
37.3
44.0
44.3
Gaze-LLE [ 35 ] (Mass)
GazeFollow
79.6
51.5
89.1
MTGS [ 15 ] (LAH)
VSGaze
75.2
40.6
86.7
Ours
SORBET
83.3
48.9
89.4
Ours + S
SORBET
85.2
51.5
90.1
Ours + S + T ASR
SORBET
86.4 (86.7)
52.6
91.0
Table 7: Results on the SORBET, VGS, and AVA-LAEO test sets , in identity accuracy (%). The Train column gives the training data, and the baselines use their released checkpoints. S denotes the speaker cue, and T ASR denotes the Whisper transcript. Numbers in parentheses use human transcripts instead. The test sets have 843, 266, and 322 samples.
Paul G. Allen School of Computer Science and Engineering, University of Washington, WA, USA · Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Bunkyo, Tokyo, Japan