Learning to Refer from Estimated Listener Gaze

Téa Wright; Alane Suhr

cs.CLSep 13, 2026

Learning to Refer from Estimated Listener Gaze

Authors: Téa Wright, Alane Suhr

Organizations: Department of Computer Science University of California, Berkeley Berkeley, CA 94704, USA

Abstract

We propose to finetune vision-language models to generate more pragmatically optimal referring expressions by transforming observations of incremental listener comprehension, in the form of gaze scanpaths, into learning signals. During training, referring expressions are sampled from the speaker policy being optimized, conditioned on images and target referents; then, a neural listener estimating human gaze behavior maps from images and sampled referring expressions to scanpaths, each represented by a sequence of fixations, with each fixation corresponding to a word in the referring expression. We experiment with several approaches to convert fixation sequences and target referents into token- and sequence-level rewards, which are used to optimize policy parameters. Through evaluation with human listeners, we find that speaker policies trained with gaze-estimating listeners result in significantly more pragmatically-optimal references than base models, reducing sequence length from 15.4 down to 4.0 words while increasing referential success from 75.2 up to 80.0%. Our work demonstrates a promising opportunity for learning to generate utterances through language-based interaction, not only from the explicit signal of communicative success, but also from implicitly-available observations of a listener's process of comprehension.

Figures & tables

Figure 1: Learning to refer from interaction with incremental listeners. During training, we iterate between (a) deploying a speaker policy $\pi^{\text{S}}_{\theta}$ in a reference game with a listener policy $\pi^{\text{G}}$ , which is trained to estimate gaze behavior of human listeners, and (b) updating the speaker policy by transforming observations of listener gaze behavior to rewards.

Speaker	$d_{\pi^{s}}$ (Human) $\downarrow$	$d_{\pi^{s}}$ (Qwen) $\downarrow$	Acc. $\uparrow$	Length	Time to target (s) $\downarrow$
Molmo	1.41	1.41	75.2	15.4	0 8.22
Gaze-BeforeFirstHit	1.11	0.81	73.3	0 4.2	0 4.86
Gaze-SeqAnyHit	0.88	0.95	80.0	0 9.5	0 7.04
Gaze-SeqLPHit	1.07	1.03	77.7	11.0	0 8.29
Gaze-Shaping	0.99	0.87	75.5	0 4.0	0 5.13
REC-BeforeFirstHit	1.49	0.99	66.8	0 2.0	0 3.98

Table 1: Human evaluation averaged over all datasets with

d_{\pi^{s}}

from Qwen2.5-VL for comparison. Time to target refers to the average time after speech onset when the human evaluators can accurately identify the target by clicking on it.

Figure 2: Comparison of communicative success and average reference length in words with human and Qwen2.5-VL listeners with 95% confidence ellipses.

Figure 3: Examples of generated referring expressions conditioned on an image and bounding box from several different speaker policies. See Appendix D for examples from all speaker models.

Figure 4: Comparison of referential strategies utilized by different speakers. The keywords used for this analysis can be found in Appendix D.1 .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Example of the iterative gaze prediction for gold and finetuned Molmo. The orange sequence represents the human scanpath and the blue is that produced by our gaze predictor model. Each point indicates the location of the gaze and the word heard at the time the fixation was recorded.

Word Index	Molmo Avg Dist	ART Avg Dist
0	4.25	4.43
1	12.17	9.81
2	12.85	12.56
3	13.59	14.08
4	12.10	11.56
5	12.77	13.11

Appendix

Table 2: Average Euclidean distance per word index and total sum of distances, averaged over all gold sequences.

Word Index	Molmo Avg Dist	ART Avg Dist
0	2.04	2.01
1	10.56	6.59
2	8.63	6.54
3	9.92	8.56
4	9.39	7.70
5	11.78	9.99

Appendix

Table 3: Average Euclidean distance per word index and total sum of distances, using the closest gold path (Best Match).

Figure 6: (A) Comparison of length distributions of human and Molmo-generated references, (B) accuracy of the gaze predictor on human-generated (blue) and Molmo-generated references (red) over different reference lengths, and (C) comparison of the proportion of words not assigned fixations.

Speaker	Acc. $\uparrow$	Length	Time (s) $\downarrow$	$d_{\pi^{s}}$ $\downarrow$
RefCOCO testA
Human (Gold)	95.67	0 3.59	0 3.97	0.00
Molmo	84.17	15.15	0 7.99	1.41
REC-BeforeFirstHit	68.50	0 2.01	0 4.23	2.37
REC-SeqAnyHit	93.17	25.27	11.40	1.89
REC-Success	93.50	18.58	0 9.21	1.31

Appendix

Table 4: Full human evaluation results by dataset.

Overall
Speaker	Time (s) $\downarrow$	Reveal Rate $\downarrow$	Early Click Rate $\downarrow$
Human (Gold)	0 3.87	0.03	0.05
Molmo	0 8.22	0.05	0.27
REC-BeforeFirstHit	0 3.98	0.04	0.04
REC-SeqAnyHit	11.26	0.05	0.45
REC-Success	0 8.79	0.03	0.41

Appendix

Table 5: Behavioral efficiency metrics grouped by dataset, including overall averages.

Speaker	Acc. $\uparrow$	Length	$d_{\pi^{s}}$ $\downarrow$
RefCOCO testA
Human (Gold)	94.60	0 3.51	0.00
Molmo	48.20	15.71	1.41
REC-BeforeFirstHit	40.07	0 3.03	1.18
REC-SeqAnyHit	60.20	25.81	1.97
REC-Success	62.85	20.27	1.53

Appendix

Table 6: QwenVL-based evaluation results by dataset.

Speaker	Acc. $\uparrow$	Length	$d_{\pi^{s}}$ $\downarrow$
Overall (Averaged)
Molmo	45.65	16.35	1.41
REC-BeforeFirstHit	46.18	0 3.05	0.99
REC-SeqAnyHit	57.96	25.29	1.84
REC-Success	61.94	19.70	1.41
REC-Shaping	52.12	0 5.17	0.86

Appendix

Table 7: QwenVL-based evaluation results averaged across all datasets.

Figure 7: Comparison of REC results with QwenVL and CogVLM-Grounding.

Figure 8: Comparison of Gaze-BFH training performance with and without the KL penalty. Red shows training with a KL penalty ( $\beta=0.02$ ) and blue shows the ablated training run. For comparison, we include Molmo and human speakers in the rightmost chart.

Speaker	Acc. $\uparrow$	Length	$d_{\pi^{s}}$ $\downarrow$
Overall (Averaged)
Molmo	45.65	16.35	1.41
REC-BeforeFirstHit	46.18	0 3.05	0.99
REC-SeqAnyHit	57.96	25.29	1.84
REC-Success	61.94	19.70	1.41
REC-Shaping	52.12	0 5.17	0.86

Appendix

Table 8: QwenVL-based evaluation comparing standard models against the length-penalized REC-Success variant.

Overall (Averaged)
Speaker	Acc. $\uparrow$	Length	$d_{\pi^{s}}$ $\downarrow$
Molmo	45.65	16.35	1.41
REC-BeforeFirstHit	46.18	0 3.05	0.99
REC-SeqAnyHit	57.96	25.29	1.84
REC-Success	61.94	19.70	1.41
REC-Shaping	52.12	0 5.17	0.86

Appendix

Table 9: Evaluation comparing speakers trained on full incremental gaze versus models predicting only the final gaze point.

Overall (Averaged)
Speaker	Acc. $\uparrow$	Length	$d_{\pi^{s}}$ $\downarrow$
Molmo	45.65	16.35	1.41
REC-BeforeFirstHit	46.18	0 3.05	0.99
REC-SeqAnyHit	57.96	25.29	1.84
REC-Success	61.94	19.70	1.41
REC-Shaping	52.12	0 5.17	0.86

Appendix

Table 10: Performance comparison when training the speaker against a listener finetuned directly on RefCOCO.

Speaker	Acc. $\uparrow$	Length	$d_{\pi^{s}}$ $\downarrow$
Overall (Averaged)
Molmo	45.65	16.35	1.41
REC-BeforeFirstHit	46.18	0 3.05	0.99
REC-SeqAnyHit	57.96	25.29	1.84
REC-Success	61.94	19.70	1.41
REC-Shaping	52.12	0 5.17	0.86

Appendix

Table 11: Comparison of Supervised Finetuning (SFT) baselines and SFT+RL hybrid models against the default RL/Gaze speakers.

Speaker	Acc. $\uparrow$	Length	$d_{\pi^{s}}$ $\downarrow$
Human (Gold)	88.98	0 3.52	0.00
Molmo-7B
Molmo	45.65	16.35	1.41
REC-BeforeFirstHit	46.18	0 3.05	0.99
REC-SeqAnyHit	57.96	25.29	1.84
REC-Success	61.94	19.70	1.41

Appendix

Table 12: Cross-architecture evaluation using LLaVA-1.5-7B and PaliGemma-3B as speaker backbones.

Figure 9: Mention frequency of the red bounding box across speaker models.

Figure 10: Analysis of ambiguity and incorrect reference rates across four selected speaker models.

Figure 11: Examples of generated references from all speaker models.

Speaker Model	Gaze Retention Rate (%)
REC-Success	100.0%
REC-BFH	90.6%
REC-Shaping	90.4%
REC-SeqAnyHit	85.7%
Gaze-Shaping	54.3%
Gaze-SeqLPHit	52.8%

Appendix

Table 13: Gaze trajectory retention metrics across speaker model configurations, demonstrating post-hit retention across reward setups.

Speaker Model	Mid-Ref Period Rate (%) $\downarrow$	Double Space Rate (%) $\downarrow$	Mean Periods
Molmo	18.58	0 0.00	1.22
REC-BeforeFirstHit	0 0.78	0 0.00	0.02
REC-Shaping	0 6.93	53.17	0.84
REC-Success	17.12	0 0.00	1.20
REC-SeqAnyHit	39.31	0 0.00	1.46
Gaze-BeforeFirstHit	0 0.52	0 0.00	0.04

Appendix

Table 14: Frequency of mid-reference period punctuation, double spacing, and mean period counts across speaker model configurations.

Speaker Model	Coordinate Occurrence Rate (%) $\downarrow$
Molmo	0.00%
REC-BeforeFirstHit	0.78%
REC-Shaping	0.02%
REC-Success	0.00%
REC-SeqAnyHit	0.01%
Gaze-BeforeFirstHit	0.00%

Appendix

Table 15: Frequency of coordinate artifact occurrences across speaker model outputs.

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 424 papers

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation
Aug 30, 2026Alvin Wei Ming Tan, Ben Prystawski, Veronica BoycePragmatic Reasoning in Language Models VLM Evaluation
Department of Psychology, Stanford University, USA · Department of Brain and Cognitive Sciences, MIT, USA
Object Referring-Guided Scanpath Prediction with Perception-Enhanced Vision-Language Models
Apr 22, 2026Rong Quan, Yantao Lai, Dong Liang +1Scanpath Prediction
Nanjing University of Aeronautics and Astronautics Nanjing, China
Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention
Jul 27, 2026Sounak Mondal, Dimitris Samaras, Gregory Zelinsky +1Grounded Text Generation
Stony Brook University, Stony Brook, New York, USA · Australian Institute for Machine Learning, Adelaide University, Australia

Aug 30, 2026J/K move · Enter open · S save

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation

Alvin Wei Ming Tan, Ben Prystawski, Veronica Boyce

Department of Psychology, Stanford University, USA · Department of Brain and Cognitive Sciences, MIT, USA

Pragmatic Reasoning in Language Models VLM Evaluation VLM Reasoning Referring Expression Comprehension

Open paper AskPDF arXiv

Flexible adaptation to context and shared pragmatic intuitions contribute to smooth human conversation. Iterated reference games---in which players repeatedly pick out novel referents using language---present a test case for agents' ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision--language models on their ability to identify the intended meaning of descriptions produced in iterated reference games, varying the provided context in terms of amount, order, and relevance. While humans performed well consistently, the models we evaluated could make use of prior context to interpret humans' referring expressions, but they struggled to build up the relevant context to interpret those expressions effectively. Our results suggest that the models we evaluated lack core skills needed for efficient linguistic collaboration.

Abstract

Text Generation

Explore similar work

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation

Object Referring-Guided Scanpath Prediction with Perception-Enhanced Vision-Language Models

Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention