From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation
Authors: Shayan Nasiriboukani, Sara Atito, Mohammad Nezamipour, Muhammad Awais
Organizations: Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK · Surrey Institute for People-Centred AI, University of Surrey, UK
Understanding human attention is fundamental for scene interpretation, yet existing approaches often rely on heavily trained models that lack interpretability. Prior methods struggle to jointly reason about gaze targets, attended objects, and visual grounding without extensive supervision. To the best of our knowledge, this work introduces the first training-free Gaze Target Agent (GTA) for gaze-guided reasoning across tasks such as gaze target prediction, attention localization, and object identification. This is achieved by leveraging pretrained vision-language models, augmenting them with visually guided prompts, and employing a memory-based retrieval strategy for high-uncertainty samples to improve performance without additional training. We evaluate our approach using both quantitative metrics and qualitative results. Quantitatively, our method achieves state of the art performance on the GazeFollow and GazeHOI benchmarks. Qualitatively, our agent provides detailed semantic predictions, predicts the correct targets even when ground truth labels are wrong, and remains flexible without vocabulary constraints.
Figures & tables
Figure 1: Comparison with prior approaches: prior methods predict gaze heatmaps indicating where a person is looking, whereas our approach not only estimates gaze but also grounds the attended object and identifies it.
Figure 2: Overview of the proposed gaze-target reasoning agent. Given a raw image, the agent first detects head bounding boxes and estimates the gaze direction of each person to infer where they are looking. The predicted gaze is converted into a visual prompt that highlights the attended region and is provided to a vision language model (VLM). The VLM predicts the object being observed and outputs an uncertainty score. When the prediction uncertainty is high, a retrieval augmented generation (RAG) module retrieves visually similar images and their labels from a memory bank and feeds them, together with the query image, back to the VLM to improve reasoning. Finally, a grounding module localizes the predicted object in the image.
Table 1: Performance comparison on GazeFollow and GazeHOI. We compare GGVL and the proposed GTA against the previous state-of-the-art (SOTA). For GazeFollow , we report Acc@1, Acc@3, and MultiAcc@1, while for GazeHOI we report Acc@1, Acc@3, AP@50, and mIoU. Grey marks the previous SOTA, while blue highlights GTA. Here, I and V denote image and vocabulary, respectively.
Method
GazeHOI
GazeFollow
Acc@1 ↑
Acc@3 ↑
Acc@1 ↑
Acc@3 ↑
MultiAcc@1 ↑
Public
Qwen3 [2b (GGVL)]
0.517
0.721
0.450
0.684
0.506
Qwen3 [2b (GTA)]
0.545
0.745
0.488
0.717
0.550
Llava 1.5 [4b (GGVL)]
0.528
0.691
0.380
0.630
0.437
Llava 1.5 [4b (GTA)]
0.572
0.750
0.466
0.700
0.535
Table 2: Ablation study on GazeFollow and GazeHOI comparing GGVL and the proposed GTA across different public and private VLM backbones. Results are reported under identical settings to show the effect of reasoning across model scales and architectures.
Threshold
Acc@1 ↑
Acc@3 ↑
MultiAcc@1 ↑
0
42.64
71.25
49.23
15
47.85
71.77
54.33
25
48.49
71.83
54.77
35
48.47
71.54
54.75
45
47.76
70.85
53.97
55
47.09
70.03
53.09
Table 3: Uncertainty threshold ablation. We compare fixed thresholds with the dynamic uncertainty threshold used in GTA.
Prompt
Acc@1 ↑
Acc@3 ↑
MultiAcc@1 ↑
Head bbox
28.69
67.57
34.36
Blue dot
40.99
66.90
47.49
Gray scale
38.39
69.72
44.42
Blur
38.98
67.75
44.92
Red arrow
48.29
72.08
54.45
Green arrow
43.91
71.98
50.17
Table 4: Visual prompt ablation. We compare different prompt types, arrow colors, and arrow widths. The color arrows without a stated width use 4 px.
k
Acc@1 ↑
Acc@3 ↑
MultiAcc@1 ↑
3
48.68
72.75
55.02
4
48.56
72.69
54.89
5
49.21
72.81
55.58
6
49.29
72.81
55.79
10
49.04
72.27
55.37
Table 5: Retrieval-size ablation. We vary the number of retrieved examples used during retrieval-based refinement.
Vocabulary
Acc@1 ↑
Acc@3 ↑
MultiAcc@1 ↑
E5 Acc@1 ↑
E5 Acc@3 ↑
Fixed + GGVL
45.75
69.78
51.48
48.43
72.40
Fixed + GTA
49.29
72.81
55.79
51.48
76.29
Open + GGVL
30.84
43.75
35.59
40.11
57.28
Open + GTA
40.63
54.27
46.07
47.85
64.62
Table 6: Vocabulary-setting ablation. We compare fixed and open-vocabulary evaluation for GGVL and GTA using both exact matching and E5 semantic matching.
Figure 3: Examples illustrate GTA’s semantic reasoning and robustness to annotation errors. The ground-truth label and top-3 model predictions are shown for each image.
Figure 4: Examples where GTA is not restricted to a fixed vocabulary. The predicted labels frequently provide more precise descriptions than the ground truth annotations.