End-to-End Shared Attention Estimation via Group Detection with Feedback Refinement

Chihiro Nakatani; Norimichi Ukita; Jean-Marc Odobez

cs.CVApr 2, 2026

End-to-End Shared Attention Estimation via Group Detection with Feedback Refinement

Authors: Chihiro Nakatani, Norimichi Ukita, Jean-Marc Odobez

Organizations: Toyota Technological Institute, Japan · Idiap Research Institute, Switzerland · ´Ecole Polytechnique F´ed´erale de Lausanne, Switzerland

Abstract

This paper proposes an end-to-end shared attention estimation method via group detection. Most previous methods estimate shared attention (SA) without detecting the actual group of people focusing on it, or assume that there is a single SA point in a given image. These issues limit the applicability of SA detection in practice and impact performance. To address them, we propose to simultaneously achieve group detection and shared attention estimation using a two step process: (i) the generation of SA heatmaps relying on individual gaze attention heatmaps and group membership scalars estimated in a group inference; (ii) a refinement of the initial group memberships allowing to account for the initial SA heatmaps, and the final prediction of the SA heatmap. Experiments demonstrate that our method outperforms other methods in group detection and shared attention estimation. Additional analyses validate the effectiveness of the proposed components. Code: https://github.com/chihina/sagd-CVPRW2026.

Figures & tables

Figure 1: Difference between previous and our shared attention estimation methods. (a) Shared attention estimation using simple integration of the individual attention of all people in the scene as post-processing. (b) Direct shared attention estimation without group detection. (c) Our shared attention estimation via group detection, in which shared attention is estimated by integration of individual attention based on detected groups.

Figure 2: Overview of our network. First, the individual attention heatmaps $A$ are estimated for each person. They are then exploited to derive group memberships per each group token $\bm{M}$ and integrated to infer the shared attention heatmap associated with each group token $\bm{S}$ . Both the group memberships and shared attention heatmaps are further refined in a second step.

Figure 3: Membership-based shared attention heatmap estimation. Individual attentions heatmaps are integrated by weighting them using the group membership obtained in the group detection step.

Figure 4: Overview of group detection. The dot product between the

e

-th updated group token and the

n

-th person token is computed as a group membership coefficient (i.e.,

\bm{M}_{e,n}

Figure 5: Overview of refined group detection in which spatial argmax is applied to the initial SA heatmaps

\bm{S}

for refinement.

	GroupAP
	$\theta_{\mathrm{IoU}}$ =0.5			$\theta_{\mathrm{IoU}}=1.0$
Method	$\theta_{\mathrm{Dist}}=0.05$	$\theta_{\mathrm{Dist}}=0.1$	$\theta_{\mathrm{Dist}}=\infty$	$\theta_{\mathrm{Dist}}=0.05$	$\theta_{\mathrm{Dist}}=0.1$	$\theta_{\mathrm{Dist}}=\infty$
MTGS-PP [ 9 ]	16.4	20.7	37.5	7.1	8.6	11.3
MTGS-Soc. [ 9 ]	5.7	7.8	28.6	2.9	3.6	9.8
Gaze-LLE-PP [ 23 ]	15.6	19.5	26.3	5.7	6.9	8.2
Ours	32.4	41.0	61.7	12.5	15.9	17.7

Table 1: Comparison of previous and our methods on the VideoCoAtt dataset. The best result in each column is colored by red . Regarding the group detection,

\theta_{\mathrm{IoU}}=0.5

and

\theta_{\mathrm{IoU}}=1.0

are used for lenient and strict criteria. For the shared attention estimation,

\theta_{\mathrm{Dist}}=0.05

\theta_{\mathrm{Dist}}=0.1

, and

\theta_{\mathrm{Dist}}=\infty

are used for evaluation.

\theta_{\mathrm{Dist}}=0.05

and

\theta_{\mathrm{Dist}}=0.1

judge whether estimated shared attention is located within a pre-defined distance threshold from ground-truth shared attentions.

\theta_{\mathrm{Dist}}=\infty

is used to only evaluate group detection while ignoring shared attention estimation errors.

Figure 6: Visual comparison on the VideoCoAtt dataset. People joining the same group are enclosed within a rectangle of the same color. The estimated shared attention point (i.e., the peak of the estimated shared attention heatmap) for each group is visualized as each group’s color point. Due to space limitations, we compare our method with MTGS-PP, having the second-best performance in Table 1 .

	GroupAP
	$\theta_{\mathrm{IoU}}$ =0.5			$\theta_{\mathrm{IoU}}=1.0$
Method	$\theta_{\mathrm{Dist}}=0.05$	$\theta_{\mathrm{Dist}}=0.1$	$\theta_{\mathrm{Dist}}=\infty$	$\theta_{\mathrm{Dist}}=0.05$	$\theta_{\mathrm{Dist}}=0.1$	$\theta_{\mathrm{Dist}}=\infty$
MTGS-PP [ 9 ]	30.3	35.6	66.0	5.1	6.3	9.0
MTGS-Soc. [ 9 ]	7.6	9.9	37.2	1.5	1.9	4.7
Gaze-LLE-PP [ 23 ]	7.5	8.5	12.4	2.9	3.0	3.3
Ours	27.2	33.5	52.0	7.5	10.1	12.0

Table 2: Comparison of previous and our methods on the VideoAttentionTarget dataset. The best result in each column is colored by red .

Figure 7: Visual comparison on the VideoAttentionTarget dataset.

	GroupAP
	$\theta_{\mathrm{IoU}}$ =0.5			$\theta_{\mathrm{IoU}}=1.0$
Method	$\theta_{\mathrm{Dist}}=0.05$	$\theta_{\mathrm{Dist}}=0.1$	$\theta_{\mathrm{Dist}}=\infty$	$\theta_{\mathrm{Dist}}=0.05$	$\theta_{\mathrm{Dist}}=0.1$	$\theta_{\mathrm{Dist}}=\infty$
MTGS-PP [ 9 ]	7.8	13.9	25.4	4.6	5.2	5.6
MTGS-Soc. [ 9 ]	0.8	1.8	9.1	0.7	0.7	1.3
Gaze-LLE-PP [ 23 ]	5.8	8.1	14.1	2.4	2.8	4.4
Ours	9.0	15.6	36.3	2.1	2.1	2.4

Table 3: Comparison of previous and our methods on the ChildPlay dataset. The best result in each column is colored by red .

Figure 8: Failure cases on the ChildPlay dataset.

	GroupAP
	$\theta_{\mathrm{IoU}}$ =1.0
Method	$\theta_{\mathrm{Dist}}=0.05$	$\theta_{\mathrm{Dist}}=0.1$	$\theta_{\mathrm{Dist}}=\infty$
w/o refinement	2.5	3.3	3.6
w/o $\mathcal{L}_{social}$	0.9	1.6	1.9
Ours (Full)	12.5	15.9	17.7

Table 4: Ablation study on the VideoCoAtt dataset.

Figure 9: Effectiveness of refined shared attention estimation (Sec. 3.3 ) on the VideoCoAtt dataset.

Visual Attention

Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.

See all 61 papers

PAGE: Towards Practical Human-level Gaze Target Estimation
Jul 6, 2026Zhoutong Ye, Chengwen Zhang, Zhaibin Cui +10Gaze Estimation
1Tsinghua University · 2Jinan University
TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues
Jul 11, 2026Junhui She, Fei Wang, Kun Li +4Gaze Estimation
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · 2Hefei University of Technology +2
SGAP-Gaze: Scene Grid Attention Based Point-of-Gaze Estimation Network for Driver Gaze
Apr 21, 2026Pavan Kumar Sharma, Pranamesh ChakrabortyGaze Estimation Autonomous Driving
Department of Civil Engineering, Indian Institute of Technology Kanpur, Kanpur-208016, India

Jul 6, 2026J/K move · Enter open · S save

PAGE: Towards Practical Human-level Gaze Target Estimation

Zhoutong Ye, Chengwen Zhang, Zhaibin Cui, Mingze Sun+9

1Tsinghua University · 2Jinan University

Gaze Estimation

Open paper AskPDF arXiv

Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code and model checkpoints are available at our project page.

Abstract

Visual Attention

Explore similar work

PAGE: Towards Practical Human-level Gaze Target Estimation

TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues

SGAP-Gaze: Scene Grid Attention Based Point-of-Gaze Estimation Network for Driver Gaze