Gaze following aims to predict where a person is looking in a scene, by predicting the target location, or indicating that the target is located outside the image. Recent works detect the gaze target by training a heatmap regression task with a pixel-wise mean-square error (MSE) loss, while formulating the in/out prediction task as a binary classification task. This training formulation puts a strict, pixel-level constraint in higher resolution on the single annotation available in training, and does not consider annotation variance and the correlation between the two subtasks. To address these issues, we introduce the patch distribution prediction (PDP) method. We replace the in/out prediction branch in previous models with the PDP branch, by predicting a patch-level gaze distribution that also considers the outside cases. Experiments show that our model regularizes the MSE loss by predicting better heatmap distributions on images with larger annotation variances, meanwhile bridging the gap between the target prediction and in/out prediction subtasks, showing a significant improvement in performance on both subtasks on public gaze following datasets. Code is available at: https://github.com/qiaomu-miao/gazefollowing_pdp
Figures & tables
Figure 1: Variance of gaze target annotations. (a) Annotations from multiple annotators on example images in the GazeFollow [ 33 ] test set. Each red dot is the annotation from one annotator. Note that annotators often disagree on the exact gaze target. (b) The distribution of annotation variance for images in the GazeFollow test set. Please refer to Sec. 4.4.2 for the calculation of the variance score.
Figure 2: Visualizations of the ground truth patch distributions created from different images. Note that our method can create various patterns of distributions, compared to the Gaussian ground truth which is always circular.
Figure 3: Overall structure of our model. The feature extraction module extracts feature encoding fenc from the input, which is then flattened in the spatial dimension and concatenated with an added ‘outside token’. The tokens go through the patch attention and temporal attention modules for information aggregation. Finally, the ‘inside tokens’ are regrouped in the spatial dimension for heatmap prediction, while all tokens go into another patch prediction head for PDP.
Figure 4: Details of the (a) patch attention module and (b) temporal attention module.
Figure 5: Procedure of Ground truth PD creation.
Method
Dep.
AUC ↑
Dist. ↓
Avg.
Min.
Random [ 33 ]
×
0.504
0.484
0.391
Center [ 33 ]
×
0.633
0.313
0.230
Fixed Bias [ 33 ]
×
0.674
0.306
0.219
GazeFollow [ 33 ]
×
0.878
0.190
0.113
Chong et al. [ 7 ]
×
0.896
0.187
0.112
Table 1: Evaluation of the spatial model part on GazeFollow dataset. Best numbers are marked as bold and 2nd best are underlined. Dep. indicate depth input.
Method
Dep.
In frame
Out of frame
AUC ↑
Dist ↓
AP ↑
Random [ 8 ]
×
0.505
0.458
0.621
Fixed Bias [ 8 ]
×
0.728
0.326
0.624
Chong et al. [ 7 ]
×
0.830
0.193
0.705
VideoAtt [ 8 ]
×
0.860
0.134
0.853
VideoAtt_depth
✓
0.911
0.118
0.861
Table 2: Evaluation of the full model on VideoAttentionTarget dataset.
Figure 6: Visualizations of output of our model on GazeFollow (1st and 2nd rows) and VideoAttentionTarget dataset (3rd row). Ground truth annoatations are visualized in red and the predicted point is visualized in yellow.
Method
GazeFollow
VideoAttentionTarget
AUC ↑
Dist. ↓
In frame
Out of frame
Avg.
Min.
AUC ↑
Dist ↓
AP ↑
No patch pred
0.927
0.138
0.077
0.889
0.130
0.885
KL → MSE
0.928
0.131
0.072
0.908
0.118
0.892
KL → BCE
0.931
0.127
0.067
0.904
0.116
0.890
No patch att
0.929
0.125
0.066
0.902
0.119
0.900
Table 3: Ablation Study Results
Figure 7: Comparison of AUC in each quantile interval in the GazeFollow test set. Our model shows obviously higher AUC in intervals with larger annotation variance, which cannot be observed in (c) when adding depth input only.
Figure 8: Predicted heatmaps from our model and VideoAtt_depth on images with larger “variance scores”. The predicted (yellow) and ground truth (red) target points are also plotted. Our predicted heatmaps are more aligned with the group-level annotations.
Method
AUC ↑
Dist. ↓
Avg.
Min.
VideoAtt [ 8 ]
0.921
0.137
0.077
VideoAtt w. in/out
0.921
0.147
0.083
VideoAtt_depth
0.927
0.131
0.071
VideoAtt_depth w. in/out
0.927
0.145
0.083
Ours w.o. dep.
0.928
0.131
0.072
Table 4: Effect of the In/out Prediction Task on Heatmap Prediction Task for VideoAtt model on GazeFollow dataset
Figure S1 : Structure of the feature extraction module. ⊗ indicates element-wise multiplication. Please see text for details.
Method
GazeFollow
VideoAttentionTarget
AUC ↑
Dist. ↓
In frame
Out of frame
Avg.
Min.
AUC ↑
Dist ↓
AP ↑
Patch: 4×4
0.921
0.129
0.071
0.911
0.110
0.897
Patch: 14×14
0.930
0.127
0.067
0.908
0.113
0.892
MaxPool → AvgPool
0.931
0.124
0.066
0.913
0.112
0.903
One-hot
0.925
0.128
0.071
0.903
0.110
0.884
Table S1 : Ablations of Patch Distribution Creation Method
Figure S2 : Visualizations of patch distributions generated in different ways from a sampled ground truth annotation (blue) on an example image in the GazeFollow test set. Annotations from other annotators are visualized in red. In the case of an annotation point being close to the patch boundary, the one-hot design simply regards the neighboring patches as unattended, while our max pooling method can generate much better distribution with high response in different patches, encouraging the model to predict multi-modal heatmaps in inference.
Dataset
Bhat. Coef. ↑
JS Div. ↓
GazeFollow
0.976
0.142
VideoAttentionTarget
0.971
0.158
Table S2 : Analysis of Consistencies between Patch and Heatmap Predictions.
Figure S3 : Visualizations of the output heatmaps of our model and the VideoAtt model with and without a depth map as input. The left two columns are the model predictions without using a depth map as input, and the right two columns are the model predictions using a depth map as input. Predicted targets are plotted in yellow and the ground truth annotations are plotted in red. The average annotation is plotted in blue.
Figure S4 : Example failure cases of our model on the GazeFollow (row 1 & 2) and VideoAttentionTarget (row 3 & 4) datasets. Predicted targets are plotted in yellow and the ground truth annotations are plotted in red. The average annotations for images in the GazeFollow dataset are plotted in blue