Gaze-Regularized VLMs for Ego-Centric Behavior Understanding

Anupam Pani; Yanchao Yang

Figure 1: Illustration of future event prediction. The input is a sequence of image frames, and the output is an action-related text prediction. Unlike the gaze-regularized model, the base model misidentifies the object to be picked up (bowl). Predicted future annotations for both models are shown on the right, with ground-truth annotations and immediate future frames displayed below for reference.

Figure 2: Overview of the Architecture A ViT encoder extracts image features, which are enhanced using gaze-based queries (obtained from gaze/psuedo gaze-overlaid images) in a gaze-regularized attention block. A Perceiver Resampler generates a fixed-size representation for the language module to predict text annotations. The gaze regularizer aligns model attention with human gaze patterns by minimizing the Kullback–Leibler divergence between the model attention distribution and gaze distribution during training.

Figure 3: Heatmap creation. Illustration of gaze data collection for generating gray-scale heatmaps and gaze-overlaid images. On the left, the aggregated gaze model incorporates multiple gaze points collected over the interval

[t-\delta,t]

to generate the heatmap. On the right, the singular gaze model uses a single gaze point collected at time

t

. Both utilize Gaussian smoothing to generate the heatmap.

Figure 4: Future event prediction results for the base model (without gaze) and our gaze-regularized model are presented for an observation horizon (

\tau_{o}

) of 5 seconds. Past frames are omitted, but ground-truth annotations and future frames with a prediction duration (

\tau_{a})

of 2 seconds are provided as references. Keywords from each set of annotations are highlighted for easier reading

Model	Annotation	Semantic	Rouge-L
	Source	score ( $\uparrow$ )	F-score
Base (no gaze)	GPT-4V	0.6525	0.4318
Singular Gaze	GPT-4V	0.7316	0.5054
Aggregated-Gaze	GPT-4V	0.7826	0.5405
Base (no gaze)	ShareCaptioner	0.6437	0.5646
Singular Gaze	ShareCaptioner	0.8212	0.6905

Table 1: Evaluation of base and gaze-regularized models for future prediction with varying annotation granularity (i.e., GPT-4V and ShareCaptioner).

Regularization	Semantic	Rouge-L
Coefficient	Score( $\uparrow$ )	F-score( $\uparrow$ )
0	0.6317	0.3738
100	0.7826	0.5405
1000	0.7798	0.5330

Table 2: Effect of regularization on aggregated gaze-regularized models.

Gaze-based	Semantic	Rouge-L
Queries	Score ( $\uparrow$ )	F-score( $\uparrow$ )
Gaze-Overlay-Dependent	0.7368	0.5034
RGB-Only	0.7505	0.5173
Gaze-Overlay-Supported	0.7826	0.5405
Pseudo-Gaze-Overlay-Supported	0.7704	0.5213

Table 3: Comparison of various gaze-based querying methods.

Model	Semantic	Rouge-L
	Score ( $\uparrow$ )	F-score( $\uparrow$ )
Base	0.6897	0.4832
Gaze-aggregated ( $\lambda$ = 100)	0.6843	0.4816
Gaze-aggregated ( $\lambda$ = 1000)	0.6621	0.4793
Gaze-aggregated ( $\lambda$ = 10)	0.7021	0.5001

Table 4: Comparison between base model and gaze-regularized models (with different regularization coefficients) for activity understanding tasks.

Model	Inputs			Semantic
	RGB	Gaze	Gaze	Score ( $\uparrow$ )
		(Text)	(Image)
\multirow 3*InternVL-2-1B	$\checkmark$			0.1572
	$\checkmark$	$\checkmark$		0.1627
(one-shot)	$\checkmark$		$\checkmark$	0.1621
\multirow 3*InternVL-2-2B	$\checkmark$			0.1601

Table 5: Comparison of semantic scores for other VLM Models with different gaze inputs on our test set.

Figure 5: GPT-4V Iterative Prompting Workflow. The process begins with a sequence of images and an initial prompt, which are input into GPT-4V to generate annotations for each image. The user then evaluates these annotations and provides feedback, which is incorporated into the prompt using a language model. This modified prompt is used to refine the annotations in a continuous cycle. The objective is to improve the quality and relevance of the output with each iteration until the user is satisfied. For GPT-4V, a set of 10 images is provided at once, ensuring that the annotations maintain contextual coherence.

Figure 6: ShareCaptioner Iterative Prompting Workflow. The process begins with a sequence of images accompanied by a starter prompt, which is processed by ShareCaptioner to generate initial annotations. The user then reviews these annotations and provides feedback. This feedback is integrated into the prompt using a language model. The updated prompt is subsequently used to enhance the annotations through an iterative cycle. This cycle continues until the output meets the user’s satisfaction, focusing on improving both quality and relevance. Additionally, a sliding window mechanism is employed to traverse the image sequence, capturing and describing changes as they occur to ensure that all actions are accurately recorded.

Figure 7: Left : Examples of annotations generated for a sample image by three different annotators: ShareCaptioner, Open Llava, and GPT-4V. The annotations from ShareCaptioner and Open Llava are more fine-grained compared to those produced by GPT-4V in our experimental setup. Right : A graph comparing the performance of the base model and gaze-regularized models, illustrating the impact of using text annotations of varying granularity.

Figure 8: An example of how images are processed by the Vision Transformer in our model. In this case, each image is tokenized into 9 patches. Positional embeddings are then added to the patches, which are subsequently passed through an attention module. In this model, the final MLP head is omitted, as we are focused solely on extracting image features for further analysis.

Figure 9: The architecture for the gaze-overlaid supported model which takes in RGB scene image and gaze-overlaid images as input and outputs fine-grained text annotations. The gaze-overlaid supported model derives gaze-based features from the gaze-overlaid images.

Figure 10: Closer look at the gaze-guided attention block in the pipeline. Gaze overlaid images are passed as queries, whereas the corresponding RGB images are passed as key and value. The binary heatmap image is used to obtain the gaze distribution, which is used in the regulariser by comparing it with the attention distribution obtained from the gaze-guided attention block. The regulariser attempts to align the attention distribution more towards the gaze distribution.

Figure 11: An example of gaze aggregation on the final image frame is shown. Bottom left : Result without occlusion check. Bottom right : Result with occlusion check applied. The fridge is absent in the final frame, indicating significant occlusion. Therefore, the gaze points from the first two frames, which are associated with the fridge, should not be overlaid on the final frame as the object is no longer visible.

Figure 12: Formation of pseudo-gaze-overlaid images and gaze-based queries. If gaze-overlaid images are not available during test time, an encoder-decoder network dynamically generates a heatmap which is then superimposed on the RGB scene image to form a pseudo-gaze-overlaid image, which is supervised with the ground truth gaze-overlaid image

Gaze	Semantic	Meteor	Rouge-L ( $\uparrow$ )
Model	Score( $\uparrow$ )	Score( $\uparrow$ )	Precision	Recall	F-score
Singular	0.7316	0.4501	0.4822	0.5309	0.5054
Singular (larger overlays)	0.7683	0.4922	0.5123	0.5508	0.5304
Aggregated	0.7826	0.5033	0.5193	0.5644	0.5405
Aggregated (larger overlays)	0.7816	0.5060	0.5128	0.5556	0.5330

Table 6: Effect of the size of the gaze point overlays on model accuracy

Attention	Semantic	Rouge-L
Blocks	Score ( $\uparrow$ )	F-score ( $\uparrow$ )
1	0.7434	0.5328
2	0.7826	0.5405
5	0.7765	0.5301

Table 7: Effect of the number of gaze-regularized attention blocks on the performance of the aggregated gaze-regularized models.

Aggregated	Semantic	Rouge-L
Gaze Model	Score( $\uparrow$ )	F-score ( $\uparrow$ )
w/o occlusion-check	0.7616	0.5286
with occlusion-check	0.7826	0.5405

Table 8: Comparison of the aggregated gaze-regularized models with and without occlusion check.

Model	Score( $\uparrow$ )	Score( $\uparrow$ )	Precision	Recall	F-score
	Semantic	Meteor	Rouge-L ( $\uparrow$ )
Base	0.6525	0.4075	0.4335	0.4301	0.4318
Base (w self-attention)	0.6701	0.4215	0.4292	0.4508	0.4393
Aggregated gaze	0.7826	0.5033	0.5193	0.5644	0.5405

Table 9: Comparison of base model with attention block against gaze regularized models

Model	Score( $\uparrow$ )	Score( $\uparrow$ )	Precision	Recall	F-score
	Semantic	Meteor	Rouge-L ( $\uparrow$ )
Base	0.6525	0.4075	0.4335	0.4301	0.4318
Aggregated gaze(in text form)	0.7021	0.4428	0.4642	0.4621	0.4630
Aggregated gaze	0.7826	0.5033	0.5193	0.5644	0.5405

Table 10: Effect of using gaze information in text form and comparison with other models

Model	Score( $\uparrow$ )	Score( $\uparrow$ )	Precision	Recall	F-score
	Semantic	Meteor	Rouge-L ( $\uparrow$ )
Base	0.6297	0.3972	0.4393	0.4065	0.4220
Aggregated Gaze	0.7519	0.4891	0.5332	0.5410	0.5386

Table 11: On increasing the prediction horizon (from 2 seconds) to predict future actions up to 5 seconds, the gaze-regularized model is able to outperform the base model.

Model	Score( $\uparrow$ )	Score( $\uparrow$ )	Precision	Recall	F-score
	Semantic	Meteor	Rouge-L ( $\uparrow$ )
Base	0.6716	0.4284	0.4544	0.4627	0.4581
Aggregated Gaze	0.7855	0.5132	0.5399	0.5523	0.5457

Table 12: On decreasing the observation horizon (from 5 to 3 seconds) to predict future actions up to 2 seconds, the gaze-regularized model is again able to outperform the base model.

	Semantic	Rouge-L ( $\uparrow$ )
Model	Score( $\uparrow$	Precision	Recall	F-score
Base w RGB image	0.6525	0.4335	0.4301	0.4318
Base w gaze-overlaid image	0.6873	0.4332	0.4553	0.4435
Aggregated gaze w RGB image	0.7826	0.5193	0.5644	0.5405

Table 13: Comparison for egocentric event prediction when gaze-overlaid images are provided to base VLM.

	Semantic	Rouge-L ( $\uparrow$ )
Model	Score( $\uparrow$	Precision	Recall	F-score
0	0.7826	0.5193	0.5644	0.5405
0.2	0.7801	0.5191	0.5607	0.5390
0.6	0.7757	0.5160	0.5372	0.5264
1	0.7368	0.4958	0.5113	0.5034

Table 14: Comparison for egocentric event prediction when gaze-overlaid images are corrupted

Model	Semantic	RGB	Gaze-Overlaid
	Score( $\uparrow$ )	Input	Input
Base Model	0.6514	$\checkmark$
Aggregated Gaze	0.7425	$\checkmark$	$\checkmark$
Aggregated gaze	0.7005	$\checkmark$
using RGB queries
Pseudo-gaze	0.7215	$\checkmark$

Table 15: Performance of models using out-of-distribution data.

	Semantic	Rouge-L ( $\uparrow$ )
Number of points	Score( $\uparrow$	Precision	Recall	F-score
6	0.7826	0.5193	0.5644	0.5405
12	0.7808	0.5143	0.5541	0.5334

Table 16: Comparison when more aggregated points are used for the gaze-regularized model

Gaze-Regularized VLMs for Ego-Centric Behavior Understanding

Abstract

Figures & tables

Vision-Language Models

Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models

OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following

How Much Future Helps? A Controlled Study of Future-Privileged Supervision for Causal Egocentric Gaze Estimation

Abstract

Vision-Language Models

Explore similar work

Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models

OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following

How Much Future Helps? A Controlled Study of Future-Privileged Supervision for Causal Egocentric Gaze Estimation