G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding

Marko Haralović; Akash Ramakrishnan; Estefania Talavera Martinez

cs.CVAug 20, 2026

G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding

Authors: Marko Haralović, Akash Ramakrishnan, Estefania Talavera Martinez

Organizations: University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands

Abstract

Egocentric action understanding is often addressed using large video models pretrained on extensive exocentric datasets. However, many first-person actions depend on a small number of hand-object interactions involving only a few relevant entities. We propose G3Ego, a graph-based framework for egocentric action understanding that uses gaze as a structural cue to identify action-relevant entities in the scene. From sparsely sampled frames, G3Ego constructs action scene graphs from vision-language descriptions, grounded objects, and hand cues, and then prunes irrelevant entities using the camera wearer's gaze. The resulting graph embeddings are temporally aggregated for action recognition and anticipation. Unlike prior work that uses gaze primarily as an auxiliary modality or attention signal, G3Ego incorporates gaze directly into graph construction, producing efficient and interpretable representations focused on action-relevant interactions. Experiments on EGTEA Gaze+ and MECCANO show that G3Ego achieves competitive performance compared with video-based approaches and consistently improves Macro-F1 under class-imbalanced evaluation, while avoiding reliance on computationally expensive video pretraining. These results demonstrate the effectiveness of gaze-guided graph representations for egocentric action understanding.

Figures & tables

Figure 1: Motivational overview of gaze-guided graph construction. Given an input egocentric frame and the camera wearer’s gaze coordinates, a vision-language model generates a caption describing the camera wearer’s interactions with the scene. Gaze is used to prune the caption, retaining only action-relevant objects. In this example, the gaze falls on the piece of bread. An action scene graph is then constructed using the global image descriptors from the image encoder as node features for the main verb, and local descriptors from the object detector, including bounding-box features, for the retained objects.

Figure 2: Overview of the proposed G3Ego framework. Given a clip of $N$ frames, we extract global, object, and hand features, construct a graph of the camera wearer’s interactions with the scene, and prune it using gaze. The resulting graph descriptors are temporally aggregated for action recognition.

Graph Type	Temporal Modeling	N Frames	Global	Object	Gaze Pruning	Hand	Top-1	Top-5	Macro F1
Effect of Object Features
FG	LSTM	10	✓	✗	✗	✗	25.58	60.75	8.29
FG	LSTM	10	✓	✓	✗	✗	37.34	70.39	8.61
Effect of Gaze-Guided Graph Pruning
FG	LSTM	10	✓	✓	✗	✗	37.34	70.39	8.61
G3Ego	LSTM	10	✓	✓	✓	✗	37.90	70.88	10.63

Table 1: Component-wise ablation of G3Ego on MECCANO. Global, Object, and Hand denote DINOv3 frame features, Grounding-DINO object-region features, and a 20-dimensional hand/hand–object interaction descriptor, respectively. Gaze is used only for graph pruning, not as a recognition feature. FG denotes the full graph. Macro-F1 is the primary selection metric.

Method	Venue	Modality	Top-1	Top-5	Macro-F1	Train. Params
Majority class baseline	–	–	27.31	51.22	0.72	–
SlowFast [ 28 ]	CVPRW’23	RGB	45.16	73.75	18.91	68M
SlowFast [ 28 ]	CVPRW’23	RGB+Depth+Gaze	49.66	77.82	–	68M
LUBECK UniFormer ens. [ 28 ]	CVPRW’23	RGB+Depth	51.82	83.35	–	540M
UCF Swin3D-B [ 28 ]	CVPRW’23	RGB+Depth	52.82	83.85	–	176M
UNIBZ 8-model ens. [ 8 ]	ISIEA’24	RGB+Depth	52.57	81.53	–	194M

Table 2: Performance comparison with SOTA on MECCANO action recognition task. AH: action history; HOCL: hand-object contact labels; OSL: object state labels.

Method	Venue	Modality	S1	S2	S3	Avg mAcc	Avg Top-1
Two Stream [ 68 ]	NeurIPS’14	RGB + Optical Flow	43.78	41.47	40.28	41.84	–
EgoIDT+Gaze [ 36 ]	CVPR’15	RGB + IDT + Gaze	42.55	37.30	37.60	39.13	–
I3D+EgoConv [ 69 ]	CVPR’16	RGB + Hands + Head + Gaze	54.19	51.45	49.41	51.68	–
TSN [ 75 ]	ECCV’16	RGB + Optical Flow	58.01	55.01	54.78	55.93	–
Sudhakaran et al. [ 71 ]	ECCVW’18	RGB	52.40	50.09	49.11	50.53	–
LSTA [ 70 ]	CVPR’19	RGB + Optical Flow	53.00	–	–	–	–

Table 3: EGTEA Gaze+ action-recognition results on the official three splits. S1–S3 report mAcc when available; otherwise they report per-split Top-1/overall accuracy. MTL: multi-task learning; IDT:improved dense trajectories.

Method	Venue	Modality	Top-1	Top-5	Macro-F1	Train. Params
Majority class baseline	–	–	27.21	51.15	0.71	–
TempAgg [ 65 ]	ECCV’20	OHG	19.69	25.37	3.11	123M
AVT [ 16 ]	ICCV’21	OHG	27.43	53.38	3.68	392M
RULSTM [ 28 ]	CVIU’23	OHG	24.08	58.23	2.47	67M
VLMAH [ 44 ]	ICCV-W’23	OHG	28.90	58.13	0.72	45M
MMTF-RU [ 49 ]	T-ASE’25	OHG	29.75	64.46	–	33.2M

Table 4: MECCANO next-action prediction results for reported methods and our graph-based models. Results are reported for action anticipation at

t_{a}=1

s before the next action. Bold values indicate the best result across the full table.

Dataset	Graph	Nodes $\downarrow$	Edges $\downarrow$	Avg. max distance $\downarrow$	Global Efficiency $\uparrow$
MECCANO	FG	13.57	8.13	3.29	0.257
	G3Ego	3.75	2.75	2.00	0.771
EGTEA	FG	4.51	3.41	2.03	0.703
	G3Ego	3.77	2.77	2.00	0.769

Table 5: Full graphs (FG) versus G3Ego graphs efficiency statistics. EGTEA values denote averages over the three official splits.

Figure 3: Qualitative examples of action classification predictions with the ground-truth labels. For each sample, we visualize the full action scene graph (FG) and the corresponding gaze-guided graph. Gaze-guided pruning preserves action-relevant entities and relationships while removing irrelevant context, yielding more discriminative graph representations and improved action recognition.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Output	Params	GFLOPs	Memory
Qwen3-VL-32B	160 tokens	33.36B	27,500.8	63.91 GiB
DINOv3 ViT-L/16	1024-D	303.13M	121.76	1,176.52 MiB
GroundingDINO	256-D	232.90M	1,312.93	2,160.50 MiB
TempAgg, pruned	352-D	105M	0.435	421.84 MiB
TempAgg, full	352-D	105M	0.47	422.44 MiB

Appendix

Table 6: Computational metrics of the frozen feature extraction components and the trainable temporal model. Latency is measured per frame for the frozen visual components and per graph sequence for the temporal model. Peak memory denotes measured peak allocated GPU memory.

Input Representation	Object Selection	Gaze Aux.	Temporal Model	N Frames	Top-1	Top-5	F1
Effect of VLM Semantic Prior
Majority class baseline	–	✗	–	–	27.31	51.22	0.72
VLM caption CLIP [ 58 ] embeddings	–	✗	LSTM	10	3.86	24.76	0.46
Effect of Pruning Strategy
FG	–	✗	LSTM	10	37.34	70.39	8.61
G3Ego	Random-2	✗	LSTM	10	21.22	57.07	7.95

Appendix

Table 7: Additional ablations on MECCANO action recognition. All variants use an LSTM over the same 10 uniformly sampled frames and exclude hand features. VLM caption embeddings are obtained using the CLIP [ 58 ] text encoder. Random-2 retains two randomly selected object nodes per frame. Gaze Aux. indicates that gaze coordinates are explicitly provided as an additional recognition feature.

Method	Venue	Pretraining	Modality / Supervision	S1	S2	S3	Avg mAcc	Avg Top-1
Two Stream [ 68 ]	NeurIPS’14	–	RGB + Optical Flow	43.78	41.47	40.28	41.84	–
EgoIDT+Gaze [ 36 ]	CVPR’15	–	RGB + IDT + Gaze	42.55	37.30	37.60	39.13	–
I3D+EgoConv [ 69 ]	CVPR’16	–	RGB + Hands + Head Motion + Gaze	54.19	51.45	49.41	51.68	–
TSN [ 75 ]	ECCV’16	–	RGB + Optical Flow	58.01	55.01	54.78	55.93	–
Sudhakaran et al. [ 71 ]	ECCVW’18	–	RGB	52.40	50.09	49.11	50.53	–
LSTA [ 70 ]	CVPR’19	–	RGB + Optical Flow	53.00	–	–	–	–

Appendix

Table 8: EGTEA Gaze+ action-recognition results on the official three splits. S1–S3 report mAcc when available; otherwise, they report per-split Top-1 or overall accuracy. The upper block contains methods without exocentric video pretraining, while the lower block contains methods using Kinetics [ 9 ] or EPIC-KITCHENS-100 [ 10 ] pretraining. Results across these blocks are provided for context and are not directly comparable because of differences in pretraining data, backbone design, and supervision.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
Jun 13, 2026Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla +3Egocentric Video Understanding In-Context Learning
University of Alicante · ETH Zürich · Microsoft
Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs
Jul 2, 2026Francesca Pistilli, Simone Alberto Peirone, Giuseppe AvertaScene Graph Generation Egocentric Video Understanding
Politecnico di Torino · Torino, Italy
Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
Aug 19, 2026Mohammad Zamani, Fatemeh ZiaeetabarHand-Object Interaction Detection Egocentric Video Understanding
Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran

Jun 13, 2026J/K move · Enter open · S save

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla, Marc Pollefeys+2

University of Alicante · ETH Zürich · Microsoft

Egocentric Video Understanding In-Context Learning VLM Reasoning

Open paper AskPDF arXiv

Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame-based inference. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.

Abstract

Egocentric Video Understanding

Explore similar work

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI