Leveraging Synthetic Data for Enhancing Egocentric Hand-Object Interaction Detection

Rosario Leonardi; Antonino Furnari; Francesco Ragusa; Giovanni Maria Farinella

cs.CVMar 31, 2026

Leveraging Synthetic Data for Enhancing Egocentric Hand-Object Interaction Detection

Authors: Rosario Leonardi, Antonino Furnari, Francesco Ragusa, Giovanni Maria Farinella

Organizations: Department of Mathematics and Computer Science, University of Catania, Catania, Italy. · Next Vision s.r.l., Catania, Italy.

Abstract

In this work, we explore the role of synthetic data in improving the detection of Hand-Object Interactions from egocentric images. Through extensive experimentation and comparative analysis on VISOR, EgoHOS, and ENIGMA-51 datasets, our findings demonstrate the potential of synthetic data to significantly improve HOI detection, particularly when real labeled data are scarce or unavailable. By using synthetic data and only 10% of the real labeled data, we achieve improvements in Overall AP over models trained exclusively on real data, with gains of +5.67% on VISOR, +8.24% on EgoHOS, and +11.69% on ENIGMA-51. Furthermore, we systematically study how aligning synthetic data to specific real-world benchmarks with respect to objects, grasps, and environments, showing that the effectiveness of synthetic data consistently improves with better synthetic-real alignment. As a result of this work, we release a new data generation pipeline and the new HOI-Synth benchmark, which augments existing datasets with synthetic images of hand-object interaction. These data are automatically annotated with hand-object contact states, bounding boxes, and pixel-wise segmentation masks. All data, code, and tools for synthetic data generation are available at: https://fpv-iplab.github.io/HOI-Synth/.

Figures & tables

Figure 1: We explore the role of synthetic data in egocentric hand-object interaction detection by generating and automatically labeling synthetic datasets (left). We then analyze domain adaptation scenarios where models are trained using both synthetic and real unlabeled data, with varying amounts of labeled real data (right).

Figure 2: The proposed data generation pipeline. (a) An object-grasp pair is selected from DexGraspNet Wang et al. (2023) and integrated with a randomly generated human model. (b) The human + object model is placed in an environment randomly selected from the Habitat-Matterport 3D dataset Ramakrishnan et al. (2021) . (c) Egocentric data of hand-object interactions is generated and automatically labeled. Labels include bounding boxes and segmentation masks of hands and interacted objects, contact-state, and hand-object relations.

Figure 3: Qualitative examples generated by our HOI-Synth Data Generation Pipeline . For each sample, the pipeline automatically generates rich ground-truth annotations, including 2D bounding boxes, pixel-wise segmentation masks, and interaction metadata describing the relationship between hands and manipulated objects.

Figure 4: A ENIGMA-51 image (left), a synthetic in-domain image (center), and a synthetic out-domain image (right).

Dataset	Images	Hands	Objects	HOI
VISOR Darkhalil et al. (2022)	32,857	52,906	42,785	42,787
Synthetic	30,259	60,098	45,219	45,353
Synthetic aligned Objects	4,951	9,810	8,311	8,263
Synthetic aligned Grasps	5,000	9,899	8,355	8,306
Synthetic aligned Environments	4,938	9,789	8,034	8,002
Synthetic aligned All	31,000	61,408	52,279	52,045

Table 1: Statistics of the training sets included in our HOI-Synth benchmark.

Figure 5: The architecture of the domain adaptation approach used in our analysis. The method is based on the Adaptive Teacher Li et al. (2022b) framework, extended with HOS recognition modules Darkhalil et al. (2022) .

% Real Labeled Data	Approach	Overall	H	H+S	H+C	O
0%	Synthetic-Only	09.88	28.41	24.89	08.64	01.23
0%	UDA	33.33	80.16	65.98	33.47	8.35
Absolute Improvement		+23.45	$+51.75$	$+41.09$	$+24.83$	$+7.12$

Table 2: Results on the VISOR validation set considering different real data settings available in training. Gray rows indicate baseline models in each configuration, while white rows highlight models trained with synthetic and real data. In each group, the best results are in bold, while the best results among the models trained with synthetic and real data are underlined. Overall enhancements are shown in blue, indicating improvements of the models trained with synthetic and real data over the baseline .

Figure 6: Qualitative examples on VISOR . Columns show predictions from Real-Only , Synthetic+Real , and SSDA models. Colors denote the left hand (blue), right hand (red), and interacted objects (yellow). Yellow arrows highlight regions of interest where the SSDA model demonstrates superior performance compared to the other baselines.

Method	Overall	H	H+S	H+C	O
Synthetic + Real	38.19	89.98	84.67	55.88	18.49
MT Tarvainen and Valpola (2017)	43.69	88.78	84.40	60.94	21.89
MT+GRL Ganin and Lempitsky (2015)	43.97	88.64	84.27	58.21	21.82
UT Liu et al. (2021)	44.32	90.60	84.49	52.55	22.11
AT Li et al. (2022b)	45.55	90.37	84.42	52.59	22.15

Table 3: Results of different semi-supervised adaptation approaches trained with synthetic data and

25\%

VISOR labeled training data.

Figure 7: UDA Overall AP on VISOR validation set for different amounts of training synthetic data. The green indicator shows the best result.

Figure 8: The green row shows the selected synthetic objects aligned with those in VISOR (black row). The red row contains the discarded examples, filtered out due to misalignment or poor quality. Note that these objects are common kitchen items, such as mugs, containers, and bottles.

Approach	Objects	Grasps	Environments	#Images	Overall	H	H+C	O
Baseline				$\sim 5k$	31.52	79.58	30.19	7.35
Target Objects	✓			$\sim 5k$	32.60	82.87	34.32	7.56
Target Grasps		✓		$\sim 5k$	31.84	82.01	31.95	7.97
Target Environments			✓	$\sim 5k$	31.59	81.22	32.68	7.16
Target All	✓	✓	✓	$\sim 5k$	32.59	82.86	34.37	7.53
Baseline				$\sim 30k$	33.33	80.16	33.47	8.35

Table 4: Performance comparison of the UDA Approach with aligned features on the VISOR dataset.

Figure 9: The green row shows examples of selected grasps aligned with real grasps from VISOR (black row). The red row contains discarded cases due to misalignment or unrealistic grasp configurations.

Figure 10: Green rows contain some examples of selected synthetic images aligned with VISOR images (black row). Note that the selected environments mainly present kitchens. The red row shows discarded examples due to misalignment or unsuitable viewpoints.

% Real Labeled Data	Approach	Overall	H	H+S	H+C	O
0%	Synthetic-Only	07.16	18.25	15.93	05.33	01.24
0%	UDA	28.16	70.30	59.21	20.84	09.65
Absolute Improvement		+21.00	$+52.05$	$+43.28$	$+15.51$	$+8.41$

Table 5: Results on the EgoHOS Zhang et al. (2022) test set.

Figure 11: Qualitative examples on EgoHOS .

% Real Labeled Data	Approach	In-domain	Overall	H	H+S	H+C	O
10% (347 images)	Real-Only	✓	45.39	81.25	76.22	37.96	39.53
	SSDA		57.08	85.40	78.62	43.56	46.97
	SSDA	✓	56.69	84.58	78.42	41.17	46.50
Absolute Improvement			+11.69	$+4.15$	$+2.40$	$+5.60$	$+7.44$
25% (870 images)	Real-Only	✓	51.83	82.95	78.70	43.52	45.25
	SSDA		58.17	84.99	80.41	46.31	49.34

Table 6: Results on the ENIGMA-51 Ragusa et al. (2024) test set.

Figure 12: Qualitative examples on ENIGMA-51 .

Approach	Target FOV	In-domain	Overall	H	H+C	O
Synthetic-Only			00.21	01.07	00.03	00.99
Synthetic-Only	✓		05.67	15.78	02.66	02.31
Synthetic-Only	✓	✓	12.85	56.05	15.24	4.79

Table 7: Performance comparison when aligning the FOV of out-domain synthetic data to that of ENIGMA-51 .

Background	Overall	H	H+C	O
Baseline	31.52	79.58	30.19	7.35
ImageNet	28.86	77.38	27.16	5.47
VISOR	27.41	82.30	6.09	2.56

Table 8: Performance comparison on the VISOR validation dataset using the UDA model trained on 5k synthetic images with different backgrounds. The baseline with coherent backgrounds outperforms the models trained with ImageNet and VISOR backgrounds, highlighting the importance of background alignment for cross-domain performance.

Dataset	Approach	Contact F1	Side F1
VISOR	Real-Only	79.46	98.92
VISOR	FSDA	81.65	99.06
EgoHOS	Real-Only	78.78	98.33
EgoHOS	FSDA	79.25	98.87
ENIGMA-51	Real-Only	77.28	98.80
ENIGMA-51	FSDA	77.83	99.11

Table 9: Comparison of FSDA vs Real-Only for F1 Scores of Contact and Side.

Figure 13: Simple HOI generation approach: Object models and hands are randomly projected onto ImageNet backgrounds.

Generation approach	Overall	H	H+C	O	Contact F1
Synthetic augmentation	19.62	62.40	11.91	3.60	8.78
Simulation (Ours)	30.54	79.05	31.51	7.33	22.75

Table 10: UDA Results on VISOR : Comparison of simulation-based image generation and classic synthetic augmentations.

Hand-Object Interaction Detection

See all 10 papers

Improving and Evaluating Hand-Object Interaction Detection
Jun 16, 2026Ahmad Darkhalil, Dima Damen, David FouheyDetection Transformer Hand-Object Interaction Detection
School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US
EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation
May 18, 2026Rosario Leonardi, Francesco Ragusa, Daniele Materia +4Temporal Action Segmentation Next-Action Prediction
Department of Mathematics and Computer Science, University of Catania, Italy · Next Vision s.r.l., Catania, Italy · Reality Labs Research, Meta, USA
Train, Test, Re-evaluate: Schedule-Sensitive Evaluation of Generative Data for Hand Detection
Jun 1, 2026Atmika Bhardwaj, Silvia Vock, Nico SteckhanSynthetic-to-Real Domain Adaptation Synthetic Data Augmentation
Federal Institute for Occupational Safety and Health (BAuA), Germany

Jun 16, 2026J/K move · Enter open · S save

Improving and Evaluating Hand-Object Interaction Detection

Ahmad Darkhalil, Dima Damen, David Fouhey

School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US

Detection Transformer Hand-Object Interaction Detection Human-Object Interaction Hand-Object Interaction

Open paper AskPDF arXiv

Understanding hands and the objects they interact with, both directly and through tools, is a key step for tasks ranging from action perception to 3D reconstruction and robotics. Our paper provides several contributions to the Hand-Object Interaction (HOI) understanding literature: (1) HOI-DETR, a new framework that introduces hand-object and object-object interactions to the Co-DETR architecture to produce a state-of-the-art method; (2) a comprehensive HOI evaluation suite of 4 diverse datasets, including a video benchmark derived from the HD-EPIC dataset and fresh annotations that improve the Hands23 benchmark and (3) a trained checkpoint that significantly improves the state of the art across Hands23, HOIST, FineBio, and HD-EPIC, including mAP gains of over 20 percentage points on Hands23 and FineBio. Our ablations confirm the contributions of each model component.

Abstract

Hand-Object Interaction Detection

Explore similar work

Improving and Evaluating Hand-Object Interaction Detection

EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation

Train, Test, Re-evaluate: Schedule-Sensitive Evaluation of Generative Data for Hand Detection