A fine-grained understanding of egocentric human-environment interactions is crucial for developing next-generation embodied agents. One fundamental challenge in this area involves accurately parsing hands and active objects. While transformer-based architectures have demonstrated considerable potential for such tasks, several key limitations remain unaddressed: 1) existing query initialization mechanisms rely primarily on semantic cues or learnable parameters, demonstrating limited adaptability to changing active objects across varying input scenes; 2) previous transformer-based methods utilize pixel-level semantic features to iteratively refine queries during mask generation, which may introduce interaction-irrelevant content into the final embeddings; and 3) prevailing models are susceptible to "interaction illusion", producing physically inconsistent predictions. To address these issues, we propose an end-to-end Interaction-aware Transformer (InterFormer), which integrates three key components, i.e., a Dynamic Query Generator (DQG), a Dual-context Feature Selector (DFS), and the Conditional Co-occurrence (CoCo) loss. The DQG explicitly grounds query initialization in the spatial dynamics of hand-object contact, enabling targeted generation of interaction-aware queries for hands and various active objects. The DFS fuses coarse interactive cues with semantic features, thereby suppressing interaction-irrelevant noise and emphasizing the learning of interactive relationships. The CoCo loss incorporates hand-object relationship constraints to enhance physical consistency in prediction. Our model achieves state-of-the-art performance on both the EgoHOS and the challenging out-of-distribution mini-HOI4D datasets, demonstrating its effectiveness and strong generalization ability. Code and models are publicly available at https://github.com/yuggiehk/InterFormer.
Figures & tables
Figure 1
Figure 3: Architecture of our end-to-end InterFormer. Given an input egocentric image, a backbone network first extracts global and multi-scale pixel-level features. We add an additional IPP branch to extract coarse boundary-guided representations that characterize the interaction. Subsequently, the DQG produces robust and dynamic queries by integrating interaction-relevant contextual information with learnable parameters. Finally, these queries and extracted features are fed into the InterFormer decoder, which employs the DFS to refine interaction-aware representations and generate the final segmentation masks. The overall end-to-end architecture is supervised by the classification loss Lcls , dice loss Ldic , cross entropy loss Lce , IPP loss Lb , and CoCo loss Lco .
Figure 4: Detailed architecture of Dual-context Feature Selector (DFS).
Method
Type
Left
Right
Left-hand
Right-hand
Two-hand
Overall
Hand
Hand
Object
Object
Object
Segformer ( Xie et al., 2021 )
T
62.49
64.77
4.03
3.01
5.13
27.89(+45.33)
SCTNet ( Xu et al., 2024b )
T
81.94
82.12
17.77
16.60
21.74
44.03(+29.19)
Para ( Zhang et al., 2022 )
T
69.08
73.50
48.67
36.21
37.46
52.98(+20.24)
Segmenter ( Strudel et al., 2021 )
T
82.20
83.28
46.22
34.79
51.10
59.52(+13.70)
UperNet ( Xiao et al., 2018 )
C
89.88
91.39
36.22
40.55
45.54
60.71(+12.51)
Table 1: Comparison with SOTA methods on the EgoHOS in-domain test set measured by IoU ↑ .
Method
Type
Left
Right
Left-hand
Right-hand
Two-hand
Overall
Hand
Hand
Object
Object
Object
Segformer ( Xie et al., 2021 )
T
71.97
71.44
7.60
5.00
4.91
32.18(+40.64)
SCTNet ( Xu et al., 2024b )
T
87.12
86.29
31.18
19.70
13.32
47.52(+25.30)
UperNet ( Xiao et al., 2018 )
C
93.17
93.96
42.53
28.88
24.35
56.58(+16.24)
Multi-UNet ( Zhao et al., 2025 )
C
92.76
83.08
44.31
39.07
37.15
59.27(+13.55)
Maskformer ( Cheng et al., 2022a )
T
92.69
94.02
51.81
39.84
39.43
63.56(+9.26)
Table 2: Comparison results on the EgoHOS out-of-domain test set measured by IoU ↑ .
Method
Type
Left
Right
Right-hand
Two-hand
Overall
Hand
Hand
Object
Object
Object
Segformer ( Xie et al., 2021 )
T
30.16
56.44
5.17
12.02
25.95(+40.12)
SCTNet ( Xu et al., 2024b )
T
35.83
66.29
17.72
20.98
35.21(+30.86)
Multi-UNet ( Zhao et al., 2025 )
C
52.15
83.64
25.70
41.60
42.36(+23.71)
UperNet ( Xiao et al., 2018 )
C
54.82
84.43
20.34
29.34
47.23(+18.84)
MaskFormer ( Cheng et al., 2022a )
T
58.50
83.66
35.28
56.91
58.59(+7.48)
Table 3: Comparison with SOTA methods on the mini-HOI4D dataset measured by IoU ↑ .
Figure 5: Visualization results on (a) EgoHOS in-domain test set, (b) EgoHOS out-of-domain test set, and (c) out-of-distribution mini-HOI4D dataset.
#
IPP
DQG
DFS
CoCo
Performance (%)
mIoU ↑
mAcc ↑
1
–
–
–
–
70.72
77.48
2
–
–
–
✓
70.95
79.02
3
✓
–
–
–
71.23
79.97
4
✓
✓
–
–
71.50
79.68
5
✓
–
✓
–
71.26
79.11
Table 4: Ablation study results on the EgoHOS in-domain test set.
#
Hyper Parameter
Performance
τ
mIoU
mAcc
1
50
71.62
80.62
2
100
73.22
80.68
3
150
71.97
80.51
4
200
71.62
80.94
5
250
71.10
78.78
Table 5: Hyperparameter experiments of τ on the EgoHOS in-domain test set.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Type
Left
Right
Left-hand
Right-hand
Two-hand
Overall
Hand ↑
Hand ↑
Objects ↑
Objects ↑
Objects ↑
mAcc ↑
Segformer ( Xie et al., 2021 )
T
75.47
78.13
4.57
3.17
5.57
33.38
SCTNet ( Xu et al., 2024b )
T
90.25
89.92
24.49
20.79
29.08
50.91
Para ( Zhang et al., 2022 )
T
75.57
75.93
39.50
39.33
42.58
54.58
Segmenter ( Strudel et al., 2021 )
T
89.87
91.92
62.69
45.59
62.78
70.57
UperNet ( Xiao et al., 2018 )
T
89.86
91.32
37.24
42.26
49.27
61.99
Appendix
Table 6: Comparison with SOTA methods on the EgoHOS in-domain test set measured by Acc ↑ (%) mAcc ↑ (%). The best results are in bold and the second best is underlined . T: transformer-based methods, C: convolution-based methods, L: MLLM-based methods.
Method
Type
Left
Right
Left-hand
Right-hand
Two-hand
Overall
Hand
Hand
Object
Object
Object
Segformer ( Xie et al., 2021 )
T
86.80
79.44
21.77
9.56
11.67
41.85
SCTNet ( Xu et al., 2024b )
T
94.62
90.92
49.14
27.47
17.12
55.85
UperNet ( Xiao et al., 2018 )
C
96.89
96.00
64.83
54.59
27.83
68.03
Multi-UNet ( Zhao et al., 2025 )
C
94.75
95.80
45.16
47.89
16.75
60.07
Maskformer ( Cheng et al., 2022a )
T
95.58
96.10
70.53
60.49
46.52
73.84
Appendix
Table 7: Comparison with SOTA methods on the EgoHOS out-of-domain test set using Acc ↑ (%) mAcc ↑ (%). The best results are in bold , and the second best is underlined . T: transformer-based methods, C: convolution-based methods, L: MLLM-based methods.
Method
Type
Left
Right
Right-hand
Two-hand
Overall
Hand
Hand
Object
Object
Object
Segformer ( Xie et al., 2021 )
T
92.13
72.42
5.52
13.41
45.87
SCTNet ( Xu et al., 2024b )
T
95.25
71.27
22.68
29.98
54.90
Multi-UNet ( Zhao et al., 2025 )
C
94.37
70.25
30.41
47.65
60.67
UperNet ( Xiao et al., 2018 )
C
97.71
86.04
25.77
36.58
61.53
MaskFormer ( Cheng et al., 2022a )
T
96.63
87.83
44.81
73.47
75.69
Appendix
Table 8: Comparison with SOTA methods on the mini-HOI4D dataset measured by Acc ↑ (%) mAcc ↑ (%). The best results are in bold , and the second best is underlined . T: transformer-based methods, C: convolution-based methods, L: MLLM-based methods.
Figure 6: The MLLM-based methods (ANNEXE) show limited spatial precision in mask generation for parsing hands and interacting objects.
Method
Type
FLOPs
mIoU (%)
SegFormer
Transformer-based
71.961G
27.89
Segmenter
Transformer-based
70.074G
59.52
Mask2Former
Transformer-based
96.093G
64.88
Seq
Transformer-based
392.483G
67.17
ANNEXE
MLLM-based
610.500G
71.38
InterFormer (Ours)
Transformer-based
122.996G
73.22
Appendix
Table 9: Comparison of computational complexity and performance on EgoHOS test set
#
λb
λco
λcls
λdic
λce
Performance (%)
mIoU ↑
mAcc ↑
1
1
1
1
1
1
71.74
78.40
2
5
1
1
1
1
71.79
78.21
3
1
5
1
1
1
71.88
78.59
4
5
5
1
1
1
71.08
79.71
5
1
1
5
1
1
71.30
78.79
Appendix
Table 10: Hyperparameter study of different loss weights on the EgoHOS in-domain test set. The best results are shown in bold .
#
Method
Rate ↓
1
Seq Zhang et al. (2022)
9.80%
2
Care-Ego Su et al. (2025a)
5.45%
3
Ours w/o CoCo
2.19%
4
Ours w/ CoCo
1.55%
Appendix
Table 11: Quantitive results of CoCo loss.
Figure 7: Qualitative comparison with/without employing CoCo Loss on EgoHOS (left) and mini-HOI4D (right) datasets.
Figure 8: Qualitative comparison of our InterFormer against baseline on OOD mini-HOI4D (left) and EgoHOS out-of-domain (right) test sets.
Understanding person-level bi-manual interactions requires not only detecting hands, but also identifying which two hands belong to the same person and what each hand interacts with. Existing hand--object interaction methods are mostly hand-centric: they treat each hand as an independent instance, which can lead to ambiguous ownership in multi-person scenes. We propose a person-centric formulation in which a single query predicts a structured output for one person, including the human box, body pose, hand boxes and states, and interaction targets. We introduce part-aware deformable attention to allocate attention across human, hand, and pose-specific reference regions, enabling one query to capture the full person structure. We further unify detection and interaction reasoning with a hand-to-query relationship matrix, where each hand selects its interaction target from the detected query set plus a learnable off token, directly recovering the target's box and class without separate object regression. We build a COCO-based dataset with person-centric bi-manual interaction annotations and define structured metrics for evaluating hand states and complete hand--object tuples. Experiments with a transformer-based detector show that our formulation improves person-level bi-manual interaction parsing and provides an effective unified framework for joint detection, pose estimation, and hand reasoning.
Understanding hands and the objects they interact with, both directly and through tools, is a key step for tasks ranging from action perception to 3D reconstruction and robotics. Our paper provides several contributions to the Hand-Object Interaction (HOI) understanding literature: (1) HOI-DETR, a new framework that introduces hand-object and object-object interactions to the Co-DETR architecture to produce a state-of-the-art method; (2) a comprehensive HOI evaluation suite of 4 diverse datasets, including a video benchmark derived from the HD-EPIC dataset and fresh annotations that improve the Hands23 benchmark and (3) a trained checkpoint that significantly improves the state of the art across Hands23, HOIST, FineBio, and HD-EPIC, including mAP gains of over 20 percentage points on Hands23 and FineBio. Our ablations confirm the contributions of each model component.
Ahmad Darkhalil, Dima Damen, David Fouhey
School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US
Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand-object masked training, which enables robust reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and object-centric embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific reasoning, we introduce Cue-Isolated HOI (CI-HOI), a new evaluation that assesses models' ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand- and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand- and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even robot manipulation action recognition, leading to more robust HOI understanding.