Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing

Yuejiao Su; Yi Wang; Lei Yao; Yawen Cui; Lap-Pui Chau

cs.CVFeb 24, 2026

Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing

Authors: Yuejiao Su, Yi Wang, Lei Yao, Yawen Cui, Lap-Pui Chau

Organizations: The Hong Kong Polytechnic University

Abstract

A fine-grained understanding of egocentric human-environment interactions is crucial for developing next-generation embodied agents. One fundamental challenge in this area involves accurately parsing hands and active objects. While transformer-based architectures have demonstrated considerable potential for such tasks, several key limitations remain unaddressed: 1) existing query initialization mechanisms rely primarily on semantic cues or learnable parameters, demonstrating limited adaptability to changing active objects across varying input scenes; 2) previous transformer-based methods utilize pixel-level semantic features to iteratively refine queries during mask generation, which may introduce interaction-irrelevant content into the final embeddings; and 3) prevailing models are susceptible to "interaction illusion", producing physically inconsistent predictions. To address these issues, we propose an end-to-end Interaction-aware Transformer (InterFormer), which integrates three key components, i.e., a Dynamic Query Generator (DQG), a Dual-context Feature Selector (DFS), and the Conditional Co-occurrence (CoCo) loss. The DQG explicitly grounds query initialization in the spatial dynamics of hand-object contact, enabling targeted generation of interaction-aware queries for hands and various active objects. The DFS fuses coarse interactive cues with semantic features, thereby suppressing interaction-irrelevant noise and emphasizing the learning of interactive relationships. The CoCo loss incorporates hand-object relationship constraints to enhance physical consistency in prediction. Our model achieves state-of-the-art performance on both the EgoHOS and the challenging out-of-distribution mini-HOI4D datasets, demonstrating its effectiveness and strong generalization ability. Code and models are publicly available at https://github.com/yuggiehk/InterFormer.

Figures & tables

Figure 3: Architecture of our end-to-end InterFormer. Given an input egocentric image, a backbone network first extracts global and multi-scale pixel-level features. We add an additional IPP branch to extract coarse boundary-guided representations that characterize the interaction. Subsequently, the DQG produces robust and dynamic queries by integrating interaction-relevant contextual information with learnable parameters. Finally, these queries and extracted features are fed into the InterFormer decoder, which employs the DFS to refine interaction-aware representations and generate the final segmentation masks. The overall end-to-end architecture is supervised by the classification loss $\mathcal{L}_{cls}$ , dice loss $\mathcal{L}_{dic}$ , cross entropy loss $\mathcal{L}_{ce}$ , IPP loss $\mathcal{L}_{b}$ , and CoCo loss $\mathcal{L}_{co}$ .

Figure 4: Detailed architecture of Dual-context Feature Selector (DFS).

Method	Type	Left	Right	Left-hand	Right-hand	Two-hand	Overall
Method	Type	Hand	Hand	Object	Object	Object	Overall
Segformer ( Xie et al., 2021 )	T	62.49	64.77	4.03	3.01	5.13	$27.89_{(+45.33)}$
SCTNet ( Xu et al., 2024b )	T	81.94	82.12	17.77	16.60	21.74	$44.03_{(+29.19)}$
Para ( Zhang et al., 2022 )	T	69.08	73.50	48.67	36.21	37.46	$52.98_{(+20.24)}$
Segmenter ( Strudel et al., 2021 )	T	82.20	83.28	46.22	34.79	51.10	$59.52_{(+13.70)}$
UperNet ( Xiao et al., 2018 )	C	89.88	91.39	36.22	40.55	45.54	$60.71_{(+12.51)}$

Table 1: Comparison with SOTA methods on the EgoHOS in-domain test set measured by IoU

\uparrow

Method	Type	Left	Right	Left-hand	Right-hand	Two-hand	Overall
Method	Type	Hand	Hand	Object	Object	Object
Segformer ( Xie et al., 2021 )	T	71.97	71.44	7.60	5.00	4.91	$32.18_{(+40.64)}$
SCTNet ( Xu et al., 2024b )	T	87.12	86.29	31.18	19.70	13.32	$47.52_{(+25.30)}$
UperNet ( Xiao et al., 2018 )	C	93.17	93.96	42.53	28.88	24.35	$56.58_{(+16.24)}$
Multi-UNet ( Zhao et al., 2025 )	C	92.76	83.08	44.31	39.07	37.15	$59.27_{(+13.55)}$
Maskformer ( Cheng et al., 2022a )	T	92.69	94.02	51.81	39.84	39.43	$63.56_{(+9.26)}$

Table 2: Comparison results on the EgoHOS out-of-domain test set measured by IoU

\uparrow

Method	Type	Left	Right	Right-hand	Two-hand	Overall
Method	Type	Hand	Hand	Object	Object	Object
Segformer ( Xie et al., 2021 )	T	30.16	56.44	5.17	12.02	$25.95_{(+40.12)}$
SCTNet ( Xu et al., 2024b )	T	35.83	66.29	17.72	20.98	$35.21_{(+30.86)}$
Multi-UNet ( Zhao et al., 2025 )	C	52.15	83.64	25.70	41.60	$42.36_{(+23.71)}$
UperNet ( Xiao et al., 2018 )	C	54.82	84.43	20.34	29.34	$47.23_{(+18.84)}$
MaskFormer ( Cheng et al., 2022a )	T	58.50	83.66	35.28	56.91	$58.59_{(+7.48)}$

Table 3: Comparison with SOTA methods on the mini-HOI4D dataset measured by IoU

\uparrow

Figure 5: Visualization results on (a) EgoHOS in-domain test set, (b) EgoHOS out-of-domain test set, and (c) out-of-distribution mini-HOI4D dataset.

#	IPP	DQG	DFS	CoCo	Performance (%)
#	IPP	DQG	DFS	CoCo	mIoU $\uparrow$	mAcc $\uparrow$
1	–	–	–	–	70.72	77.48
2	–	–	–	✓	70.95	79.02
3	✓	–	–	–	71.23	79.97
4	✓	✓	–	–	71.50	79.68
5	✓	–	✓	–	71.26	79.11

Table 4: Ablation study results on the EgoHOS in-domain test set.

#	Hyper Parameter	Performance
#	$\tau$	mIoU	mAcc
1	50	71.62	80.62
2	100	73.22	80.68
3	150	71.97	80.51
4	200	71.62	80.94
5	250	71.10	78.78

Table 5: Hyperparameter experiments of

\tau

on the EgoHOS in-domain test set.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Type	Left	Right	Left-hand	Right-hand	Two-hand	Overall
Method	Type	Hand $\uparrow$	Hand $\uparrow$	Objects $\uparrow$	Objects $\uparrow$	Objects $\uparrow$	mAcc $\uparrow$
Segformer ( Xie et al., 2021 )	T	75.47	78.13	4.57	3.17	5.57	33.38
SCTNet ( Xu et al., 2024b )	T	90.25	89.92	24.49	20.79	29.08	50.91
Para ( Zhang et al., 2022 )	T	75.57	75.93	39.50	39.33	42.58	54.58
Segmenter ( Strudel et al., 2021 )	T	89.87	91.92	62.69	45.59	62.78	70.57
UperNet ( Xiao et al., 2018 )	T	89.86	91.32	37.24	42.26	49.27	61.99

Appendix

Table 6: Comparison with SOTA methods on the EgoHOS in-domain test set measured by Acc

\uparrow

(%) mAcc

\uparrow

(%). The best results are in bold and the second best is underlined . T: transformer-based methods, C: convolution-based methods, L: MLLM-based methods.

Method	Type	Left	Right	Left-hand	Right-hand	Two-hand	Overall
Method	Type	Hand	Hand	Object	Object	Object
Segformer ( Xie et al., 2021 )	T	86.80	79.44	21.77	9.56	11.67	41.85
SCTNet ( Xu et al., 2024b )	T	94.62	90.92	49.14	27.47	17.12	55.85
UperNet ( Xiao et al., 2018 )	C	96.89	96.00	64.83	54.59	27.83	68.03
Multi-UNet ( Zhao et al., 2025 )	C	94.75	95.80	45.16	47.89	16.75	60.07
Maskformer ( Cheng et al., 2022a )	T	95.58	96.10	70.53	60.49	46.52	73.84

Appendix

Table 7: Comparison with SOTA methods on the EgoHOS out-of-domain test set using Acc

\uparrow

(%) mAcc

\uparrow

(%). The best results are in bold , and the second best is underlined . T: transformer-based methods, C: convolution-based methods, L: MLLM-based methods.

Method	Type	Left	Right	Right-hand	Two-hand	Overall
Method	Type	Hand	Hand	Object	Object	Object
Segformer ( Xie et al., 2021 )	T	92.13	72.42	5.52	13.41	45.87
SCTNet ( Xu et al., 2024b )	T	95.25	71.27	22.68	29.98	54.90
Multi-UNet ( Zhao et al., 2025 )	C	94.37	70.25	30.41	47.65	60.67
UperNet ( Xiao et al., 2018 )	C	97.71	86.04	25.77	36.58	61.53
MaskFormer ( Cheng et al., 2022a )	T	96.63	87.83	44.81	73.47	75.69

Appendix

Table 8: Comparison with SOTA methods on the mini-HOI4D dataset measured by Acc

\uparrow

(%) mAcc

\uparrow

(%). The best results are in bold , and the second best is underlined . T: transformer-based methods, C: convolution-based methods, L: MLLM-based methods.

Figure 6: The MLLM-based methods (ANNEXE) show limited spatial precision in mask generation for parsing hands and interacting objects.

Method	Type	FLOPs	mIoU (%)
SegFormer	Transformer-based	71.961G	27.89
Segmenter	Transformer-based	70.074G	59.52
Mask2Former	Transformer-based	96.093G	64.88
Seq	Transformer-based	392.483G	67.17
ANNEXE	MLLM-based	610.500G	71.38
InterFormer (Ours)	Transformer-based	122.996G	73.22

Appendix

Table 9: Comparison of computational complexity and performance on EgoHOS test set

#	$\lambda_{b}$	$\lambda_{co}$	$\lambda_{cls}$	$\lambda_{dic}$	$\lambda_{ce}$	Performance (%)
#	$\lambda_{b}$	$\lambda_{co}$	$\lambda_{cls}$	$\lambda_{dic}$	$\lambda_{ce}$	mIoU $\uparrow$	mAcc $\uparrow$
1	1	1	1	1	1	71.74	78.40
2	5	1	1	1	1	71.79	78.21
3	1	5	1	1	1	71.88	78.59
4	5	5	1	1	1	71.08	79.71
5	1	1	5	1	1	71.30	78.79

Appendix

Table 10: Hyperparameter study of different loss weights on the EgoHOS in-domain test set. The best results are shown in bold .

#	Method	Rate $\downarrow$
1	Seq Zhang et al. (2022)	9.80%
2	Care-Ego Su et al. (2025a)	5.45%
3	Ours w/o CoCo	2.19%
4	Ours w/ CoCo	1.55%

Appendix

Table 11: Quantitive results of CoCo loss.

Figure 7: Qualitative comparison with/without employing CoCo Loss on EgoHOS (left) and mini-HOI4D (right) datasets.

Figure 8: Qualitative comparison of our InterFormer against baseline on OOD mini-HOI4D (left) and EgoHOS out-of-domain (right) test sets.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

Single-Query Person-Centric Bimanual Hand-Object Interaction Detection
Sep 10, 2026Jonghyun Kim, Junho Roh, Yubin Yoon +5Hand-Object Interaction Detection Human-Object Interaction
AI Lab, CTO Division, LG Electronics, South Korea
Improving and Evaluating Hand-Object Interaction Detection
Jun 16, 2026Ahmad Darkhalil, Dima Damen, David FouheyDetection Transformer Hand-Object Interaction Detection
School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US
Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
Jul 9, 2026Masatoshi Tateno, Alexandros Stergiou, Risa Shinoda +2Human Activity Recognition Object-Centric Representation Learning
1The University of Tokyo · University of Twente · University of Bristol

Sep 10, 2026J/K move · Enter open · S save

Single-Query Person-Centric Bimanual Hand-Object Interaction Detection

Jonghyun Kim, Junho Roh, Yubin Yoon, Hyotae Lee+4

AI Lab, CTO Division, LG Electronics, South Korea

Hand-Object Interaction Detection Human-Object Interaction Hand-Object Interaction

Open paper AskPDF arXiv

Understanding person-level bi-manual interactions requires not only detecting hands, but also identifying which two hands belong to the same person and what each hand interacts with. Existing hand--object interaction methods are mostly hand-centric: they treat each hand as an independent instance, which can lead to ambiguous ownership in multi-person scenes. We propose a person-centric formulation in which a single query predicts a structured output for one person, including the human box, body pose, hand boxes and states, and interaction targets. We introduce part-aware deformable attention to allocate attention across human, hand, and pose-specific reference regions, enabling one query to capture the full person structure. We further unify detection and interaction reasoning with a hand-to-query relationship matrix, where each hand selects its interaction target from the detected query set plus a learnable off token, directly recovering the target's box and class without separate object regression. We build a COCO-based dataset with person-centric bi-manual interaction annotations and define structured metrics for evaluating hand states and complete hand--object tuples. Experiments with a transformer-based detector show that our formulation improves person-level bi-manual interaction parsing and provides an effective unified framework for joint detection, pose estimation, and hand reasoning.

Abstract

Human-Object Interaction

Explore similar work

Single-Query Person-Centric Bimanual Hand-Object Interaction Detection

Improving and Evaluating Hand-Object Interaction Detection

Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?