FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

Francesco Tonini; Lorenzo Vaquero; Mohammad Mahdi Derakhshani; Cees Snoek; Elisa Ricci; Cigdem Beyan

cs.CVSep 5, 2026

FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

Authors: Francesco Tonini, Lorenzo Vaquero, Mohammad Mahdi Derakhshani, Cees Snoek, Elisa Ricci, Cigdem Beyan

Organizations: University of Trento Trento, Italy · Fondazione Bruno Kessler Trento, Italy · University of Amsterdam Amsterdam, The Netherlands · Department of Computer Science University of Verona Verona, Italy

Abstract

Human-Object Interaction (HOI) detection aims to localize humans and objects in images and classify their interactions. Zero-shot HOI focuses on recognizing interactions that are not observed during training, requiring models to generalize beyond seen verb-object compositions. Recent approaches leverage Vision-Language Models (VLMs), benefiting from rich semantic representations. However, they often rely on global or detector-centric features that compress interaction cues and hinder fine-grained spatial reasoning. To overcome this limitation, we propose FineHOI, a zero-shot HOI framework that explicitly models interactions from dense patch-level features. Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships, which are not preserved by global and detector-centric representations. To this end, we introduce an Adaptive Part-Level Attention module that decomposes humans and objects into semantically coherent parts via unsupervised clustering, and re-weights them based on their interaction relevance. These representations are then integrated through a Region-Aware Interaction Transformer that integrates part-aware and global features and produces the final HOI embedding. Extensive experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods, achieving particularly strong gains on unseen interactions. Code is available at https://github.com/francescotonini/fine-hoi.

Figures & tables

Figure 1. Comparison of zero-shot HOI approaches. Traditional methods, e.g. ( Kim et al., 2025 ; Lei et al., 2024a ; Lei et al., 2024b ) rely on object-level features which do not capture the fine-grained details needed for interaction classification. In contrast, FineHOI explicitly models dense patch-level interactions through human/object parts decomposition, and can dynamically select which parts are most relevant for an interaction ( e.g. hands, upper-body, object handle), enabling finer understanding of human-object interactions.

Figure 2. FineHOI consists of three stages: Object Detection and Soft-Mask Extraction , which uses an object detector $\mathcal{OD}$ to localize the human and object instances, while a visual encoder $\mathcal{V}$ extracts dense spatial feature map used for human and object soft masks construction; Adaptive Part-Level Attention , which decomposes the human and object patches into structurally coherent parts and refines them based on interaction relevance; and Region-Aware Interaction Transformer , which fuses the refined human-object representations with contextual patch features for HOI classification. Ⓟ stands for feature pooling ( Eq. 3 ), Ⓜ for mask-guided feature pooling ( Eq. 9 ), and Ⓒ for concatenation.

			UV				UO				UC-RF				UC-NF				UC
Model		$\Delta$	Full	Seen	Unseen	HM	Full	Seen	Unseen	HM	Full	Seen	Unseen	HM	Full	Seen	Unseen	HM	Full	Seen	Unseen	HM
w/ LLM supervision
ADA-CM ( Lei et al., 2023 )	CLIP ViT-B/16	✓	-	-	-	-	-	-	-	-	33.01	34.35	27.36	30.86	31.39	31.13	32.41	31.77	-	-	-	-
EZ-HOI ( Lei et al., 2024a )	CLIP ViT-L/14	✓	32.32	33.49	25.10	29.30	32.27	32.06	33.28	32.67	33.13	34.15	29.02	31.59	31.17	30.55	33.66	32.11	-	-	-	-
DYSCO ( Tonini et al., 2025 )	CLIP ViT-L/14	✓	-	-	-	-	-	-	-	-	25.24	23.96	30.36	27.16	25.18	24.58	27.56	26.07	-	-	-	-
VDRP ( Yang et al., 2025 )	CLIP ViT-L/14	✓	37.18	38.16	31.16	34.31	37.81	37.50	39.39	38.41	38.13	38.48	36.72	37.58	36.46	36.21	37.48	36.83	-	-	-	-

Table 1. State-of-the-art comparison of zero-shot HOI detection methods on the HICO-DET ( Chao et al., 2018 ) dataset. Results for prior methods are taken from ( Kim et al., 2025 ; Yang et al., 2025 ) , while results marked with

\dagger

are reproduced using the official code.

\Delta

indicates the use of detector-centric features. Best results are highlighted in bold, and second-best results are underlined .

Model		$\Delta$	HICO-DET				V-COCO
Model		$\Delta$	Full	Rare	Non-rare	HM	$\text{{AP}}^{\text{{S2}}}$
w/ LLM supervision
ADA-CM ( Lei et al., 2024b )	CLIP ViT-L/14	✓	38.40	37.52	38.66	38.09	63.9
EZ-HOI ( Lei et al., 2024a )	CLIP ViT-L/14	✓	38.61	37.70	38.89	38.30	66.1
DYSCO ( Tonini et al., 2025 )	CLIP ViT-L/14	✓	28.24	34.22	26.46	30.34	47.8
VDRP ( Yang et al., 2025 ) †	CLIP ViT-L/14	✓	38.90	39.51	38.71	39.12	66.2

Table 2. Evaluation of zero-shot methods on closed-set HOI detection on HICO-DET ( Chao et al., 2018 ) and V-COCO ( Gupta and Malik, 2015 ) datasets. Results for prior methods are taken from ( Kim et al., 2025 ; Yang et al., 2025 ) , while results marked with

\dagger

are reproduced using the official code.

\Delta

indicates the use of detector-centric features. Best results are in bold, second-best are underlined .

SM	APA	$\mathcal{I}$	Full	Seen	Unseen	HM
✗	✗	✗	33.39	34.53	26.42	30.48
✓	✗	✗	35.36	36.05	31.13	33.59
✓	✓	✗	35.36	35.55	34.21	34.88
✓	✓	✓	38.10	38.51	35.58	37.04

Table 3. Ablation of FineHOI on HICO-DET dataset under the UV setting. SM stands for soft-masks ( Section 3.1 ), APA for Adaptive Part-Level Attention ( Section 3.2 ), and

\mathcal{I}

for the Region-Aware Interaction Transformer ( Section 3.3 ). Best results in bold.

	Full	Seen	Unseen	HM
RoIAlign	35.61	36.34	31.13	33.73
Uniform	35.41	35.68	33.80	34.74
Soft-mask	38.10	38.51	35.58	37.04

Table 4. Effectiveness of soft-mask on the HICO-DET dataset under the UV setting. Best results in bold.

	Full	Seen	Unseen	HM
HO token ( $\tilde{\mathbf{h}}_{ho}$ )	35.49	36.21	31.07	33.64
+ global ( $\bar{\mathbf{e}}_{g}$ )	36.18	37.05	30.79	33.92
+ global ( $\bar{\mathbf{e}}_{g}$ ) + mask-guided ( $\bar{\mathbf{e}}_{m}$ )	38.10	38.51	35.58	37.04

Table 5. Effectiveness of multimodal embeddings on HICO-DET under the UV setting. Best results in bold.

Figure 3. Visualization of the part-level attention module with separate or unified human and object parts decoder.

	Full	Seen	Unseen	HM
Unified human/object	36.36	37.01	32.34	34.66
Separate human/object	38.10	38.51	35.58	37.04

Table 6. Comparison between unified and separate part-level attention on the HICO-DET dataset under the UV setting.

Figure 4. Attention maps of the interaction transformer $\mathcal{I}$ in FineHOI. Warmer colors indicate higher attention to visual patches. Blue and green boxes denote the human and object detections, respectively.

	$\Delta$	Full	Seen	Unseen	HM
LAIN ( Kim et al., 2025 )	✓	36.01	34.72	38.11	36.41
LAIN ( Kim et al., 2025 )		34.85	34.38	37.18	35.78
FineHOI (Ours)	✓	36.84	35.93	41.42	38.67
FineHOI (Ours)		36.62	35.92	40.13	38.03

Table 7. Performance of FineHOI and LAIN with DINOtxt and detector features (

\Delta

) on HICO-DET under the UO setting.

		Full	Seen	Unseen	HM
UV	SAM2	37.57	38.20	33.68	35.94
UV	Soft-mask	38.10	38.51	35.58	37.04
UO	SAM2	36.61	36.38	37.70	37.04
UO	Soft-mask	36.62	35.92	40.13	38.03
UC-RF	SAM2	37.90	37.95	37.69	37.82
UC-RF	Soft-mask	38.39	38.54	37.79	38.17

Table 8. Comparison between our proposed soft-mask and SAM2 on the HICO-DET dataset under all zero-shot settings.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Comparison between the original LAIN ( Kim et al., 2025 ) and our adapted version. In LAIN, DETR ( $\mathcal{OD}$ ) provides both the detections ( $\mathbf{b}_{h}$ and $\mathbf{b}_{o}$ ) and the human/object features ( $\mathbf{f}_{h}$ and $\mathbf{f}_{o}$ ) used to construct the interaction representation, which is then processed jointly with the image by a CLIP-based visual encoder ( $\mathcal{V}_{CLIP}$ ). In our adaptation, we replace CLIP with DINOtxt ( $\mathcal{V}$ ) and use DETR only for bounding box prediction. Human and object features are obtained by average pooling (Ⓟ) the DINOtxt patch features within the corresponding detected boxes.

			UO				UC-RF				UC-NF				UC
SM	APA	$\mathcal{I}$	Full	Seen	Unseen	HM	Full	Seen	Unseen	HM	Full	Seen	Unseen	HM	Full	Seen	Unseen	HM
✗	✗	✗	33.00	33.40	30.98	32.19	34.02	35.72	27.27	31.50	30.56	30.54	30.65	30.60	33.70	33.51	31.17	32.34
✓	✗	✗	34.11	33.56	36.88	35.22	35.13	35.36	34.20	34.88	34.70	34.90	33.87	34.38	35.42	36.16	32.68	34.42
✓	✓	✗	35.97	35.47	38.48	36.98	37.06	37.55	35.09	36.32	35.40	35.71	34.15	34.93	36.06	36.79	33.16	34.98
✓	✓	✓	36.62	35.92	40.13	38.03	38.39	38.55	37.79	38.17	36.58	36.30	38.11	37.15	36.88	38.00	34.79	36.10

Appendix

Table 9. Ablation of FineHOI on HICO-DET dataset across UO, UC-RF, UC-NF, and UC settings. UV setting is shown in Table 3 SM stands for soft-masks ( Section 3.1 ), APA for Adaptive Part-Level Attention ( Section 3.2 ), and

\mathcal{I}

for the Region-Aware Interaction Transformer ( Section 3.3 ). Best results in bold.

Figure 6. Qualitative results of the unsupervised part decomposition ( Section 3.2 ) with $K=\{1,5,10,20\}$ .

$K$	Full	Seen	Unseen	HM
1	36.76	37.40	32.90	35.15
5	37.37	37.90	34.18	36.04
10	38.10	38.51	35.58	37.04
20	37.02	37.50	34.13	35.81

Appendix

Table 10. Effectiveness of

K

on HICO-DET under the UV setting. Best results in bold.

Figure 7. Attention maps of the interaction transformer $\mathcal{I}$ in FineHOI. Warmer colors indicate higher attention to visual patches.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition
Feb 16, 2026Shiyu Xuan, Dongkai Wang, Zechao Li +1Human-Object Interaction Zero-Shot Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University
Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection
Apr 2, 2026Soo Won Seo, KyungChae Lee, Hyungchan Cho +3Human-Object Interaction
Seoul National University · Hanyang University
Streamlined Open-Vocabulary Human-Object Interaction Detection
Mar 29, 2026Chang Sun, Dongliang Liao, Changxing DingHuman-Object Interaction
South China University of Technology

Feb 16, 2026J/K move · Enter open · S save

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Shiyu Xuan, Dongkai Wang, Zechao Li, Jinhui Tang

School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University

Human-Object Interaction Zero-Shot Learning

Open paper AskPDF arXiv

Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. The codes are publicly available at https://github.com/SY-Xuan/DA-HOI.

Abstract

Human-Object Interaction

Explore similar work

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection

Streamlined Open-Vocabulary Human-Object Interaction Detection