Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Shiyu Xuan; Dongkai Wang; Zechao Li; Jinhui Tang

cs.CVFeb 16, 2026

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Authors: Shiyu Xuan, Dongkai Wang, Zechao Li, Jinhui Tang

Organizations: School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University

Abstract

Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. The codes are publicly available at https://github.com/SY-Xuan/DA-HOI.

Figures & tables

Figure 1: (a) Existing methods, including two-stage methods, couple object detection and interaction recognition together. Their performance are constrained by the limited generalization of detector features and coarse-grained VLM features. (b) Our method fully decouples these two processes and harnesses the powerful MLLMs for interaction recognition. This design benefits from the generalization of MLLMs and advanced detectors.

Figure 2: The overall framework of our method and the spatial-aware pooling (SAP). (a) The proposed method decouples the object detection and interaction recognition for HOI detection. With the detected human-object pair, a MLLM is used to recognize their interaction. To enhance both performance and inference efficiency, SAP integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method enables the prediction of all candidate interactions in a single forward pass. (b) SAP takes the human and object features as input. The cross attention layer aggregates features beyond the area of bounding box, enhancing robust to the noise in the detection results. Spatial Embedding encodes the useful pairwise information into the interaction features.

Methods	RF-UC			NF-UC			UO			UV			Avg
Methods	Unseen	Seen	Full	Unseen	Seen	Full	Unseen	Seen	Full	Unseen	Seen	Full	Full
ADA-CM	27.63	34.35	33.01	32.41	31.13	31.39	-	-	-	-	-	-	-
ADA-CM $\dagger$	20.82	30.12	28.26	26.23	27.03	26.87	-	-	-	-	-	-	-
ADA-CM $\ddagger$	20.43	29.23	27.47	24.89	27.69	27.13	-	-	-	-	-	-	-
BCOM	28.52	35.04	33.74	33.12	31.76	32.03	-	-	-	-	-	-	-
BCOM $\dagger$	9.23	23.08	20.31	16.38	20.73	19.86	-	-	-	-	-	-	-

Table 1: Performance comparison on HICO-DET under zero-shot, cross-detector and training-free settings.

\diamond

indicates BLIP2 ( Li et al., 2023 ) features is used.

\dagger

and

\ddagger

indicate Grounding-DINO ( Liu et al., 2024 ) and Yolo-World ( Cheng et al., 2024 ) are used as the detector, respectively.

\star

is the training-free setting.

Methods	H $\rightarrow$ V-COCO
Methods	mAP ${}^{\#2}_{role}$
GEN-VLKT	42.34
ADA-CM	46.39
HOICLIP	41.82
BCOM	48.87
CMMP	47.65

Table 2: Performance comparison on V-COCO under cross-dataset setting.

Settings	Full	Format Error Rate	Single Output Rate
Simple	14.23	36.78	80.91
Multiple Choice Questions	18.91	23.75	80.17
In Context	19.57	18.75	78.82
Simple $*$	16.19	36.78	80.91
Multiple Choice Questions $*$	20.36	23.75	80.17
In Context $*$	20.97	18.75	78.82

Table 3: Ablation studies on the deterministic generation. “Simple”, “Multiple Choice Questions”, “In Context” are the different formulations of question prompts. Some examples are shown in A.5 . “*” denotes leveraging the conditional likelihood of the answer as the confidence score of the prediction. “Format Error Rate” denotes the rate at which the format of the answer is error. “Single Output Rate” indicates the rate at which the answer only contains one interaction. “SFT” indicates the model fine-tuned on the HICO-DET with supervised fine-tuning.

Methods	UO			UV			Inference
Methods	Unseen	Seen	Full	Unseen	Seen	Full	Time ( ms )
Baseline	43.66	38.36	39.24	34.06	38.46	37.84	569
w/ SAP	45.91	41.59	42.31	36.47	42.84	41.95	217
w/ DM	45.63	39.47	40.50	36.14	39.75	39.24	189
w/o Pairwise Spatial	44.31	41.52	41.98	35.32	41.66	40.77	86
w/o Cross Attention	43.61	40.92	41.37	34.50	41.76	40.74	87

Table 4: Ablation studies on each proposed component. The model trained with SFT and inference with deterministic generation serves as the baseline. “SAP” and “DM” are the spatial-aware pooling module and deterministic matching, respectively. “Pairwise Spatial” and “Cross Attention” are the integration of pairwise spatial information and cross attention to aggregate image features, respectively. “UPT” indicates the unary-pairwise transformer is used instead of SAP.

Unseen	Seen	Full
48.65 $\pm$ 0.03	42.59 $\pm$ 0.02	43.61 $\pm$ 0.02

Table 5: Impact of the Candidate Order.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	UO			UV
Methods	Unseen	Seen	Full	Unseen	Seen	Full
Full Tuning LLM	47.59	43.03	43.79	35.39	43.67	42.51
Full Tuning Visual + LLM	46.47	42.78	43.39	34.10	43.05	41.80
LoRA Visual + LLM	48.26	42.74	43.66	35.66	43.38	42.30
Ours	48.67	42.58	43.60	36.89	43.84	42.88

Appendix

Table 6: Ablation studies on training settings. “Full Tuning” and “LoRA” indicate full tuning or LoRA tuning of the model, respectively. “Visual” and “LLM” denote tuning the visual encoder or LLM of MLLM, respectively.

Methods	UO			UV
Methods	Unseen	Seen	Full	Unseen	Seen	Full
LLaVA Onevision 0.5B	45.29	41.34	42.00	34.35	42.29	41.18
Qwen 2.5-VL 3B	48.67	42.58	43.60	36.89	43.84	42.88
Qwen 2.5-VL 7B	50.78	45.03	45.99	38.73	45.77	44.78

Appendix

Table 7: Ablation studies on different MLLMs.

Methods	HICO-DET			V-COCO
Methods	Full	Rare	Non-Rare	mAP ${}^{\#1}_{role}$	mAP ${}^{\#2}_{role}$
GEN-VLKT	33.75	29.25	35.10	62.4	64.5
HOICLIP	34.69	31.12	35.74	63.5	64.8
CLIP4HOI	35.33	33.95	35.74	-	66.3
ADA-CM	33.80	31.72	34.42	-	61.2
CMMP	38.14	37.75	38.25	-	64.0

Appendix

Table 8: Performance comparison on HICO-DET and V-COCO under fully supervised setting.

\diamond

indicates BLIP2 ( Li et al., 2023 ) features is used.

\dagger

and

\ddagger

indicate Grounding-DINO ( Liu et al., 2024 ) and Yolo-World ( Cheng et al., 2024 ) are used as the detector, respectively.

Settings	RF-UC			NF-UC
Settings	Unseen	Seen	Full	Unseen	Seen	Full
ADA-CM	27.63	34.35	33.01	32.41	31.13	31.39
w/ Ground-Truth	26.89	35.50	33.78	30.13	31.18	30.97
BCOM	28.52	35.04	33.74	33.12	31.76	32.03
w/ Ground-Truth	10.78	20.83	18.82	17.60	20.16	19.65
Ours	41.79	44.01	43.56	43.12	39.63	40.33

Appendix

Table 9: Performance comparison with ground-truth detection results. ADA-CM and BCOM are trained with ResNet50 DETR as the detector. “Ground-Truth” indicates the annotated bounding box is used as the detection results.

Figure 3: Visualization of successful examples. Humans (subject) are marked with red rectangles, and objects with yellow rectangles.

Figure 4: Visualization of failure cases. Humans (subject) are marked with red rectangles, and objects with yellow rectangles.

Methods	Inference Time ( ms )	Avg mAP
UniHOI	86	32.08
BC-HOI	102	37.87
ADA-CM	93 (27 + 66)	-
EZ-HOI	100 (27 + 73)	36.20
Ours + ResNet50 DETR	118 (27 + 91)	42.59
Ours + Grounding-DINO	187 (96 + 91)	44.00

Appendix

Table 10: Comparisons in inference time.

Figure 5: Visualization of cross attention map obtained from the spatial-aware pooling. The human is marked with red rectangle, while the object is marked with yellow rectangle.

Figure 6: Visualization of more examples about cross attention map obtained from the spatial-aware pooling. The human is marked with red rectangle, while the object is marked with yellow rectangle.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection
Sep 5, 2026Francesco Tonini, Lorenzo Vaquero, Mohammad Mahdi Derakhshani +3Human-Object Interaction Zero-Shot Learning
University of Trento Trento, Italy · Fondazione Bruno Kessler Trento, Italy · University of Amsterdam Amsterdam, The Netherlands +1
Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interaction
Oct 13, 2026Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi +1Human-Object Interaction Multimodal Large Language Models
Milestone Systems A/S, Banemarksvej 50, 2605 Broendby, Denmark · Universitat de Barcelona, Gran Via de les Corts Catalanes 585, 08007, Barcelona, Spain · Aalborg University, Rendsburggade 14, 9000 Aalborg, Denmark
Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
Oct 5, 2026Zhaolin Cai, Huiyu Duan, Liu Yang +5Multimodal Large Language Models MLLM Reasoning
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1

Sep 5, 2026J/K move · Enter open · S save

FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

Francesco Tonini, Lorenzo Vaquero, Mohammad Mahdi Derakhshani, Cees Snoek+2

University of Trento Trento, Italy · Fondazione Bruno Kessler Trento, Italy · University of Amsterdam Amsterdam, The Netherlands+1

Human-Object Interaction Zero-Shot Learning Representation Learning

Open paper AskPDF arXiv

Human-Object Interaction (HOI) detection aims to localize humans and objects in images and classify their interactions. Zero-shot HOI focuses on recognizing interactions that are not observed during training, requiring models to generalize beyond seen verb-object compositions. Recent approaches leverage Vision-Language Models (VLMs), benefiting from rich semantic representations. However, they often rely on global or detector-centric features that compress interaction cues and hinder fine-grained spatial reasoning. To overcome this limitation, we propose FineHOI, a zero-shot HOI framework that explicitly models interactions from dense patch-level features. Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships, which are not preserved by global and detector-centric representations. To this end, we introduce an Adaptive Part-Level Attention module that decomposes humans and objects into semantically coherent parts via unsupervised clustering, and re-weights them based on their interaction relevance. These representations are then integrated through a Region-Aware Interaction Transformer that integrates part-aware and global features and produces the final HOI embedding. Extensive experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods, achieving particularly strong gains on unseen interactions. Code is available at https://github.com/francescotonini/fine-hoi.

Abstract

Human-Object Interaction

Explore similar work

FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interaction

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection