Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interaction

Oriol Rabasseda; Zenjie Li; Kamal Nasrollahi; Sergio Escalera

cs.CVOct 13, 2026

Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interaction

Authors: Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi, Sergio Escalera

Organizations: Milestone Systems A/S, Banemarksvej 50, 2605 Broendby, Denmark · Universitat de Barcelona, Gran Via de les Corts Catalanes 585, 08007, Barcelona, Spain · Aalborg University, Rendsburggade 14, 9000 Aalborg, Denmark

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong zero-shot reasoning across diverse domains, but their potential for Human-Object Interaction (HOI) remains unexplored. Unlike traditional HOI approaches that rely on extensive labeled datasets and fixed interaction taxonomies, MLLMs offer flexible reasoning, making them well-suited for dynamic, real-world scenarios. In this work, we present the first systematic study of MLLMs for HOI recognition and detection within a pipeline that combines an external object detector for entity localization with zero-shot multimodal reasoning for interaction understanding. We introduce a benchmark framework for principled zero-shot evaluation, consisting of (i) a unified taxonomy of zero-shot prompting strategies, and (ii) a methodology that transforms token-level logits into interaction scores, enabling fair comparison with specialized HOI models under standard metrics. To support prompt-based evaluation, we additionally construct new question-answer annotations for standard HOI datasets, transforming them into a format suitable for MLLMs. Building on this, we propose two visual prompting strategies, namely Objects as Context and Black Other Objects, that guide MLLMs to target objects while preserving scene context. Experiments on HICO, HICO-DET, and V-COCO demonstrate state-of-the-art zero-shot performance, achieving 53.50 mAP on HICO and 23.69 mAP on HICO-DET. Overall, our work provides a principled benchmarking framework for MLLMs in HOI, opening the door to new models and datasets and enabling flexible open-world interaction understanding.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Visual example of Objects as Context: cropped relevant objects are input alongside text prompts to enrich visual tokens. (b) Visual example of Black Other Objects: all non-target objects are masked to remove distractions.

Figure 2: Taxonomy of zero-shot methods leveraging MLLMs divided into textual, visual, and hybrid strategies.

Taxonomy	Prompting methods
Ensemble	Resampling, Self-consistency ( Wang et al. [2022] ), Tree-of-thoughts ( Yao et al. [2023] ), Graph-of-thoughts ( Besta et al. [2024] )
Generated Knowledge	Multimodal Chain-of-thought ( Zhang et al. [2023b] ), Generated knowledge ( Liu et al. [2021] )
Task Decomposition	Prompt chaining, Least-to-most prompting ( Zhou et al. [2022] )
Templates	Role-prompting, Multimodal Chain-of-thought ( Zhang et al. [2023b] ), Meta prompting ( Zhang et al. [2023a] ), Prompt Pattern Catalog ( White et al. [2023] )
External Knowledge	FLARE ( Jiang et al. [2023] ), In-Context RALM ( Ram et al. [2023] )
Saliency	RedCircle ( Shtedritski et al. [2023] ), FGVP ( Yang et al. [2023c] )

Table 1: Position in zero-shot taxonomy of relevant methods.

Figure 3: Zero-shot HOI recognition method using MLLMs. The output $\hat{y}_{ij}$ represents the classification score for object class $i$ and interaction $j$ , $P(c_{ij})$ denotes the classification probability computed according to Equation ( 1 ), and $\tau$ is the detection threshold for querying the MLLM. The image is taken from the HICO test set.

Figure 4: Objects as Context architecture. Step 1: Detect all objects. Step 2a: Filter relevant objects ( e.g. “backpacks”) based on the question. Step 2b: Filter all persons. Step 3: Input the information into the LLM, specifying visual tokens related to relevant objects/persons and the original image.

Category (from Taxonomy)	Strategy	HOI Recognition	HOI Detection
Hybrid	Multimodal Chain-of-Thought ( Zhang et al. [2023b] )	✓	–
Textual – Generated Knowledge	Caption as Context	✓	–
Textual – Ensemble	Ensemble of Prompts	✓	–
Textual – External Knowledge	Textual Center Coordinates, Textual Boundary Coordinates	–	✓
Visual – Contextual Information	Objects as Context (ours)	✓	✓
Visual – Markers	Overlay Bounding Boxes, Overlay Elliptic Markers	–	✓

Table 2: Summary of prompting strategies evaluated in our benchmark, grouped by the taxonomy in Section 3 .

Panel of Figure 5 — Figure 5: Example of all visual prompting methods side-by-side compared to the base image (a). (b) Overlay Bounding Boxes highlights humans and objects with rectangular boxes. (c) Overlay Elliptic Markers uses ellipses to indicate the queried regions. (d) Blur the Background removes contextual clutter by blurring everything outside the target entities. (e) Crop focuses only on the cropped human–object pair. (f) Black Other Objects masks all objects except the queried pair while preserving the background. (g) Gray the Background reduces distraction by desaturating non-target regions.

Model	mAP $\uparrow$	Diff. Answers	Incoherences
MiniCPM-V 1	48.20	0%	10.06%
MiniCPM-V 2	47.48	0%	2.74%
LLaVA-OV-0.5B (OV)	46.15	0%	1.25%
LLaVA-OV-0.5B (SI)	44.61	0%	2.27%
Phi-3.5-Vision	40.44	7.05%	0.00%
InternVL2.5-1B	41.81	7.44%	0.48%

Table 3: Comparison of MLLMs on the HICO dataset: performance and score soundness study.

Strategy	MiniCPM-V 1	Qwen2-VL 2B	LLaVA-OV 0.5B (OV)
MCoT	39.45	30.94	-
Caption as Context	44.81	49.43	-
Objects as Context	44.54	53.50	47.79

Table 4: Evaluations on HICO dataset using zero-shot strategies.

Panel of Figure 6 — Figure 6: Evaluation on HICO dataset with different number of prompts using Ensemble of Prompts.

Method	mAP
Supervised methods
R*CNN ( Gkioxari et al. [2015] )	28.5
PaStaNet ( Li et al. [2020] )	46.3
DEFR ( Jin et al. [2022] )	65.6
Partially zero-shot
VCL † ( Hou et al. [2020] )	26.7

Table 5: Comparison with prior work for HOI recognition on HICO. Note: † results are reported in Ma et al. [2022] .

Method	HICO-DET			V-COCO
Method	Full	Rare	Non-rare	${AP}_{role}^{\#1}$	${AP}_{role}^{\#2}$
MiniCPM-V 1
Baseline	20.78	26.63	19.03	19.95	21.98
Objects as Context	22.17	27.53	20.57	21.46	23.09
Overlay Bounding Boxes	15.90	19.55	14.81	5.65	6.11
Overlay Elliptic Marker	17.70	22.55	16.25	5.94	6.24

Table 6: Pure zero-shot evaluation on HICO-DET and V-COCO of the prompting strategies for object specification.

Method	HICO-DET			V-COCO
Method	Full	Rare	Non-rare	${AP}_{role}^{\#1}$	${AP}_{role}^{\#2}$
Fully supervised
GEN-VLKT ( Liao et al. [2022] )	34.95	31.18	36.08	63.6	65.9
SICHOI ( Luo et al. [2024] )	45.04	45.61	44.88	71.1	75.6
Pose-Aware ( Wu et al. [2024a] )	46.01	46.74	45.80	63.0	68.7
Pre-trained object detector

Table 7: Comparison with other methods on HICO-DET and V-COCO. Fully supervised methods are trained end-to-end with HOI annotations, while methods with a pre-trained object detector reuse a pre-trained object detector and train only the interaction head. Partially zero-shot methods hold out 120 action–object pairs during training and report HICO-DET performance over the full set of 600 classes, without separate Rare and Non-rare results. V-COCO does not define an analogous held-out setting. Pure zero-shot methods do not use any HOI training data. Not all methods are evaluated on both datasets; - denotes results that are not reported. We report the best overall configuration from Table 6 as our method.

Method	mAP
Method	Full	Rare	Non-rare
MiniCPM-V 1
Baseline	46.35	47.79	45.32
Objects as Context	48.41	50.85	47.68
Overlay Bounding Boxes	39.31	42.06	38.49
Overlay Elliptic Marker	42.89	45.38	42.14

Table 8: Pure zero-shot evaluation on HICO-DET removing the effect of the object detector.

Panel of Figure 7 — Figure 7: Image examples of the HOIs that exhibit the highest performance gains when applying the identified prompting methods. Objects as Context improves detection for interactions Sign baseball bat (+47.69 AP), Wash sheep (+29.44 AP) and Repair oven (+27.26 AP), while Black Other Objects enhances detection for Inspect backpack (+34.43 AP), Sign baseball bat (+28.83 AP) and Eat cake (+28.75 AP).

Panel of Figure 8 — Figure 8: Image examples of the HOIs that exhibit the highest performance losses when applying the identified prompting methods. Objects as Context suffers detection decrease for interactions No_interaction spoon (-28.90 AP), No_interaction carrot (-21.41 AP) and Fill wine glass (-17.62 AP), while Black Other Objects downgrades detection for Lie on surfboard (-48.34 AP), No_interaction bicycle (-36.23 AP) and Buy pizza (-35.97 AP).

Panel of Figure 9 — Figure 9: Impact of prompting methods on HICO-DET using ground truth bounding boxes. (a) Number of HOIs showing an increase or decrease in AP compared to the baseline for all prompting methods. (b) Performance of the proposed methods relative to the number of detected objects of the target class. (c) Performance gains of the proposed methods on the most challenging classes. Most challenging classes are considered those that have a lower average AP over all object specification approaches.

Panel of Figure 10 — Figure 10: Examples where Objects as Context (a-d) and Black Other Objects (e-f) correctly classify a HOI that all other methods fail to detect: (a) a focused crop of the pilot, (b) a crop of the person holding the STOP sign, (c) a crop of the small and partially occluded cell phone, and (d) a crop of the laptop and the interacting person presumably help the model with more specific visual context. (e) Masking other cyclists and (f) masking irrelevant overlapping objects presumably direct model’s attention to the correct interaction.

Object Detector	MiniCPM-V 1	Qwen2-VL 2B	#Questions
YOLOX-x ( Ge et al. [2021] )	48.59	52.45	275,746
YOLO11-x ( Khanam and Hussain [2024] )	48.20	51.39	251,593
Faster R-CNN ( Ren et al. [2015] )	48.31	52.28	351,466
DINO ( Zhang et al. [2022a] )	51.27	55.49	352,653

Table 9: Comparison of object detectors. Evaluation is conducted on HICO. # Questions denotes the number of retained queries after the object detector filtering step. All models are trained on MS-COCO.

$\tau$	MiniCPM-V 1	Qwen2-VL 2B	#Questions
0.00	50.98	53.59	990,910
0.05	49.33	52.61	273,668
0.10	48.20	51.39	251,593
0.25	47.92	51.67	223,304
0.40	47.06	50.82	206,492
0.50	46.23	49.63	195,441

Table 10: Effect of varying class score thresholds in the pipeline. Evaluation conducted on HICO.

Table 21

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	MiniCPM-V 1	Qwen2-VL 2B	LLaVA-OneVision 7B
Baseline	98.88	99.10	98.68
MCoT	99.15	98.99	-
Caption as Context	99.03	99.21	-
Objects as Context	98.67	99.16	99.51

Appendix

Table D.1 : Thresholded accuracy (%) on HICO. Binary predictions are obtained by thresholding the predicted probabilities at

0.5

Hyperparameter	Value
Attention implementation	Flash Attention 2 from Dao [2023]
Generation strategy	Greedy
Maximum number of new tokens	1
Maximum number of output tokens for generated knowledge	256
Minimum size of cropped objects 3 3 3 Only apply to Qwen2-VL and LLaVA-OneVision models.	28 per side

Appendix

Table E.1 : List of hyperparameters for MLLMs in inference mode.

Model	MiniCPM-V 1	Qwen2-VL 2B	#Questions
nano	43.37	47.16	240,581
small	45.86	49.73	249,280
medium	47.28	51.15	251,102
large	48.55	52.32	252,084
extra-large	48.20	51.39	251,593
MLLM (Qwen2-VL 2B)	50.96	14.45	264,782

Appendix

Table F.1 : Performance across YOLO11 model sizes. Evaluation conducted on HICO. All detectors are trained on MS-COCO.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition
Feb 16, 2026Shiyu Xuan, Dongkai Wang, Zechao Li +1Human-Object Interaction Zero-Shot Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University
Towards Unconstrained Human-Object Interaction
Apr 15, 2026Francesco Tonini, Alessandro Conti, Lorenzo Vaquero +2Human-Object Interaction Multimodal Large Language Models
Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy · Department of Computer Science, University of Verona, Verona, Italy
Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
Oct 5, 2026Zhaolin Cai, Huiyu Duan, Liu Yang +5Multimodal Large Language Models MLLM Reasoning
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1

Feb 16, 2026J/K move · Enter open · S save

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Shiyu Xuan, Dongkai Wang, Zechao Li, Jinhui Tang

School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University

Human-Object Interaction Zero-Shot Learning

Open paper AskPDF arXiv

Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. The codes are publicly available at https://github.com/SY-Xuan/DA-HOI.

Abstract

Human-Object Interaction

Explore similar work

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Towards Unconstrained Human-Object Interaction

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection