Towards Unconstrained Human-Object Interaction

Francesco Tonini; Alessandro Conti; Lorenzo Vaquero; Cigdem Beyan; Elisa Ricci

cs.CVApr 15, 2026

Towards Unconstrained Human-Object Interaction

Authors: Francesco Tonini, Alessandro Conti, Lorenzo Vaquero, Cigdem Beyan, Elisa Ricci

Organizations: Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy · Department of Computer Science, University of Verona, Verona, Italy

Abstract

Human-Object Interaction (HOI) detection is a longstanding computer vision problem concerned with predicting the interaction between humans and objects. Current HOI models rely on a vocabulary of interactions at training and inference time, limiting their applicability to static environments. With the advent of Multimodal Large Language Models (MLLMs), it has become feasible to explore more flexible paradigms for interaction recognition. In this work, we revisit HOI detection through the lens of MLLMs and apply them to in-the-wild HOI detection. We define the Unconstrained HOI (U-HOI) task, a novel HOI domain that removes the requirement for a predefined list of interactions at both training and inference. We evaluate a range of MLLMs on this setting and introduce a pipeline that includes test-time inference and language-to-graph conversion to extract structured interactions from free-form text. Our findings highlight the limitations of current HOI detectors and the value of MLLMs for U-HOI. Code will be available at https://github.com/francescotonini/anyhoi

Figures & tables

Fig. 1: Comparison between traditional HOI settings and our U-HOI. Unlike prior art, U-HOI does not require training and eliminates the need for a predefined list of interactions. We tackle U-HOI using off-the-shelf Multimodal Large Language Models (MLLMs), processing their free-form outputs and applying a post-generation refinement that extracts HOI triplets, all without being constrained by a fixed vocabulary.

	Interaction Vocabulary	Object-Verb Priors
Closed-set , e.g , [ 68 , 18 , 69 ]	$\mathcal{V}_{train}=\mathcal{V}_{test}$	Required
Open-vocabulary , e.g , [ 25 , 43 , 50 , 66 , 64 ]	$\mathcal{V}_{train}\neq\mathcal{V}_{test}$	Required
Unconstrained HOI (Ours)	Not provided	Not required

TABLE I: Unlike conventional methods with predefined object-verb vocabularies, our Unconstrained HOI task infers interactions without prior combinations.

Fig. 2: AnyHOI first detects human-object pairs using an object detector. Then, it crops the region encompassing both the human and its paired object and feeds it into an MLLM, along with a prompt to generate free-form scene descriptions. A post-generation refinement step then analyzes these descriptions to produce a relationship graph. Finally, the graph is refined by filtering out the triplets that are irrelevant.

Fig. 3: The test-time compute strategy generates multiple responses using its baseline MLLM, producing a diverse set of interaction proposals, which are aggregated and sampled based on their frequency, yielding the final predictions.

Method		Annotated-box		Computed-box
Method		mAP (%) $\uparrow$	SR (x100) $\uparrow$	mAP (%) $\uparrow$	SR (x100) $\uparrow$
Random		0.30	35.89	0.05	6.68
	VG	9.80	55.83	1.45	9.71
CLIP [ 52 ]	WN	2.95	39.75	0.37	7.22
	VG	8.73	52.85	1.06	9.89
DHD [ 64 ]	WN	8.86	50.36	1.98	10.31

TABLE II: U-HOI performance on VG-HOI [ 64 ] dataset under both annotated and computed-box settings.

Method		mAP (%) $\uparrow$			SR $\mathbf{\uparrow}$
Method		Full	Rare	Non-rare	(x100)
Random		0.69	0.27	0.81	22.22
	VG	3.44	7.42	2.26	26.46
CLIP [ 52 ]	WN	3.13	7.90	1.70	23.57
GEN-VLKT [ 34 ]	VG	4.99	5.18	4.93	29.77
GEN-VLKT [ 34 ]	WN	3.67	2.09	4.14	28.89

TABLE III: U-HOI performance on HICO-DET [ 5 ] dataset under computed-box setting.

Method		mAP (%) $\uparrow$			SR $\mathbf{\uparrow}$
Method		Full	Rare	Non-rare	(x100)
Random		0.79	0.27	0.95	31.68
	VG	5.49	11.47	3.44	37.55
CLIP [ 52 ]	WN	4.51	11.25	2.50	33.86
CogVLM2 19B [ 13 ]		2.08	1.57	2.23	9.94
+ AnyHOI (Ours)		13.01 ↑10.93	12.45 ↑10.88	13.18 ↑10.95	40.96 ↑31.02

TABLE IV: U-HOI performance on HICO-DET [ 5 ] dataset under annotated-box setting.

Method		mAP Avg. (%) $\uparrow$			SR $\mathbf{\uparrow}$
Method		Full	Rare	Non-rare	(x100)
	CLIP [ 52 ] (VG)	5.49	11.47	3.44	37.55
	AnyHOI	13.46	15.12	12.96	39.91
Annot.	+ TT	20.75 ↑7.29	25.25 ↑10.13	19.40 ↑6.44	63.54 ↑23.63
	LAIN ${}_{\text{CVPR25}}$ [ 21 ] (VG)	6.97	2.72	8.24	30.92
	AnyHOI	6.43	5.09	6.84	20.04

TABLE V: Performance of AnyHOI with test-time using LLaVA OV 0.5B on HICO-DET [ 5 ] dataset.

Method		mAP Avg. (%) $\uparrow$	SR (x100) $\mathbf{\uparrow}$
	DHD ${}_{\text{AAAI24}}$ [ 64 ] (WN)	8.86	50.36
	AnyHOI	12.76	42.70
Annot.	+ TT	24.46 ↑11.70	72.82 ↑30.12
	DHD ${}_{\text{AAAI24}}$ [ 64 ] (WN)	1.98	10.31
	AnyHOI	1.99	8.10
Comp.	+ TT	3.47 ↑1.48	12.99 ↑4.89

TABLE VI: Performance of AnyHOI with test-time using LLaVA OV 0.5B on VG-HOI [ 64 ] dataset.

Method		Method	mAP Avg. (%) $\uparrow$			SR $\mathbf{\uparrow}$
Method		Method	Full	Rare	Non-rare	(x100)
	0.5B [ 27 ]	Sampling	18.90	21.71	18.06	57.31
	0.5B [ 27 ]	Top-k	20.75	25.25	19.40	63.54
		Sampling	21.32	26.86	19.67	56.46
LLaVA	7B [ 27 ]	Top-k	23.01	29.03	21.20	61.14

TABLE VII: Performance of AnyHOI + TT using different test-time sampling strategies on HICO-DET [ 5 ] dataset under annotated-box setting.

Fig. 4: Qualitative results of AnyHOI and CLIP [ 52 ] on the HICO-DET [ 5 ] dataset

Vision	mAP Avg. (%) $\uparrow$			SR $\mathbf{\uparrow}$
Prompts	Full	Rare	Non-rare	(x100)
Blind	4.12	0.53	5.19	37.25
Red circle	12.44	13.76	12.04	38.25
Reverse blur	13.06	13.55	12.92	38.33
Crop w/ mask	13.01	13.90	12.74	38.21
Crop	13.46	15.12	12.96	39.91

TABLE VIII: Evaluation of visual prompting techniques using LLaVA OV 0.5B on HICO-DET [ 5 ] dataset under annotated-box setting.

“PP” stands for our post-processing pipeline ( Sec. IV ).
Method	PP	mAP Avg. (%) $\uparrow$			SR $\mathbf{\uparrow}$
		Full	Rare	Non-rare	(x100)
Structured	✗	6.20	8.40	5.54	26.89
CoT	✓	10.24	11.04	9.71	31.18
Descriptive	✓	10.90	11.60	10.69	33.53
Direct	✓	13.46	15.12	12.96	39.91

TABLE IX: Performance of AnyHOI using different prompts on HICO-DET [ 5 ] dataset under annotated-box setting.

Fig. 5: Evaluation of DHD [ 64 ] on the VG-HOI [ 64 ] dataset with and without the occurrence matrix

\mathcal{M}

Method	Vision Tower		Text Tower
	Model	Num. params.	Model	Num. params.
Idefics2 [ 23 ]	SigLIP	400M	Mistral	7B
InstructBLIP [ 16 ]	ViT-g (BLIP-2)	1.1B	Vicuna	7B/13B
InternVL2 [ 7 ]	InternViT	300M/6B	Qwen2 500MB or InternLM2 8B	500M/8B
LLaVA-OV [ 27 ]	SOViT (SigLIP)	400M	Qwen2	500M/7B
Phi3V [ 1 ]	ViT-L (CLIP)	400M	3.8B	MIT

TABLE X: Summary of the vision and language towers of the MLLMs we tested in this work.

Prompts
	What are the interactions between the person
Direct	and the obj ?
CoT	What are the interactions between the person
CoT	and the obj ? Think step by step.
	Describe all the interactions occurring between the person
	and the obj . If no interactions are being performed

TABLE XI: The prompts used in the ablation study in Sec. V .

Method		mAP Avg. (%) $\uparrow$			SR $\mathbf{\uparrow}$
Method		Full	Rare	Non-rare	(x100)
CoT	Idefics2 8B [ 23 ]	14.04	14.96	13.77	42.87
	InternVL 2B [ 7 ]	8.46	9.72	8.08	28.51
	InternVL 8B [ 7 ]	9.40	9.82	9.27	30.21
	InstructBLIP 7B [ 16 ]	11.46	14.52	10.54	41.68
	InstructBLIP 13B [ 16 ]	11.12	13.32	10.47	42.70

TABLE XII: Comparison of AnyHOI under different prompts on the HICO-DET [ 5 ] dataset and the annotated-box setting. Best results are in bold .

Method		$k$	mAP Avg. (%) $\uparrow$			SR $\mathbf{\uparrow}$
Method		$k$	Full	Rare	Non-rare	(x100)
CLIP [ 52 ]	VG-HOI [ 64 ]	1	2.09	5.44	1.10	21.79
		2	3.11	7.77	1.71	26.60
		5	4.51	10.70	2.66	32.80
		10	5.49	11.47	3.44	37.55
		1	1.32	3.16	0.76	21.25

TABLE XIII: The impact of

k

, i.e , the number of interaction triplets at test time on the HICO-DET [ 5 ] dataset and the annotated-box setting.

Setting	Method	mAP Avg. (%) $\uparrow$			SR $\mathbf{\uparrow}$
Setting	Method	Full	Rare	Non-rare	(x100)
	LLaVA OV 0.5B [ 27 ]	13.46	15.12	12.96	39.91
Annotated-box	+ LoRA [ 15 ]	27.06	17.02	30.06	58.64
Computed-box	LLaVA OV 0.5B [ 27 ]	6.43	15.12	12.96	36.25
Computed-box	+ LoRA [ 15 ]	16.72	10.89	18.46	43.65

TABLE XIV: Performance of AnyHOI on LLaVA OV 0.5B on HICO-DET [ 5 ] using the pre-trained weights or after a LoRA finetuning stage with the HICO-DET [ 5 ] annotated-box setting training set. Best results are in bold .

Setting	Method	mAP Avg. (%) $\uparrow$	SR (x100) $\mathbf{\uparrow}$
	LLaVA OV 0.5B [ 27 ]	12.76	42.70
Annotated-box	+ LoRA [ 15 ]	21.04	66.49
Computed-box	LLaVA OV 0.5B [ 27 ]	1.99	8.10
Computed-box	+ LoRA [ 15 ]	2.76	12.55

TABLE XV: Performance of AnyHOI on LLaVA OV 0.5B on VG-HOI [ 64 ] using the pre-trained weights or after a LoRA finetuning stage with the HICO-DET [ 5 ] annotated-box setting training set. Best results are in bold .

Fig. 6: Results of LLaVA + AnyHOI + TT varying the number of generations on the HICO-DET [ 5 ] dataset under the annotated-box setting. Other methods using AnyHOI without TT are plotted for reference.

Model	Inference time (s)	Full mAP
CogVLM2	1.03	1.34
+ AnyHOI	1.13	8.10
InternVL2-8B	0.77	1.91
+ AnyHOI	0.87	6.47
Qwen2-VL	0.65	4.03
+ AnyHOI	0.77	9.35

TABLE XVI: Inference times per interaction of MLLMs and AnyHOI.

Fig. 7: mAP of MLLMs on grouped verbs by topic.

Fig. 8: mAP of MLLMs on opposite verbs.

Fig. 9: mAP of MLLMs of social interaction verbs.

Fig. 10: mAP of MLLMs on object manipulation verbs.

Fig. 11: mAP of MLLMs performance on verbs with propositions

Method		mAP @0.6 (%) $\uparrow$			mAP @0.7 (%) $\uparrow$			mAP @0.8 (%) $\uparrow$			mAP @0.9 (%) $\uparrow$			mAP @0.95 (%) $\uparrow$
Method		Full	Rare	Non-rare	Full	Rare	Non-rare	Full	Rare	Non-rare	Full	Rare	Non-rare	Full	Rare	Non-rare
	VG-HOI	9.69	16.24	7.73	8.14	17.05	5.48	5.86	15.93	2.85	2.78	8.14	1.18	0.00	0.00	0.00
CLIP [ 52 ]	WordNet	7.34	16.46	4.61	6.88	16.51	4.00	5.83	15.54	2.92	2.52	7.73	0.97	0.00	0.00	0.00
CogVLM2 19B [ 13 ]		3.03	2.10	3.30	2.67	2.10	2.84	1.73	1.37	1.84	1.51	1.15	1.61	1.49	1.15	1.59
+ AnyHOI (Ours)		14.27	12.78	14.72	13.51	12.64	13.77	12.55	12.51	12.56	12.39	12.17	12.45	12.34	12.17	12.39
Idefics2 8B [ 23 ]		8.19	7.28	8.46	7.85	6.89	8.14	6.07	5.50	6.24	5.46	5.11	5.57	5.40	5.11	5.49

TABLE XVII: Performance on the HICO-DET [ 5 ] dataset for U-HOI under the annotated-box setting, using different thresholds for mAP calculation.

Method		mAP @0.6 (%) $\uparrow$			mAP @0.7 (%) $\uparrow$			mAP @0.8 (%) $\uparrow$			mAP @0.9 (%) $\uparrow$			mAP @0.95 (%) $\uparrow$
Method		Full	Rare	Non-rare	Full	Rare	Non-rare	Full	Rare	Non-rare	Full	Rare	Non-rare	Full	Rare	Non-rare
	VG-HOI	6.36	10.71	5.06	5.47	11.39	3.70	3.54	9.37	1.79	1.85	5.61	0.73	0.00	0.00	0.00
CLIP [ 52 ]	WordNet	4.86	11.51	2.88	4.72	11.56	2.68	4.12	10.67	2.16	1.94	5.77	0.80	0.00	0.00	0.00
GEN-VLKT ${}_{\text{CVPR22}}$ [ 34 ]	VG-HOI	8.18	7.67	8.34	6.07	7.13	5.75	4.90	6.20	4.52	3.00	2.43	3.18	2.82	2.51	2.91
GEN-VLKT ${}_{\text{CVPR22}}$ [ 34 ]	WordNet	6.28	4.76	6.73	4.65	2.48	5.29	3.37	1.74	3.86	2.14	0.73	2.56	1.91	0.73	2.27
	VG-HOI	1.67	2.66	1.37	1.56	2.38	1.31	1.46	1.9	1.32	1.48	1.91	1.35	1.48	1.91	1.35

TABLE XVIII: Performance on the HICO-DET [ 5 ] dataset for U-HOI under the computed-box setting, using different thresholds for mAP calculation

Method		mAP @0.6 (%) $\uparrow$	mAP @0.7 (%) $\uparrow$	mAP @0.8 (%) $\uparrow$	mAP @0.9 (%) $\uparrow$	mAP @0.95 (%) $\uparrow$
	VG-HOI	27.70	15.33	3.63	1.29	1.07
CLIP [ 52 ]	WordNet	4.96	3.30	2.78	1.97	1.72
DHD ${}_{\text{AAAI24}}$ [ 64 ]	VG-HOI	21.41	18.54	2.57	0.65	0.50
DHD ${}_{\text{AAAI24}}$ [ 64 ]	WordNet	11.34	9.34	8.79	7.57	7.27
CogVLM2 [ 13 ]		4.82	3.59	3.00	0.00	0.00
+ AnyHOI (Ours)		32.66	28.99	26.02	2.45	0.02

TABLE XIX: Performance on the VG-HOI [ 64 ] dataset for U-HOI under the annotated-box setting, using different thresholds for mAP calculation.

Method		mAP @0.6 (%) $\uparrow$	mAP @0.7 (%) $\uparrow$	mAP @0.8 (%) $\uparrow$	mAP @0.9 (%) $\uparrow$	mAP @0.95 (%) $\uparrow$
	VG-HOI	3.54	2.30	0.77	0.38	0.27
CLIP [ 52 ]	WordNet	0.65	0.39	0.37	0.30	0.14
DHD [ 64 ]	VG-HOI	2.16	1.91	0.74	0.26	0.23
DHD [ 64 ]	WordNet	2.19	2.17	2.05	1.76	1.76
CogVLM2 [ 13 ]		0.53	0.40	0.19	0.00	0.00
+ AnyHOI (Ours)		4.04	3.34	2.55	0.61	0.00

TABLE XX: Performance on the VG-HOI [ 64 ] dataset for U-HOI under the computed-box setting, using different thresholds for mAP calculation.

Fig. 12: Qualitative results of LLaVA OV 0.5B [ 27 ] + AnyHOI + TT and CLIP [ 52 ] on the HICO-DET [ 5 ] annotated-box setting. Our proposed AnyHOI surpasses the CLIP baseline in predicting interactions.

Fig. 13: Qualitative results of LLaVA OV 0.5B [ 27 ] + AnyHOI + TT, and the intermediate outputs, on the HICO-DET [ 5 ] annotated-box setting.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
Oct 5, 2026Zhaolin Cai, Huiyu Duan, Liu Yang +5Multimodal Large Language Models MLLM Reasoning
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1
Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition
Feb 16, 2026Shiyu Xuan, Dongkai Wang, Zechao Li +1Human-Object Interaction Zero-Shot Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University
Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interaction
Oct 13, 2026Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi +1Human-Object Interaction Multimodal Large Language Models
Milestone Systems A/S, Banemarksvej 50, 2605 Broendby, Denmark · Universitat de Barcelona, Gran Via de les Corts Catalanes 585, 08007, Barcelona, Spain · Aalborg University, Rendsburggade 14, 9000 Aalborg, Denmark

Oct 5, 2026J/K move · Enter open · S save

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

Zhaolin Cai, Huiyu Duan, Liu Yang, Yanjun Qin+4

Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University+1

Multimodal Large Language Models MLLM Reasoning Active Visual Perception Human-Object Interaction

Open paper AskPDF arXiv

Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform strongly but rely on task-specific training. Recent multimodal large language models (MLLMs) offer a promising route to training-free HOI detection through their broad visual-semantic knowledge and versatile perceptual and reasoning capabilities. However, existing approaches largely invoke these capabilities through loosely coordinated inference stages. This fragmented execution restricts the role of interaction hypotheses in guiding visual exploration, leaving key participants overlooked and local ambiguities unresolved. Furthermore, propagating early semantic assumptions through subsequent visual grounding and relation prediction induces self-reinforcing semantic circularity. To resolve these challenges, we propose HarnessHOI, a training-free framework that transforms passive MLLM inference into an active interaction-centric harness. Specifically, we introduce an interaction-guided perception mechanism that projects emerging interaction hypotheses back into the visual space to discover missing participants and refine ambiguous evidence through targeted observation. Furthermore, a relation-agnostic geometric adjudication module reconciles multi-source evidence to establish a unified spatial basis for grounded interaction reasoning across multiple actions and semantic roles. Extensive experiments on HICO-DET and V-COCO demonstrate that HarnessHOI achieves state-of-the-art performance among training-free methods, confirming the effectiveness of the proposed harness for complex interaction understanding. Code will be released upon publication.

Abstract

Human-Object Interaction

Explore similar work

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interaction