RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection

Jihwan Park; Chanhyeong Yang; Jinyoung Park; Taehoon Song; Hyunwoo J. Kim

cs.CVApr 1, 2026

RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection

Authors: Jihwan Park, Chanhyeong Yang, Jinyoung Park, Taehoon Song, Hyunwoo J. Kim

Organizations: KAIST · LG Energy Solution

Abstract

Weakly-supervised Human-Object Interaction (HOI) detection is essential for scalable scene understanding, as it learns interactions from only image-level annotations. Due to the lack of localization signals, prior works typically rely on an external object detector to generate candidate pairs and then infer their interactions through pairwise reasoning. However, this framework often struggles to scale due to the substantial computational cost incurred by enumerating numerous instance pairs. In addition, it suffers from false positives arising from non-interactive combinations, which hinder accurate instance-level HOI reasoning. To address these issues, we introduce Relational Grounding Transformer (RegFormer), a versatile interaction recognition module for efficient and accurate HOI reasoning. Under image-level supervision, RegFormer leverages spatially grounded signals as guidance for the reasoning process and promotes locality-aware interaction learning. By learning localized interaction cues, our module distinguishes humans, objects, and their interactions, enabling direct transfer from image-level interaction reasoning to precise and efficient instance-level reasoning without additional training. Our extensive experiments and analyses demonstrate that RegFormer effectively learns spatial cues for instance-level interaction reasoning, operates with high efficiency, and even achieves performance comparable to fully supervised models. Our code is available at https://github.com/mlvlab/RegFormer.

Figures & tables

Figure 1: Comparison of weakly-supervised HOI detection frameworks. (A) As the number of instance pairs increases, RegFormer shows only a marginal increase in inference time, whereas the ML-Decoder becomes significantly slower. (B) In addition, RegFormer effectively suppresses non-interactive human–object pairs, producing less false positives.

Figure 2: Overall framework of RegFormer. RegFormer unifies image-level and instance-level reasoning within a single framework by learning to capture spatial cues for interaction reasoning using only image-level labels. During training, pairwise instance encoder constructs a human–object (HO) query $q^{\text{ho}}$ by aggregating spatial features $x$ based on patch importance score $\alpha(p)$ associated with each human and object class. The resulting HO queries are processed by interaction decoder , which outputs the interaction classification scores $\hat{s}^{\text{a}}$ . To further support locality-aware learning, we introduce a spatially aggregated interactiveness score ${r}$ , which acts as a gating signal for the interaction score $\hat{s}^{\text{a}}$ and receives explicit supervision. During instance-level HOI inference, given detected human and object instances from the object detector, region-aware mask $m(p)$ is applied to constrain both the HO queries and the interactiveness scores within their corresponding regions, enabling HOI detection without additional training.

Figure 3: Visualization of interactiveness score in HOI detection. Top row shows the interactiveness score for the human ( red box ), and the bottom row shows the score for the object ( blue box ). In row 1, for the human, the masked global interactiveness (0.01) corrects the inflated local interactiveness (0.768) caused by strong semantic alignment between the human and the patch, reducing the pairwise score of the non-interactive region ( red box ) to 0.008.

	HO $\rightarrow$ I	SG	IA	HICO	HICO-DET			# Backbone Forward
				mAP	Full	Rare	Non-rare
(a)	ML-Decoder [ 8 , 9 ]			52.6	17.49	18.19	17.28	$N_{\tilde{\text{h}}}N_{\tilde{\text{o}}}$
(b)	✓			53.7	17.63	18.20	17.46	$N_{\tilde{\text{h}}}N_{\tilde{\text{o}}}$
(c)	✓	✓		54.4	22.08	23.91	21.53	1
(d)	✓		✓	56.2	23.38	24.45	23.07	1
(e)	✓	✓	✓	57.6	30.01	32.05	29.39	1

Table 1: Contribution of each component in RegFormer. Each component is defined as follows: HO

\rightarrow

I for our reasoning process in the interaction decoder, SG for Spatially Grounded pairwise query, and IA for InterActiveness scoring. Without these components, the model is equivalent to ML-Decoder [ 8 , 9 ] .

Method	Detector	Backbone		Full	Rare	Non-rare
Method	Detector	Vision	Text	Full	Rare	Non-rare
Fully Supervised
HOTR [ 14 ]	DETR	RN50	-	25.10	17.34	27.42
QPIC [ 17 ]	DETR	RN50	-	29.07	21.85	31.23
GEN-VKLT [ 18 ]	DETR	RN50	CLIP-B	33.75	29.25	35.10
ADA-CM [ 10 ]	DETR	CLIP-B	CLIP-B	33.80	31.72	34.42

Table 2: Weakly & Fully supervised HOI detection on HICO-DET benchmark dataset .

Method	Detector	AP ${}_{\text{role2}}$
AlignFormer [ 3 ]	DETR	14.2
OpenCat [ 7 ]	Faster R-CNN	15.2
OpenCat † [ 7 ]	Faster R-CNN	36.1
Weakly HOI-CLIP [ 6 ]	Faster R-CNN	48.1
RegFormer	Faster R-CNN	56.5
RegFormer	DETR	57.5

Table 3: Weakly supervised HOI detection on V-COCO .

\dagger

indicates additional large-scale weakly supervised pre-training.

	RF-UC				NF-UC
Method	Unseen	Seen	Full	HM	Unseen	Seen	Full	HM
Fully Supervised
FCL [ 27 ]	13.16	24.23	22.01	17.06	18.66	19.55	19.37	19.09
ATL [ 28 ]	9.18	24.67	21.57	13.38	18.25	18.78	18.67	18.51
RLIP [ 20 ]	19.19	33.35	30.52	24.36	20.27	27.67	26.19	23.40
GEN-VLKT [ 18 ]	21.36	32.91	30.56	25.91	25.05	23.38	23.71	24.19

Table 4: Zero-shot HOI detection. RF-UC and NF-UC denote rare-first and non-rare-first unseen compositions, respectively. † indicates the use of large-scale pretraining.

Local	Global	Full	Rare	Non-rare
Without Interactiveness-aware learning
		22.08	23.91	21.53
	✓	26.02	26.81	25.79
With Interactiveness-aware learning
✓		23.44	25.77	22.75
✓	✓	30.01	32.05	29.39

Table 5: Ablation on Interactiveness Scoring on HICO-DET . Local and Global stand for local interactiveness and masked global interactiveness, respectively.

Figure 4: Qualitative results on patch importance score . We visualize the human patch importance score, $\alpha^{\text{h}}(p)$ .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Efficiency comparison. (a) ML-Decoder baseline, (b) adding our HO $\rightarrow$ I reasoning, (c) (b) + spatially grounded pairwise query, (d) (b) + interactiveness scoring, (e) full RegFormer with all components.

Figure 6: Sensitivity analysis on scaling factors.

$\tau_{p}$	HICO	HICO-DET
$\tau_{p}$	mAP	Full	Rare	Non-rare
0.01	56.4	27.32	29.44	26.68
0.05	57.6	30.01	32.05	29.39
0.1	57.6	29.95	31.89	29.39
1.0	53.1	22.75	24.78	22.14

Appendix

Table 6: Sensitivity analysis of

\tau_{p}

Backbone $\mathcal{V}$	Full	Rare	Non-rare
CLIP-RN50	24.89	26.86	24.31
CLIP-B	28.00	29.99	27.40
SigLIP-B	30.84	35.62	29.41
DINO-B	32.90	35.18	32.21

Appendix

Table 7: Performance comparison across different vision backbones. DETR [ 32 ] is used to extract instance proposals.

Figure 7: Prediction comparison on non-interactive pairs.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
Oct 5, 2026Zhaolin Cai, Huiyu Duan, Liu Yang +5Multimodal Large Language Models MLLM Reasoning
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1
Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection
Apr 2, 2026Soo Won Seo, KyungChae Lee, Hyungchan Cho +3Human-Object Interaction
Seoul National University · Hanyang University
FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection
Sep 5, 2026Francesco Tonini, Lorenzo Vaquero, Mohammad Mahdi Derakhshani +3Human-Object Interaction Zero-Shot Learning
University of Trento Trento, Italy · Fondazione Bruno Kessler Trento, Italy · University of Amsterdam Amsterdam, The Netherlands +1

Oct 5, 2026J/K move · Enter open · S save

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

Zhaolin Cai, Huiyu Duan, Liu Yang, Yanjun Qin+4

Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University+1

Multimodal Large Language Models MLLM Reasoning Active Visual Perception Human-Object Interaction

Open paper AskPDF arXiv

Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform strongly but rely on task-specific training. Recent multimodal large language models (MLLMs) offer a promising route to training-free HOI detection through their broad visual-semantic knowledge and versatile perceptual and reasoning capabilities. However, existing approaches largely invoke these capabilities through loosely coordinated inference stages. This fragmented execution restricts the role of interaction hypotheses in guiding visual exploration, leaving key participants overlooked and local ambiguities unresolved. Furthermore, propagating early semantic assumptions through subsequent visual grounding and relation prediction induces self-reinforcing semantic circularity. To resolve these challenges, we propose HarnessHOI, a training-free framework that transforms passive MLLM inference into an active interaction-centric harness. Specifically, we introduce an interaction-guided perception mechanism that projects emerging interaction hypotheses back into the visual space to discover missing participants and refine ambiguous evidence through targeted observation. Furthermore, a relation-agnostic geometric adjudication module reconciles multi-source evidence to establish a unified spatial basis for grounded interaction reasoning across multiple actions and semantic roles. Extensive experiments on HICO-DET and V-COCO demonstrate that HarnessHOI achieves state-of-the-art performance among training-free methods, confirming the effectiveness of the proposed harness for complex interaction understanding. Code will be released upon publication.

Abstract

Human-Object Interaction

Explore similar work

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection

FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection