Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation

Yuanhao Luo; Di Wen; Kunyu Peng; Ruiping Liu; Junwei Zheng; Yufan Chen; Jiale Wei; Rainer Stiefelhage

cs.CVApr 12, 2026

Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation

Authors: Yuanhao Luo, Di Wen, Kunyu Peng, Ruiping Liu, Junwei Zheng, Yufan Chen, Jiale Wei, Rainer Stiefelhage

Organizations: Karlsruhe Institute of Technology, Karlsruhe, Germany · INSAIT, Sofia University, Sofia, Bulgaria · ETH Zurich, Zurich, Switzerland

Abstract

Video-based human-object interaction (HOI) understanding requires both detecting ongoing interactions and anticipating their future evolution. However, existing methods usually treat anticipation as a downstream forecasting task built on externally constructed human-object pairs, limiting joint reasoning between detection and prediction. In addition, sparse keyframe annotations in current benchmarks can temporally misalign nominal future labels from actual future dynamics, reducing the reliability of anticipation evaluation. To address these issues, we introduce DETAnt-HOI, a temporally corrected benchmark derived from VidHOI and Action Genome for more faithful multi-horizon evaluation, and HOI-DA, a pair-centric framework that jointly performs subject-object localization, present HOI detection, and future anticipation by modeling future interactions as residual transitions from current pair states. Experiments show consistent improvements in both detection and anticipation, with larger gains at longer horizons. Our results highlight that anticipation is most effective when learned jointly with detection as a structural constraint on pair-level video representation learning. Benchmark and code will be publicly available.

Figures & tables

Figure 1. Comparison between conventional multi-stage video HOI pipelines and HOI-DA. Unlike prior methods that separate detection, association, and interaction reasoning, HOI-DA uses shared pair hypotheses to jointly predict present interactions and future HOIs from the observed clip. The figure contrasts conventional multi-stage video HOI pipelines with the proposed unified formulation. The top row illustrates a typical two-stage pipeline, where human and object instances are first detected and tracked, followed by post-hoc association and interaction classification. In this design, interaction reasoning operates on externally constructed pairs, and future prediction is treated as a separate downstream task. The bottom row shows HOI-DA, a one-stage pair-centric framework that directly models a set of persistent human–object pair hypotheses from the observed clip. These pair representations are maintained across time and are used to jointly predict present interactions and future HOIs without explicit tracking or association. Unlike multi-stage approaches, the same pair identity is preserved from detection to anticipation, enabling temporally consistent reasoning over interaction evolution. This unified formulation allows anticipation to be learned as a structured extension of current interaction states, rather than as an independent prediction task.

Figure 2. Overview of our model. Given an observed clip, HOI-DA builds a shared spatio-temporal visual memory and uses a unified pair-centric decoder for subject–object localization, present HOI prediction, and multi-horizon future verb anticipation. A language-guided semantic branch regularizes object and verb prediction, while dual residual orthogonality regularization separates future dynamics from present interaction states and across future horizons. The figure illustrates the overall architecture of HOI-DA for unified human–object interaction (HOI) detection and anticipation. Given an observed video clip, a visual backbone and spatio-temporal encoder extract a shared video representation, forming a global memory for interaction reasoning. \parA unified pair-centric decoder constructs a fixed set of subject–object pair slots, where pair identity is preserved across time without relying on external detection, tracking, or post-hoc association. These pair representations are used to jointly perform instance localization and present-time HOI prediction through the interaction decoder. \parTo enable future anticipation, the Temporal Summary Module aggregates information over the full interaction history and produces horizon-specific queries, which are further processed by the anticipation decoder to predict future verb labels at multiple time horizons. \parIn parallel, a language-guided semantic branch injects object and verb priors into the shared representation, improving generalization under long-tail distributions. Dual orthogonality regularization is applied to decouple present interaction grounding from future dynamics and to enforce diversity across different anticipation horizons. \parOverall, the architecture unifies detection and anticipation within a shared pair-centric representation, enabling temporally consistent reasoning over persistent interaction instances.

Figure 3. Temporal Summary Module. Learnable horizon anchors attend over the full shared pair representation $\mathbf{Q}_{\mathrm{pair}}$ and produce compact horizon-specific anticipation queries. The figure illustrates the Temporal Summary Module (TSM) used to construct horizon-specific anticipation queries from the shared pair representation. The module takes as input the full temporal sequence of pair representations $\mathbf{Q}_{\mathrm{pair}}$ (serving as keys and values) and a set of learnable horizon anchors $\mathbf{Q}_{\mathrm{sum}}^{(h)}$ for each future horizon. Each horizon anchor attends over the entire observed interaction trajectory through multi-head cross-attention, aggregating information across all time steps rather than relying on the last frame alone. The resulting summarized representation is then combined with task and horizon embeddings to produce the final anticipation query $\mathbf{Q}_{\mathrm{ant}}^{(h)}$. By conditioning future prediction on the full interaction history, the module enables history-aware anticipation and avoids degenerate solutions where future states collapse to the observation boundary. This design allows the model to capture temporally evolving interaction dynamics and generate distinct predictions for different anticipation horizons. — Figure 3. Temporal Summary Module. Learnable horizon anchors attend over the full shared pair representation $\mathbf{Q}_{\mathrm{pair}}$ and produce compact horizon-specific anticipation queries. The figure illustrates the Temporal Summary Module (TSM) used to construct horizon-specific anticipation queries from the shared pair representation. The module takes as input the full temporal sequence of pair representations $\mathbf{Q}_{\mathrm{pair}}$ (serving as keys and values) and a set of learnable horizon anchors $\mathbf{Q}_{\mathrm{sum}}^{(h)}$ for each future horizon. Each horizon anchor attends over the entire observed interaction trajectory through multi-head cross-attention, aggregating information across all time steps rather than relying on the last frame alone. The resulting summarized representation is then combined with task and horizon embeddings to produce the final anticipation query $\mathbf{Q}_{\mathrm{ant}}^{(h)}$ . By conditioning future prediction on the full interaction history, the module enables history-aware anticipation and avoids degenerate solutions where future states collapse to the observation boundary. This design allows the model to capture temporally evolving interaction dynamics and generate distinct predictions for different anticipation horizons.

Figure 4. Temporal non-continuity in the VidHOI evaluation protocol. We analyze temporal gaps between annotated keyframes in VidHOI from two perspectives: (a) the number of temporal discontinuities per video (

N_{\text{step}}

), and (b) the duration of these gaps in seconds (

L_{\text{step}}

). A substantial portion of videos exhibits frequent and long temporal gaps, indicating that nominal “future” labels may not correspond to temporally coherent continuations of observed interactions. The figure presents a statistical analysis of temporal discontinuities in the VidHOI evaluation protocol. The left histogram shows the distribution of the number of time gaps per video, while the right histogram shows the distribution of gap durations in seconds. Most videos contain at least one temporal discontinuity, and a non-negligible portion exhibits multiple gaps with large temporal spans. These gaps arise from the keyframe-based annotation scheme, where intermediate non-interactive frames are omitted, breaking temporal continuity. As a result, annotated ``future'' frames may be separated from the observed clip by irregular and sometimes long intervals, making them unreliable as true future supervision signals. This observation motivates a temporally corrected evaluation protocol that preserves instance continuity and ensures that anticipation is evaluated on genuinely future interaction dynamics rather than annotation artifacts.

	VidHOI		Action Genome
	Original	DETAnt-HOI	Original	DETAnt-HOI
Train clips	162952	165140	139453	145470
Test clips	19299	19542	48163	52748
Valid pairs @ $h{=}1$	50386 / 5564		22212 / 6936
Valid pairs @ $h{=}3$	46658 / 5160		20570 / 6677
Valid pairs @ $h{=}5$	43039 / 4711		18799 / 6332

Table 1. DETAnt-HOI Statistics. “Original” and “DETAnt-HOI” denote clip counts before and after temporal continuity correction. For “Valid pairs” and “Supplementary frames added”, values are reported as train/test .

Model	Paradigm	Detector	VidHOI Component							Action Genome Component
			Det. mAP $\uparrow$			Ant. mAP $\uparrow$				Det. mAP $\uparrow$			Ant. mAP $\uparrow$
			Full	Rare	Non-rare	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	Full	Rare	Non-rare	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
ST-HOI ( Chiou et al., 2021 )	Two-Stage	Faster R-CNN ( Ren et al., 2016 )	3.10	2.10	5.90	–	–	–	–	–	–	–	–	–	–	–
STTran ( Cong et al., 2021 )	Two-Stage	Faster R-CNN ( Ren et al., 2016 )	7.61	3.33	13.18	8.80	8.32	8.67	8.75	6.11	0.20	7.30	6.07	5.52	5.24	4.92
Gaze-Tran ( Ni et al., 2023 )	Two-Stage	YOLOv5 ( Khanam and Hussain, 2024 )	10.40	5.46	16.83	11.30	10.65	10.19	10.14	6.99	0.50	9.86	8.02	7.17	6.70	6.47
HOI-DA	One-Stage	–	16.27	12.21	22.35	16.40	16.02	16.63	18.73	9.70	1.88	13.32	9.22	8.48	8.08	7.60

Table 2. Comparison with prior video HOI methods on DETAnt-HOI. We report present-time detection mAP on the Full, Rare, and Non-rare splits, and future anticipation mAP at horizons

h\in\mathcal{H}

, on both the VidHOI and Action Genome components.

Model	Paradigm	Recall@10 $\uparrow$					Recall@20 $\uparrow$					Recall@50 $\uparrow$
Model	Paradigm	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
STTran ( Cong et al., 2021 )	Two-Stage	39.56	38.96	38.18	37.35	37.13	40.06	38.76	38.22	39.38	40.31	42.75	41.90	42.69	42.13	47.02
Gaze-Tran ( Ni et al., 2023 )	Two-Stage	46.20	47.14	46.98	48.08	47.66	48.94	50.00	50.03	50.90	50.66	49.62	50.78	50.90	51.65	51.41
HOI-DA	One-Stage	57.54	58.08	59.01	59.60	59.92	61.50	62.08	62.98	63.56	64.03	63.29	63.97	64.73	65.25	65.68

Table 3. Recall-based evaluation on the VidHOI component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction (

h{=}0

) and future anticipation at horizons

h\in\mathcal{H}

Model	Paradigm	Recall@10 $\uparrow$					Recall@20 $\uparrow$					Recall@50 $\uparrow$
Model	Paradigm	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
STTran ( Cong et al., 2021 )	Two-Stage	19.86	20.42	18.24	17.78	17.11	28.76	29.23	28.02	27.14	26.89	33.53	34.94	33.51	32.64	32.04
Gaze-Tran ( Ni et al., 2023 )	Two-Stage	21.24	21.16	20.13	19.55	19.02	30.88	31.15	30.56	29.64	28.92	36.74	37.52	36.35	35.35	34.53
HOI-DA	One-Stage	28.89	29.06	28.29	27.68	27.35	34.70	35.17	34.53	34.14	33.85	39.99	40.98	40.58	40.38	40.08

Table 4. Recall-based evaluation on the Action Genome component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction (

h{=}0

) and future anticipation at horizons

h\in\mathcal{H}

Figure 5. Qualitative results of unified HOI detection and multi-horizon anticipation on VidHOI. Given an observed video clip (6 frames, 3 shown), the model jointly predicts present interactions ( $h=0$ ) and future HOIs at horizons $h\in\mathcal{H}$ . Each row corresponds to a persistent human-object pair hypothesis maintained across time. Detection results are reported at $h=0$ for consistency. Verb colors: black—true positives, red—false positives, gray—false negatives. \Description{The figure visualizes joint human-object interaction (HOI) detection and multi-horizon anticipation on a video sequence. The top row shows sampled frames across time, including both observed frames ($h \leq0$) and future horizons ($h > 0$). The bottom panels present predicted HOI triplets for each horizon. Each interaction is organized by persistent human–object pair hypotheses, where the same pair identity is maintained across all time steps. Present-time detection is performed at $h=0$, and future interactions are predicted for increasing horizons based on the observed clip. The visualization highlights that dominant interactions (e.g., feeding and holding) remain stable across time, while transient interactions (e.g., watching) emerge at short horizons and disappear at longer horizons. This demonstrates that the model performs history-conditioned forecasting rather than naive last-frame extrapolation. Overall, the figure provides qualitative evidence that the proposed unified formulation preserves pair consistency and produces temporally coherent interaction predictions across multiple future horizons. } — Figure 5. Qualitative results of unified HOI detection and multi-horizon anticipation on VidHOI. Given an observed video clip (6 frames, 3 shown), the model jointly predicts present interactions ( $h=0$ ) and future HOIs at horizons $h\in\mathcal{H}$ . Each row corresponds to a persistent human-object pair hypothesis maintained across time. Detection results are reported at $h=0$ for consistency. Verb colors: black—true positives, red—false positives, gray—false negatives. \Description{The figure visualizes joint human-object interaction (HOI) detection and multi-horizon anticipation on a video sequence. The top row shows sampled frames across time, including both observed frames ( $h \leq0$ ) and future horizons ( $h > 0$ ). The bottom panels present predicted HOI triplets for each horizon. Each interaction is organized by persistent human–object pair hypotheses, where the same pair identity is maintained across all time steps. Present-time detection is performed at $h=0$ , and future interactions are predicted for increasing horizons based on the observed clip. The visualization highlights that dominant interactions (e.g., feeding and holding) remain stable across time, while transient interactions (e.g., watching) emerge at short horizons and disappear at longer horizons. This demonstrates that the model performs history-conditioned forecasting rather than naive last-frame extrapolation. Overall, the figure provides qualitative evidence that the proposed unified formulation preserves pair consistency and produces temporally coherent interaction predictions across multiple future horizons. }

Variant	Det. mAP $\uparrow$			Ant. mAP $\uparrow$
Variant	Full	Rare	Non-rare	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
Separate Detection and Anticipation	14.29	10.83	19.00	14.09	13.28	13.36	14.20
Unified One-Decoder	13.08	9.26	18.64	12.61	13.79	14.63	14.66
w/o Temporal Summary Module	15.58	11.76	21.31	14.95	14.96	15.66	16.05
w/o Decoder-Side Text Guidance	16.11	12.43	21.64	16.31	15.26	15.67	16.38
w/o Language-Guided Semantic Branch	13.93	10.70	18.77	14.26	14.09	16.09	16.48

Table 5. Ablation study on the VidHOI of DETAnt-HOI.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Paradigm	Modality	Det. mAP $\uparrow$
Model	Paradigm	Modality	Full	Rare	Non-rare
QPIC ( Tamura et al., 2021 )	One-Stage	Image	10.04	4.76	16.92
RLIPv2-ParSeDA ( Yuan et al., 2023 )	One-Stage	Image + Text	13.52	7.87	20.86
HOI-DA (ResNet-50)	One-Stage	Video + Text	16.27	12.21	22.35

Appendix

Table 6. Detection mAP comparison with image-based HOI methods on the VidHOI component of DETAnt-HOI. Image-based methods are evaluated by treating each key-frame as an independent image input.

Model	Paradigm	Modality	Recall@10 $\uparrow$	Recall@20 $\uparrow$	Recall@50 $\uparrow$
QPIC ( Tamura et al., 2021 )	One-Stage	Image	50.94	58.90	65.06
RLIPv2-ParSeDA ( Yuan et al., 2023 )	One-Stage	Image + Text	53.53	61.92	68.88
HOI-DA (ResNet-50)	One-Stage	Video + Text	57.54	61.50	63.29

Appendix

Table 7. Present-time (

h{=}0

) Recall@

k

comparison with image-based HOI methods on the VidHOI component of DETAnt-HOI.

Backbone	Modality	VidHOI Component							Action Genome Component
		Det. mAP $\uparrow$			Ant. mAP $\uparrow$				Det. mAP $\uparrow$			Ant. mAP $\uparrow$
		Full	Rare	Non-rare	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	Full	Rare	Non-rare	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
ResNet-50 ( He et al., 2016 )	Video + Text	16.27	12.21	22.35	16.40	16.02	16.63	18.73	9.70	1.88	13.32	9.22	8.48	8.08	7.60
Swin-T ( Liu et al., 2021 )	Video + Text	14.38	10.00	11.39	15.28	15.01	16.07	16.69	9.50	1.79	11.62	8.18	7.83	7.45	7.32

Appendix

Table 8. Backbone sensitivity of HOI-DA on DETAnt-HOI. Both variants use identical hyperparameters.

Model	Data	Recall@10 $\uparrow$					Recall@20 $\uparrow$					Recall@50 $\uparrow$
Model	Data	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
HOI-DA (ResNet-50 ( He et al., 2016 ) )	Video(+Text)	57.54	58.08	59.01	59.60	59.92	61.50	62.08	62.98	63.56	64.03	63.29	63.97	64.73	65.25	65.68
HOI-DA (Swin-T ( Liu et al., 2021 ) )	Video(+Text)	54.35	54.94	49.58	56.28	56.43	58.32	58.99	59.82	60.33	60.45	59.98	60.53	61.26	61.78	61.92

Appendix

Table 9. Recall-based evaluation on the VidHOI component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction (

h{=}0

) and future anticipation at horizons

h\in\mathcal{H}

Model	Data	Recall@10 $\uparrow$					Recall@20 $\uparrow$					Recall@50 $\uparrow$
Model	Data	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
HOI-DA (ResNet-50 ( He et al., 2016 ) )	Video(+Text)	28.89	29.06	28.29	27.68	27.35	34.70	35.17	34.53	34.14	33.85	39.99	40.98	40.58	40.38	40.08
HOI-DA (Swin-T ( Liu et al., 2021 ) )	Video(+Text)	27.69	27.52	27.08	26.53	26.20	33.68	33.97	33.65	33.06	32.67	38.52	39.29	39.16	38.74	38.53

Appendix

Table 10. Recall-based evaluation on the Action Genome component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction (

h{=}0

) and future anticipation at horizons

h\in\mathcal{H}

Method	Recall@10 $\uparrow$					Recall@20 $\uparrow$					Recall@50 $\uparrow$
Method	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$	$h{=}0$	$h{=}1$	$h{=}3$	$h{=}5$	$h{=}7$
Separate Det. & Ant.	49.30	49.65	51.04	51.23	50.40	57.53	58.03	59.19	59.38	59.52	60.80	61.22	61.93	62.32	62.76
Unified One-Decoder	48.69	46.33	47.89	48.83	49.52	54.28	54.62	56.01	57.02	57.54	58.72	58.73	59.63	60.51	61.01
w/o Temporal Summary Module	55.97	56.78	57.66	58.36	58.89	59.65	60.35	61.21	61.82	62.28	61.11	61.82	62.59	63.10	63.59
w/o Decoder-Side Text Guidance	56.37	57.18	58.06	58.76	59.29	60.05	60.75	61.61	62.22	62.68	61.51	62.22	62.99	63.50	63.99
w/o Lang.-Guided Semantic Branch	54.02	54.32	55.52	56.01	56.65	57.54	57.83	58.89	59.55	60.14	58.82	59.79	60.13	60.70	61.31

Appendix

Table 11. Recall@

k

results for the ablation study on the VidHOI component of DETAnt-HOI.

h{=}0

denotes present-time detection;

h{>}0

denotes future anticipation. The mAP counterpart of this table appears in Table 5 of the main paper.

Figure 6. Pair tracking under occlusion and camera motion on VidHOI. Numbered bounding boxes denote persistent human–object pair hypotheses maintained by HOI-DA across the full observation window. Top: a person and an object temporarily exit the field of view due to instance motion. Bottom: rapid camera movement causes severe occlusion and motion blur. Red-bordered frames are supplementary non-interactive key-frames introduced by DETAnt-HOI to preserve instance positional continuity. Despite disappearance and occlusion, HOI-DA correctly maintains pair identity and resumes accurate interaction prediction upon reappearance, in contrast to two-stage pipelines that must re-identify instances from scratch. Two video scenarios showing temporary instance disappearance and occlusion. Numbered bounding boxes track persistent human–object pairs. Red-bordered frames mark supplementary non-interactive key-frames from DETAnt-HOI. HOI-DA maintains correct pair identity and interaction predictions throughout both scenarios.

Figure 7. Joint HOI detection and multi-horizon anticipation across diverse scenes on VidHOI. For each scenario, we visualize predictions for persistent human–object pair hypotheses at multiple temporal offsets, including an observed past frame at $h{=}{-}2$ , present-time detection at $h{=}0$ , and future anticipation at $h{=}1$ and $h{=}5$ . Here, $h{=}{-}2$ denotes the frame two steps before the current observation boundary within the six-frame input window. The model captures systematic temporal transitions such as grab $\to$ hold and lift $\to$ hold , while errors concentrate on sustained or cyclic interactions where continuation duration is ambiguous. Both successful and failed predictions are shown. Verb colors: black (true positives), red (false positives), gray (false negatives). Multiple video scenarios with HOI detection and anticipation results at three horizons. Each row tracks a persistent human–object pair. True positives are in black, false positives in red, false negatives in gray.

Figure 8. Cross-attention heatmaps at three decoder stages of HOI-DA on VidHOI. Top (instance localization): attention concentrates on texture-discriminative regions, including faces and limbs for humans and shape boundaries for objects. Center (present interaction detection): attention shifts to the hand–object contact zone, which carries the strongest signal for fine-grained predicate classification. Bottom (future anticipation): attention broadens to incorporate gaze direction and environmental context beyond the current contact zone. This progression is consistent with the residual formulation in Eq. (6) of the main paper, where the localization and detection stages anchor the present pair state and the anticipation stage attends to cues that predict how the interaction will evolve. Three columns of cross-attention heatmaps for the localization, detection, and anticipation decoder stages. Attention progresses from instance-level texture regions, to hand-object contact zones, to broader gaze and environmental context.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection
Sep 20, 2026Junwen Chen, Keiji YanaiEfficient Object Detection Object Detection
Department of Informatics, The University of Electro-Communications, Tokyo, Japan
Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
Oct 5, 2026Zhaolin Cai, Huiyu Duan, Liu Yang +5Multimodal Large Language Models MLLM Reasoning
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1
Improving and Evaluating Hand-Object Interaction Detection
Jun 16, 2026Ahmad Darkhalil, Dima Damen, David FouheyDetection Transformer Hand-Object Interaction Detection
School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US

Sep 20, 2026J/K move · Enter open · S save

HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection

Junwen Chen, Keiji Yanai

Department of Informatics, The University of Electro-Communications, Tokyo, Japan

Efficient Object Detection Object Detection Human-Object Interaction

Open paper AskPDF arXiv

Human-object interaction (HOI) detection requires grounding an interacting human-object pair and recognizing the verb that links them, often under severe long-tail supervision. Recent methods improve accuracy with stronger detectors and vision-language priors, but many still stack heavy transformer encoders, intricate denoising schedules, or post-hoc semantic calibration on top of the detector. We present HOIBlender, an efficient HOI detector named after its core design principle: blending detector-grounded visual tokens, spatial subject-object reasoning, and BLIP-2 semantic priors inside one lightweight decoding pipeline. HOIBlender builds on an RF-DETR/LW-DETR-style foundation with a DINOv2 backbone and selects top-

K

image-conditioned tokens directly from the multi-scale projector as subject and object candidates, removing the dedicated encoder stage retained by prior HOI methods. A dual-stage decoder first stabilizes human-object geometry and then performs verb and HOI classification through progressive BLIP-2 prior fusion, with classifier weights initialized from BLIP-2 text embeddings for long-tail categories. Grouped-query training further enriches optimization without increasing inference cost. Across three model scales (Nano, Small, 2XL), HOIBlender consistently outperforms SOV-STG-VLA and Hybrid-SOV-VLA on HICO-DET, reaching

44.49

Default Full mAP in only

9

training epochs while maintaining competitive latency and parameter budgets. These results show that lightweight detection, structured spatial-semantic decoding, and deeply integrated vision-language priors can be blended into a single efficient HOI pipeline.

Abstract

Human-Object Interaction

Explore similar work

HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

Improving and Evaluating Hand-Object Interaction Detection