Video-based human-object interaction (HOI) understanding requires both detecting ongoing interactions and anticipating their future evolution. However, existing methods usually treat anticipation as a downstream forecasting task built on externally constructed human-object pairs, limiting joint reasoning between detection and prediction. In addition, sparse keyframe annotations in current benchmarks can temporally misalign nominal future labels from actual future dynamics, reducing the reliability of anticipation evaluation. To address these issues, we introduce DETAnt-HOI, a temporally corrected benchmark derived from VidHOI and Action Genome for more faithful multi-horizon evaluation, and HOI-DA, a pair-centric framework that jointly performs subject-object localization, present HOI detection, and future anticipation by modeling future interactions as residual transitions from current pair states. Experiments show consistent improvements in both detection and anticipation, with larger gains at longer horizons. Our results highlight that anticipation is most effective when learned jointly with detection as a structural constraint on pair-level video representation learning. Benchmark and code will be publicly available.
Figures & tables
Figure 1. Comparison between conventional multi-stage video HOI pipelines and HOI-DA. Unlike prior methods that separate detection, association, and interaction reasoning, HOI-DA uses shared pair hypotheses to jointly predict present interactions and future HOIs from the observed clip. The figure contrasts conventional multi-stage video HOI pipelines with the proposed unified formulation. The top row illustrates a typical two-stage pipeline, where human and object instances are first detected and tracked, followed by post-hoc association and interaction classification. In this design, interaction reasoning operates on externally constructed pairs, and future prediction is treated as a separate downstream task. The bottom row shows HOI-DA, a one-stage pair-centric framework that directly models a set of persistent human–object pair hypotheses from the observed clip. These pair representations are maintained across time and are used to jointly predict present interactions and future HOIs without explicit tracking or association. Unlike multi-stage approaches, the same pair identity is preserved from detection to anticipation, enabling temporally consistent reasoning over interaction evolution. This unified formulation allows anticipation to be learned as a structured extension of current interaction states, rather than as an independent prediction task.
Figure 2. Overview of our model. Given an observed clip, HOI-DA builds a shared spatio-temporal visual memory and uses a unified pair-centric decoder for subject–object localization, present HOI prediction, and multi-horizon future verb anticipation. A language-guided semantic branch regularizes object and verb prediction, while dual residual orthogonality regularization separates future dynamics from present interaction states and across future horizons. The figure illustrates the overall architecture of HOI-DA for unified human–object interaction (HOI) detection and anticipation. Given an observed video clip, a visual backbone and spatio-temporal encoder extract a shared video representation, forming a global memory for interaction reasoning. \parA unified pair-centric decoder constructs a fixed set of subject–object pair slots, where pair identity is preserved across time without relying on external detection, tracking, or post-hoc association. These pair representations are used to jointly perform instance localization and present-time HOI prediction through the interaction decoder. \parTo enable future anticipation, the Temporal Summary Module aggregates information over the full interaction history and produces horizon-specific queries, which are further processed by the anticipation decoder to predict future verb labels at multiple time horizons. \parIn parallel, a language-guided semantic branch injects object and verb priors into the shared representation, improving generalization under long-tail distributions. Dual orthogonality regularization is applied to decouple present interaction grounding from future dynamics and to enforce diversity across different anticipation horizons. \parOverall, the architecture unifies detection and anticipation within a shared pair-centric representation, enabling temporally consistent reasoning over persistent interaction instances.
Figure 3. Temporal Summary Module. Learnable horizon anchors attend over the full shared pair representation Qpair and produce compact horizon-specific anticipation queries. The figure illustrates the Temporal Summary Module (TSM) used to construct horizon-specific anticipation queries from the shared pair representation. The module takes as input the full temporal sequence of pair representations Qpair (serving as keys and values) and a set of learnable horizon anchors Qsum(h) for each future horizon. Each horizon anchor attends over the entire observed interaction trajectory through multi-head cross-attention, aggregating information across all time steps rather than relying on the last frame alone. The resulting summarized representation is then combined with task and horizon embeddings to produce the final anticipation query Qant(h). By conditioning future prediction on the full interaction history, the module enables history-aware anticipation and avoids degenerate solutions where future states collapse to the observation boundary. This design allows the model to capture temporally evolving interaction dynamics and generate distinct predictions for different anticipation horizons.
Figure 4. Temporal non-continuity in the VidHOI evaluation protocol. We analyze temporal gaps between annotated keyframes in VidHOI from two perspectives: (a) the number of temporal discontinuities per video ( Nstep ), and (b) the duration of these gaps in seconds ( Lstep ). A substantial portion of videos exhibits frequent and long temporal gaps, indicating that nominal “future” labels may not correspond to temporally coherent continuations of observed interactions. The figure presents a statistical analysis of temporal discontinuities in the VidHOI evaluation protocol. The left histogram shows the distribution of the number of time gaps per video, while the right histogram shows the distribution of gap durations in seconds. Most videos contain at least one temporal discontinuity, and a non-negligible portion exhibits multiple gaps with large temporal spans. These gaps arise from the keyframe-based annotation scheme, where intermediate non-interactive frames are omitted, breaking temporal continuity. As a result, annotated ``future'' frames may be separated from the observed clip by irregular and sometimes long intervals, making them unreliable as true future supervision signals. This observation motivates a temporally corrected evaluation protocol that preserves instance continuity and ensures that anticipation is evaluated on genuinely future interaction dynamics rather than annotation artifacts.
VidHOI
Action Genome
Original
DETAnt-HOI
Original
DETAnt-HOI
Train clips
162952
165140
139453
145470
Test clips
19299
19542
48163
52748
Valid pairs @ h=1
50386 / 5564
22212 / 6936
Valid pairs @ h=3
46658 / 5160
20570 / 6677
Valid pairs @ h=5
43039 / 4711
18799 / 6332
Table 1. DETAnt-HOI Statistics. “Original” and “DETAnt-HOI” denote clip counts before and after temporal continuity correction. For “Valid pairs” and “Supplementary frames added”, values are reported as train/test .
Model
Paradigm
Detector
VidHOI Component
Action Genome Component
Det. mAP ↑
Ant. mAP ↑
Det. mAP ↑
Ant. mAP ↑
Full
Rare
Non-rare
h=1
h=3
h=5
h=7
Full
Rare
Non-rare
h=1
h=3
h=5
h=7
ST-HOI ( Chiou et al., 2021 )
Two-Stage
Faster R-CNN ( Ren et al., 2016 )
3.10
2.10
5.90
–
–
–
–
–
–
–
–
–
–
–
STTran ( Cong et al., 2021 )
Two-Stage
Faster R-CNN ( Ren et al., 2016 )
7.61
3.33
13.18
8.80
8.32
8.67
8.75
6.11
0.20
7.30
6.07
5.52
5.24
4.92
Gaze-Tran ( Ni et al., 2023 )
Two-Stage
YOLOv5 ( Khanam and Hussain, 2024 )
10.40
5.46
16.83
11.30
10.65
10.19
10.14
6.99
0.50
9.86
8.02
7.17
6.70
6.47
HOI-DA
One-Stage
–
16.27
12.21
22.35
16.40
16.02
16.63
18.73
9.70
1.88
13.32
9.22
8.48
8.08
7.60
Table 2. Comparison with prior video HOI methods on DETAnt-HOI. We report present-time detection mAP on the Full, Rare, and Non-rare splits, and future anticipation mAP at horizons h∈H , on both the VidHOI and Action Genome components.
Model
Paradigm
Recall@10 ↑
Recall@20 ↑
Recall@50 ↑
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
STTran ( Cong et al., 2021 )
Two-Stage
39.56
38.96
38.18
37.35
37.13
40.06
38.76
38.22
39.38
40.31
42.75
41.90
42.69
42.13
47.02
Gaze-Tran ( Ni et al., 2023 )
Two-Stage
46.20
47.14
46.98
48.08
47.66
48.94
50.00
50.03
50.90
50.66
49.62
50.78
50.90
51.65
51.41
HOI-DA
One-Stage
57.54
58.08
59.01
59.60
59.92
61.50
62.08
62.98
63.56
64.03
63.29
63.97
64.73
65.25
65.68
Table 3. Recall-based evaluation on the VidHOI component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction ( h=0 ) and future anticipation at horizons h∈H .
Model
Paradigm
Recall@10 ↑
Recall@20 ↑
Recall@50 ↑
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
STTran ( Cong et al., 2021 )
Two-Stage
19.86
20.42
18.24
17.78
17.11
28.76
29.23
28.02
27.14
26.89
33.53
34.94
33.51
32.64
32.04
Gaze-Tran ( Ni et al., 2023 )
Two-Stage
21.24
21.16
20.13
19.55
19.02
30.88
31.15
30.56
29.64
28.92
36.74
37.52
36.35
35.35
34.53
HOI-DA
One-Stage
28.89
29.06
28.29
27.68
27.35
34.70
35.17
34.53
34.14
33.85
39.99
40.98
40.58
40.38
40.08
Table 4. Recall-based evaluation on the Action Genome component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction ( h=0 ) and future anticipation at horizons h∈H .
Figure 5. Qualitative results of unified HOI detection and multi-horizon anticipation on VidHOI. Given an observed video clip (6 frames, 3 shown), the model jointly predicts present interactions ( h=0 ) and future HOIs at horizons h∈H . Each row corresponds to a persistent human-object pair hypothesis maintained across time. Detection results are reported at h=0 for consistency. Verb colors: black—true positives, red—false positives, gray—false negatives. \Description{The figure visualizes joint human-object interaction (HOI) detection and multi-horizon anticipation on a video sequence. The top row shows sampled frames across time, including both observed frames (h≤0) and future horizons (h>0). The bottom panels present predicted HOI triplets for each horizon. Each interaction is organized by persistent human–object pair hypotheses, where the same pair identity is maintained across all time steps. Present-time detection is performed at h=0, and future interactions are predicted for increasing horizons based on the observed clip. The visualization highlights that dominant interactions (e.g., feeding and holding) remain stable across time, while transient interactions (e.g., watching) emerge at short horizons and disappear at longer horizons. This demonstrates that the model performs history-conditioned forecasting rather than naive last-frame extrapolation. Overall, the figure provides qualitative evidence that the proposed unified formulation preserves pair consistency and produces temporally coherent interaction predictions across multiple future horizons. }
Variant
Det. mAP ↑
Ant. mAP ↑
Full
Rare
Non-rare
h=1
h=3
h=5
h=7
Separate Detection and Anticipation
14.29
10.83
19.00
14.09
13.28
13.36
14.20
Unified One-Decoder
13.08
9.26
18.64
12.61
13.79
14.63
14.66
w/o Temporal Summary Module
15.58
11.76
21.31
14.95
14.96
15.66
16.05
w/o Decoder-Side Text Guidance
16.11
12.43
21.64
16.31
15.26
15.67
16.38
w/o Language-Guided Semantic Branch
13.93
10.70
18.77
14.26
14.09
16.09
16.48
Table 5. Ablation study on the VidHOI of DETAnt-HOI.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Paradigm
Modality
Det. mAP ↑
Full
Rare
Non-rare
QPIC ( Tamura et al., 2021 )
One-Stage
Image
10.04
4.76
16.92
RLIPv2-ParSeDA ( Yuan et al., 2023 )
One-Stage
Image + Text
13.52
7.87
20.86
HOI-DA (ResNet-50)
One-Stage
Video + Text
16.27
12.21
22.35
Appendix
Table 6. Detection mAP comparison with image-based HOI methods on the VidHOI component of DETAnt-HOI. Image-based methods are evaluated by treating each key-frame as an independent image input.
Model
Paradigm
Modality
Recall@10 ↑
Recall@20 ↑
Recall@50 ↑
QPIC ( Tamura et al., 2021 )
One-Stage
Image
50.94
58.90
65.06
RLIPv2-ParSeDA ( Yuan et al., 2023 )
One-Stage
Image + Text
53.53
61.92
68.88
HOI-DA (ResNet-50)
One-Stage
Video + Text
57.54
61.50
63.29
Appendix
Table 7. Present-time ( h=0 ) Recall@ k comparison with image-based HOI methods on the VidHOI component of DETAnt-HOI.
Backbone
Modality
VidHOI Component
Action Genome Component
Det. mAP ↑
Ant. mAP ↑
Det. mAP ↑
Ant. mAP ↑
Full
Rare
Non-rare
h=1
h=3
h=5
h=7
Full
Rare
Non-rare
h=1
h=3
h=5
h=7
ResNet-50 ( He et al., 2016 )
Video + Text
16.27
12.21
22.35
16.40
16.02
16.63
18.73
9.70
1.88
13.32
9.22
8.48
8.08
7.60
Swin-T ( Liu et al., 2021 )
Video + Text
14.38
10.00
11.39
15.28
15.01
16.07
16.69
9.50
1.79
11.62
8.18
7.83
7.45
7.32
Appendix
Table 8. Backbone sensitivity of HOI-DA on DETAnt-HOI. Both variants use identical hyperparameters.
Model
Data
Recall@10 ↑
Recall@20 ↑
Recall@50 ↑
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
HOI-DA (ResNet-50 ( He et al., 2016 ) )
Video(+Text)
57.54
58.08
59.01
59.60
59.92
61.50
62.08
62.98
63.56
64.03
63.29
63.97
64.73
65.25
65.68
HOI-DA (Swin-T ( Liu et al., 2021 ) )
Video(+Text)
54.35
54.94
49.58
56.28
56.43
58.32
58.99
59.82
60.33
60.45
59.98
60.53
61.26
61.78
61.92
Appendix
Table 9. Recall-based evaluation on the VidHOI component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction ( h=0 ) and future anticipation at horizons h∈H .
Model
Data
Recall@10 ↑
Recall@20 ↑
Recall@50 ↑
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
HOI-DA (ResNet-50 ( He et al., 2016 ) )
Video(+Text)
28.89
29.06
28.29
27.68
27.35
34.70
35.17
34.53
34.14
33.85
39.99
40.98
40.58
40.38
40.08
HOI-DA (Swin-T ( Liu et al., 2021 ) )
Video(+Text)
27.69
27.52
27.08
26.53
26.20
33.68
33.97
33.65
33.06
32.67
38.52
39.29
39.16
38.74
38.53
Appendix
Table 10. Recall-based evaluation on the Action Genome component of DETAnt-HOI. We report Recall@10, Recall@20, and Recall@50 for present-time prediction ( h=0 ) and future anticipation at horizons h∈H .
Method
Recall@10 ↑
Recall@20 ↑
Recall@50 ↑
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
h=0
h=1
h=3
h=5
h=7
Separate Det. & Ant.
49.30
49.65
51.04
51.23
50.40
57.53
58.03
59.19
59.38
59.52
60.80
61.22
61.93
62.32
62.76
Unified One-Decoder
48.69
46.33
47.89
48.83
49.52
54.28
54.62
56.01
57.02
57.54
58.72
58.73
59.63
60.51
61.01
w/o Temporal Summary Module
55.97
56.78
57.66
58.36
58.89
59.65
60.35
61.21
61.82
62.28
61.11
61.82
62.59
63.10
63.59
w/o Decoder-Side Text Guidance
56.37
57.18
58.06
58.76
59.29
60.05
60.75
61.61
62.22
62.68
61.51
62.22
62.99
63.50
63.99
w/o Lang.-Guided Semantic Branch
54.02
54.32
55.52
56.01
56.65
57.54
57.83
58.89
59.55
60.14
58.82
59.79
60.13
60.70
61.31
Appendix
Table 11. Recall@ k results for the ablation study on the VidHOI component of DETAnt-HOI. h=0 denotes present-time detection; h>0 denotes future anticipation. The mAP counterpart of this table appears in Table 5 of the main paper.
Figure 6. Pair tracking under occlusion and camera motion on VidHOI. Numbered bounding boxes denote persistent human–object pair hypotheses maintained by HOI-DA across the full observation window. Top: a person and an object temporarily exit the field of view due to instance motion. Bottom: rapid camera movement causes severe occlusion and motion blur. Red-bordered frames are supplementary non-interactive key-frames introduced by DETAnt-HOI to preserve instance positional continuity. Despite disappearance and occlusion, HOI-DA correctly maintains pair identity and resumes accurate interaction prediction upon reappearance, in contrast to two-stage pipelines that must re-identify instances from scratch. Two video scenarios showing temporary instance disappearance and occlusion. Numbered bounding boxes track persistent human–object pairs. Red-bordered frames mark supplementary non-interactive key-frames from DETAnt-HOI. HOI-DA maintains correct pair identity and interaction predictions throughout both scenarios.
Figure 7. Joint HOI detection and multi-horizon anticipation across diverse scenes on VidHOI. For each scenario, we visualize predictions for persistent human–object pair hypotheses at multiple temporal offsets, including an observed past frame at h=−2 , present-time detection at h=0 , and future anticipation at h=1 and h=5 . Here, h=−2 denotes the frame two steps before the current observation boundary within the six-frame input window. The model captures systematic temporal transitions such as grab → hold and lift → hold , while errors concentrate on sustained or cyclic interactions where continuation duration is ambiguous. Both successful and failed predictions are shown. Verb colors: black (true positives), red (false positives), gray (false negatives). Multiple video scenarios with HOI detection and anticipation results at three horizons. Each row tracks a persistent human–object pair. True positives are in black, false positives in red, false negatives in gray.
Figure 8. Cross-attention heatmaps at three decoder stages of HOI-DA on VidHOI. Top (instance localization): attention concentrates on texture-discriminative regions, including faces and limbs for humans and shape boundaries for objects. Center (present interaction detection): attention shifts to the hand–object contact zone, which carries the strongest signal for fine-grained predicate classification. Bottom (future anticipation): attention broadens to incorporate gaze direction and environmental context beyond the current contact zone. This progression is consistent with the residual formulation in Eq. (6) of the main paper, where the localization and detection stages anchor the present pair state and the anticipation stage attends to cues that predict how the interaction will evolve. Three columns of cross-attention heatmaps for the localization, detection, and anticipation decoder stages. Attention progresses from instance-level texture regions, to hand-object contact zones, to broader gaze and environmental context.
Human-object interaction (HOI) detection requires grounding an interacting human-object pair and recognizing the verb that links them, often under severe long-tail supervision. Recent methods improve accuracy with stronger detectors and vision-language priors, but many still stack heavy transformer encoders, intricate denoising schedules, or post-hoc semantic calibration on top of the detector. We present HOIBlender, an efficient HOI detector named after its core design principle: blending detector-grounded visual tokens, spatial subject-object reasoning, and BLIP-2 semantic priors inside one lightweight decoding pipeline. HOIBlender builds on an RF-DETR/LW-DETR-style foundation with a DINOv2 backbone and selects top-K image-conditioned tokens directly from the multi-scale projector as subject and object candidates, removing the dedicated encoder stage retained by prior HOI methods. A dual-stage decoder first stabilizes human-object geometry and then performs verb and HOI classification through progressive BLIP-2 prior fusion, with classifier weights initialized from BLIP-2 text embeddings for long-tail categories. Grouped-query training further enriches optimization without increasing inference cost. Across three model scales (Nano, Small, 2XL), HOIBlender consistently outperforms SOV-STG-VLA and Hybrid-SOV-VLA on HICO-DET, reaching 44.49 Default Full mAP in only 9 training epochs while maintaining competitive latency and parameter budgets. These results show that lightweight detection, structured spatial-semantic decoding, and deeply integrated vision-language priors can be blended into a single efficient HOI pipeline.
Junwen Chen, Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo, Japan
Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform strongly but rely on task-specific training. Recent multimodal large language models (MLLMs) offer a promising route to training-free HOI detection through their broad visual-semantic knowledge and versatile perceptual and reasoning capabilities. However, existing approaches largely invoke these capabilities through loosely coordinated inference stages. This fragmented execution restricts the role of interaction hypotheses in guiding visual exploration, leaving key participants overlooked and local ambiguities unresolved. Furthermore, propagating early semantic assumptions through subsequent visual grounding and relation prediction induces self-reinforcing semantic circularity. To resolve these challenges, we propose HarnessHOI, a training-free framework that transforms passive MLLM inference into an active interaction-centric harness. Specifically, we introduce an interaction-guided perception mechanism that projects emerging interaction hypotheses back into the visual space to discover missing participants and refine ambiguous evidence through targeted observation. Furthermore, a relation-agnostic geometric adjudication module reconciles multi-source evidence to establish a unified spatial basis for grounded interaction reasoning across multiple actions and semantic roles. Extensive experiments on HICO-DET and V-COCO demonstrate that HarnessHOI achieves state-of-the-art performance among training-free methods, confirming the effectiveness of the proposed harness for complex interaction understanding. Code will be released upon publication.
Zhaolin Cai, Huiyu Duan, Liu Yang +5
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1
Understanding hands and the objects they interact with, both directly and through tools, is a key step for tasks ranging from action perception to 3D reconstruction and robotics. Our paper provides several contributions to the Hand-Object Interaction (HOI) understanding literature: (1) HOI-DETR, a new framework that introduces hand-object and object-object interactions to the Co-DETR architecture to produce a state-of-the-art method; (2) a comprehensive HOI evaluation suite of 4 diverse datasets, including a video benchmark derived from the HD-EPIC dataset and fresh annotations that improve the Hands23 benchmark and (3) a trained checkpoint that significantly improves the state of the art across Hands23, HOIST, FineBio, and HD-EPIC, including mAP gains of over 20 percentage points on Hands23 and FineBio. Our ablations confirm the contributions of each model component.
Ahmad Darkhalil, Dima Damen, David Fouhey
School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US