Human-object interaction (HOI) detection aims to detect interactions between humans and objects in images. While recent advances have improved performance on existing benchmarks, their evaluations mainly focus on overall prediction accuracy and provide limited insight into the underlying causes of model failures. In particular, modern models often struggle in complex scenes involving multiple people and rare interaction combinations. In this work, we present a study to better understand the failure modes of two-stage HOI models, which form the basis of many current HOI detection approaches. Rather than constructing a large-scale benchmark, we instead decompose HOI detection into multiple interpretable perspectives and analyze model behavior across these dimensions to study different types of failure patterns. We curate a subset of images from an existing HOI dataset organized by human-object-interaction configurations (e.g., multi-person interactions and object sharing), and analyze model behavior under these configurations to examine different failure modes. This design allows us to analyze how these HOI models behave under different scene compositions and why their predictions fail. Importantly, high overall benchmark performance does not necessarily reflect robust visual reasoning about human-object relationships. We hope that this study can provide useful insights into the limitations of HOI models and offer observations for future research in this area.
Figures & tables
Figure 1: Qualitative examples of representative failure modes in HOI detection.
Figure 2: Overview of how we organize the HICO-DET test set for analysis. We first divide images into single-person and multi-person scenarios. For single-person images, we define two subsets: single-person single-object (SPSO) and single-person multiple-object (SPMO) , which isolate object ambiguity without subject ambiguity. For multi-person images, we categorize samples along two axes: object relation (same vs. different object instances) and interaction relation (same vs. different interaction). Their combination yields six categories (A–F), capturing different levels of ambiguity in human–object pairing and interaction attribution.
Figure 3: Distribution of images and HOIs across the subsets.
Figure 4: mAP comparison between single-person and multi-person scenarios. All models show a performance drop in multi-person settings, suggesting increased difficulty in this setting.
Figure 5: mAP across categories. While most categories achieve performance comparable to single-person settings (SPSO, SPMO), category C tends to show lower performance across all models, suggesting a consistent source of error.
Figure 6: Distribution of false positive error types across categories (A–D, SPSO, SPMO). We decompose incorrect predictions into six types: human box error, object box error, verb classification error, object classification error, human–object pairing error, and duplicate prediction (repeated). These error types are not mutually exclusive, and a single prediction may contain multiple errors.
Figure 7: Example of human–object pairing errors: the interaction ( toast ) is assigned to the wrong wine glass even when human and object detections are correct across the evaluated models.
Figure 8: Top- 10 HOI distribution and performance across categories. We visualize the top- 10 HOIs ranked by frequency. Bars indicate the number of training instances (orange) and category-specific occurrences (blue), while lines denote AP across different models.
Figure 9: Training verb distributions conditioned on object (orange), measured by instance counts, alongside AP from four models, suggesting object-centric bias in HOI detection.
Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform strongly but rely on task-specific training. Recent multimodal large language models (MLLMs) offer a promising route to training-free HOI detection through their broad visual-semantic knowledge and versatile perceptual and reasoning capabilities. However, existing approaches largely invoke these capabilities through loosely coordinated inference stages. This fragmented execution restricts the role of interaction hypotheses in guiding visual exploration, leaving key participants overlooked and local ambiguities unresolved. Furthermore, propagating early semantic assumptions through subsequent visual grounding and relation prediction induces self-reinforcing semantic circularity. To resolve these challenges, we propose HarnessHOI, a training-free framework that transforms passive MLLM inference into an active interaction-centric harness. Specifically, we introduce an interaction-guided perception mechanism that projects emerging interaction hypotheses back into the visual space to discover missing participants and refine ambiguous evidence through targeted observation. Furthermore, a relation-agnostic geometric adjudication module reconciles multi-source evidence to establish a unified spatial basis for grounded interaction reasoning across multiple actions and semantic roles. Extensive experiments on HICO-DET and V-COCO demonstrate that HarnessHOI achieves state-of-the-art performance among training-free methods, confirming the effectiveness of the proposed harness for complex interaction understanding. Code will be released upon publication.
Zhaolin Cai, Huiyu Duan, Liu Yang +5
Shanghai Jiao Tong University · Xinjiang University · Beijing Jiaotong University +1
Human-Object Interaction (HOI) detection aims to localize humans and objects in images and classify their interactions. Zero-shot HOI focuses on recognizing interactions that are not observed during training, requiring models to generalize beyond seen verb-object compositions. Recent approaches leverage Vision-Language Models (VLMs), benefiting from rich semantic representations. However, they often rely on global or detector-centric features that compress interaction cues and hinder fine-grained spatial reasoning. To overcome this limitation, we propose FineHOI, a zero-shot HOI framework that explicitly models interactions from dense patch-level features. Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships, which are not preserved by global and detector-centric representations. To this end, we introduce an Adaptive Part-Level Attention module that decomposes humans and objects into semantically coherent parts via unsupervised clustering, and re-weights them based on their interaction relevance. These representations are then integrated through a Region-Aware Interaction Transformer that integrates part-aware and global features and produces the final HOI embedding. Extensive experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods, achieving particularly strong gains on unseen interactions. Code is available at https://github.com/francescotonini/fine-hoi.
Francesco Tonini, Lorenzo Vaquero, Mohammad Mahdi Derakhshani +3
University of Trento Trento, Italy · Fondazione Bruno Kessler Trento, Italy · University of Amsterdam Amsterdam, The Netherlands +1
Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. The codes are publicly available at https://github.com/SY-Xuan/DA-HOI.
Shiyu Xuan, Dongkai Wang, Zechao Li +1
School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University