Human-object interaction (HOI) detection aims to detect interactions between humans and objects in images. While recent advances have improved performance on existing benchmarks, their evaluations mainly focus on overall prediction accuracy and provide limited insight into the underlying causes of model failures. In particular, modern models often struggle in complex scenes involving multiple people and rare interaction combinations. In this work, we present a study to better understand the failure modes of two-stage HOI models, which form the basis of many current HOI detection approaches. Rather than constructing a large-scale benchmark, we instead decompose HOI detection into multiple interpretable perspectives and analyze model behavior across these dimensions to study different types of failure patterns. We curate a subset of images from an existing HOI dataset organized by human-object-interaction configurations (e.g., multi-person interactions and object sharing), and analyze model behavior under these configurations to examine different failure modes. This design allows us to analyze how these HOI models behave under different scene compositions and why their predictions fail. Importantly, high overall benchmark performance does not necessarily reflect robust visual reasoning about human-object relationships. We hope that this study can provide useful insights into the limitations of HOI models and offer observations for future research in this area.
Figures & tables
Figure 1: Qualitative examples of representative failure modes in HOI detection.
Figure 2: Overview of how we organize the HICO-DET test set for analysis. We first divide images into single-person and multi-person scenarios. For single-person images, we define two subsets: single-person single-object (SPSO) and single-person multiple-object (SPMO) , which isolate object ambiguity without subject ambiguity. For multi-person images, we categorize samples along two axes: object relation (same vs. different object instances) and interaction relation (same vs. different interaction). Their combination yields six categories (A–F), capturing different levels of ambiguity in human–object pairing and interaction attribution.
Figure 3: Distribution of images and HOIs across the subsets.
Figure 4: mAP comparison between single-person and multi-person scenarios. All models show a performance drop in multi-person settings, suggesting increased difficulty in this setting.
Figure 5: mAP across categories. While most categories achieve performance comparable to single-person settings (SPSO, SPMO), category C tends to show lower performance across all models, suggesting a consistent source of error.
Figure 6: Distribution of false positive error types across categories (A–D, SPSO, SPMO). We decompose incorrect predictions into six types: human box error, object box error, verb classification error, object classification error, human–object pairing error, and duplicate prediction (repeated). These error types are not mutually exclusive, and a single prediction may contain multiple errors.
Figure 7: Example of human–object pairing errors: the interaction ( toast ) is assigned to the wrong wine glass even when human and object detections are correct across the evaluated models.
Figure 8: Top- 10 HOI distribution and performance across categories. We visualize the top- 10 HOIs ranked by frequency. Bars indicate the number of training instances (orange) and category-specific occurrences (blue), while lines denote AP across different models.
Figure 9: Training verb distributions conditioned on object (orange), measured by instance counts, alongside AP from four models, suggesting object-centric bias in HOI detection.
School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Nanjing Forestry University