Open-vocabulary human-object interaction (HOI) detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason about human-object relationships. However, standard evaluation metrics, such as mean Average Precision (mAP), treat HOI classes as discrete categorical labels and fail to credit semantically valid but lexically different predictions (e.g., "lean on couch" vs. "sit on couch"), limiting their applicability for evaluating open-vocabulary predictions that go beyond any predefined set of HOI labels. We introduce SHOE (Semantic HOI Open-Vocabulary Evaluation), a new evaluation framework that incorporates semantic similarity between predicted and ground-truth HOI labels. SHOE decomposes each HOI prediction into its verb and object components, estimates their semantic similarity using the average of multiple large language models (LLMs), and combines them into a similarity score to evaluate alignment beyond exact string match. This enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models using standard benchmarks such as HICO-DET. Experimental results show that SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines, achieving an agreement of 85.73% with the average human ratings. Our work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions. We will release our evaluation metric to the public to facilitate future research.
Figures & tables
Figure 1: (a) Standard mAP metric considers all mismatches as false positives, even when predictions are semantically similar to ground truths. (b) Our SHOE metric assigns soft credit to such cases based on similarity scores, resulting in partial true positives.
Figure 2: SHOE Framework Overview. SHOE evaluates closed and open-vocabulary HOI predictions splitting by verb and object mapping to WordNet synsets and computing LLM agreement based pairwise similarity with ground-truth interactions. Predictions get matched to the ground truth based on the highest pair similarity. SHOE mAP is calculated if confidence score is available.
Figure 3: Pearson correlation between LLMs for verb (upper triangle) and object (lower triangle) similarity ratings.
Figure 4: Pairwise Spearman rank correlation ( ρ ) between annotators (A1–A5) on HOI similarity ratings. Higher values indicate stronger alignment in their relative rankings of interaction pairs.
Figure 5: (a) Comparison of the number of pairwise similarity computations required by brute-force HOI evaluation versus our approach. The brute-force method grows as (V×O)2 , while our method grows as V2+O2 . (b) SHOE expands the 600 HOI classes from HICO-DET to over 38M semantically related HOI labels.
Category
Metric
Agreement (%)
SHOE
Standard
85.73
Geometric Mean
84.29
Minimum
84.01
LLMs
DeepSeek-V3-0324
83.34
Qwen3-32B
82.03
Llama-4-Mav-17B
82.10
Table 1: Agreement between HOI similarity metrics and average human judgments collected via a user study. Each metric scores the similarity of human-object interaction (HOI) pairs. For methods that compute cosine similarity, they either use embeddings of HOI phrases (pair) or embeddings of their WordNet glosses (gloss). WordNet-based metrics are semantic graph distances between verb-object synset pairs.
Figure 6: Optimizing the arithmetic-mean weight w via MAE for each of the three categories in our user study. Category 1 (same verb, different object), Category 2 (different verb, same object), Category 3 (both different)
Method Type
Model
mAP
SHOE mAP
Default
LAIN (ViT-B)
35.30
36.58
CMMP (ViT-L)
37.77
39.06
ADA-CM (ViT-L)
38.40
39.72
HOLA (ViT-L)
39.05
39.92
Zero-shot RF-UC
CMMP (ViT-B)
32.18
32.95
EZ-HOI (ViT-B)
33.13
33.83
Table 2: Performance comparison across different HOI models and evaluation metrics on HICO-DET, including our proposed SHOE mAP and standard mAP. For VLMs, token probabilities are used as a proxy for the confidence score.
#
DS
G.
Mv.
Qw.
Yi
Agree.%
1
1
0
0
0
0
86.46
1
0
1
0
0
0
84.56
1
0
0
1
0
0
84.11
1
0
0
0
1
0
85.60
1
0
0
0
0
1
84.87
Avg.
85.12 ± 0.92
Table 1: Combined LLM ablation: left = sizes 1-2, right = sizes 3-5. Agreement scores for LLMs with human annotation. 1 = model included, 0 = excluded. # = number of models included. LLMs in order: DeepSeek-V3-0324, Gemini-2.5-pro, Llama-4-Maverick-17B, Qwen3-32B, Yi-1.5-34B-Chat.
Method Type
Model
GT Miss Rate (FN %)
Prediction Miss Rate (FP %)
SHOE mF1
Default
LAIN [ 14 ] (ViT-B)
21.04
37.77
45.80
CMMP [ 22 ] (ViT-L)
20.06
37.64
50.47
ADA-CM [ 20 ] (ViT-L)
19.18
36.21
51.93
HOLA [ 17 ] (ViT-L)
19.45
35.83
52.21
Zero-shot RF-UC
CMMP [ 22 ] (ViT-B)
21.15
39.13
44.84
EZ-HOI [ 18 ] (ViT-B)
20.45
38.10
46.94
Table 2: Performance comparison across different HOI models on the single HOI HICO-DET [ 6 ] subset (575 classes) using GT Miss Rate, Prediction Miss Rate and SHOE mF1 scores on HICO-DET, along with GT miss rate (false negatives) and prediction miss rate (false positives). confidence score rank i=0.5 was chosen for all models that provide confidence scores to make them comparable to the VLM and MLLM methods.
Figure 1: Example of the user study interface. Annotators are shown two HOI interactions along with their WordNet glosses and asked to rate their semantic similarity on a 5-point scale.
Figure 2: Per-annotator distribution of HOI similarity scores ranging from 0 (no similarity) to 4 (high similarity), showing individual rating tendencies across the annotation set.
Figure 3: Distribution of Annotator Votes by Majority Scores. Each bar shows the number of times a given annotator (A1–A5) assigned each score from 0 to 4. The numeric labels on top of each bar indicate the raw count.
Figure 4: Pairwise Quadratic Weighted Kappa (QWK) between annotators. All values are between 0.51 and 0.65, indicating moderate agreement. Diagonal entries represent perfect self-agreement ( κ=1.00 ).
Metric Model
Verb Agreement (%)
Object Agreement (%)
Standard SHOE
78.82
80.09
DeepSeek-V3-0324
79.89
81.21
Qwen3-32B
78.86
81.41
Yi-1.5-34B-Chat
79.01
80.61
Llama-4-Maverick-17B
77.92
80.99
Gemini-2.5-pro
78.34
81.34
Table 3: Average LLM agreement with human annotator ratings on HOI pairs where either the verb or object differs from the ground truth while the other is identical.
Model
mAP
SHOE-mAP
GPT-4.1
50.47
57.82
Qwen2.5-VL-32B
41.12
64.77
InternVL3-38B
46.34
50.79
Table 4: Comparison of mAP and SHOE-mAP on V-COCO for three common VLMs. VLMs token probabilities are used as a proxy for the confidence score and DETR is used for object detection.
Figure 5: Qualitative comparison of interaction annotations. HICO-DET captures fine-grained, specific interactions (e.g., ”eat”, ”wash”, ”cut with”), whereas SWIG-HOI primarily provides broad scene descriptions (e.g., ”cook”, ”clean”).
Figure 6: Qualitative results for GPT4.1 on HICO-DET single HOI subset.
Open-vocabulary human-object interaction detectors describe interactions in free text, and this creates a problem for evaluation. Category scoring rejects valid paraphrases, while semantic similarity scoring can reward predictions that name the wrong action or reverse the roles of the participants. We present a diagnostic protocol that holds every prediction fixed and scores it under five views, namely official category evaluation, an identity control, lexical normalization, graded semantic credit, and strict propositional equivalence. Comparing these views separates the effect of the matching engine from the effect of semantic credit. The protocol pairs controlled transformations that preserve or break meaning with a blinded audit of natural predictions, and we derive consistency checks that link mean credit, invalid acceptance, and transformation counts. Applied to six systems on HICO-DET, the views disagree on which system leads. GEN-VLKT leads under official evaluation, SL-HOI leads under SHOE, and GEN-VLKT leads again under strict equivalence. Semantic gains range from 1.6 to 20.3 mAP, and AgentHOI keeps less than half of its gain once equivalence is enforced. We recommend that semantic scores be reported together with relational validity and paired uncertainty, so that recovered credit is not read as better interaction understanding.
S M Asif Hossain, Ruksat Khan Shayoni, Md Kishor Morol
Open-vocabulary human-object interaction (HOI) detection requires recognizing interaction phrases that may not appear as annotated categories during training. Recent vision-language HOI detectors improve semantic transfer by matching human-object features with text embeddings, but their predictions are often dominated by object affordance and phrase-level co-occurrence. As a result, a model may predict cut cake from the presence of a knife and a cake without verifying whether the hand, tool, target, contact pattern, and object state jointly support the action. We propose ScriptHOI, a structured framework that represents each interaction phrase as a soft scripted state transition. Rather than treating a phrase as a single class token, ScriptHOI decomposes it into body-role, contact, geometry, affordance, motion, and object-state slots. A visual state tokenizer parses each detected human-object pair into corresponding state tokens, and a slot-wise matcher estimates both script coverage and script conflict. These two quantities calibrate HOI logits, expose missing visual evidence, and provide training constraints for incomplete annotations. To avoid suppressing valid but unannotated interactions, we further introduce interval partial-label learning, which constrains unannotated candidates with script-derived lower and upper probability bounds instead of assigning closed-world negatives. A counterfactual script contrast loss swaps individual script slots to discourage object-only shortcuts. Experiments on HICO-DET, V-COCO, and open-vocabulary HOI splits show that ScriptHOI improves rare and unseen interaction recognition while substantially reducing affordance-conflict false positives.
Open-vocabulary human-object interaction (HOI) detection aims to localize and recognize all human-object interactions in an image, including those unseen during training. Existing approaches usually rely on the collaboration between a conventional HOI detector and a Vision-Language Model (VLM) to recognize unseen HOI categories. However, feature fusion in this paradigm is challenging due to significant gaps in cross-model representations. To address this issue, we introduce SL-HOI, a StreamLined open-vocabulary HOI detection framework based solely on the powerful DINOv3 model. Our design leverages the complementary strengths of DINOv3's components: its backbone for fine-grained localization and its text-aligned vision head for open-vocabulary interaction classification. Moreover, to facilitate smooth cross-attention between the interaction queries and the vision head's output, we propose first feeding both the interaction queries and the backbone image tokens into the vision head, effectively bridging their representation gaps. All DINOv3 parameters in our approach are frozen, with only a small number of learnable parameters added, allowing a fast adaptation to the HOI detection task. Extensive experiments show that SL-HOI achieves state-of-the-art performance on both the SWiG-HOI and HICO-DET benchmarks, demonstrating the effectiveness of our streamlined model architecture. Code is available at https://github.com/MPI-Lab/SL-HOI.