Open-vocabulary human-object interaction (HOI) detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason about human-object relationships. However, standard evaluation metrics, such as mean Average Precision (mAP), treat HOI classes as discrete categorical labels and fail to credit semantically valid but lexically different predictions (e.g., "lean on couch" vs. "sit on couch"), limiting their applicability for evaluating open-vocabulary predictions that go beyond any predefined set of HOI labels. We introduce SHOE (Semantic HOI Open-Vocabulary Evaluation), a new evaluation framework that incorporates semantic similarity between predicted and ground-truth HOI labels. SHOE decomposes each HOI prediction into its verb and object components, estimates their semantic similarity using the average of multiple large language models (LLMs), and combines them into a similarity score to evaluate alignment beyond exact string match. This enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models using standard benchmarks such as HICO-DET. Experimental results show that SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines, achieving an agreement of 85.73% with the average human ratings. Our work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions. We will release our evaluation metric to the public to facilitate future research.
Figures & tables
Figure 1: (a) Standard mAP metric considers all mismatches as false positives, even when predictions are semantically similar to ground truths. (b) Our SHOE metric assigns soft credit to such cases based on similarity scores, resulting in partial true positives.
Figure 2: SHOE Framework Overview. SHOE evaluates closed and open-vocabulary HOI predictions splitting by verb and object mapping to WordNet synsets and computing LLM agreement based pairwise similarity with ground-truth interactions. Predictions get matched to the ground truth based on the highest pair similarity. SHOE mAP is calculated if confidence score is available.
Figure 3: Pearson correlation between LLMs for verb (upper triangle) and object (lower triangle) similarity ratings.
Figure 4: Pairwise Spearman rank correlation ( ρ ) between annotators (A1–A5) on HOI similarity ratings. Higher values indicate stronger alignment in their relative rankings of interaction pairs.
Figure 5: (a) Comparison of the number of pairwise similarity computations required by brute-force HOI evaluation versus our approach. The brute-force method grows as (V×O)2 , while our method grows as V2+O2 . (b) SHOE expands the 600 HOI classes from HICO-DET to over 38M semantically related HOI labels.
Category
Metric
Agreement (%)
SHOE
Standard
85.73
Geometric Mean
84.29
Minimum
84.01
LLMs
DeepSeek-V3-0324
83.34
Qwen3-32B
82.03
Llama-4-Mav-17B
82.10
Table 1: Agreement between HOI similarity metrics and average human judgments collected via a user study. Each metric scores the similarity of human-object interaction (HOI) pairs. For methods that compute cosine similarity, they either use embeddings of HOI phrases (pair) or embeddings of their WordNet glosses (gloss). WordNet-based metrics are semantic graph distances between verb-object synset pairs.
Figure 6: Optimizing the arithmetic-mean weight w via MAE for each of the three categories in our user study. Category 1 (same verb, different object), Category 2 (different verb, same object), Category 3 (both different)
Method Type
Model
mAP
SHOE mAP
Default
LAIN (ViT-B)
35.30
36.58
CMMP (ViT-L)
37.77
39.06
ADA-CM (ViT-L)
38.40
39.72
HOLA (ViT-L)
39.05
39.92
Zero-shot RF-UC
CMMP (ViT-B)
32.18
32.95
EZ-HOI (ViT-B)
33.13
33.83
Table 2: Performance comparison across different HOI models and evaluation metrics on HICO-DET, including our proposed SHOE mAP and standard mAP. For VLMs, token probabilities are used as a proxy for the confidence score.
#
DS
G.
Mv.
Qw.
Yi
Agree.%
1
1
0
0
0
0
86.46
1
0
1
0
0
0
84.56
1
0
0
1
0
0
84.11
1
0
0
0
1
0
85.60
1
0
0
0
0
1
84.87
Avg.
85.12 ± 0.92
Table 1: Combined LLM ablation: left = sizes 1-2, right = sizes 3-5. Agreement scores for LLMs with human annotation. 1 = model included, 0 = excluded. # = number of models included. LLMs in order: DeepSeek-V3-0324, Gemini-2.5-pro, Llama-4-Maverick-17B, Qwen3-32B, Yi-1.5-34B-Chat.
Method Type
Model
GT Miss Rate (FN %)
Prediction Miss Rate (FP %)
SHOE mF1
Default
LAIN [ 14 ] (ViT-B)
21.04
37.77
45.80
CMMP [ 22 ] (ViT-L)
20.06
37.64
50.47
ADA-CM [ 20 ] (ViT-L)
19.18
36.21
51.93
HOLA [ 17 ] (ViT-L)
19.45
35.83
52.21
Zero-shot RF-UC
CMMP [ 22 ] (ViT-B)
21.15
39.13
44.84
EZ-HOI [ 18 ] (ViT-B)
20.45
38.10
46.94
Table 2: Performance comparison across different HOI models on the single HOI HICO-DET [ 6 ] subset (575 classes) using GT Miss Rate, Prediction Miss Rate and SHOE mF1 scores on HICO-DET, along with GT miss rate (false negatives) and prediction miss rate (false positives). confidence score rank i=0.5 was chosen for all models that provide confidence scores to make them comparable to the VLM and MLLM methods.
Figure 1: Example of the user study interface. Annotators are shown two HOI interactions along with their WordNet glosses and asked to rate their semantic similarity on a 5-point scale.
Figure 2: Per-annotator distribution of HOI similarity scores ranging from 0 (no similarity) to 4 (high similarity), showing individual rating tendencies across the annotation set.
Figure 3: Distribution of Annotator Votes by Majority Scores. Each bar shows the number of times a given annotator (A1–A5) assigned each score from 0 to 4. The numeric labels on top of each bar indicate the raw count.
Figure 4: Pairwise Quadratic Weighted Kappa (QWK) between annotators. All values are between 0.51 and 0.65, indicating moderate agreement. Diagonal entries represent perfect self-agreement ( κ=1.00 ).
Metric Model
Verb Agreement (%)
Object Agreement (%)
Standard SHOE
78.82
80.09
DeepSeek-V3-0324
79.89
81.21
Qwen3-32B
78.86
81.41
Yi-1.5-34B-Chat
79.01
80.61
Llama-4-Maverick-17B
77.92
80.99
Gemini-2.5-pro
78.34
81.34
Table 3: Average LLM agreement with human annotator ratings on HOI pairs where either the verb or object differs from the ground truth while the other is identical.
Model
mAP
SHOE-mAP
GPT-4.1
50.47
57.82
Qwen2.5-VL-32B
41.12
64.77
InternVL3-38B
46.34
50.79
Table 4: Comparison of mAP and SHOE-mAP on V-COCO for three common VLMs. VLMs token probabilities are used as a proxy for the confidence score and DETR is used for object detection.
Figure 5: Qualitative comparison of interaction annotations. HICO-DET captures fine-grained, specific interactions (e.g., ”eat”, ”wash”, ”cut with”), whereas SWIG-HOI primarily provides broad scene descriptions (e.g., ”cook”, ”clean”).
Figure 6: Qualitative results for GPT4.1 on HICO-DET single HOI subset.