Adaptive Test-Time Memory Allocation for Long-Tailed Human-Object Interaction Detection
Organizations: College of Future Information Technology, Fudan University, Shanghai, 200438, China. · Institute of Artificial Intelligence (TeleAI), China Telecom. · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China.
Abstract
Long-tailed human-object interaction (HOI) detection remains challenging because rare interactions are sparsely represented in training data. Although test observations provide additional visual evidence, existing cache-based methods typically assign uniform memory capacity across classes, limiting their ability to capture informative features of underrepresented interactions. To address this issue, we propose Adaptive Diversity Cache (ADC), a training-free, plug-and-play framework that adaptively constructs test-time caches through frequency-aware capacity allocation and confidence-diversity joint sample selection. Specifically, Frequency-Aware Cache Adaptation assigns larger capacities to underrepresented interactions using frequency-dependent priorities and a weighted logarithmic allocation objective, and fills remaining slots with selected augmented views. Confidence-Diversity Joint Cache Selection further combines prediction confidence with local feature diversity to retain informative cache entries. Cache retrieval then refines the predictions of the frozen detector. Experiments on HICO-DET and V-COCO across three base detectors demonstrate consistent improvements. On HICO-DET, ADC improves mean average precision (mAP) for rare interactions by 4.02--8.57 percentage points, while also improving performance on non-rare interactions. Online adaptation further achieves gains of 2.87--7.36 percentage points on rare interactions. These results demonstrate that adaptive test-time memory effectively complements existing HOI detectors and improves long-tailed detection performance.
Figures & tables
| Method | Backbone | HICO-DET | V-COCO | ||
|---|---|---|---|---|---|
| Full | Rare | Non-rare | |||
| iCAN [ 1 ] | R50 | 14.84 | 10.45 | 16.15 | 45.3 |
| DRG [ 37 ] | R50-F | 24.53 | 19.47 | 26.04 | 51.0 |
| HOTR [ 17 ] | R50 | 25.10 | 17.34 | 27.42 | 55.2 |
| QPIC [ 18 ] | R50 | 29.07 | 21.85 | 31.23 | 58.8 |
| CPC [ 38 ] | R50 | 29.63 | 23.14 | 31.57 | 63.1 |
| CJCS | FACA | Full | Rare | Non-rare | ||
|---|---|---|---|---|---|---|
| Completion | ||||||
| ✓ | 38.49 | 38.42 | 38.51 | |||
| ✓ | ✓ | 38.79 | 39.63 | 38.54 | ||
| ✓ | ✓ | ✓ | 39.47 | 40.32 | 39.21 | |
| ✓ | ✓ | ✓ | ✓ | 39.81 | 41.48 | 39.30 |
| Method | RF-UC Setting | NF-UC Setting | ||||
|---|---|---|---|---|---|---|
| Full | Seen | Unseen | Full | Seen | Unseen | |
| GEN-VLKT [ 2 ] | 30.56 | 32.91 | 21.36 | 23.71 | 23.38 | 25.05 |
| HOICLIP [ 9 ] | 32.99 | 34.85 | 25.53 | 27.75 | 28.10 | 26.39 |
| CLIP4HOI [ 23 ] | 34.08 | 35.48 | 28.47 | 28.90 | 28.26 | 31.44 |
| LOGICHOI [ 42 ] | 33.17 | 34.93 | 25.97 | 27.95 | 27.86 | 26.84 |
| ADA-CM [ 25 ] | 33.01 | 34.35 | 27.63 | 31.39 | 31.13 | 32.41 |
| Method | Backbone | SG1 | SG2 | SG3 |
|---|---|---|---|---|
| EZ-HOI [ 10 ] | R50 | 16.57 | 14.62 | 15.73 |
| + ADC | R50 | 17.64 | 15.53 | 16.42 |
| - | +1.07 | +0.91 | +0.69 | |
| ADA-CM [ 25 ] | R50 | 15.64 | 15.02 | 15.76 |
| + ADC | R50 | 17.16 | 16.93 | 17.01 |
| - | +1.52 | +1.91 | +1.25 |
| Method | HICO-DET (Online) | V-COCO (Online) | |||||
|---|---|---|---|---|---|---|---|
| mAP | Cost | mAP | Cost | ||||
| Full | Rare | Time | Mem. | Time | Mem. | ||
| EZ-HOI [ 10 ] | 38.33 | 36.70 | 1h 23m | 7619M | 60.50 | 23m | 6822M |
| + ADC | 38.81 | 39.57 | 2h 20m | 8926M | 60.97 | 39m | 8410M |
| ADA-CM [ 25 ] | 38.39 | 37.46 | 11m | 4886M | 58.50 | 8m | 4758M |
| + ADC | 39.48 | 40.60 | 39m | 6908M | 61.90 | 14m | 6916M |
Human-Object Interaction
Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.
See all 114 papers