cs.CVNov 24, 2025

Adaptive Test-Time Memory Allocation for Long-Tailed Human-Object Interaction Detection

Authors: Yuqiu Jiang, Xiaozhen Qiao, Yifan Chen, Ye Zheng, Zhe Sun, Xuelong Li

Organizations: College of Future Information Technology, Fudan University, Shanghai, 200438, China. · Institute of Artificial Intelligence (TeleAI), China Telecom. · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China.

Abstract

Long-tailed human-object interaction (HOI) detection remains challenging because rare interactions are sparsely represented in training data. Although test observations provide additional visual evidence, existing cache-based methods typically assign uniform memory capacity across classes, limiting their ability to capture informative features of underrepresented interactions. To address this issue, we propose Adaptive Diversity Cache (ADC), a training-free, plug-and-play framework that adaptively constructs test-time caches through frequency-aware capacity allocation and confidence-diversity joint sample selection. Specifically, Frequency-Aware Cache Adaptation assigns larger capacities to underrepresented interactions using frequency-dependent priorities and a weighted logarithmic allocation objective, and fills remaining slots with selected augmented views. Confidence-Diversity Joint Cache Selection further combines prediction confidence with local feature diversity to retain informative cache entries. Cache retrieval then refines the predictions of the frozen detector. Experiments on HICO-DET and V-COCO across three base detectors demonstrate consistent improvements. On HICO-DET, ADC improves mean average precision (mAP) for rare interactions by 4.02--8.57 percentage points, while also improving performance on non-rare interactions. Online adaptation further achieves gains of 2.87--7.36 percentage points on rare interactions. These results demonstrate that adaptive test-time memory effectively complements existing HOI detectors and improves long-tailed detection performance.

Figures & tables

Explore similar work

CardsList
  1. HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection

    Sep 20, 2026Junwen Chen, Keiji YanaiEfficient Object DetectionObject Detection

  2. Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

    Feb 16, 2026Shiyu Xuan, Dongkai Wang, Zechao Li +1Human-Object InteractionZero-Shot Learning

  3. Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

    Oct 5, 2026Zhaolin Cai, Huiyu Duan, Liu Yang +5Multimodal Large Language ModelsMLLM Reasoning