cs.CVNov 24, 2025

Adaptive Test-Time Memory Allocation for Long-Tailed Human-Object Interaction Detection

Authors: Yuqiu Jiang, Xiaozhen Qiao, Yifan Chen, Ye Zheng, Zhe Sun, Xuelong Li

Organizations: College of Future Information Technology, Fudan University, Shanghai, 200438, China. · Institute of Artificial Intelligence (TeleAI), China Telecom. · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China.

Abstract

Long-tailed human-object interaction (HOI) detection remains challenging because rare interactions are sparsely represented in training data. Although test observations provide additional visual evidence, existing cache-based methods typically assign uniform memory capacity across classes, limiting their ability to capture informative features of underrepresented interactions. To address this issue, we propose Adaptive Diversity Cache (ADC), a training-free, plug-and-play framework that adaptively constructs test-time caches through frequency-aware capacity allocation and confidence-diversity joint sample selection. Specifically, Frequency-Aware Cache Adaptation assigns larger capacities to underrepresented interactions using frequency-dependent priorities and a weighted logarithmic allocation objective, and fills remaining slots with selected augmented views. Confidence-Diversity Joint Cache Selection further combines prediction confidence with local feature diversity to retain informative cache entries. Cache retrieval then refines the predictions of the frozen detector. Experiments on HICO-DET and V-COCO across three base detectors demonstrate consistent improvements. On HICO-DET, ADC improves mean average precision (mAP) for rare interactions by 4.02--8.57 percentage points, while also improving performance on non-rare interactions. Online adaptation further achieves gains of 2.87--7.36 percentage points on rare interactions. These results demonstrate that adaptive test-time memory effectively complements existing HOI detectors and improves long-tailed detection performance.

Figures & tables

Explore similar work

Sep 20, 2026

HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection

Human-object interaction (HOI) detection requires grounding an interacting human-object pair and recognizing the verb that links them, often under severe long-tail supervision. Recent methods improve accuracy with stronger detectors and vision-language priors, but many still stack heavy transformer encoders, intricate denoising schedules, or post-hoc semantic calibration on top of the detector. We present HOIBlender, an efficient HOI detector named after its core design principle: blending detector-grounded visual tokens, spatial subject-object reasoning, and BLIP-2 semantic priors inside one lightweight decoding pipeline. HOIBlender builds on an RF-DETR/LW-DETR-style foundation with a DINOv2 backbone and selects top-KK image-conditioned tokens directly from the multi-scale projector as subject and object candidates, removing the dedicated encoder stage retained by prior HOI methods. A dual-stage decoder first stabilizes human-object geometry and then performs verb and HOI classification through progressive BLIP-2 prior fusion, with classifier weights initialized from BLIP-2 text embeddings for long-tail categories. Grouped-query training further enriches optimization without increasing inference cost. Across three model scales (Nano, Small, 2XL), HOIBlender consistently outperforms SOV-STG-VLA and Hybrid-SOV-VLA on HICO-DET, reaching 44.4944.49 Default Full mAP in only 99 training epochs while maintaining competitive latency and parameter budgets. These results show that lightweight detection, structured spatial-semantic decoding, and deeply integrated vision-language priors can be blended into a single efficient HOI pipeline.
Feb 16, 2026

Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. The codes are publicly available at https://github.com/SY-Xuan/DA-HOI.
Oct 5, 2026

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform strongly but rely on task-specific training. Recent multimodal large language models (MLLMs) offer a promising route to training-free HOI detection through their broad visual-semantic knowledge and versatile perceptual and reasoning capabilities. However, existing approaches largely invoke these capabilities through loosely coordinated inference stages. This fragmented execution restricts the role of interaction hypotheses in guiding visual exploration, leaving key participants overlooked and local ambiguities unresolved. Furthermore, propagating early semantic assumptions through subsequent visual grounding and relation prediction induces self-reinforcing semantic circularity. To resolve these challenges, we propose HarnessHOI, a training-free framework that transforms passive MLLM inference into an active interaction-centric harness. Specifically, we introduce an interaction-guided perception mechanism that projects emerging interaction hypotheses back into the visual space to discover missing participants and refine ambiguous evidence through targeted observation. Furthermore, a relation-agnostic geometric adjudication module reconciles multi-source evidence to establish a unified spatial basis for grounded interaction reasoning across multiple actions and semantic roles. Extensive experiments on HICO-DET and V-COCO demonstrate that HarnessHOI achieves state-of-the-art performance among training-free methods, confirming the effectiveness of the proposed harness for complex interaction understanding. Code will be released upon publication.