ImagineAgent: What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation
Organizations: Dream-X Team · Stanford University · National University of Singapore · Independent Researcher ·
Abstract
Multimodal Large Language Models have shown promising capabilities in bridging visual and textual reasoning, yet their reasoning capabilities in Open-Vocabulary Human-Object Interaction (OV-HOI) are limited by cross-modal hallucinations and limited viewpoints of images. To address this, we propose ImagineAgent, an agentic framework that integrates cognitive mapping, tool-augmented reinforcement learning (RL), and generative world modeling for robust OV-HOI understanding. Specifically, we first propose an innovative CoT dataset named hicodet-6K for supervised fine-tuning (SFT), which effectively bridges the perception-to-cognition gap by structuring perceived entities into interaction pairs for comprehensive predictions. Subsequently, we develop a multimodal tool library integrating online retrieval, image cropping, and generative modeling, enabling the agent to dynamically augment reasoning with domain-specific tools to resolve visual-semantic ambiguities and hallucinations during inference. Moreover, we incorporate a generative model to reconstruct alternative viewpoints, enabling the agent to 'imagine' under limited viewpoints. Finally, we propose a composite reward mechanism to jointly optimize prediction accuracy and tool efficiency. Evaluations on both SWIG-HOI and HICO-DET datasets demonstrate that our method achieves state-of-the-art performance while requiring merely 36.7% of the training data compared to existing methods, validating our robustness, empirical effectiveness and efficiency.
Figures & tables
| Method | Pretrained Detector | Unseen | Seen | Full |
| Zero-shot Methods | - | - | - | - |
| FCL [ 13 ] | ✓ | 13.16 | 24.23 | 22.01 |
| GEN-VLKT [ 25 ] | ✓ | 21.36 | 32.91 | 30.56 |
| HOICLIP [ 32 ] | ✓ | 23.48 | 34.47 | 32.26 |
| CLIP4HOI [ 31 ] | ✓ | 28.47 | 35.48 | 34.08 |
| HOIGen [ 10 ] | ✓ | 31.01 | 34.57 | 33.86 |
| Method | Non-rare | Rare | Unseen | Full |
| QPIC [ 37 ] | 16.95 | 10.84 | 6.21 | 11.12 |
| GEN-VLKT [ 25 ] | 20.91 | 10.41 | - | 10.87 |
| MP-HOI [ 42 ] | 20.28 | 14.78 | - | 12.61 |
| THID [ 38 ] | 17.67 | 12.82 | 10.04 | 13.26 |
| CMD-SE [ 23 ] | 21.46 | 14.64 | 10.70 | 15.26 |
| INP-CC [ 22 ] | 22.84 | 16.74 | 11.02 | 16.74 |
| Method | Training Steps | Unseen | Seen | Full |
| (a) Qwen3-VL-8B | 0 steps | 19.41 | 18.24 | 18.47 |
| (b) w/ SFT ( hicodet-6K ) | 750 steps | 28.24 | 26.98 | 27.23 |
| (c) w/ RL (GRPO) | 1,000 steps | 25.33 | 24.34 | 24.54 |
| ImagineAgent (Ours) | 1,750 steps | 37.85 | 36.49 | 36.76 |
| Method | Unseen | Seen | Full |
| (a) w/o image crop | 37.69 | 35.19 | 35.69 |
| (b) w/o action explanation | 37.37 | 34.17 | 34.81 |
| (c) w/o scene explanation | 37.31 | 35.96 | 36.23 |
| (d) w/o view transform | 37.82 | 34.90 | 35.48 |
| (e) w/o outpaint | 37.58 | 35.10 | 35.61 |
| ImagineAgent (Ours) | 37.85 | 36.49 | 36.76 |
| Method | Unseen | Seen | Full |
| (a) w/o HOI Reward | 33.75 | 32.34 | 32.62 |
| (b) w/o Precision & Recall | 34.56 | 33.42 | 33.65 |
| (c) w/o Hungarian | 35.62 | 34.53 | 34.75 |
| (d) w/o Verb Similarity | 37.25 | 36.03 | 36.27 |
| (e) w/o Tool Reward | 36.62 | 35.42 | 35.66 |
| ImagineAgent (Ours) | 37.85 | 36.49 | 36.76 |
| Editing Model | Unseen | Seen | Full |
| (a) FLUX.1 Kontext [ 20 ] | 38.34 | 36.95 | 37.23 |
| (b) Qwen-Image [ 41 ] | 38.19 | 36.83 | 37.10 |
| (c) BAGEL [ 7 ] ( Ours ) | 37.85 | 36.49 | 36.76 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Unseen | Seen | Full |
| GPT-4o | 22.15 | 21.45 | 21.78 |
| Gemini 3.0 Pro | 22.45 | 21.75 | 22.06 |
| GPT-5 | 23.45 | 22.85 | 23.18 |
| ImagineAgent (Ours) | 33.55 | 32.40 | 32.84 |
| Metric | Num. Images | Rate |
| MLLM Hallucination | 9,546 | 38.7% |
| Object Occlusion | 600 | 12.3% |
| Viewpoint Limitation | 600 | 19.7% |
| Error Type | Proportion |
| Action classification error | 23.2% |
| Grounding error | 20.8% |
| Object classification error | 25.5% |
| Incomplete HOI matching | 39.7% |
| Tool | Input | Output | Failure Handling |
| Image Crop | predicted human/object boxes | cropped image patch | ignored if boxes are invalid |
| Action Explanation | candidate action phrase | textual action definition | fallback to original query |
| Scene Explanation | image and scene query | textual scene context | fallback to original query |
| Viewpoint Transform | image and editing prompt | transformed image | used only as auxiliary evidence |
| Outpaint | image and editing prompt | expanded image | used only as auxiliary evidence |
Tool-Using Vision-Language Agents
Tool-using vision-language agents combine visual and language models with external tools to gather evidence, act, and solve multi-step tasks.
See all 147 papers