We present HippoCamp, a new benchmark designed to evaluate agents' capabilities on multimodal file management. Unlike existing agent benchmarks that focus on tasks like web interaction, tool use, or software automation in generic settings, HippoCamp evaluates agents in user-centric environments to model individual user profiles and search massive personal files for context-aware reasoning. Our benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files. Building upon the raw files, we construct 581 QA pairs to assess agents' capabilities in search, evidence perception, and multi-step reasoning. To facilitate fine-grained analysis, we provide 46.1K densely annotated structured trajectories for step-wise failure diagnosis. We evaluate a wide range of state-of-the-art multimodal large language models (MLLMs) and agentic methods on HippoCamp. Our comprehensive experiments reveal a significant performance gap: even the most advanced commercial models achieve only 48.3% accuracy in user profiling, struggling particularly with long-horizon retrieval and cross-modal reasoning within dense personal file systems. Furthermore, our step-wise failure diagnosis identifies multimodal perception and evidence grounding as the primary bottlenecks. Ultimately, HippoCamp exposes the critical limitations of current agents in realistic, user-centric environments and provides a robust foundation for developing next-generation personal AI assistants.
Figures & tables
Figure 1: Overview of Task in HippoCamp Benchmark. HippoCamp is a benchmark designed to evaluate agents’ ability to search, perceive, and reason over long-term, realistic, large-scale personal file systems. It reproduces multimodal digital environments containing tens of gigabytes of user-specific assets across diverse modalities. Through tasks that demand grounded retrieval, cross-file pattern inference, and long-horizon reasoning, the benchmark rigorously evaluates the personalized multimodal memory of contextual agents.
Benchmark
Source
Modalities
#Samples
Multimodal
User Profile
File-System
HotpotQA ( Yang et al., 2018 ) / KILT ( Petroni et al., 2021 )
Wikipedia
∼ 100k
✗
✗
✗
BrowseComp ( Wei et al., 2025 )
Web
1,266
✗
✗
✗
MetaTool ( Huang et al., 2024 ) / MINT ( Wang et al., 2024a )
Web + Tools
(w/ code)
∼ 3k–20k
✗
✗
✗
WebQA ( Chang and Bisk, 2022 )
Web
7.5k
✓
✗
✗
GAIA ( Mialon et al., 2024 ) / WebShop ( Yao et al., 2022 ) / PaperBench ( Starace et al., 2025 )
Web
∼ 1k–10k
✓
✗
✗
MultiModalQA ( Talmor et al., 2021 )
Web
(Table-Only)
29k
✓
✗
✗
Table 1: Comparison of Benchmarks Related to Multimodal Context and Agentic Reasoning. For modalities, denotes text, denotes images, denotes documents, denotes videos, and denotes audio. HippoCamp introduces a personalized multimodal file-system environment that spans all these modalities, enabling cross-file and cross-modal reasoning tasks at real-world scale.
Figure 2: HippoCamp hierarchical annotation schema. The pyramid organizes benchmark supervision from low-level atomic grounding and action traces to structured trajectories and task-level QA, with increasing abstraction and aggregation toward user-level memory. It highlights the benchmark’s multi-granularity diagnostic structure: factual retention is grounded in localized evidence and intermediate reasoning steps, while profiling sits at the top, requiring long-horizon, cross-modal synthesis of multiple grounded facts into coherent user-level inference.
Figure 3: Archetypal user profiles in HippoCamp. Each profile instantiates a distinct personal-device environment characterized along multiple dimensions, and is paired with a representative word-cloud summary of text-converted file contents , together with factual retention and profiling QA examples for (a) Bei, (b) Adam, and (c) Victoria.
Figure 4: Benchmark overview. (a) Data collection and human–LLM collaborative annotation pipeline for grounded trajectory construction. (b) Distribution of factual retention and profiling tasks. (c) Distribution of annotated agent capability labels over search , perception , and reasoning .
Figure 5: Shared evaluation framework for HippoCamp. All methods are evaluated under common profile-local constraints and a shared evaluator, supporting three execution regimes: (A) native retrieval methods, (B) vacuum-Docker terminal agents, and (C) hosted commercial agent modes. The framework standardizes control policies, metric computation, and output artifacts for downstream analysis.
Figure 6: Representative failure and success patterns on a cross-modal profiling query. The query requires aligning evidence across heterogeneous personal files and modalities. Standard RAG exhibits retrieval mismatch (retrieving unrelated finance documents), Search-R1 shows grounding avoidance (generic advice despite available evidence), Terminal Agent (GPT-5.2) triggers hard evidence hallucination (fabricated file paths/metadata), and ChatGPT Agent Mode makes an entity misattribution error (shifting the referent from the user to the pet), while iterative exploration yields the strongest overall behavior.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Modality composition across HippoCamp profiles (by file count). Percentage breakdown over five modalities for (a) Bei Weiwei, (b) Adam Turner, and (c) Victoria Anne Clarke.
Ext.
Modality
Bei
Adam
Victoria
mp3
Audio
384.80 MB
21.16 MB
1.20 GB
csv
Documents
–
11.92 KB
435 B
docx
Documents
2.26 MB
44.98 MB
4.15 MB
eml
Documents
14.47 MB
21.84 MB
107.91 KB
ics
Documents
1.08 KB
13.28 KB
25.99 KB
pdf
Documents
1.08 GB
82.07 MB
960.08 MB
Appendix
Table 4: File type and storage footprint by profile. Extension-level size statistics across Bei, Adam, and Victoria, highlighting storage asymmetry and long-tail format coverage.
Figure 8: Temporal coverage of file activity. Creation and modification timestamp distributions for (a) Bei, (b) Adam, and (c) Victoria.
Field
Type
Allowed values / format
Role
Primary use
id
string
String identifier
Record identifier for one QA instance.
Per-instance alignment; error analysis.
question
string
Free-form natural language
User query defining the information need.
LLM-judge input; token statistics; difficulty.
answer
string
Normalized textual answer; optional file refs in [...]
Table 5: Top-level fields in one HippoCamp trajectory record. Stored fields are explicitly present in the released JSON.
Modality
AU unit
Typical resolution
Example
Text
token span / sentence span
contiguous text span
a clause in a note or email body
Document
page-equivalent; optional region
page; optional table/figure region
page 4 of a PDF; a row in a regulatory table
Image
spatial patch / region
patch grid or bounding box
the face region or background region in a photo
Video
frame or temporal chunk
frame or timestamp window
00:07–00:08 in an advertisement clip
Audio
temporal segment
timestamp window / frame span
00:42:56–00:43:26 in a recording
Appendix
Table 6: Atomic-unit (AU) definition by modality. AUs normalize heterogeneous evidence locators into modality-aligned minimal units.
Figure 11: Factual retention example (cross-modal asset retrieval). Given a vlog script, the agent extracts the required assets and identifies the matching photos, with ground-truth answer and evidence visualizations.
Figure 12: Factual retention example (document–video compliance). The agent verifies logo clearspace by extracting the rule from the manual and checking it against video frames, with ground-truth answer and evidence visualizations.
Figure 13: Profiling subtask distribution. Proportions of the five profiling subtasks (preferences, behavioral patterns, scheduling information, retrospective reflections, and workflows) across the three HippoCamp profiles.
Figure 14: Profiling subtask: Preferences. Inferring stable photo-editing preferences from an email thread with annotated visual feedback and user confirmation.
Figure 15: Profiling subtask: Behavioral Patterns. Inferring a stable stress-regulation routine by aligning a training plan, logs, messages, diary notes, and multimodal evidence over time.
Figure 16: Profiling subtask: Scheduling Information. Conditional planning under conflicting commitments, grounded in calendars/emails and historical conflict-resolution evidence.
Figure 17: Profiling subtask: Retrospective Reflections. The agent reconstructs a graduation-day itinerary by aligning calendar events, photo metadata, and personal notes into a coherent timeline.
Figure 18: Profiling subtask: Workflows. Reconstructing a recurring “collaborative learning loop” by aligning calendar events, logs, consultation audio, messages, and document updates.
Figure 19: Evidence breadth distribution. Number of ground-truth evidence files required per query, shown for each profile and overall. The distribution is heavy-tailed, with many multi-file queries beyond 20 evidence files.
Figure 20: Modality breadth distribution. Number of distinct modalities required per query, shown for each profile and overall.
Figure 21: Reasoning depth distribution. Number of annotated reasoning steps required per query for each profile and overall. The distribution peaks at medium depth (6-8 steps) with a long tail of deep multi-step queries.
Figure 22: Difficulty distributions on HippoCamp. Histograms of the difficulty score for each profile and overall, with mean/median markers and task-type overlays (factual retention vs. profiling).
Figure 23: Difficulty vs. performance (question-level). For each profile and for the full benchmark, mean LLM-judge score (0–5) for nine methods as a function of binned difficulty (5-point bins). Scores are generally low-to-moderate and decline with increasing difficulty, with pronounced degradation in the hard tail.
Figure 24: Profile (a) Bei Weiwei example. A factual-retention query requiring extraction of visa photo constraints from an official document and verification against personal images, with grounded evidence.
Figure 25: Profile (b) Adam Turner example. An evidence-grounded reply drafting task that integrates an email inquiry with supporting precedent passages to produce a procedurally accurate response.
Figure 26: Profile (c) Victoria Anne Clarke example. A factual-retention query requiring cross-quarter attribution by aligning earnings-call transcripts and filings to identify changing revenue-growth drivers.
(a) Bei; ID 178; 111 files : How long was my last stay in Japan? (broad retrieval, pruning, cross-file corroboration). (b) Adam; ID 93; 56 files : Filter transferred companies with Date Incorporation after 2015. (large evidence-set management and structured verification).
Modality breadth
(a) Bei; ID 11; 6 modalities : How should I structure my folders for organizing reinforcement learning study materials? (cross-modal grounding across heterogeneous artifacts). (c) Victoria; ID 5; 5 modalities : I’m thinking about picking up the guitar. How might I get started? (multi-modal evidence alignment with user-specific context).
Reasoning depth
(c) Victoria; ID 34; 17 steps : How might Apple’s buyback cadence evolve if cash balances trend lower or rates shift? (long-horizon, verification-centric synthesis). (b) Adam; ID 95; 15 steps : Check if there are missing/incorrect parts in my Manufacturing company info. (iterative checking, error finding, and evidence-linked correction).
Appendix
Table 7: Diagnostic edge cases in HippoCamp. A compact set of tail instances that instantiate extreme evidence breadth , modality breadth , and reasoning depth .
Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources. Due to this complexity, agentic task failures arise from various sources, and pinpointing these failure causes is essential to diagnose and improve agentic systems. Existing benchmarks, however, tend to focus on a single leaderboard score, leaving the underlying failure modes opaque. To fill this gap, we introduce AgentHop, a diagnostic benchmark of 1,011 multiple-choice questions paired with a controlled seven-tool sandbox under fixed token, turn, and tool-call constraints. AgentHop reveals model vulnerabilities by dissecting a single accuracy score along four axes of agent operation: retrieval, synthesis, tool-call, and resource management. Across 19 models, we find that behavior clusters by model family, with tool-call signatures revealing distinct family fingerprints: GPT models commit early, Anthropic and GLM checkpoints verify before committing, DeepSeek and Kimi over-search, and Gemini-3 Pro stays balanced. Decomposed axes further expose within-family structure: Claude Opus 4.6 and Sonnet 4.6 land within one accuracy point yet diverge on retrieval-versus-synthesis emphasis, with Opus retrieving more and Sonnet synthesizing better. We release the full benchmark set and the harness to support diagnostic agent benchmarking.
The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and tools, and has been rapidly adopted across personal applications and development platforms. However, existing benchmarks predominantly focus on generic information-seeking tools and fail to capture the practical challenges posed by personal social applications, where tools interact with individual accounts or local databases. To bridge this critical gap, we introduce MCP-Persona, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools. MCP-Persona encompasses a diverse set of widely-used applications, ranging from social media platforms like Reddit and Xiaohongshu (Rednote) to enterprise collaboration suites such as Lark (Feishu) and Slack. Our extensive experiments on various state-of-the-art (SOTA) agents demonstrate their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations. MCP-Persona is publicly available at https://github.com/wwh0411/MCP-Persona}{https://github.com/wwh0411/MCP-Persona.
Wenhao Wang, Peizhi Niu, Gongyi Zou +9
Multi-Agent Governance & Intelligence Crew (MAGIC), Shanghai Jiao Tong University, Shanghai, China · Department of Computer Science and Technology, Zhejiang University, Hangzhou, China · University of Illinois at Urbana-Champaign, Illinois, USA +2
Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants. However, existing memory benchmarks largely focus on single-user, text-only interactions, failing to capture these challenges. To address this gap, we introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions. H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.
Shiping Zhu, Yibo Yang, Zhengyang Wang +3
Jilin University · Shanghai Jiao Tong University · University of California at Merced