cs.AIApr 1, 2026
SaveHippoCamp: Benchmarking Contextual Agents on Personal Computers
Organizations: S-Lab, Nanyang Technological University, Singapore · Synvo AI
Abstract
We present HippoCamp, a new benchmark designed to evaluate agents' capabilities on multimodal file management. Unlike existing agent benchmarks that focus on tasks like web interaction, tool use, or software automation in generic settings, HippoCamp evaluates agents in user-centric environments to model individual user profiles and search massive personal files for context-aware reasoning. Our benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files. Building upon the raw files, we construct 581 QA pairs to assess agents' capabilities in search, evidence perception, and multi-step reasoning. To facilitate fine-grained analysis, we provide 46.1K densely annotated structured trajectories for step-wise failure diagnosis. We evaluate a wide range of state-of-the-art multimodal large language models (MLLMs) and agentic methods on HippoCamp. Our comprehensive experiments reveal a significant performance gap: even the most advanced commercial models achieve only 48.3% accuracy in user profiling, struggling particularly with long-horizon retrieval and cross-modal reasoning within dense personal file systems. Furthermore, our step-wise failure diagnosis identifies multimodal perception and evidence grounding as the primary bottlenecks. Ultimately, HippoCamp exposes the critical limitations of current agents in realistic, user-centric environments and provides a robust foundation for developing next-generation personal AI assistants.
Figures & tables
Figure 1: Overview of Task in HippoCamp Benchmark. HippoCamp is a benchmark designed to evaluate agents’ ability to search, perceive, and reason over long-term, realistic, large-scale personal file systems. It reproduces multimodal digital environments containing tens of gigabytes of user-specific assets across diverse modalities. Through tasks that demand grounded retrieval, cross-file pattern inference, and long-horizon reasoning, the benchmark rigorously evaluates the personalized multimodal memory of contextual agents.
| Benchmark | Source | Modalities | #Samples | Multimodal | User Profile | File-System |
|---|---|---|---|---|---|---|
| HotpotQA ( Yang et al., 2018 ) / KILT ( Petroni et al., 2021 ) | Wikipedia | 100k | ✗ | ✗ | ✗ | |
| BrowseComp ( Wei et al., 2025 ) | Web | 1,266 | ✗ | ✗ | ✗ | |
| MetaTool ( Huang et al., 2024 ) / MINT ( Wang et al., 2024a ) | Web + Tools | (w/ code) | 3k–20k | ✗ | ✗ | ✗ |
| WebQA ( Chang and Bisk, 2022 ) | Web | 7.5k | ✓ | ✗ | ✗ | |
| GAIA ( Mialon et al., 2024 ) / WebShop ( Yao et al., 2022 ) / PaperBench ( Starace et al., 2025 ) | Web | 1k–10k | ✓ | ✗ | ✗ | |
| MultiModalQA ( Talmor et al., 2021 ) | Web | (Table-Only) | 29k | ✓ | ✗ | ✗ |
Table 1: Comparison of Benchmarks Related to Multimodal Context and Agentic Reasoning. For modalities, denotes text, denotes images, denotes documents, denotes videos, and denotes audio. HippoCamp introduces a personalized multimodal file-system environment that spans all these modalities, enabling cross-file and cross-modal reasoning tasks at real-world scale.
Figure 2: HippoCamp hierarchical annotation schema. The pyramid organizes benchmark supervision from low-level atomic grounding and action traces to structured trajectories and task-level QA, with increasing abstraction and aggregation toward user-level memory. It highlights the benchmark’s multi-granularity diagnostic structure: factual retention is grounded in localized evidence and intermediate reasoning steps, while profiling sits at the top, requiring long-horizon, cross-modal synthesis of multiple grounded facts into coherent user-level inference.
Figure 3: Archetypal user profiles in HippoCamp. Each profile instantiates a distinct personal-device environment characterized along multiple dimensions, and is paired with a representative word-cloud summary of text-converted file contents , together with factual retention and profiling QA examples for (a) Bei, (b) Adam, and (c) Victoria.
Figure 4: Benchmark overview. (a) Data collection and human–LLM collaborative annotation pipeline for grounded trajectory construction. (b) Distribution of factual retention and profiling tasks. (c) Distribution of annotated agent capability labels over search , perception , and reasoning .
Figure 5: Shared evaluation framework for HippoCamp. All methods are evaluated under common profile-local constraints and a shared evaluator, supporting three execution regimes: (A) native retrieval methods, (B) vacuum-Docker terminal agents, and (C) hosted commercial agent modes. The framework standardizes control policies, metric computation, and output artifacts for downstream analysis.
Figure 6: Representative failure and success patterns on a cross-modal profiling query. The query requires aligning evidence across heterogeneous personal files and modalities. Standard RAG exhibits retrieval mismatch (retrieving unrelated finance documents), Search-R1 shows grounding avoidance (generic advice despite available evidence), Terminal Agent (GPT-5.2) triggers hard evidence hallucination (fabricated file paths/metadata), and ChatGPT Agent Mode makes an entity misattribution error (shifting the referent from the user to the pet), while iterative exploration yields the strongest overall behavior.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Modality composition across HippoCamp profiles (by file count). Percentage breakdown over five modalities for (a) Bei Weiwei, (b) Adam Turner, and (c) Victoria Anne Clarke.
| Ext. | Modality | Bei | Adam | Victoria |
|---|---|---|---|---|
| mp3 | Audio | 384.80 MB | 21.16 MB | 1.20 GB |
| csv | Documents | – | 11.92 KB | 435 B |
| docx | Documents | 2.26 MB | 44.98 MB | 4.15 MB |
| eml | Documents | 14.47 MB | 21.84 MB | 107.91 KB |
| ics | Documents | 1.08 KB | 13.28 KB | 25.99 KB |
| Documents | 1.08 GB | 82.07 MB | 960.08 MB |
Table 4: File type and storage footprint by profile. Extension-level size statistics across Bei, Adam, and Victoria, highlighting storage asymmetry and long-tail format coverage.
Figure 8: Temporal coverage of file activity. Creation and modification timestamp distributions for (a) Bei, (b) Adam, and (c) Victoria.
| Field | Type | Allowed values / format | Role | Primary use |
|---|---|---|---|---|
| id | string | String identifier | Record identifier for one QA instance. | Per-instance alignment; error analysis. |
| question | string | Free-form natural language | User query defining the information need. | LLM-judge input; token statistics; difficulty. |
| answer | string | Normalized textual answer; optional file refs in [...] | Gold answer target in normalized form. | LLM-judge reference; token statistics; difficulty. |
| QA_type | string | factual_retention , profiling | Top-level task family label. | Main results grouping; task-type breakdowns. |
| profiling_type | string | Profiling subtype labels | Profiling subtask label when applicable. | Profiling-subtype breakdowns. |
| data_source | string | Source tag from annotation ontology | Provenance/source tag for the instance. | Source audit; source-based analyses. |
Table 5: Top-level fields in one HippoCamp trajectory record. Stored fields are explicitly present in the released JSON.
| Modality | AU unit | Typical resolution | Example |
|---|---|---|---|
| Text | token span / sentence span | contiguous text span | a clause in a note or email body |
| Document | page-equivalent; optional region | page; optional table/figure region | page 4 of a PDF; a row in a regulatory table |
| Image | spatial patch / region | patch grid or bounding box | the face region or background region in a photo |
| Video | frame or temporal chunk | frame or timestamp window | 00:07–00:08 in an advertisement clip |
| Audio | temporal segment | timestamp window / frame span | 00:42:56–00:43:26 in a recording |
Table 6: Atomic-unit (AU) definition by modality. AUs normalize heterogeneous evidence locators into modality-aligned minimal units.
Figure 11: Factual retention example (cross-modal asset retrieval). Given a vlog script, the agent extracts the required assets and identifies the matching photos, with ground-truth answer and evidence visualizations.
Figure 12: Factual retention example (document–video compliance). The agent verifies logo clearspace by extracting the rule from the manual and checking it against video frames, with ground-truth answer and evidence visualizations.
Figure 13: Profiling subtask distribution. Proportions of the five profiling subtasks (preferences, behavioral patterns, scheduling information, retrospective reflections, and workflows) across the three HippoCamp profiles.
Figure 14: Profiling subtask: Preferences. Inferring stable photo-editing preferences from an email thread with annotated visual feedback and user confirmation.
Figure 15: Profiling subtask: Behavioral Patterns. Inferring a stable stress-regulation routine by aligning a training plan, logs, messages, diary notes, and multimodal evidence over time.
Figure 16: Profiling subtask: Scheduling Information. Conditional planning under conflicting commitments, grounded in calendars/emails and historical conflict-resolution evidence.
Figure 17: Profiling subtask: Retrospective Reflections. The agent reconstructs a graduation-day itinerary by aligning calendar events, photo metadata, and personal notes into a coherent timeline.
Figure 18: Profiling subtask: Workflows. Reconstructing a recurring “collaborative learning loop” by aligning calendar events, logs, consultation audio, messages, and document updates.
Figure 19: Evidence breadth distribution. Number of ground-truth evidence files required per query, shown for each profile and overall. The distribution is heavy-tailed, with many multi-file queries beyond 20 evidence files.
Figure 20: Modality breadth distribution. Number of distinct modalities required per query, shown for each profile and overall.
Figure 21: Reasoning depth distribution. Number of annotated reasoning steps required per query for each profile and overall. The distribution peaks at medium depth (6-8 steps) with a long tail of deep multi-step queries.
Figure 22: Difficulty distributions on HippoCamp. Histograms of the difficulty score for each profile and overall, with mean/median markers and task-type overlays (factual retention vs. profiling).
Figure 23: Difficulty vs. performance (question-level). For each profile and for the full benchmark, mean LLM-judge score (0–5) for nine methods as a function of binned difficulty (5-point bins). Scores are generally low-to-moderate and decline with increasing difficulty, with pronounced degradation in the hard tail.
Figure 24: Profile (a) Bei Weiwei example. A factual-retention query requiring extraction of visa photo constraints from an official document and verification against personal images, with grounded evidence.
Figure 25: Profile (b) Adam Turner example. An evidence-grounded reply drafting task that integrates an email inquiry with supporting precedent passages to produce a procedurally accurate response.
Figure 26: Profile (c) Victoria Anne Clarke example. A factual-retention query requiring cross-quarter attribution by aligning earnings-call transcripts and filings to identify changing revenue-growth drivers.
| Axis | Selected tail instances (Profile; ID; complexity; diagnostic requirement) |
|---|---|
| Evidence breadth | (a) Bei; ID 178; 111 files : How long was my last stay in Japan? (broad retrieval, pruning, cross-file corroboration). (b) Adam; ID 93; 56 files : Filter transferred companies with Date Incorporation after 2015. (large evidence-set management and structured verification). |
| Modality breadth | (a) Bei; ID 11; 6 modalities : How should I structure my folders for organizing reinforcement learning study materials? (cross-modal grounding across heterogeneous artifacts). (c) Victoria; ID 5; 5 modalities : I’m thinking about picking up the guitar. How might I get started? (multi-modal evidence alignment with user-specific context). |
| Reasoning depth | (c) Victoria; ID 34; 17 steps : How might Apple’s buyback cadence evolve if cash balances trend lower or rates shift? (long-horizon, verification-centric synthesis). (b) Adam; ID 95; 15 steps : Check if there are missing/incorrect parts in my Manufacturing company info. (iterative checking, error finding, and evidence-linked correction). |
Table 7: Diagnostic edge cases in HippoCamp. A compact set of tail instances that instantiate extreme evidence breadth , modality breadth , and reasoning depth .
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,486 papers