Code agents increasingly work on evolving real repositories, and memory systems aim to help them reuse earlier experience. However, such experience is diverse and evolving, so it remains unclear how memory systems organize it and thereby affect code agents. Existing benchmarks cannot answer this reliably because they lack either temporally valid histories or control over prior-task relevance. In this paper, we present MemDev-Bench to evaluate how memory systems affect code agents over diverse repository histories. It pairs each task with relevance-annotated earlier tasks from the same repository to build histories with controlled relevance. We also build an automated pipeline that reliably makes these earlier tasks executable and verifiable, producing 4,869 historical tasks for 555 base tasks. Based on MemDev-Bench, our systematic evaluation of memory systems across diverse development scenarios yields three main findings. (1) Memory mainly improves fine-grained localization and editing, increasing functional correctness. (2) Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures. These failures mainly stem from agents reusing memory without reassessing its applicability. (3) Compared with using memory systems, directly providing agents with raw historical trajectories substantially reduces their steps. Further analysis suggests that agents save these steps by reusing process information that compressed memories tend to omit. These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context.
Figures & tables
Figure 1: Base tasks distribution. Left: SWE-bench Verified (344 tasks, 12 Python repositories). Right: SWE-bench Pro (211 tasks, 10 repositories, four languages).
Figure 2: Benchmark construction and evaluation pipeline. Top: we mine candidate PRs, annotate their relevance, and convert them into executable historical tasks. Bottom: an agent solves historical tasks chronologically to build memory, then solves the base task with the frozen memory.
Relevance level
Verified
Pro
Strong Related
19.6
13.5
Weak Related
20.5
16.9
Unrelated, merged
47.0
65.2
Unrelated, unmerged
12.9
0 4.4
Table 1: Relevance composition (%).
Figure 3: Empirical cumulative distributions of historical task properties. (a) Time gap to the base task; (b) non-test patch size in characters; (c) number of changed files.
Resolved (%)
Rescue (%)
Harm (%)
Tok. In (M)
Tok. Out (K)
Steps
System
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
NoMem
35.3 ± 2.7
–
–
6.45 ± 0.23
59.5 ± 3.6
185.1 ± 5.3
FullTraj
39.1
37.3
39.6
12.2
11.2
11.8
8.4
9.2
7.5
16.31
24.11
32.86
50.2
41.2
37.4
141.7
114.5
113.1
± 0.8
± 1.6
± 1.7
8.9–15.9
7.8–14.9
8.4–15.4
5.3–11.9
6.4–12.0
5.1–10.0
± 0.15
± 1.46
± 2.18
± 1.3
± 1.2
± 3.8
± 1.7
± 6.0
± 7.6
A-Mem
40.4
38.1
41.6
11.4
10.4
13.2
6.3
7.6
6.9
6.11
6.47
6.51
61.9
63.0
62.7
178.0
180.6
183.8
± 2.4
± 4.3
± 1.6
8.1–14.8
7.1–13.9
9.6–17.1
4.0–9.0
5.1–10.3
4.6–9.6
± 0.07
± 0.36
± 0.31
± 1.6
± 2.2
± 1.7
± 0.8
± 5.0
± 4.1
Table 2: Results on 133 Improvement Tasks under Relevant (Rel.), Noisy (Noi.), and Repo history. Second lines give three-run SDs, or 95% bootstrap CIs for Rescue/Harm. Bold marks the best non-NoMem result; lower is better for Harm and cost. Appendix H.1 defines the metrics.
Resolved (%)
Rescue (%)
Harm (%)
Tok. In (M)
Tok. Out (K)
Steps
System
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
NoMem
94.0 ± 2.0
–
–
3.08 ± 0.24
34.9 ± 3.3
130.5 ± 7.1
FullTraj
97.3
92.7
90.0
4.0
2.9
2.7
0.7
4.2
6.7
7.22
11.80
22.99
29.6
24.4
22.5
95.1
83.4
81.2
± 1.2
± 2.3
± 2.0
0.7–8.0
0.4–6.2
0.7–5.3
0.0–2.0
1.6–7.3
3.1–10.4
± 0.43
± 0.25
± 1.03
± 0.9
± 3.1
± 2.5
± 5.6
± 2.0
± 3.6
A-Mem
95.3
94.0
93.3
4.4
3.1
2.7
3.1
3.1
3.3
3.30
3.50
3.55
37.8
41.6
39.1
128.9
131.2
134.8
± 3.1
± 2.0
± 3.1
1.1–8.9
0.7–6.0
0.7–5.3
0.7–5.8
0.9–5.8
0.9–6.2
± 0.34
± 0.30
± 0.13
± 3.0
± 1.3
± 1.0
± 5.8
± 2.7
± 2.3
Table 3: Results on 50 Preservation Tasks . Scenarios and reporting conventions follow Table 2 .
Resolved (%)
File Recall (%)
Line Recall (%)
Edit Recall (%)
F2P (%)
P2P (%)
System
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
Rel.
Noi.
Repo
NoMem
0.00 ± 0.00
74.22 ± 0.37
18.41 ± 2.19
11.02 ± 0.58
18.10 ± 1.01
97.39 ± 0.22
FullTraj
11.27
12.75
11.27
71.61
70.59
70.96
22.16
20.59
19.64
14.77
13.90
12.09
26.10
25.81
27.53
97.45
98.12
97.42
± 4.73
± 1.70
± 1.70
± 2.95
± 2.73
± 1.24
± 1.76
± 1.05
± 1.85
± 1.69
± 1.64
± 1.01
± 4.22
± 0.19
± 3.16
± 0.28
± 0.72
± 0.16
A-Mem
10.29
10.78
12.75
72.02
74.59
70.42
17.65
18.67
18.82
11.83
12.38
11.21
25.55
26.99
27.29
97.28
97.37
97.54
± 3.89
± 4.25
± 0.85
± 3.55
± 2.09
± 0.67
± 1.23
± 1.70
± 2.85
± 0.77
± 0.83
± 1.59
± 2.97
± 3.55
± 0.55
± 0.32
± 0.29
± 0.16
Table 4: Task-solving outcomes and repair metrics on 68 tasks where all three NoMem runs fail.
Figure 4: Counts of failure modes.
Figure 5: FullTraj reductions in steps, output tokens, and action calls. Comparisons with NoMem and five memory systems use pooled means over 183 instances. Super and Claude denote Supermemory and Claude-Mem.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Relation type
Definition
Strong Related
reusable implementation pattern
The historical PR contains an implementation pattern, code structure, algorithm, or repair strategy that can be reused directly, or applied to the base task after local adaptation.
same code path
The historical PR involves the same critical code path as the base task, or one directly connected to it, so its changes help localize where and how to fix it.
introducing change
The historical PR introduced or substantially modified the behavior, assumption, or implementation that later gave rise to the base task’s problem, making it an important source for understanding the current issue.
same problem
The historical PR addresses essentially the same problem or requirement as the base task, possibly in a different version, component, or usage scenario.
same root cause
The surface symptoms of the historical PR and the base task differ, but the two share the same underlying technical cause, so the historical analysis or fix transfers directly to the current task.
Appendix
Table 5: Fine-grained relation types between a historical PR and a base task. Each relevance level has its own set of types, and an annotation must pick a type that belongs to the level it assigns.
Figure 6: Distribution of fine-grained relation types within each relevance level.
Figure 7: Executable historical task construction pipeline. (a) PR changes are split into test patch T and solution patch S , and the task inputs are frozen. (b) The original commit c is resolved and validated to construct a shared Base image. (c) The two patched source states independently invoke the shared resolver, and their plans and environment artifacts are frozen as Post A and Post B for offline restoration. (d) Stable outcomes are obtained from three fresh runs on each side.
Setting
Configuration
Agent and base model
Agent
Claude Code 2.1.170
Base model
DeepSeek-V4-Flash
Model routing
The primary route, all Claude Code model aliases, and the subagent route use DeepSeek-V4-Flash
Reasoning effort
High
Interaction budget
At most 200 turns per task
Appendix
Table 6: Shared agent and execution configuration. Every setting is identical across all evaluated systems and all six evaluation conditions.
Tool category
Availability
Enforcement
Bash
Available
The guard validates each request before execution and inspects the complete command string.
Read, Write, Edit, NotebookEdit, Glob, Grep
Available
The guard validates each request for protected paths, Git metadata, and alternate copies of the target project.
WebSearch, WebFetch
Disabled
Turned off in the CLI and additionally rejected by the guard if invoked, removing the direct route to external information.
EnterWorktree, ExitWorktree
Disabled
Turned off in the CLI and additionally rejected by the guard if invoked, keeping the agent inside the current checkout.
Appendix
Table 7: Tool availability under the information-isolation protocol. Tools that remain available are checked request by request rather than turned off, so normal local development is preserved.
Rule family
Restricted behavior
Preserved behavior
Target-project copies
Inspecting the target project under site-packages or dist-packages , package-metadata queries that locate such copies, and imports of the target project from outside the current checkout.
Genuinely separate third-party dependencies may still be inspected.
Ordinary installation of named third-party dependencies.
Git and metadata
Git history, any Git operation other than the permitted commands, direct access to .git , Git plumbing utilities, and Git-metadata environment variables.
git status and git diff on the current working tree or index, optionally against HEAD .
Benchmark-private paths
Access to the benchmark-private /tests and /solution paths.
Repository-owned source files and tests inside the current checkout.
Explicit network access
Explicit URLs in curl , wget , and recognized inline-Python calls ( requests , httpx , aiohttp , urllib , http.client , socket ) that target code hosts (GitHub and githubusercontent.com , GitLab, Bitbucket), Q&A sites (Stack Overflow and Stack Exchange), search engines (Google, Bing, DuckDuckGo, Brave Search), or SWE-bench.
Networking needed for ordinary dependency installation and task execution.
Guard bypass
Attempts to unset or override LD_PRELOAD or network-guard endpoint variables, or to bypass the execution guard by any other means.
Requests that satisfy the isolation checks proceed.
Appendix
Table 8: Main request-level checks enforced by the PreToolUse guard. Each rule family blocks a route to external answers or repository history while keeping the corresponding development operation available.
Task performance
Solving cost
Memory system
Resolved
File Recall
F2P
P2P
Tokens
Steps
(%)
(%)
(%)
(%)
In (M)
Out (K)
causal-memory ( JingxuanC, 2026 )
52.67
81.56
79.84
99.47
6.18
56.50
186.35
ChronoHybridMem ( Tin11Mn, 2026 )
52.00
83.34
81.29
99.42
6.62
59.31
186.68
Memory Bear ( Wen et al., 2026 )
52.00
79.90
80.50
99.46
6.62
55.95
180.39
TencentDB Agent Memory ( Tencent Cloud, 2026 )
51.33
82.23
80.76
99.45
5.86
55.94
184.69
Appendix
Table 9: Evaluation of 29 memory systems under the Relevant History scenario on the same 150 base tasks.
System
Implementation
Memory LLM
Embedding Model
Retrieval Configuration
A-Mem
ceffb860f0712 ; MCP 0.2.6 ( dce0ac62cba5 )
DeepSeek-V4-Flash
all-MiniLM-L6-v2 (SentenceTransformers, Chroma)
k=5 for standard and agentic search; k=10 for temporal search
Mem0
Plugin 0.2.9 ( 366945965df4 ); official API
Provider-managed
Provider-managed
Official API defaults
MemOS
MCP 1.1.2; official API
Provider-managed
Provider-managed
memory_limit_number=6 ; retry with relativity=0 when no result is returned
Supermemory
Plugin 0.0.13 ( 193290b5ccba ); official API
Provider-managed
Provider-managed
limit=5 ; searchMode=hybrid ; threshold=0.6
Claude-Mem
v12.1.1 ( 0b6997e3d03e )
DeepSeek-V4-Flash
all-MiniLM-L6-v2 (ONNX, 384-d)
search limit=20 ; 100 semantic candidates
Appendix
Table 10: Memory-system configurations used in our evaluation.
Figure 8: Shared trajectory. An initial change breaks default metadata preservation in the existing methods. The same five tests expose the regression and pass after the repair.
Figure 9: A-Mem with Claude Code. Claude composes a note from e3 – e5 , which A-Mem enriches and links. Retrieval combines semantic previews, selective reading, and optional link expansion.
Figure 10: Mem0 with Claude Code. Selected interaction evidence is distilled into persistent memories for scoped retrieval.
Figure 11: MemOS with Claude Code. MemReader extracts textual memories into MemCube for later semantic retrieval.
Figure 12: Supermemory with Claude Code. Documents yield linked atomic memories for project profiles and subsequent retrieval.
Figure 13: Claude-Mem with Claude Code. Observations and progress summaries support startup context and selective retrieval.
Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluations do not show whether those systems improve executable repository work. Repository benchmarks test code changes but do not isolate memory, while memory benchmarks score recall without measuring downstream coding outcomes. We introduce VibeMemBench, a benchmark for evaluating memory systems on 111 coding targets from 90 SWE-rebench V2 repositories and 3,634 history trajectories from the target repositories. The targets follow the SWE benchmark style and cover bug fixes, feature requests, interface changes, and configuration work. An agent edits each target codebase under a declared memory condition. Executable tests decide task resolution. Each target is retained only when injected history experience improves its executable outcome in a reference setting, so every target carries a prior experience whose usefulness is verified by execution in that setting. The frozen verified experience is then transferred to five held-out solvers. Direct injection raises observed task resolution on four of them by 1.1 to 4.5 percentage points while lowering agent steps on all five. Yet when four existing memory systems must construct and retrieve experience from the same history, eleven of twelve solver and system pairings fail to exceed the matched memory-off baseline. VibeMemBench exposes the gap between the useful experience that repository history holds and the experience existing memory systems deliver for repository coding tasks.
Liyang Fan, Yingcheng Shi, Yongbin Li +7
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · SUAT · Alibaba Token Hub, Alibaba Group +1
As coding agents take on long-horizon software evolution tasks spanning multiple files and stages, longer execution trajectories introduce two coupled challenges: (1) accumulated histories strain context budgets, and (2) repository changes can invalidate earlier execution evidence. Existing approaches address these challenges through techniques like larger context windows, compression, retrieval, or repository representations, but often fail to reconstruct a consistent task state after a context refresh or verify whether recalled evidence remains valid. Thus, we introduce MemTrace, a provenance-aware memory system that preserves execution history and aligns its reuse with the evolving task (e.g., iterative cross-file repair) and repository state. MemTrace stores history as immutable Memory Traces anchored to key information (e.g., files, symbols, tests), and organizes their execution order and dependencies in a Memory Trace Graph. When context is constrained, working memory retains only compact Memory Anchors, from which the agent can reconstruct the latest execution state and locate evidence relevant to its next action. Before restoring historical evidence, MemTrace checks its validity against the current repository state and retrieves only what the next action requires. Across three complementary long-horizon coding benchmarks, MemTrace consistently outperforms all fully evaluated baselines under the same backbone and harness, improving DeepSWE pass@1 by 21.2 points, SWE-EVO Resolved Rate by 4.4 points, and SWE-Milestone Score by 17.8 points under Codex CLI.
Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.