MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories

Guorui Chen; Tao Liu; Zhenzhen Huang; Yidan Zhang; Liyu Jiang; Shuoyuan Wang; Hanchen Sun; Xu He

cs.AIOct 20, 2026

MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories

Authors: Guorui Chen, Tao Liu, Zhenzhen Huang, Yidan Zhang, Liyu Jiang, Shuoyuan Wang, Hanchen Sun, Xu He,

Organizations: Southeast University, Nanjing, China · Southern University of Science and Technology, Shenzhen, China · MemoraX AI, Shenzhen, China

Abstract

Code agents increasingly work on evolving real repositories, and memory systems aim to help them reuse earlier experience. However, such experience is diverse and evolving, so it remains unclear how memory systems organize it and thereby affect code agents. Existing benchmarks cannot answer this reliably because they lack either temporally valid histories or control over prior-task relevance. In this paper, we present MemDev-Bench to evaluate how memory systems affect code agents over diverse repository histories. It pairs each task with relevance-annotated earlier tasks from the same repository to build histories with controlled relevance. We also build an automated pipeline that reliably makes these earlier tasks executable and verifiable, producing 4,869 historical tasks for 555 base tasks. Based on MemDev-Bench, our systematic evaluation of memory systems across diverse development scenarios yields three main findings. (1) Memory mainly improves fine-grained localization and editing, increasing functional correctness. (2) Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures. These failures mainly stem from agents reusing memory without reassessing its applicability. (3) Compared with using memory systems, directly providing agents with raw historical trajectories substantially reduces their steps. Further analysis suggests that agents save these steps by reusing process information that compressed memories tend to omit. These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context.

Figures & tables

Figure 1: Base tasks distribution. Left: SWE-bench Verified (344 tasks, 12 Python repositories). Right: SWE-bench Pro (211 tasks, 10 repositories, four languages).

Figure 2: Benchmark construction and evaluation pipeline. Top: we mine candidate PRs, annotate their relevance, and convert them into executable historical tasks. Bottom: an agent solves historical tasks chronologically to build memory, then solves the base task with the frozen memory.

Relevance level	Verified	Pro
Strong Related	19.6	13.5
Weak Related	20.5	16.9
Unrelated, merged	47.0	65.2
Unrelated, unmerged	12.9	0 4.4

Table 1: Relevance composition (%).

Figure 3: Empirical cumulative distributions of historical task properties. (a) Time gap to the base task; (b) non-test patch size in characters; (c) number of changed files.

	Resolved (%)			Rescue (%)			Harm (%)			Tok. In (M)			Tok. Out (K)			Steps
System	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo
NoMem	35.3 $\pm$ 2.7			–			–			6.45 $\pm$ 0.23			59.5 $\pm$ 3.6			185.1 $\pm$ 5.3
FullTraj	39.1	37.3	39.6	12.2	11.2	11.8	8.4	9.2	7.5	16.31	24.11	32.86	50.2	41.2	37.4	141.7	114.5	113.1
	$\pm$ 0.8	$\pm$ 1.6	$\pm$ 1.7	8.9–15.9	7.8–14.9	8.4–15.4	5.3–11.9	6.4–12.0	5.1–10.0	$\pm$ 0.15	$\pm$ 1.46	$\pm$ 2.18	$\pm$ 1.3	$\pm$ 1.2	$\pm$ 3.8	$\pm$ 1.7	$\pm$ 6.0	$\pm$ 7.6
A-Mem	40.4	38.1	41.6	11.4	10.4	13.2	6.3	7.6	6.9	6.11	6.47	6.51	61.9	63.0	62.7	178.0	180.6	183.8
	$\pm$ 2.4	$\pm$ 4.3	$\pm$ 1.6	8.1–14.8	7.1–13.9	9.6–17.1	4.0–9.0	5.1–10.3	4.6–9.6	$\pm$ 0.07	$\pm$ 0.36	$\pm$ 0.31	$\pm$ 1.6	$\pm$ 2.2	$\pm$ 1.7	$\pm$ 0.8	$\pm$ 5.0	$\pm$ 4.1

Table 2: Results on 133 Improvement Tasks under Relevant (Rel.), Noisy (Noi.), and Repo history. Second lines give three-run SDs, or

95\%

bootstrap CIs for Rescue/Harm. Bold marks the best non-NoMem result; lower is better for Harm and cost. Appendix H.1 defines the metrics.

	Resolved (%)			Rescue (%)			Harm (%)			Tok. In (M)			Tok. Out (K)			Steps
System	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo
NoMem	94.0 $\pm$ 2.0			–			–			3.08 $\pm$ 0.24			34.9 $\pm$ 3.3			130.5 $\pm$ 7.1
FullTraj	97.3	92.7	90.0	4.0	2.9	2.7	0.7	4.2	6.7	7.22	11.80	22.99	29.6	24.4	22.5	95.1	83.4	81.2
	$\pm$ 1.2	$\pm$ 2.3	$\pm$ 2.0	0.7–8.0	0.4–6.2	0.7–5.3	0.0–2.0	1.6–7.3	3.1–10.4	$\pm$ 0.43	$\pm$ 0.25	$\pm$ 1.03	$\pm$ 0.9	$\pm$ 3.1	$\pm$ 2.5	$\pm$ 5.6	$\pm$ 2.0	$\pm$ 3.6
A-Mem	95.3	94.0	93.3	4.4	3.1	2.7	3.1	3.1	3.3	3.30	3.50	3.55	37.8	41.6	39.1	128.9	131.2	134.8
	$\pm$ 3.1	$\pm$ 2.0	$\pm$ 3.1	1.1–8.9	0.7–6.0	0.7–5.3	0.7–5.8	0.9–5.8	0.9–6.2	$\pm$ 0.34	$\pm$ 0.30	$\pm$ 0.13	$\pm$ 3.0	$\pm$ 1.3	$\pm$ 1.0	$\pm$ 5.8	$\pm$ 2.7	$\pm$ 2.3

Table 3: Results on 50 Preservation Tasks . Scenarios and reporting conventions follow Table 2 .

	Resolved (%)			File Recall (%)			Line Recall (%)			Edit Recall (%)			F2P (%)			P2P (%)
System	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo	Rel.	Noi.	Repo
NoMem	0.00 $\pm$ 0.00			74.22 $\pm$ 0.37			18.41 $\pm$ 2.19			11.02 $\pm$ 0.58			18.10 $\pm$ 1.01			97.39 $\pm$ 0.22
FullTraj	11.27	12.75	11.27	71.61	70.59	70.96	22.16	20.59	19.64	14.77	13.90	12.09	26.10	25.81	27.53	97.45	98.12	97.42
	$\pm$ 4.73	$\pm$ 1.70	$\pm$ 1.70	$\pm$ 2.95	$\pm$ 2.73	$\pm$ 1.24	$\pm$ 1.76	$\pm$ 1.05	$\pm$ 1.85	$\pm$ 1.69	$\pm$ 1.64	$\pm$ 1.01	$\pm$ 4.22	$\pm$ 0.19	$\pm$ 3.16	$\pm$ 0.28	$\pm$ 0.72	$\pm$ 0.16
A-Mem	10.29	10.78	12.75	72.02	74.59	70.42	17.65	18.67	18.82	11.83	12.38	11.21	25.55	26.99	27.29	97.28	97.37	97.54
	$\pm$ 3.89	$\pm$ 4.25	$\pm$ 0.85	$\pm$ 3.55	$\pm$ 2.09	$\pm$ 0.67	$\pm$ 1.23	$\pm$ 1.70	$\pm$ 2.85	$\pm$ 0.77	$\pm$ 0.83	$\pm$ 1.59	$\pm$ 2.97	$\pm$ 3.55	$\pm$ 0.55	$\pm$ 0.32	$\pm$ 0.29	$\pm$ 0.16

Table 4: Task-solving outcomes and repair metrics on 68 tasks where all three NoMem runs fail.

Figure 4: Counts of failure modes.

Figure 5: FullTraj reductions in steps, output tokens, and action calls. Comparisons with NoMem and five memory systems use pooled means over 183 instances. Super and Claude denote Supermemory and Claude-Mem.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Relation type	Definition
Strong Related
reusable implementation pattern	The historical PR contains an implementation pattern, code structure, algorithm, or repair strategy that can be reused directly, or applied to the base task after local adaptation.
same code path	The historical PR involves the same critical code path as the base task, or one directly connected to it, so its changes help localize where and how to fix it.
introducing change	The historical PR introduced or substantially modified the behavior, assumption, or implementation that later gave rise to the base task’s problem, making it an important source for understanding the current issue.
same problem	The historical PR addresses essentially the same problem or requirement as the base task, possibly in a different version, component, or usage scenario.
same root cause	The surface symptoms of the historical PR and the base task differ, but the two share the same underlying technical cause, so the historical analysis or fix transfers directly to the current task.

Appendix

Table 5: Fine-grained relation types between a historical PR and a base task. Each relevance level has its own set of types, and an annotation must pick a type that belongs to the level it assigns.

Figure 6: Distribution of fine-grained relation types within each relevance level.

Figure 7: Executable historical task construction pipeline. (a) PR changes are split into test patch

T

and solution patch

S

, and the task inputs are frozen. (b) The original commit

c

is resolved and validated to construct a shared Base image. (c) The two patched source states independently invoke the shared resolver, and their plans and environment artifacts are frozen as Post A and Post B for offline restoration. (d) Stable outcomes are obtained from three fresh runs on each side.

Setting	Configuration
Agent and base model
Agent	Claude Code 2.1.170
Base model	DeepSeek-V4-Flash
Model routing	The primary route, all Claude Code model aliases, and the subagent route use DeepSeek-V4-Flash
Reasoning effort	High
Interaction budget	At most 200 turns per task

Appendix

Table 6: Shared agent and execution configuration. Every setting is identical across all evaluated systems and all six evaluation conditions.

Tool category	Availability	Enforcement
Bash	Available	The guard validates each request before execution and inspects the complete command string.
Read, Write, Edit, NotebookEdit, Glob, Grep	Available	The guard validates each request for protected paths, Git metadata, and alternate copies of the target project.
WebSearch, WebFetch	Disabled	Turned off in the CLI and additionally rejected by the guard if invoked, removing the direct route to external information.
EnterWorktree, ExitWorktree	Disabled	Turned off in the CLI and additionally rejected by the guard if invoked, keeping the agent inside the current checkout.

Appendix

Table 7: Tool availability under the information-isolation protocol. Tools that remain available are checked request by request rather than turned off, so normal local development is preserved.

Rule family	Restricted behavior	Preserved behavior
Target-project copies	Inspecting the target project under site-packages or dist-packages , package-metadata queries that locate such copies, and imports of the target project from outside the current checkout.	Genuinely separate third-party dependencies may still be inspected.
Shell and package acquisition	Remote-access and source-acquisition tools ( gh , ssh , scp , sftp , ftp , netcat , rsync ); pip download / wheel ; --target / -t installation; and URL- or VCS-based installation.	Ordinary installation of named third-party dependencies.
Git and metadata	Git history, any Git operation other than the permitted commands, direct access to .git , Git plumbing utilities, and Git-metadata environment variables.	git status and git diff on the current working tree or index, optionally against HEAD .
Benchmark-private paths	Access to the benchmark-private /tests and /solution paths.	Repository-owned source files and tests inside the current checkout.
Explicit network access	Explicit URLs in curl , wget , and recognized inline-Python calls ( requests , httpx , aiohttp , urllib , http.client , socket ) that target code hosts (GitHub and githubusercontent.com , GitLab, Bitbucket), Q&A sites (Stack Overflow and Stack Exchange), search engines (Google, Bing, DuckDuckGo, Brave Search), or SWE-bench.	Networking needed for ordinary dependency installation and task execution.
Guard bypass	Attempts to unset or override LD_PRELOAD or network-guard endpoint variables, or to bypass the execution guard by any other means.	Requests that satisfy the isolation checks proceed.

Appendix

Table 8: Main request-level checks enforced by the PreToolUse guard. Each rule family blocks a route to external answers or repository history while keeping the corresponding development operation available.

	Task performance				Solving cost
Memory system	Resolved	File Recall	F2P	P2P	Tokens		Steps
	(%)	(%)	(%)	(%)	In (M)	Out (K)
causal-memory ( JingxuanC, 2026 )	52.67	81.56	79.84	99.47	6.18	56.50	186.35
ChronoHybridMem ( Tin11Mn, 2026 )	52.00	83.34	81.29	99.42	6.62	59.31	186.68
Memory Bear ( Wen et al., 2026 )	52.00	79.90	80.50	99.46	6.62	55.95	180.39
TencentDB Agent Memory ( Tencent Cloud, 2026 )	51.33	82.23	80.76	99.45	5.86	55.94	184.69

Appendix

Table 9: Evaluation of 29 memory systems under the Relevant History scenario on the same 150 base tasks.

System	Implementation	Memory LLM	Embedding Model	Retrieval Configuration
A-Mem	ceffb860f0712 ; MCP 0.2.6 ( dce0ac62cba5 )	DeepSeek-V4-Flash	all-MiniLM-L6-v2 (SentenceTransformers, Chroma)	k=5 for standard and agentic search; k=10 for temporal search
Mem0	Plugin 0.2.9 ( 366945965df4 ); official API	Provider-managed	Provider-managed	Official API defaults
MemOS	MCP 1.1.2; official API	Provider-managed	Provider-managed	memory_limit_number=6 ; retry with relativity=0 when no result is returned
Supermemory	Plugin 0.0.13 ( 193290b5ccba ); official API	Provider-managed	Provider-managed	limit=5 ; searchMode=hybrid ; threshold=0.6
Claude-Mem	v12.1.1 ( 0b6997e3d03e )	DeepSeek-V4-Flash	all-MiniLM-L6-v2 (ONNX, 384-d)	search limit=20 ; 100 semantic candidates

Appendix

Table 10: Memory-system configurations used in our evaluation.

Figure 8: Shared trajectory. An initial change breaks default metadata preservation in the existing methods. The same five tests expose the regression and pass after the repair.

Figure 9: A-Mem with Claude Code. Claude composes a note from

e_{3}

–

e_{5}

, which A-Mem enriches and links. Retrieval combines semantic previews, selective reading, and optional link expansion.

Figure 10: Mem0 with Claude Code. Selected interaction evidence is distilled into persistent memories for scoped retrieval.

Figure 11: MemOS with Claude Code. MemReader extracts textual memories into MemCube for later semantic retrieval.

Figure 12: Supermemory with Claude Code. Documents yield linked atomic memories for project profiles and subsequent retrieval.

Figure 13: Claude-Mem with Claude Code. Observations and progress summaries support startup context and selective retrieval.

Memory Agent Benchmarks

Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.

See all 164 papers

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks
Sep 20, 2026Liyang Fan, Yingcheng Shi, Yongbin Li +7Benchmark Design Software Engineering Benchmarks
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · SUAT · Alibaba Token Hub, Alibaba Group +1
MemTrace: State-Consistent Memory for Long-Horizon Coding Agents
Oct 4, 2026Hongming Xu, Le Zhou, ZhongHe Jin +5Agent Memory Agent Memory Management
Shanghai Jiao Tong University · MemTensor (Shanghai) Technology Co., Ltd. · Theseus Lab
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Sep 21, 2026Soumil Rathi, Deshraj Yadav, Taranjeet SinghAgent Memory Persistent Memory for Language Models
Mem0

Sep 20, 2026J/K move · Enter open · S save

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

Liyang Fan, Yingcheng Shi, Yongbin Li, Chenghao Sun+6

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · SUAT · Alibaba Token Hub, Alibaba Group+1

Benchmark Design Software Engineering Benchmarks Memory-Augmented Agents Agent Memory Management

Open paper AskPDF arXiv

Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluations do not show whether those systems improve executable repository work. Repository benchmarks test code changes but do not isolate memory, while memory benchmarks score recall without measuring downstream coding outcomes. We introduce VibeMemBench, a benchmark for evaluating memory systems on 111 coding targets from 90 SWE-rebench V2 repositories and 3,634 history trajectories from the target repositories. The targets follow the SWE benchmark style and cover bug fixes, feature requests, interface changes, and configuration work. An agent edits each target codebase under a declared memory condition. Executable tests decide task resolution. Each target is retained only when injected history experience improves its executable outcome in a reference setting, so every target carries a prior experience whose usefulness is verified by execution in that setting. The frozen verified experience is then transferred to five held-out solvers. Direct injection raises observed task resolution on four of them by 1.1 to 4.5 percentage points while lowering agent steps on all five. Yet when four existing memory systems must construct and retrieve experience from the same history, eleven of twelve solver and system pairings fail to exceed the matched memory-off baseline. VibeMemBench exposes the gap between the useful experience that repository history holds and the experience existing memory systems deliver for repository coding tasks.

Abstract

Memory Agent Benchmarks

Explore similar work

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

MemTrace: State-Consistent Memory for Long-Horizon Coding Agents

DolphinBench: Mapping the Pareto Frontier of Agent Memory