Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

Jiaqi Liu; Zipeng Ling; Shi Qiu; Yanqing Liu; Siwei Han; Peng Xia; Haoqin Tu; Zeyu Zheng

cs.AIApr 1, 2026

Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

Authors: Jiaqi Liu, Zipeng Ling, Shi Qiu, Yanqing Liu, Siwei Han, Peng Xia, Haoqin Tu, Zeyu Zheng,

Organizations: UNC-Chapel Hill · University of Pennsylvania · University of California, Santa Cruz · University of California Berkeley ·

Abstract

AI agents increasingly operate over extended time horizons, yet their ability to retain, organize, and recall multimodal experiences remains a critical bottleneck. Building effective lifelong memory requires navigating a vast design space spanning architecture, retrieval strategies, prompt engineering, and data pipelines; this space is too large and interconnected for manual exploration or traditional AutoML to explore effectively. We deploy an autonomous research pipeline to discover Omni-SimpleMem, a unified multimodal memory framework for lifelong AI agents. Starting from a naïve baseline (F1=0.117 on LoCoMo), the pipeline autonomously executes ${\sim}50$ experiments across two benchmarks, diagnosing failure modes, proposing architectural modifications, and repairing data pipeline bugs, all without human intervention in the inner loop. The resulting system achieves state-of-the-art on both benchmarks, improving F1 by +411% on LoCoMo (0.117 $\to$ 0.598) and +214% on Mem-Gallery (0.254 $\to$ 0.797) relative to the initial configurations. Critically, the most impactful discoveries are not hyperparameter adjustments: bug fixes (+175%), architectural changes (+44%), and prompt engineering (+188% on specific categories) each individually exceed the cumulative contribution of all hyperparameter tuning, demonstrating capabilities fundamentally beyond the reach of traditional AutoML. We provide a taxonomy of six discovery types and identify four properties that make multimodal memory particularly suited for autoresearch, offering guidance for applying autonomous research pipelines to other AI system domains. Code is available at this https://github.com/aiming-lab/SimpleMem.

Figures & tables

Figure 1: Overview of the Discovery Process of Omni-SimpleMem . (a) Discovered architecture: multimodal inputs are filtered for novelty, compressed into MAUs, and retrieved via hybrid dense-sparse-graph search with pyramid expansion. (b) Autonomous optimization trajectory on Mem-Gallery: 39 experiments improve F1 from 0.254 to 0.793 (+214%).

Figure 2: Omni-SimpleMem architecture overview. Left: Selective ingestion filters multimodal inputs (text, image, audio, video) via modality-specific novelty detectors and creates MAUs with LLM-generated summaries and embeddings. Center: MAUs are stored in hot storage (summaries, embeddings, metadata) and cold storage (raw content), with entity extraction building a knowledge graph with typed entities and relations. Right: Retrieval combines dense (FAISS), sparse (BM25), and graph ( $h$ -hop) search via set-union merging, then progressively expands results through a pyramid mechanism (summaries $\to$ full text $\to$ raw content) under a token budget $B$ .

Panel of Figure 3 — Figure 3: Optimization trajectories on LoCoMo (top, 9 iterations) and Mem-Gallery (bottom, 39 experiments across 7 phases). Solid lines denote accepted iterations; failed/reverted experiments are marked with $\times$ . The dashed red line indicates the previous SOTA. Key discoveries at each stage are annotated.

		LoCoMo						Mem-Gallery
Backbone	Method	MH	SH	Tmp	Open	Adv	All	F1	EM	B	B-1	B-2
GPT-4o	MemVerse	0.260	0.157	0.196	0.192	0.944	0.365	0.505	0.330	0.270	0.440	0.355
	Mem0	0.309	0.156	0.217	0.295	0.857	0.397	0.298	0.192	0.182	0.268	0.224
	Claude-Mem	0.294	0.153	0.167	0.243	0.915	0.383	0.210	0.148	0.148	0.194	0.170
	A-MEM	0.295	0.174	0.200	0.266	0.898	0.394	0.370	0.252	0.240	0.332	0.285
	MemGPT	0.305	0.188	0.246	0.305	0.843	0.404	0.435	0.298	0.275	0.390	0.335

Table 1: Comparison across five LLM backbones on LoCoMo (left) and Mem-Gallery (right). LoCoMo columns: MH = multi-hop, SH = single-hop, Tmp = temporal, Open = open-domain, Adv = adversarial, All = overall F1. Mem-Gallery columns: F1, EM = exact match, B/B-1/B-2 = BLEU/BLEU-1/BLEU-2. Best baseline (excl. Omni-SimpleMem ) is underlined ; best overall is bold .

Component Removed	$\Delta$ F1	Rel.
w/o Pyramid Expansion	$-$ 10.2	$-$ 17%
w/o BM25 Hybrid	$-$ 8.5	$-$ 14%
w/o LLM Summarization	$-$ 7.3	$-$ 12%
Reduced top- $k$ (5 vs 20)	$-$ 4.2	$-$ 7%
w/o Metadata Context	$-$ 1.4	$-$ 2%

Table 2: Component ablation on LoCoMo (mean

\Delta

\times

100 across 4 backbones).

Figure 4: Throughput vs. F1. Omni-SimpleMem with 8 workers achieves 3.5

\times

higher throughput.

Ret./Gen. in milliseconds.
Method	q/s	Ret.	Gen.
SimpleMem	1.68	45	550
MemVerse	1.48	70	596
Mem0	1.46	18	665
Omni-SimpleMem (w=1)	1.05	118	846
Omni-SimpleMem (w=8)	5.81	461	821

Table 3: Latency breakdown.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Parameter	Value
Visual Ingestion	CLIP threshold $\tau_{\mathrm{high}}$	0.9
	CLIP threshold $\tau_{\mathrm{low}}$	0.7
	Frame buffer size	3
Audio Ingestion	VAD threshold	0.5
Text Ingestion	Jaccard threshold $\tau_{\mathrm{dup}}$	0.8
Entity Resolution	Merge threshold $\tau_{\mathrm{res}}$	0.85

Appendix

Table 4: Hyperparameter specification of Omni-SimpleMem .

Parameter	LoCoMo	Mem-Gallery
Embedding	text-emb-3-large	MiniLM-L6-v2
Embedding dim	3072	384
top- $k$	20–30	20–40 †
Token budget	6,000	2K–8K †
Per-doc memory	No	Yes
System prompt	Yes	Yes

Appendix

Table 5: Per-benchmark configurations discovered by the autonomous pipeline ( † varies per question category).

Iter	Key Discovery	F1	$\Delta$	Type
0	Naïve baseline	0.117	—	—
1	JSON response_format missing	0.322	+175%	Bug fix
2	BM25 hybrid	0.464	+44%	Architecture
3	Anti-hallucination prompting	0.516	+11%	Prompt
4b	Evaluation format alignment	0.543	+5%	Format
5	MAU timestamp correction	0.580	+7%	Data repair

Appendix

Table 6: Optimization trajectory on LoCoMo. The pipeline autonomously discovers 7 successive improvements and correctly reverts 2 failed experiments.

Phase	Focus	F1 Range	$\Delta$	Key Discovery
1	Environment setup	0.254 $\to$ 0.353	+39%	LLM upgrade + local embedding
2	Architecture	0.353 $\to$ 0.690	+96%	Full-text retrieval + image BM25
3	Fine-tuning	0.690 $\to$ 0.717	+4%	Constraint position sensitivity
4	Scale validation	0.717 $\to$ 0.726	+1%	Data completeness $>$ algorithms
5	Exact citation	0.726 $\to$ 0.771	+6%	BM25 tokenization fix (+0.018)
6	Visual reasoning	0.771 $\to$ 0.789	+2%	Image catalog + context

Appendix

Table 7: Optimization trajectory on Mem-Gallery (39 experiments, 7 phases). Each phase represents a qualitative shift in the pipeline’s optimization strategy.

Iter	Cat1 (MH)	Cat2 (T)	Cat3 (O)	Cat4 (SH)	Cat5 (A)
Baseline	0.093	0.047	0.116	0.219	0.000
Iter 1	0.206	0.117	0.229	0.381	0.447
Iter 2	0.292	0.117	0.282	0.418	1.000
Iter 3	0.351	0.335	0.346	0.444	1.000
Iter 4b	0.389	0.309	0.382	0.466	1.000
Iter 5	0.398	0.487	0.388	0.445	1.000

Appendix

Table 8: LoCoMo per-category F1 across iterations (conv-26, 199 QA pairs).

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 191 papers

EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents
May 13, 2026Jiaqi Liu, Xinyu Ye, Peng Xia +4LLM Agent Memory LLM Memory
UNC-Chapel Hill · UC Berkeley · UCSC
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Agent Evaluation Multimodal Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3
Towards Autonomous Memory Agents
Feb 25, 2026Xinle Wu, Rui Zhang, Mustafa Anis Hussain +1Memory-Augmented Language Model Agents Memory Management for LLM Agents
National University of Singapore, Singapore

May 13, 2026J/K move · Enter open · S save

EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents

Jiaqi Liu, Xinyu Ye, Peng Xia, Zeyu Zheng+3

UNC-Chapel Hill · UC Berkeley · UCSC

LLM Agent Memory LLM Memory Long-Term Conversational Memory Agent Memory Management Self-Evolving Agents

Open paper AskPDF arXiv

Long-term memory is essential for LLM agents that operate across multiple sessions, yet existing memory systems treat retrieval infrastructure as fixed: stored content evolves while scoring functions, fusion strategies, and answer-generation policies remain frozen at deployment. We argue that truly adaptive memory requires co-evolution at two levels: the stored knowledge and the retrieval mechanism that queries it. We present EvolveMem, a self-evolving memory architecture that exposes its full retrieval configuration as a structured action space optimized by an LLM-powered diagnosis module. In each evolution round, the module reads per-question failure logs, identifies root causes, and proposes targeted configuration adjustments; a guarded meta-analyzer applies them with automatic revert-on-regression and explore-on-stagnation safeguards. This closed-loop self-evolution realizes an AutoResearch process: the system autonomously conducts iterative research cycles on its own architecture, replacing manual configuration tuning. Starting from a minimal baseline, the process converges autonomously, discovering effective retrieval strategies including entirely new configuration dimensions not present in the original action space. On LoCoMo, EvolveMem outperforms the strongest baseline by 25.7% relative and achieves a 78.0% relative improvement over the minimal baseline. On MemBench, EvolveMem exceeds the strongest baseline by 18.9% relative. Evolved configurations transfer across benchmarks with positive rather than catastrophic transfer, indicating that the self-evolution process captures universal retrieval principles rather than benchmark-specific heuristics. Code is available at https://github.com/aiming-lab/SimpleMem.

Abstract

Agent Memory

Explore similar work

EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents

WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

Towards Autonomous Memory Agents