MemT-1.0: Towards Memory-Native Evolution

Zhiyu Li; Zeyu Zhang; Linyun Liu; Yiwei Bu; Chenyang Xi; Xichong Zhang; Yi Yu; Yu Yu

cs.CLOct 19, 2026

MemT-1.0: Towards Memory-Native Evolution

Authors: Zhiyu Li, Zeyu Zhang, Linyun Liu, Yiwei Bu, Chenyang Xi, Xichong Zhang, Yi Yu, Yu Yu, Zhichen Liu, Yuxiao Yi, Chunyu Li, Haoran Liu, Zhenkai Ding, Le Zhou, Juncheng Zhang, Boyu Chen, Zehao Lin, Guorui Li, Renyu Fu, Ji Li, Youyang Xia, Jiajun Shen, Harvey Li, Jingsong Zhou, Helan Hu, Tong Shen, Tianxiao Xu, Chenye Han, Weijie Li, Zhongquan Zhou, Hongming Xu, Yuekeng Li, Erenkai Zhu, Peng Yu, Wenhao Li, Jingyi Kang, Xixuan Hao, Xuzhou Zhu, Can Chen, Jiawei Liu, Chen Jiang, Tao Jiang, Fangwei Xiang, Zepeng Yao, Kai Chen, Guang Yang, Mingfei Lu, Chunhui Li, Zhengxuan Jiang, Bo Tang, Shiyu Zhan, Yihang Sun, Jie Chu, Wenjun Xiong, Zhenxiang Hua, Chen Lin, Detong Su, Dongyu Deng, Qiuping Ding, Guofeng Ding, Si Chen, Guohua Wen, Xu Chen, Zhi-Qin John Xu, Ying-Cong Chen, Ningyu Zhang, Haofen Wang, Muning Wen, Haotian Chen, Wentao Zhang, Xuanhe Zhou, Rita Zhang, Junchi Yan, Feiyu Xiong

Abstract

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately $1.86\times$ end-to-end inference speedup and a $66\times$ reduction in persistent storage relative to full context KV cache. By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.

Figures & tables

Figure 3: Inference efficiency and storage overhead of MemT-1.0-Nano.

Figure 4: Architecture of MemT-1.0. MemT-1.0 equips causal language models with native memory that evolves across interaction steps. Each selected Transformer block is augmented with a native memory block that performs parallel original and memory attention. The memory mechanism comprises dynamic local memory and trainable hyper memory, together with native memory storage and utilization procedures for updating and retrieving persistent information.

Figure 5: Scaling native memory capacity. MoS expands each native memory block into multiple memory experts and employs separate dense storage and utilization routing over shared expert addresses.

Window size (complete steps)	Peak memory (GiB)	Memory reduction	F+B (ms)	Time increase
2048 tokens (1 step)	$47.83\rightarrow 6.03$	87.4%	$462.7\rightarrow 493.5$	6.6%
4096 tokens (2 steps)	$47.83\rightarrow 9.49$	80.1%	$463.0\rightarrow 470.6$	1.6%

Table 1: Memory-time tradeoff for one MoS memory block on a single GPU at a 32K-token pack length. Arrows denote whole-pack to windowed execution. The values are medians.

Figure 6: Data and training scaling of MemT-1.0-Nano. Memory performance over two consecutive post-training epochs. Blue circles show the average score on the left axis, and red squares show LoCoMo accuracy on the right axis.

Figure 7: Memory capacity scaling during post-training after 400 CPT steps. We compare MoS configurations with 2 and 4 memory experts at matched post-training steps, reporting the average score and the LoCoMo score.

Figure 8: Left: End-to-end latency includes history preparation and generation of a 128-token response. Right: Query TTFT measures the time from query execution to the first generated token, excluding history preparation.

Figure 9: Results of storage overhead. Per-trajectory persistent storage on BEAM histories of increasing length.

Figure 10: Impact of online low-rank decomposition on memory performance and storage.

Figure 11: Experience-guided agentic framework with MemT-1.0-Nano. Retrieved experience is stored in textual context or native memory to guide subsequent actions.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	MemT-1.0-Nano	MemT-1.0-Flash
Backbone	Qwen3.6-35B-A3B	DeepSeek-V4-Flash, after backbone preparation
Hidden width / decoder layers	2048 / 40	4096 / 43
Memory layers	$3,7,\ldots,39$ (10 layers)	$4,6,\ldots,42$ (20 layers)
Memory-layer type	Full attention	Compressed sparse attention
Memory $H_{q}/H_{kv}$	16 / 16	64 / 64
Memory $d_{k}/d_{v}$	256 / 256	512 / 512

Appendix

Table 6: Backbone and native-memory configuration. State size counts the persistent memory matrices for one trajectory, excluding model weights and current-step KV caches.

Setting	MemT-1.0-Nano	MemT-1.0-Flash
TP / PP / CP / EP	1 / 1 / 1 / 8	1 / 4 / 2 / 8
Logical DP / accumulation steps	64 / 1	12 / 16
Global batch (packs)	64	192
Maximum pack / storage step	65536 / 2048 tokens	65536 / 2048 tokens
Post-training target	Online backbone, Top-64 KL	Online backbone, Top-64 KL
Temperature / KL / CE weight	1 / 1 / 0	1 / 1 / 0

Appendix

Table 7: Execution and distillation settings for the evaluated checkpoints. TP, PP, CP, and EP denote tensor, pipeline, context, and expert parallelism. DP counts logical data-parallel replicas. EP partitions backbone experts within that topology.

Figure 12: An analytical single-step erase example with

D=I

and no write term. Both panels use the same input

y=e

and orthonormal

\{k,e\}

basis.

Figure 13: Construction of experience-conditioned training trajectories. Retrieved experience is added at selected breakpoints in unsuccessful trajectories, and execution resumes from the same state. Continuations that pass the task tests are retained as training samples comprising the trajectory prefix, experience, and successful continuation.

Statistic	Train	Validation	Test (Verified)
Number of repositories	537	176	12
Number of tasks	2,396	266	500

Appendix

Table 9: Task and repository counts for the training and validation splits and SWE-bench Verified.

Statistic	Value	Calculation or description
Original trajectories	18,820	Target rollouts generated by Qwen3.6-35B-A3B
Unsolved original trajectories	14,649 (77.8%)	$14{,}649/18{,}820$
Breakpoint–experience variants	199,500	7,901 usable trajectories produced 19,950 retained breakpoints
Successful experience	45,481 (22.8%)	$45{,}481/199{,}500$

Appendix

Table 10: Training data construction statistics and experience intervention outcomes.

Panel of Figure 14 — Figure 14: Training loss and validation NLL under CE and KD objectives. Within each panel, the left and right columns show training loss and validation NLL, respectively. Solid blue curves denote Max-2, and dashed orange curves denote Max-3.

	Validation NLL		Avg. Success (%)
Configuration	CE	KD	CE	KD
First-Step-Max2	0.3452	0.4304	12.5	52.0
First-Step-Max3	0.3373	0.4267	34.1	52.0
Full-Suffix-Max2	0.1795	0.2005	37.5	54.3
Full-Suffix-Max3	0.1771	0.1996	22.7	52.7

Appendix

Table 11: Validation NLL and preliminary agent performance under CE and KD. Average success rates are measured on the 100-task development subset of SWE-bench Verified with three runs per task. Validation NLL uses hard-label targets; lower is better.

Configuration	Avg. Success	pass@3	pass^3
First-Step-Max2	52.00	65.00	38.00
First-Step-Max3	52.00	61.00	42.00
Full-Suffix-Max2	54.33	66.00	42.00
Full-Suffix-Max3	52.67	66.00	36.00

Appendix

Table 12: Metis KD configurations on the 100-task development subset of SWE-bench Verified under Before Execution. All values are percentages; bold values indicate the highest scores in each column.

Figure 15: Agentic coding performance on SWE-bench Verified and SWE-bench Multilingual under Before Execution (top) and Harness-Triggered (bottom). Each task is evaluated in three runs.

Method	Avg. Success	pass@3	pass^3
Before Execution
No Experience	60.00	73.67	45.33
Textual Experience: Fixed Rank-1	64.22	72.67	56.33
Textual Experience: Fixed Rank-2	61.89	72.67	51.33
Textual Experience: Fixed Rank-3	63.78	74.00	54.33
Textual Experience: Per-Task Best	70.56	78.67	61.67

Appendix

Table 13: Textual Experience selection on SWE-bench Multilingual (300 tasks). Fixed Rank-

k

uses the single experience at rank

k

; Per-Task Best selects among three candidates using evaluation outcomes. All values are percentages.

Configuration	Value
Hidden dimension	2048
Memory experts	4
Memory heads per expert / head dimension	16 / 256
Router projection rank	128
Decay-gate / output-gate projection rank	256 / 256
Parameter and projection / memory-state precision	BF16 / FP32

Appendix

Table 14: Configuration of the MoS memory-block experiment.

Pack length (tokens)	Window budget (tokens)	Whole-pack memory (GiB)	Windowed memory (GiB)	Whole-pack F+B (ms)	Windowed F+B (ms)
8192	2048	12.37	4.85	102.1	122.9
8192	4096	12.37	8.36	102.1	117.4
16384	2048	24.19	5.23	211.6	246.1
16384	4096	24.19	8.77	211.6	235.1
32768	2048	47.83	6.03	462.7	493.5
32768	4096	47.83	9.49	463.0	470.6

Appendix

Table 15: Full memory and time results for the MoS memory block. Each row is an independently measured whole-pack/window pair; values are medians.

Pack length (tokens)	Window budget (tokens)	Whole-pack median [Q25, Q75]	Windowed median [Q25, Q75]	Standard deviation (whole / windowed)
8192	2048	102.10 [102.08, 102.12]	122.87 [122.86, 122.88]	0.03 / 0.02
8192	4096	102.11 [102.10, 102.12]	117.42 [117.41, 117.45]	0.05 / 0.03
16384	2048	211.55 [211.43, 212.02]	246.11 [246.02, 246.42]	37.19 / 0.26
16384	4096	211.61 [211.45, 211.87]	235.07 [234.98, 235.39]	0.30 / 0.27
32768	2048	462.71 [462.37, 463.87]	493.48 [492.91, 494.38]	22.22 / 0.94
32768	4096	462.96 [462.42, 463.68]	470.58 [469.84, 471.46]	0.76 / 38.01

Appendix

Table 16: F+B timing variability in milliseconds. Intervals are the 25th and 75th percentiles; standard deviations use all measured samples.

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 291 papers

Metis: Memory Foundation Model
Jul 29, 2026Zeyu Zhang, Ziliang Guo, Yihang Sun +14Agent Memory Memory-Augmented Language Models
MemTensor (Shanghai) Technology Co., Ltd. · Shanghai Jiao Tong University · National University of Singapore +2
$δ$ -mem: Efficient Online Memory for Large Language Models
May 12, 2026Jingdi Lei, Di Zhang, Junxian Li +7Memory-Augmented Language Models Memorization in Language Models
Nanyang Technological University · Mind Lab · Fudan University +3
Safin-1: Safety from Within through Memory-Native State Evolution
Aug 31, 2026Ming Zhang, Kaisen Yang, Shu Yu +15Memory-Augmented Language Models LLM Safety Alignment
Shanghai AI Laboratory

Jul 29, 2026J/K move · Enter open · S save

Metis: Memory Foundation Model

Zeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang+13

MemTensor (Shanghai) Technology Co., Ltd. · Shanghai Jiao Tong University · National University of Singapore+2

Agent Memory Memory-Augmented Language Models Persistent Memory for Language Models

Open paper AskPDF arXiv

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.

Abstract

Memory-Augmented Language Models

Explore similar work

Metis: Memory Foundation Model

δδδ-mem: Efficient Online Memory for Large Language Models

Safin-1: Safety from Within through Memory-Native State Evolution

$δ$ -mem: Efficient Online Memory for Large Language Models