ProTTT: Learning to Learn Semantic User Memory with Test-Time Training

Sejun Park; Hyoungjo Bhang; Hyein Jeong; Yohan Jo

cs.AISep 26, 2026

ProTTT: Learning to Learn Semantic User Memory with Test-Time Training

Authors: Sejun Park, Hyoungjo Bhang, Hyein Jeong, Yohan Jo

Organizations: Graduate School of Data Science, Seoul National University

Abstract

Personalization requires language models to capture user-specific knowledge from a growing user history. Existing context-based approaches incur increasing inference costs as user history accumulates and rely on separate retrieval or summarization stages, while parametric-based approaches often require reconstructing user representations when new user data is added. We introduce ProTTT, a profile-supervised meta-learning framework for learning semantic user memory. The memory construction starts from a shared initialization and is updated for each user through test-time training on user history, allowing it to evolve continuously as the history grows. However, since test-time training alone does not explicitly encourage the memory to capture semantic user knowledge necessary for personalization, we learn this shared initialization using textual user profiles as supervision, so that test-time training on user history captures semantic knowledge more effectively. ProTTT consistently outperforms both full history ICL and all parametric baselines across diverse benchmarks, while substantially reducing inference cost by compressing user history into a lightweight parameterized memory. Our analysis also shows that profile supervision is a reliable objective for learning semantic user knowledge and that the resulting memory can track and retain evolving user preferences, while remaining robust across different history sizes. Overall, we demonstrate the effectiveness of test-time training for personalization and establish ProTTT as a baseline for continuously evolving user memory.

Figures & tables

Figure 1: Overview of ProTTT . (a) During training, a shared initialization

\phi_{s}

is adapted to each user through next-token prediction on the user history, producing user-specific memory

\phi_{u}

. The adapted memory is then evaluated on a semantic profile task, whose loss is backpropagated through the adaptation process to update

\phi_{s}

. (b) At inference time, the learned initialization is adapted using only the target user’s history and then frozen for personalized generation.

Figure 2: User memory architecture. A low-rank adapter reads the hidden state and adds its output to the residual stream.

	Personalization					Efficiency on CMV
Method	CMV AUC $\uparrow$	LaMP-3 MAE $\downarrow$	LaMP-3 RMSE $\downarrow$	PMem T6 $\uparrow$	PMem T7 $\uparrow$	History Proc. (s/user) $\downarrow$	Prompt (tokens/query) $\downarrow$	Latency (ms/query) $\downarrow$
Base	.466	2.103	2.602	.273	.088	–	748	15.3
In-Context Baselines
Full History	.475	.892	1.569	.273	.140	–	18.9K	273.5
RAG (Recent-5)	.456	1.800	2.434	.418	.140	–	2.2K	20.0
RAG (BM25)	.436	1.763	2.405	.491	.246	–	4.3K	39.2

Table 1: Personalization performance and efficiency. w/o user denotes the corresponding method without target-user information. For ProTTT , this denotes the shared initialization before constructing memory from the target user’s history;

\delta

-Mem uses an empty memory state, whereas P2P uses an empty user profile and history. Bold indicates the best personalization result in each column. † PAG profiles are generated offline using GPT-5-mini through an external API; profile-construction latency is therefore excluded from the H200 measurements.

Figure 3: Effect of the meta-learning objective on TTT. The figure reports downstream task performance changes before and after user memory construction for each meta-learning objective. Bold values show the signed change, and smaller values show the corresponding raw scores. Positive values indicate improvement.

Figure 4: Continual acquisition and retention of semantic user memory. (a) Mean token log-probability of each gold user profile as profile-associated histories are sequentially incorporated. (b) Change in the log-probability of Profile A as additional user histories are added, comparing ProTTT with full history ICL. Smaller degradation indicates better retention of previously acquired user information.

Figure 5: Personalization and general capability as user history accumulates. The plots show the scores after constructing user memory from CMV user histories. Curves compare ProTTT under different memory readout scales against full history ICL as the number of history items

k

increases. Horizontal lines denote the base model and the shared initialization before TTT. Full history ICL is reported only up to

k=100

because of the context limit.

	(a) General capability and generation quality					(b) Claims in generated user profiles
Method	MMLU Acc. $\uparrow$	Non- term. $\downarrow$	Seq.- rep.-2 $\downarrow$	Distinct-2 $\uparrow$	Self- BLEU $\downarrow$	Claims /gen.	Sup.	Gen.	Unsup.	Contr.
Base	.474	.000	.057	.605	.359	6.22	1.67	2.27	2.24	.04
User-specific LoRA	.397	.740	.417	.246	.463	5.11	1.24	1.49	2.13	.24
P2P	.390	.006	.064	.671	.255	3.29	.91	.13	1.89	.36
$\delta$ -Mem	.474	.003	.123	.563	.309	9.87	2.80	2.09	4.53	.44
ProTTT , $\alpha=.75$	.457	.009	.092	.605	.277	8.04	5.69	1.02	1.22	.11

Table 2: General capability, generation quality, and user-information recall. (a) Reports 5-shot MMLU accuracy after processing

k=100

history items and generation metrics on

40

WildChat prompts. (b) Reports the mean number of claims per generated user profile. Sup., Gen., Unsup., and Contr. denote supported, generic, unsupported, and contradicted claims, respectively.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Source	User history	Meta-learning target	# examples
ALPSBench	User–assistant session dialogues	Human-verified user-memory annotations	3,033
CMV	User-authored posts and comments	Generated semantic user profiles	1,321
Total			4,354

Appendix

Table 3: Composition of the meta-learning data.

Hyperparameter	Value
Backbone	Qwen3-0.6B
Memory rank	18
History chunk size	2,048 tokens
TTT optimizer	SGD
TTT learning rate	0.1
TTT gradient clipping	1.0

Appendix

Table 4: Hyperparameters used to train the main model.

In-Context Baselines
Method	Track Evolution (139)	Shared Facts (129)	Update Reasons (99)	New Ideas (93)	Mentioned Facts (17)	T6 Recommend (55)	T7 Generalize (57)	All (589)
Base	.273	.287	.444	.183	.353	.273	.088	.275
Full History	.230	.287	.313	.172	.294	.273	.140	.244
RAG (Recent-5)	.295	.318	.525	.151	.235	.418	.140	.311
RAG (BM25)	.259	.326	.455	.140	.118	.491	.246	.304
PAG	.201	.279	.394	.151	.235	.309	.123	.246

Appendix

Table 5: PersonaMem accuracy across all released question categories. Before and after denote performance without and with target-user information, respectively. For ProTTT , the two after-TTT rows differ only in the memory readout scale. Bold indicates the highest point estimate in each column; ties are all bolded. The number of questions is shown in parentheses.

Method	User-specific parameters	Training settings
User-Specific LoRA	Rank $16$ , scale $32$ , attention and feed-forward projections	AdamW, lr $10^{-3}$ , one epoch
P2P	Hypernetwork-generated rank- $8$ LoRA on query and value projections	AdamW, lr $2\times 10^{-5}$ , profile-supervised training
$\delta$ -Mem	Rank- $8$ token-wise memory on query and output projections	AdamW, lr $2\times 10^{-4}$ , one QASPER epoch

Appendix

Table 6: Configurations of the parametric baselines.

Method	Prompt Tokens mean / max	KV Cache (MiB) mean / max	Peak Memory (MiB)	Latency (ms) median / mean	GFLOPs per query
Base	748 / 1,583	82 / 173	1,383	15.3 / 17.3	732
Full History	18,919 / 29,544	2,069 / 3,231	5,166	273.5 / 218.7	63.6K
RAG (Recent-5)	2,223 / 7,023	243 / 768	2,119	20.0 / 23.7	2.7K
RAG (BM25)	4,255 / 14,189	465 / 1,552	3,088	39.2 / 49.2	6.9K
PAG	1,066 / 1,931	117 / 211	1,430	15.9 / 18.0	1.1K
PAG + Full History	19,215 / 29,927	2,102 / 3,273	5,217	278.1 / 222.0	65.3K

Appendix

Table 7: Query-time cost on CMV. Prompt length and KV-cache size are reported as mean/max, and latency as median/mean. P2P latency includes query-specific adapter generation. For P2P, the second peak-memory value corresponds to adapter generation and is measured with nvidia-smi because it runs in a separate process.

Method	User Representation	Construction	Time (s) mean / max	Peak Memory (MiB)
Full History	Raw history (17.8K tokens)	–	–	–
RAG (Recent-5)	Five most recent entries	–	–	–
RAG (BM25)	Raw history	–	–	–
PAG	268-token profile	Generate profile (GPT-5-mini)	n/a	n/a
PAG + Full History	Raw history + 268-token profile	Generate profile (GPT-5-mini)	n/a	n/a
User-Specific LoRA	10.09M parameters	One NTP epoch	11.74 / 13.05	17,450

Appendix

Table 8: User representations on CMV and the cost of constructing them. Time is reported as mean/max per user. PAG profiles are generated through the external GPT-5-mini API, so their construction time and GPU memory are not included. P2P adapter generation is performed for each query and is included in the query latency reported in Table 7 ; its profile-generation peak memory is measured using nvidia-smi .

Method	History Proc. (s/user) $\downarrow$	Prompt Tokens /query $\downarrow$	Latency (ms/query) $\downarrow$
LaMP-3
Base	–	170	14.2
Full History	–	12.5K	90.3
RAG (Recent-5)	–	750	14.9
RAG (BM25)	–	1.17K	18.6
PAG	n/a	404	14.9

Appendix

Table 9: Efficiency on LaMP-3 and PersonaMem. History processing is the mean one-time cost per user for LaMP-3 or memory group for PersonaMem. PAG profiles are generated through the external GPT-5-mini API, so their construction time is not reported. For P2P, history processing includes profile generation only; query-specific adapter generation is included in query latency. Prompt length is averaged over queries, and latency is the median.

Objective	Training prompt	Target
Profile prediction	Here is a new post from this user: [POST]. What do you remember about this user that would help persuade them on this post? List each fact you remember, one per line.	Bullet-point profile written by GPT-5-mini from the post and the user’s history.
Persuasion judgment	Here is a new post from this user: [POST]. And here is a response to it: [RESPONSE]. Would this response change this user’s view? Answer with just “yes” or “no”.	yes for a persuasive reply, no otherwise.
Response selection	Here is a new post from this user: [POST]. Here are four responses to it: [OPTIONS]. Which response would change this user’s view? Answer with just the number.	Number of the persuasive reply.
Response generation	Here is a new post from this user: [POST]. Write the response that would most likely change this user’s view.	One persuasive reply to the post.

Appendix

Table 10: Prompts and training targets of the four meta-learning tasks. [POST] is the user’s new post, [RESPONSE] is one reply to it, and [OPTIONS] lists the four candidate replies, numbered (1)–(4).

	Personalization					Efficiency on CMV
Method	CMV AUC $\uparrow$	LaMP-3 MAE $\downarrow$	LaMP-3 RMSE $\downarrow$	PMem T6 $\uparrow$	PMem T7 $\uparrow$	History Proc. (s/user) $\downarrow$	Prompt (tokens/query) $\downarrow$	Latency (ms/query) $\downarrow$
Base	.598	1.889	2.347	.255	.088	–	763	8.5
In-Context Baselines
Full History	.515	1.745	2.146	.255	.088	–	18.8K	220.7
RAG (Recent-5)	.549	1.711	2.107	.255	.088	–	2.2K	14.6
RAG (BM25)	.555	1.709	2.066	.255	.088	–	4.3K	32.6

Appendix

Table 11: Personalization performance and efficiency with Llama-3.2-1B-Instruct. w/o user denotes the shared initialization before constructing memory from the target user’s history. Efficiency is measured on CMV using a single NVIDIA H200 NVL. History processing is a one-time per-user cost, while prompt length and latency are measured per query. † PAG profiles are generated offline using GPT-5-mini through an external API; profile-construction latency is therefore excluded from the H200 measurements. Bold indicates the best personalization result for each metric.

	(a) General capability and generation quality					(b) Claims in generated user profiles
Method	MMLU Acc. $\uparrow$	Non- term. $\downarrow$	Seq.- rep.-2 $\downarrow$	Distinct-2 $\uparrow$	Self- BLEU $\downarrow$	Claims /gen.	Sup.	Gen.	Unsup.	Contr.
Base	.455	.000	.071	.609	.304	8.88	3.93	2.13	2.62	.20
ProTTT , $\alpha=.5$	.405	.014	.068	.642	.263	14.85	6.85	2.38	5.27	.35
ProTTT , $\alpha=1$	.259	.059	.143	.487	.289	11.98	5.15	.67	5.77	.40

Appendix

Table 12: General capability, generation quality, and user-information recall with Llama-3.2-1B-Instruct. (a) Reports 5-shot MMLU accuracy after processing

100

history items and generation metrics on

40

WildChat prompts. (b) Reports the mean number of claims per generated user profile. Sup., Gen., Unsup., and Contr. denote supported, generic, unsupported, and contradicted claims, respectively.

Figure 6: Continual acquisition and retention with Llama-3.2-1B-Instruct. (a) Change in the mean token log-probability of each gold user profile as the corresponding histories are sequentially incorporated. (b) Change in the mean token log-probability of Profile A as seven later histories are incorporated, comparing ProTTT with full history ICL. Smaller degradation indicates better retention of previously acquired user knowledge.

Figure 7: Effect of the memory readout scale across backbone models. Panels (a)–(c) show personalization performance on CMV, LaMP-3, and PersonaMem Types 6 and 7. Panel (d) shows 5-shot MMLU accuracy. Panels (e)–(h) show generation quality on WildChat. Solid lines represent ProTTT after user memory has been constructed; dashed lines represent the corresponding base model without user memory. Memory construction is identical across readout scales.

Personalized Language Models

Personalized language models adapt responses to individual users’ preferences, histories, writing styles, and changing needs.

See all 88 papers

Latent Personal Memory: Represent personal memory as dynamic soft prompts
Jun 18, 2026Debrup Das, Avinash Amballa, Yashas Malur Saidutta +3LLM Inference Efficiency Memory-Efficient Fine-Tuning
University of Massachusetts Amherst · Samsung Research America
Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents
May 25, 2026Yeonjun In, Wonjoong Kim, Sangwu Park +2Memorization in Language Models Persistent Memory for Language Models
KAIST
Less Is Personal: Learning Minimal Sufficient User Profiles for Personalized Language Models
Sep 8, 2026Minghang Liu, Qiang Qiu, Yuanzhuo Wang +2LLM Personalization Memory-Augmented Language Models
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences, Beijing, China

Jun 18, 2026J/K move · Enter open · S save

Latent Personal Memory: Represent personal memory as dynamic soft prompts

Debrup Das, Avinash Amballa, Yashas Malur Saidutta, Vijay Srinivasan+2

University of Massachusetts Amherst · Samsung Research America

LLM Inference Efficiency Memory-Efficient Fine-Tuning Memory-Augmented Language Models Persistent Memory for Language Models Prompt Tuning Personalized Language Models

Open paper AskPDF arXiv

Personalizing large language models (LLMs) requires encoding long-term, user-specific behavioral patterns in a way that is computationally efficient, scalable, and compatible with a frozen base model. We present Latent Personal Memory (LPM), a scalable framework that represents user-specific history as a compact, persistent matrix of N latent slots, that are interpretable. A shared cross-attention projection network maps these slots into dynamic, input-conditioned soft prompts that are prepended to the input of a frozen LLM. We evaluate LPM on PersonaMem v1 and LoCOMO benchmarks across Qwen3-1.7B, 4B, and 8B backbones. Results demonstrate that LPM outperforms LoRA and Prompt Tuning by up to 8.8% and 54.4% in overall accuracy respectively on PersonaMem v1, while reducing KV-cache usage by over 64x. On LoCoMo, LPM matches LoRA accuracy with 120x fewer trainable parameters. We also show that the efficiency of LPM grows with context length and outperforms full-context at 128K context length.

Abstract

Personalized Language Models

Explore similar work

Latent Personal Memory: Represent personal memory as dynamic soft prompts

Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents

Less Is Personal: Learning Minimal Sufficient User Profiles for Personalized Language Models