Analyzing Repetition and Interference Effects in Memory-Constrained User Simulators

Cara Su-Yi Leong; Nicholas Tomlin; Tal Linzen

cs.CLOct 16, 2026

Analyzing Repetition and Interference Effects in Memory-Constrained User Simulators

Authors: Cara Su-Yi Leong, Nicholas Tomlin, Tal Linzen

Organizations: New York University · Toyota Technological Institute at Chicago

Abstract

User simulators can help evaluate and train AI systems to interact with humans at scale. To do so, evaluation of user simulators must measure behavioral parity with people. We evaluate whether memory-constrained user simulators are sensitive to two factors known to affect human information retention: repetition of a fact and interference from competing information. We find that some simulators achieve human-like overall accuracy, and replicate the magnitude of repetition and interference effects, but none match people's accuracy on particular questions: while humans misremember facts, models forget them, leading to different error patterns. We release this data as a challenge benchmark for future work on user simulators.

Figures & tables

Figure 1: Top: Four versions of a passage vary how often facts are restated and whether interference exists from competing facts. Bottom: Each passage is followed by five multi-select questions. Interference and restatement have different effects on humans and user simulators.

Figure 2: Human accuracy by passage version and question. (a) Accuracy by version averaged over topics; error bars are bootstrapped 95% CIs. *:p < 0.05; ***:p < 0.001. (b) Each question’s accuracy in SayOnce against its accuracy in SayTwice .

Figure 3: Memory-constrained simulators are human-like on coarse-grained levels (a)–(b), but not item agreement (c). Each point represents a model–hyperparameter configuration; yellow regions represent bootstrapped 95% confidence interval. (a) Simulators other than prompt-based methods have configurations that reach listener accuracy. (b) Some memory-constrained simulators achieve human-like effects of repetition ( SayTwice

-

SayOnce ) and interference ( SayOnce

-

Interfere ). (c) No simulators are as similar as two groups of humans are to each other on their rankings of item difficulty. Correlation on item difficulty is measured by Pearson’s r between mean model and human item rankings.

Figure 4: Statement endorsement rates in humans and representative models. Gemini 3.1 Pro is representative of the median model; Qwen3-4B is one of the most human-like models at the item level.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Version	Words	Audio (s)
SayOnce	128 (114–156)	48 (44–56)
SayTwice	134 (126–152)	55 (51–60)
SayThrice	194 (167–219)	77 (63–83)
Interfere	198 (176–227)	76 (68–83)

Appendix

Table 1: Passage length by version; mean (range) over the four topics.

Figure 5: Correlation between question-level accuracy in any two passage versions. Each point represents mean accuracy on a question.

True	Qs	Acc.	Selected	Omission	Commission
0	3	35.7	1.08	–	64.3
1	5	66.9	1.07	–	33.1
2	11	42.4	1.55	39.3	18.4
3	1	26.4	2.04	62.2	11.4

Appendix

Table 2: Human accuracy on all passage versions by number of true statements in answer key (0: “None of the above” is correct). Selected: mean number of options endorsed. Omission: % of responses that chose only true statements but missed at least one. Commission: % that endorsed at least one false option. Omission is impossible for 0- and 1-answer questions.

Figure 6: Human accuracy on all passage versions by topic. Error bars represent bootstrapped 95% confidence intervals.

Model	Thinking	$T$	$p$	$k$
Qwen2.5-0.5B, 1.5B	Off	0.7	0.8	20
Qwen2.5-3B–72B	Off	0.7	0.8	20
Qwen3-0.6B–32B	Off	0.7	0.8	20
	On	0.6	0.95	20
Qwen3.8-27B	Off	0.7	0.8	20
	On (med.)	1.0	0.95	20

Appendix

Table 3: Sampling settings for each model and thinking mode.

Figure 7: Correlation with humans on item difficulty by model. Models with 2–7B parameters tend to outperform frontier models. Rows are sorted by mean.

Figure 8: Comparison between humans and models on three levels by model across all configurations—bottleneck capacity, presentation mode, thinking mode, prompt.

Figure 9: Model size against the three levels for Qwen2.5 and Qwen3 (batch encoding,

k=4

). In (b), point size scales with parameter count; the star and box show the human value and bootstrapped 95% confidence interval.

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 128 papers

Simulating Human Memory with Language Models
May 25, 2026Qihan Wang, Nicholas Tomlin, Michael Hu +2LLM Evaluation Memory-Augmented Language Models
NYU · UMass Amherst
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking
Oct 1, 2026Anjali Kantharuban, Jonas MuellerUser Simulation AI Agent Benchmarks
Handshake AI · Carnegie Mellon University

May 25, 2026J/K move · Enter open · S save

Simulating Human Memory with Language Models

Qihan Wang, Nicholas Tomlin, Michael Hu, Brian Dillon+1

NYU · UMass Amherst

LLM Evaluation Memory-Augmented Language Models Memorization in Language Models LLM Prompting LLM Memory Memorization in Generative Models+1

Open paper AskPDF arXiv

Language models are increasingly being deployed as user simulators, but their memory is far more reliable than that of real users. To measure this gap, we run a series of classic memory experiments from psychology on both humans and language models. Across tasks, we find that out-of-the-box language models exhibit better memory than humans, even when prompted to imitate human behavior. We then show that better prompting strategies and the use of a compactor can cause language models to forget content in a more human-like way. Using these methods, we show preliminary evidence that language models with human-like memory constraints can function as more effective user simulators in a downstream education task. Finally, we release human reference data and benchmarks to support future work on simulating human memory with language models.

Abstract

User Simulation

Explore similar work

Simulating Human Memory with Language Models

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking