User Simulation

Follow this topic Ask about this topic

User simulation studies how to represent people’s goals, preferences, knowledge, and behavior in interactions with AI agents, from tutoring and shopping to healthcare and web use. Common approaches include persona prompting, models grounded in real interaction logs, explicit cognitive or behavioral profiles, and learned simulators trained with supervised or reinforcement learning. Researchers evaluate these simulators against human behavior using measures of conversational and decision fidelity, behavioral diversity, long-term trajectories, and effects on agent performance. A central challenge is closing the sim-to-real gap: fluent or population-level outputs can still misrepresent individuals, omit difficult behaviors, or bias evaluations through overly cooperative simulated users.

Latest papers

128 papers29 papers in the last four weeks, up 81% on the four weeks before. 0.2% of all new papers.

Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation
Oct 26, 2026Atri Vivek Sharma, Brian Formento, Alessio LomuscioLLM Agent Evaluation User Simulation
Imperial College London · Safe Intelligence
Profile-Driven User Simulation via Reinforcement Learning with Hierarchical Rewards for Task-Oriented Dialogues
Oct 23, 2026Juan Li, Fangshu Chang, Xinyi Jiang +2User Simulation Synthetic Data Generation for LLM Agents
Bairong Inc.
Towards User Simulators That Stay On Task And Still Sound Natural
Oct 22, 2026Mustafa Omer Gul, Philippe Laban, Tanya Goyal +1User Simulation Human Behavior Simulation
Cornell University · Microsoft Research
SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots
Oct 21, 2026Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova +3Clinical Language Model Evaluation User Simulation
UserTrace · Cliniva · hearX Foundation +2
SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
Oct 21, 2026Lorenz Brehme, Benjamin Steenhoek, Ruth BreuCoding Agents AI Agent Evaluation
University of Innsbruck Innsbruck, Austria · Microsoft Redmond, WA, USA
OmniSimulator: Aligning Small Language Models for Authentic Heterogeneous Behavior Modeling
Oct 21, 2026Jiawei Chen, Ruoxi Xu, Boxi Cao +12User Simulation Human Behavior Simulation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Kuaishou Technology
Beyond Behavioral Accuracy: Evaluating Individual-Level Fidelity in LLM Investor Simulation
Oct 19, 2026Jiajie He, Yuanhong Jiang, Xintong Chen +2Human Behavior Simulation User Simulation
Hithink Research · McMaster University · Nanyang Technological University
Can LLM User-Proxies Stand In for Humans in Reliance Measurement? Replicating Rel-A.I. with Simulated Users
Oct 19, 2026Shuai Wang, Rui HuUser Simulation Large Language Model-Based Social Simulation
Zhejiang University
Analyzing Repetition and Interference Effects in Memory-Constrained User Simulators
Oct 16, 2026Cara Su-Yi Leong, Nicholas Tomlin, Tal LinzenUser Simulation
New York University · Toyota Technological Institute at Chicago
Learning User Representations to Improve Simulation Fidelity
Oct 15, 2026Shuhaib Mehri, Philippe Laban, Michel Galley +3User Simulation User Modeling
University of Illinois Urbana-Champaign · Microsoft Research
ARGUS: An Agenda-Driven Framework for Reliable User Simulation
Oct 15, 2026Hang Zheng, Nan Jiang, Zichen Zhu +6Task-Oriented Dialogue Systems User Simulation
X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence SJTU AI Institute, Shanghai Jiao Tong University, Shanghai, China · Jiangsu Key Lab of Language Computing, Suzhou, China · AISpeech Co., Ltd., Suzhou, China +1
HACHIMI-II: From Role-Play to Learner Behavior Modeling for Faithful Student Simulation
Oct 14, 2026Yilin Jiang, Fei Tan, Xuanyu Yin +2User Simulation AI in Education
East China Normal University, Shanghai, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shanghai Innovation Institute, Shanghai, China
SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems
Oct 13, 2026Erfan Shahabi, Amirhossein Roshandel, Mohammad Sadegh Sadeghi +1User Simulation Recommender Systems
School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran
EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents
Oct 12, 2026Min Cai, Dan Zhang, Gao Xing +3LLM Agent Training RL for LLM Agents
University of Alberta · National University of Singapore · Kuaishou Technology +2
Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints
Oct 12, 2026Yutao Xie, Jingbo Yang, Guanyu Yao +3LLM Agent Evaluation User Simulation
University of California, Santa Barbara · Cresta
Training User Simulators with Latent Intrinsic Feedback
Oct 9, 2026Tarek Naous, Craig Boutilier, Wei Xu +1User Simulation Large Language Model-Based Role-Play Simulation
Google Research · Georgia Institute of Technology
InterviewPlayground: A Validated Simulation Environment for Evaluating AI Interviewers
Oct 8, 2026Jonathan Ivey, Aimee Liang, Arthur Y. S. Wang +3Conversational Agents Multi-Turn Dialogue Evaluation
Johns Hopkins University · Listen Labs
UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions
Oct 8, 2026Mengze Hong, Zeyang Lei, Wenbo Shang +20Benchmark Design Multi-Turn Dialogue Evaluation
Yuanbao Team, Tencent
Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators
Oct 7, 2026Riqiang Wang, Elena Khasanova, Harsh Saini +4User Simulation Voice Agent Evaluation
Dialpad Inc.
Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation
Oct 7, 2026Lex Konnelly, Elena Khasanova, Riqiang Wang +3User Simulation Language Model Generation Evaluation
Dialpad Inc.
MIMESIS: Learning User Simulators as Training Environments for Interactive Agents
Oct 7, 2026Hoang Phan, Dat Huynh, Andrey Zhmoginov +7User Simulation LLM Agent Training
Meta Superintelligence Labs · New York University · University of Wisconsin - Madison
Multi-Objective Aligned Small Language Model Framework for SUD Patient Dialogue Generation
Oct 6, 2026Thushara Manjari Naduvilakandy, Hyeju Jang, Mohammad Al HasanSmall Language Models User Simulation
Department of Computer Science, Indiana University Indianapolis, Indianapolis, IN, USA
Joint Workflow and Prompt Optimization for User Behavior Simulation
Oct 6, 2026Nipun B Nair, Tongtong Wu, Hongzhi Yin +2Multi-Agent LLM Systems Automatic Prompt Optimization
Monash University · The University of Queensland · Xiamen University
AgentPersonaBench: Benchmarking Persona-Driven User Simulation
Oct 3, 2026Jintao Huang, Yifan Wang, Hongyu Shen +43AI Agent Benchmarks Persona-Conditioned Generation
CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking
Oct 1, 2026Anjali Kantharuban, Jonas MuellerUser Simulation AI Agent Benchmarks
Handshake AI · Carnegie Mellon University
SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling
Sep 29, 2026Yunan Lu, Shuang Xie, Meghna Allamudi +4Synthetic Data Generation User Modeling
Shopify · Columbia University
UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
Sep 29, 2026Ashish Jain, Armaan SandhuAI Agent Benchmarks User Simulation
Sarvam AI
Social Choice Foundations for Simulation-Augmented Generation
Sep 29, 2026Sonja Kraiczy, Smitha Milli, Ratip Emin Berker +6Large Language Model-Based Social Simulation User Simulation
University of Oxford · PrincInt · FAIR at Meta +4
Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation
Sep 28, 2026Jiashen Ren, Wenlin Zhang, Bohan Zhang +5Persona-Conditioned Generation User Simulation
City University of Hong Kong · Stanford University
Proactive Dialogue Policy Optimization via Cognitive-State Transition
Sep 28, 2026Minghui Ma, Mengqi Chen, Bin Guo +1Policy Optimization Conversational Agents
Northwestern Polytechnical University

Oct 26, 2026J/K move · Enter open · S save

Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation

Atri Vivek Sharma, Brian Formento, Alessio Lomuscio

Imperial College London · Safe Intelligence

LLM Agent Evaluation User Simulation Multi-Turn Dialogue Evaluation LLM Agent Reliability Prompt Optimization

Open paper AskPDF arXiv

Multi-turn agents are commonly evaluated with LLM-based simulated users, which can be overly cooperative and can therefore overestimate an agent's performance. We here introduce StressSim, an intent-preserving adversarial user simulator framework for stress-testing agents with challenging yet valid user interactions. StressSim optimises the simulator's high-level adversarial plan for each task instance. Candidate planner prompts are evolved with a genetic prompt-optimisation loop that uses rollout traces, granular task-completion signals, and validity feedback to favour conversations that reduce task success without changing the user's underlying goal. We evaluate the approach on the MultiWOZ dataset and the retail and airline domains from tau-bench with three target LLM agents, comparing against standard simulators and hand-designed non-collaborative user behaviours. We demonstrate the effectiveness of StressSim in exposing robustness gaps across frontier LLMs, with tau-bench pass^4 scores decreasing by up to 69% on gemini-3.8-flash and 62% on gpt-5.6-sol. StressSim maintains goal preservation performance (which decreases by at most 10%), using an optimisation budget of 20 rollouts per task. Sensitivity analysis suggests that increasing the budget further decreases task success. Overall our results demonstrate that StressSim exposes robustness gaps missed by standard and hand-designed user simulators while maintaining conversation realism. This is achieved through lightweight prompt optimisation alone, without model fine-tuning.