User Simulation

Follow this topic Ask about this topic

User simulation studies how to represent people’s goals, preferences, knowledge, and behavior in interactions with AI agents, from tutoring and shopping to healthcare and web use. Common approaches include persona prompting, models grounded in real interaction logs, explicit cognitive or behavioral profiles, and learned simulators trained with supervised or reinforcement learning. Researchers evaluate these simulators against human behavior using measures of conversational and decision fidelity, behavioral diversity, long-term trajectories, and effects on agent performance. A central challenge is closing the sim-to-real gap: fluent or population-level outputs can still misrepresent individuals, omit difficult behaviors, or bias evaluations through overly cooperative simulated users.

Latest papers

128 papers29 papers in the last four weeks, up 81% on the four weeks before. 0.2% of all new papers.

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
May 13, 2026Harshita Chopra, Kshitish Ghate, Aylin Caliskan +3Human Behavior Simulation LLM Agent Evaluation
University of Washington, Seattle · Georgetown University, Washington DC
PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior
May 12, 2026James Flemings, Murali AnnavaramLLM Evaluation User Modeling
University of Southern California
Controllable User Simulation
May 12, 2026Guy Tennenholtz, Ofer Meshi, Amir Globerson +3Off-Policy Evaluation Causal Inference
† Google Research · ‡ Tel Aviv University
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
May 10, 2026Joseph Suh, Ayush Raj, Minwoo Kang +1LLM Agent Evaluation Human-AI Interaction
UC Berkeley
ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions
May 10, 2026Yaoning Yu, Ye Yu, John Sciarappo +4Human Behavior Simulation LLM Agent Evaluation
University of Illinois at Urbana-Champaign
SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
May 8, 2026Yada Pruksachatkun, Elaine Wan, Lyanna Chen +2RL for Language Models Large Language Model-Based Role-Play Simulation
Salesforce Research · University of California Los Angeles
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
May 8, 2026Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar +4Human Behavior Simulation User Simulation
University of Illinois Urbana-Champaign · Microsoft Research · University of California, Berkeley
Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework
May 8, 2026Nipun B Nair, Tongtong Wu, Weiqing WangAutomatic Prompt Optimization Recommender Systems
Monash University, Australia
Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems
May 5, 2026Yuan-Chi Li, Li-Chi Chen, Sung-Yi Wu +2Human Behavior Simulation Recommender Systems
Department of Computer Science and Information Engineering National Taiwan University
Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations
May 4, 2026Yu Lu Liu, Hyokun Yun, Tanya Roosta +1Conversational Agents User Simulation
Johns Hopkins University · UC Berkeley
Fake Plastic Voters: When Political Parties Can Use AI-Simulated Focus Groups
Apr 28, 2026Claudio Novelli, Javier Argota Sanchez-Vaquerizo, Jennifer Cyr +4Human Behavior Simulation Human-AI Decision Making
Digital Ethics Center, Yale University, New Haven, CT, USA · Department of Legal Studies, University of Bologna, Bologna, Italy · Department of Humanities, Social and Political Sciences (GESS), ETH Zürich, Switzerland +3
PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulators
Apr 28, 2026Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu +4Benchmark Design Clinical Language Model Evaluation
University of Illinois Urbana-Champaign · VinUniversity
A Survey on LLM-based Conversational User Simulation
Apr 27, 2026Bo Ni, Leyao Wang, Yu Wang +27Human Behavior Simulation User Modeling
Vanderbilt University · Yale University · University of Oregon +8
Efficient Agent Evaluation via Diversity-Guided User Simulation
Apr 23, 2026Itay Nakash, George Kour, Ateret Anaby-TavorAutomated Software Testing LLM Agent Evaluation
IBM Research
Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation
Apr 16, 2026Ziyang Chen, Renbing Chen, Daowei Li +4Human Behavior Simulation User Simulation
LongCat Interaction Team, Meituan Inc. Beijing, China · Independent Researcher Changsha, China
MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignment
Apr 15, 2026Zihao Liu, Hantao Zhou, Jiguo Li +5User Simulation Large Language Model-Based Role-Play Simulation
College of Computer Science and Artificial Intelligence, Fudan University · Meituan
Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation
Apr 10, 2026Lingfeng Huang, Huizhong Guo, Tianjun Wei +2User Simulation VLM Adaptation
Singapore University of Technology and Design, Singapore · Zhejiang University, Hangzhou, China · Nanyang Technological University, Singapore
Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces
Apr 9, 2026Jiawei Chen, Ruoxi Xu, Boxi Cao +12Human Behavior Simulation User Simulation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Kuaishou Technology
RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
Apr 7, 2026Ming Zhu, Juntao Tan, Rithesh Murthy +6User Simulation Human Behavior Simulation
Salesforce AI Research
What Makes a Sale? Simulating End-to-End Seller--Buyer Retail Dynamics with LLM Agents
Apr 6, 2026Jeonghwan Choi, Jibin Hwang, Gyeonghun Sun +4User Simulation Human Behavior Simulation
Korea Advanced Institute of Science and Technology (KAIST) · GS Retail · Cluvion
Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants
Apr 1, 2026Deepak Nathani, Cheng Zhang, Chang Huan +7Proactive Agents Proactive Assistance
University of California, Santa Barbara · Independent Researcher, USA · Apple, USA +1
PolicySim: An LLM-Based Agent Social Simulation Sandbox for Proactive Policy Optimization
Mar 20, 2026Renhong Huang, Ning Tang, Jiarong Xu +6Large Language Model-Based Social Simulation Policy Optimization
Zhejiang University, Hangzhou, China · Fudan University, Shanghai, China · HKUST, Hong Kong, China +3
Interplay: Training Independent Simulators for Reference-Free Conversational Recommendation
Mar 19, 2026Jerome Ramos, Feng Xia, Xi Wang +4Recommender Systems User Simulation
University College London, London, UK · University of Sheffield, Sheffield, UK · Missouri University of Science and Technology, Rolla, USA
SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
Mar 17, 2026Jonggeun Lee, Junseong Pyo, Jeongmin Park +1Spoken Dialogue Systems Conversational Agents
Graduate School of Data Science, Seoul National University · Department of Information Systems, Hanyang University · Department of Computer Science and Engineering, Seoul National University
LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation
Mar 12, 2026Feiyu Duan, Xuanjing Huang, Zhongyu WeiUser Simulation Long-Horizon Agent Evaluation
School of Data Science, Fudan University · College of Computer Science and Artificial Intelligence, Fudan University · Shanghai Innovation Institute +2
EducaSim: Interactive Simulacra for CS1 Instructional Practice
Mar 12, 2026Cameron Mohne, Nicholas Vo, Dora Demszky +1AI in Education Computer Science Education
Stanford University, California, USA
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
Interview-Informed Generative Agents for Product Discovery: A Validation Study
Mar 10, 2026Zichao Wang, Alexa SiuUser Simulation Large Language Model-Based Social Simulation
Adobe Research San Jose, CA, USA
TriageSim: A Conversational Emergency Triage Simulation Framework from Structured Electronic Health Records
Mar 2, 2026Dipankar Srirag, Quoc Dung Nguyen, Aditya Joshi +2Clinical Triage User Simulation
University of New South Wales, Sydney, Australia
SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation
Mar 1, 2026Tim Rieder, Marian Schneider, Mario Truss +6A/b Testing User Simulation
ETH Zurich, Zurich, Switzerland · Adobe, Munich, Germany · Adobe, Basel, Switzerland +1

May 13, 2026J/K move · Enter open · S save

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno+2

University of Washington, Seattle · Georgetown University, Washington DC

Human Behavior Simulation LLM Agent Evaluation LLM Agent Training Persona-Conditioned Generation User Simulation

Open paper AskPDF arXiv

Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are unclear, impatient, or reluctant to share information. However, collecting real interaction data at scale remains expensive. The field has turned to LLM-based user simulators as stand-ins, but these simulators inherit the behavior of their underlying models: cooperative and homogeneous. As a result, agents that appear strong in simulation often fail in real human interactions. To narrow this gap, we introduce Persona Policies (PPol), a plug-and-play control layer that induces realistic behavioral variation in user simulators while preserving original task goals. Rather than hand-crafting personas, we employ an evolutionary coding agent to discover persona generation programs optimized for human-likeness and behavioral coverage over real user conversations. The evolved program generates diverse, human-like personas for any task in the domain. Across 4 benchmarks--including

τ^2

-bench Retail and Airline, ColBench, and WildChat--evolved PPol yield 28-72% absolute gains in fitness score over the baseline simulator. In blinded evaluations, annotators judged PPol users as 'human' 80.4% of the time, nearly 2x more than the baseline simulators. Training agents with PPol also improves real-world performance: our user study with live human-agent interactions showed that fine-tuning with our method boosted task success by +23% over default baselines. PPol thus offers a novel approach to strengthen simulator-based evaluation and training without changing underlying tasks.