User Simulation

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 8% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 76

All topics

Controllable User Simulation
May 12, 2026Guy Tennenholtz, Ofer Meshi, Amir Globerson +3Off-Policy Evaluation Causal Inference
† Google Research · ‡ Tel Aviv University
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
May 10, 2026Joseph Suh, Ayush Raj, Minwoo Kang +1LLM Agent Evaluation Human-AI Interaction
UC Berkeley
MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions
May 10, 2026Yaoning Yu, Ye Yu, Haojing Luo +1Human Behavior Simulation LLM Agent Evaluation
University of Illinois at Urbana-Champaign
SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
May 8, 2026Yada Pruksachatkun, Elaine Wan, Lyanna Chen +2RL for Language Models Large Language Model-Based Role-Play Simulation
Salesforce Research · University of California Los Angeles
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
May 8, 2026Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar +4Human Behavior Simulation User Simulation
University of Illinois Urbana-Champaign · Microsoft Research · University of California, Berkeley
Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework
May 8, 2026Nipun B Nair, Tongtong Wu, Weiqing WangAutomatic Prompt Optimization Recommender Systems
Monash University, Australia
Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems
May 5, 2026Yuan-Chi Li, Li-Chi Chen, Sung-Yi Wu +2Human Behavior Simulation Recommender Systems
Department of Computer Science and Information Engineering National Taiwan University
Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations
May 4, 2026Yu Lu Liu, Hyokun Yun, Tanya Roosta +1Conversational Agents User Simulation
Johns Hopkins University · UC Berkeley
Fake Plastic Voters: When Political Parties Can Use AI-Simulated Focus Groups
Apr 28, 2026Claudio Novelli, Javier Argota Sanchez-Vaquerizo, Jennifer Cyr +4Human Behavior Simulation Human-AI Decision Making
Digital Ethics Center, Yale University, New Haven, CT, USA · Department of Humanities, Social and Political Sciences (GESS), ETH Zürich, Switzerland · Department of Political Science and International Studies, Universidad Torcuato Di Tella, Buenos Aires, Argentina +3
PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulators
Apr 28, 2026Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu +4Benchmark Design Clinical Language Model Evaluation
University of Illinois Urbana-Champaign · VinUniversity
A Survey on LLM-based Conversational User Simulation
Apr 27, 2026Bo Ni, Leyao Wang, Yu Wang +27Human Behavior Simulation User Modeling
Vanderbilt University · Yale University · University of Oregon +8
Efficient Agent Evaluation via Diversity-Guided User Simulation
Apr 23, 2026Itay Nakash, George Kour, Ateret Anaby-TavorAutomated Software Testing LLM Agent Evaluation
IBM Research
Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation
Apr 16, 2026Ziyang Chen, Renbing Chen, Daowei Li +4Human Behavior Simulation User Simulation
LongCat Interaction Team, Meituan Inc. Beijing, China · Independent Researcher Changsha, China
SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
Mar 17, 2026Jonggeun Lee, Junseong Pyo, Jeongmin Park +1Spoken Dialogue Systems Conversational Agents
Graduate School of Data Science, Seoul National University · Department of Information Systems, Hanyang University · Department of Computer Science and Engineering, Seoul National University
SimRPD: Optimizing Recruitment Proactive Dialogue Agents through Simulator-Based Data Evaluation and Selection
Jan 6, 2026Zhiyong Cao, Dunqiang Liu, Qi Dai +9Synthetic Data Generation Conversational Agents
Zhaopin Limited · Institute of Automation, Chinese Academy of Sciences · School of Automation,Beijing Institute of Technology +3
ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation
Jul 16, 2025Rose Niousha, Mihran Miroyan, Abigail O'Neill +4Generative AI in Education User Simulation
University of California, Berkeley

May 12, 2026, cs.AIJ/K move · Enter open · S save

Controllable User Simulation

Guy Tennenholtz, Ofer Meshi, Amir Globerson, Uri Shalit+2

† Google Research · ‡ Tel Aviv University

Off-Policy Evaluation Causal Inference User Simulation

Open paper AskPDF arXiv

Using offline datasets to evaluate conversational agents often fails to cover rare scenarios or to support testing new policies. This has motivated the use of controllable user simulators for targeted, counterfactual evaluation, typically implemented by prompting or fine-tuning large language models. In this work, we formalize controllable simulation as a causal inference problem. By bridging natural language evaluation with off-policy evaluation methodology, we show that the standard practice of training simulators via supervised fine-tuning on post-hoc trajectory labels yields a structurally biased model. Specifically, these labels are inextricably coupled to the data-generating behavior policy, injecting a look-ahead bias that breaks causal consistency. Furthermore, we prove that under policy shift this failure causes the variance of evaluation metrics to explode geometrically, a phenomenon we term controllability collapse. To restore causal consistency, we establish theoretical conditions for accurate simulation and propose practical training mitigations: a priori controls, step-wise dynamic controls, and direct policy-conditioned learning. Empirical evaluation confirms that while standard global controls distort conversational distributions and collapse behavioral diversity, our causally grounded simulators eliminate look-ahead bias, preserve natural variance, and exhibit robust zero-shot generalization to unseen agent behaviors.

Momentum

Related topics

Latest papers 76