SimAI: Mechanistic User Simulation through Psychological Network Models
Authors: Cheng Li, Adhyyan Narang, Jiayi Yuan, John F. Buss, Jina Suh, Devon Sandel-Fernandez, Yulia Tsvetkov, Adam M. Kuczynski, Tim Althoff
Organizations: Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA · Department of Electrical & Computer Engineering, University of Washington, Seattle, WA, USA · Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine, Seattle, WA, USA
Realistic user simulation enables the study of human interactions when direct experimentation is costly or potentially harmful. However, existing simulators largely overlook the psychological mechanisms that drive user behavior. We introduce SimAI, a framework for mechanistic user simulation that models how psychological factors interact to shape user responses. To overcome the challenge of supervising open-ended behavior, we propose a graph reconstruction objective that provides graph-level rewards based on how well simulated conversations reflect the underlying psychological relationships. To support this approach, we construct PSYCHE, a dataset of 18,576 profile-network pairs derived from 516 psychotherapy cases. Automatic evaluations show that SimAI achieves greater simulation and behavioral diversity than existing patient simulators while maintaining strong profile alignment. In an interactive study with 30 mental-health professionals and trainees, SimAI receives the highest average ratings for conversational authenticity, realistic help-seeking behavior, and clinical training utility. Beyond simulation quality, SimAI enables realistic clinical training and uncovers safety failures in prolonged AI interactions with vulnerable users. Code is available at https://github.com/behavioral-data/ANGEL-UserSim.
Figures & tables
Figure 1: Overview of SimAI . (a) Unlike existing user simulators that generate conversations directly from profiles, SimAI represents underlying psychological mechanisms through a network model. (b) PSYCHE contains 18,576 paired profile–network model samples derived from 516 psychotherapy cases. (c) The Observer constructs a psychological network model from a user profile; the example is simplified for clarity. (d) Selected graph relationships are masked to condition the Actor, while graph reconstruction provides training feedback. (e) Automatic and human evaluations compare SimAI with the best baseline for each metric. Points indicate means and error bars show 95% confidence intervals. (f) Applications in clinician education and AI safety.
Figure 2: Network model and its activation during conversation. (a) Nodes represent stimuli, cognitions, emotions, behaviors, and physiological sensations; edges represent causal, triggering, or reinforcing relationships. (b) Different conversational turns activate different subgraphs of the underlying network model to ground the Actor’s responses.
Figure 3: Training framework for SimAI . (a) The Observer is trained with SFT and GRPO to construct network models from patient profiles. (b) The Actor generates conversations conditioned on masked network models; reconstruction similarity provides preference signals for DPO.
Figure 4: Automatic and human evaluation. (a–c) SimAI achieves the highest Simulation and Behavior Diversity while maintaining strong Profile Alignment. (d) In interactive human evaluation, SimAI receives the highest mean ratings for Authenticity, Resemblance, Training Readiness, and Novice Recommendation. Error bars indicate 95% confidence intervals.
Figure 5: Construct validity of the diversity metrics. (a–b) Diversity decreases as more profile attributes are fixed (shading: 95% confidence intervals). (c–d) For SimAI simulations, groups drawn from different profiles are more diverse than groups from the same profile (Simulation Diversity: AUC =1.00 , d=7.51 ; Behavior Diversity: AUC =0.79 , d=1.21 ).
Configuration
Simulation Diversity
Behavior Diversity
Profile Alignment
Authenticity
Behavioral Realism
Profile + base model
0.33
0.94
0.96
3.7
3.6
GPT-5 network model + base model
0.33
0.94
0.95
4.0
3.9
Observer network model + base model
0.36
0.96
0.95
4.0
4.0
Observer network model + Actor (Full SimAI )
0.39
0.97
0.94
4.3
4.4
Table 1: Ablation study of SimAI . Authenticity and Behavioral Realism are rated by GPT-5 on a 1–5 scale. Simulation Diversity, Behavior Diversity, and Profile Alignment are computed using the automatic metrics defined in Section 5.1.2 .
Figure 6: AI safety evaluation under the prefilled long-context attack. (a) Mean Risk and Safety Scores with 95% bootstrap confidence intervals; lower Risk and higher Safety indicate better safety. (b) Example GPT-4o failure reinforcing harmful reasoning and encouraging inappropriate behavioral action.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: PSYCHE : the first graph-grounded dataset for psychological user simulation. Starting from 516 psychotherapy cases, we construct paired patient profiles and network models and use GPT-5 to generate augmented profiles, yielding 3,096 profiles. For each profile, we apply six independent masking levels to the profile and its network model, producing complementary paired samples that capture different partial views of the same underlying psychological mechanisms. Overall, PSYCHE contains 18,576 paired profile–network model training samples.
Table 2: Disorder-specific candidate edge types used for graph masking.
SFT
Stage-1 GRPO
Stage-2 GRPO
Base model
Qwen3-8B
Stage-1 SFT
Stage-2 SFT
LoRA rank r
64
16
32
LoRA α
16
32
64
Learning rate
1×10−4
5×10−6
1×10−5
GRPO generations
–
8
4
Training length
3 epochs
100×6 steps
100×8 steps
Appendix
Table 3: Training configuration of the Observer Model.
SFT
DPO
Base model
Qwen3-8B
Qwen3-8B + SFT adapter
Quantization
4-bit NF4, bf16
4-bit NF4, bf16
LoRA rank r
64
16
LoRA α
16
32
LoRA dropout
0.05
0.05
LoRA targets
q,k,v,o,gate,up,down proj
Appendix
Table 4: Training configuration of the Actor Model.
Figure 8: Effect of run count on diversity metrics. Lines show mean scores across profiles, and shaded regions indicate 95% bootstrap confidence intervals. (a) Simulation Diversity decreases and gradually stabilizes as more runs are included, while SimAI consistently maintains the highest diversity. (b) Behavior Diversity is comparatively stable across run counts, with SimAI maintaining strong diversity throughout.
Figure 9: Stage-level model performance. (a) Simulation Diversity across interview stages. (b) Behavior Diversity across interview stages. (c) Profile Alignment across profile aspects. Points indicate mean scores, and error bars show 95% bootstrap confidence intervals. The y-axes in panels (b) and (c) are truncated for visualization.
Figure 10: Stage-level between-profile discrimination of the diversity metrics. (a) Simulation Diversity and (b) Behavior Diversity for simulations generated from the same versus different patient profiles across interview stages. Points indicate mean diversity scores, and connecting lines show the separation between conditions within each stage.
Figure 11: Conversation-level structure by patient profile. (a) Pairwise cosine distances between simulated conversations, ordered by patient profile, with white lines indicating profile boundaries. (b–c) t-SNE visualizations of the representations underlying Simulation Diversity and Behavior Diversity, respectively, with colors denoting patient profiles. These projections are provided as qualitative visualizations of representation structure rather than quantitative evidence of cluster separation.
Figure 12: Overview of the human evaluation procedure. Participants first complete a pre-study questionnaire and construct a patient profile for a mental-health condition within their knowledge or experience. The same profile is then used to evaluate four anonymized patient simulators in a within-profile design, with simulator presentation order randomized across participants. For each simulator, participants conduct a 15-minute semi-structured interview covering the presenting problem, current symptoms, and context and functional impact, followed by ratings on six criteria using a five-point scale.
Figure 13: Characteristics of human-evaluation participants ( N=30 ). Participant distributions by (a) gender, (b) age, (c) race/ethnicity, (d) professional background, (e) years of clinical experience, and (f) clinical experience across mental-health conditions. Values indicate participant counts and percentages. Participants could report clinical experience with multiple conditions in (f) .
Component
Information Captured
Example
Identity
Basic patient characteristics
Age, gender, occupation/role
Background
Relevant history, living situation, and stressors
Family environment, school/work context, reason for seeking help
Social situations, evaluation, interpersonal conflict
Current Emotions
Typical affective experiences
Anxious, ashamed, fearful, worthless
Current Behaviors
Observable behaviors and coping patterns
Avoidance, withdrawal, reassurance seeking
Appendix
Table 5: Structured patient-profile construction. Participants specify psychological, behavioral, contextual, and communication characteristics of the patient subsequently used to evaluate all four simulators. Except for required identity information, participants can provide the information most relevant to their intended case.
Stage
Time
Goal
Core Question(s)
Presenting Problem
∼ 4 min
Identify why the patient is seeking help and establish the main concern.
“What brings you in today, and what feels most important for me to understand?”
Current Symptoms
∼ 5 min
Elicit the patient’s primary emotional, cognitive, and behavioral experiences.
“When the problem is at its worst, what do you usually feel, think, and do?”
Context and Functional Impact
∼ 6 min
Identify relevant triggers and understand effects on everyday functioning.
“What situations or events tend to trigger or worsen the problem?” “How is the problem affecting your daily life, such as work or school, relationships, sleep, self-care, or everyday responsibilities?”
Appendix
Table 6: Semi-structured 15-minute interview protocol. Participants are provided with the same stage goals and core questions for each simulator. Optional prompts are available when clarification is needed, while participants remain free to adapt their questions to the ongoing conversation.
Criterion
Evaluation Focus
Conversational Authenticity
Naturalness, believability, and human-likeness of the interaction.
Role Consistency
Consistency with the assigned patient role throughout the conversation.
Typical Help-Seeking Resemblance
Similarity to plausible real-world client or help-seeking behavior.
Challenge for the Counselor
Degree to which the patient presents realistic challenges requiring interviewer adaptation.
Readiness for Training Use
Suitability of the simulator for listener or counselor training.
Recommendation for Novice Training
Willingness to recommend the simulator as a practice partner for novice listeners or counselors.
Appendix
Table 7: Human evaluation criteria. Participants rate each simulator on six criteria using a five-point scale (1: Poor, 2: Fair, 3: Adequate, 4: Good, 5: Excellent). The criteria assess simulation realism, interaction characteristics, and potential training utility.
Figure 14: Human evaluation of SimAI across patient and evaluator subgroups. Mean ratings and 95% confidence intervals across six evaluation dimensions, stratified by (a) patient disorder, (b) evaluator clinical experience, and (c) evaluator professional role. Subgroup sample sizes are reported in the legends. Ratings use a 1–5 scale, with higher values indicating higher ratings. Results for smaller subgroups should be interpreted as exploratory.
Figure 15: Safety codebook scores for GPT-4o under the prefilled long-context attack setting. Higher scores indicate greater susceptibility to the corresponding safety failure patterns.
Figure 16: Safety codebook scores for Gemini-3-Pro under the prefilled long-context attack setting. Higher scores indicate greater susceptibility to the corresponding safety failure patterns.
Figure 17: Safety codebook scores for Claude-4.5-Opus under the prefilled long-context attack setting. Higher scores indicate greater susceptibility to the corresponding safety failure patterns.
Figure 18: Highest-risk prompt categories automatically discovered for GPT-4o. The scores represent the average risk induced by each prompt category.
Figure 19: Highest-risk prompt categories automatically discovered for Gemini-3-Pro. The scores represent the average risk induced by each prompt category.
Figure 20: Highest-risk prompt categories automatically discovered for Claude-4.5-Opus. The scores represent the average risk induced by each prompt category.
Figure 21: Representative safety failure automatically discovered by our vulnerable user simulation framework. The example illustrates a misrepresentation failure in which the model reinforces false beliefs about its capabilities.
Figure 22: Representative safety failure automatically discovered by our vulnerable user simulation framework. The example illustrates a sycophancy failure in which the model validates harmful reasoning through excessive agreement.
Figure 23: Additional representative safety failure automatically discovered by our vulnerable user simulation framework. The example illustrates a behavioral advice failure in which the model encourages self-destructive behavior.
Figure 24: Additional representative safety failure automatically discovered by our vulnerable user simulation framework. The example illustrates a behavioral advice failure in which the model encourages self-destructive behavior.
Figure 25: Additional representative safety failure automatically discovered by our vulnerable user simulation framework. The example illustrates a misrepresentation failure in which the model reinforces false beliefs about its capabilities.
Figure 26: Additional representative safety failure automatically discovered by our vulnerable user simulation framework. The example illustrates a sycophancy failure in which the model validates harmful reasoning through excessive agreement.
Figure 27: Additional representative safety failure automatically discovered by our vulnerable user simulation framework. The example illustrates a validation failure in which the model accepts and reinforces delusional beliefs instead of questioning them.
LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstrained LLM defaults produce a Formalism Ceiling (style match rates of 6-8% against real users), while hand-crafted behavioral directives trigger Directive Amplification, where models hyper-interpret instructions into unnatural behavioral extremes that vary dramatically across simulator models. We present RealUserSim, the first user simulation framework grounded in real behavioral data. From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles and use them to ground LLM simulators. A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3% across five behavioral dimensions. Agent evaluation on TauBench with 6 simulator models and extensive analysis shows that grounded simulation acts as a realistic stress test, surfacing three failure mechanisms invisible to cooperative simulators (mean -3.2% to -3.5% task success degradation), while Directive Amplification in existing benchmarks produces unrealistic behavior that compromises the validity of agent evaluation.
Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. Yet current simulators often produce plausible individual responses without reproducing the intent evolution and outcomes observed in real interactions. We propose TRACER, a multi-turn user simulator that models evolving user intent and aligns simulated trajectories with real ones. TRACER is trained in two stages: supervised fine-tuning on real user dialogues, followed by multi-turn reinforcement learning. The RL stage combines hierarchical outcome- and trajectory-level rewards with deviation-aware advantage modulation, jointly addressing reward sparsity and credit assignment challenges in long dialogues. On real customer-service sessions organized into reference cohorts, TRACER-7B surpasses the strongest baseline by 11.4 conversion F1 points, while outperforming all baselines on group-level conversion-rate error and semantic trajectory distance and generalizing to out-of-distribution scenarios. In human Turing tests, annotators identified TRACER conversations at near-chance accuracy. Building on this simulator, we further introduce the Dynamic Marketing Benchmark, which jointly evaluates persuasion and response quality via simulated interactions, revealing that higher response quality does not necessarily correspond to higher conversion rates.
Geng Chen, Ruotong Pan, Zhirui Yang +9
Peking University · Kuaishou Technology · Xi’an Jiaotong University +3
User simulators enable interactive training and evaluation of AI assistants at scale. Yet current simulators struggle to reproduce the diverse behaviors of real users, leading to assistants that generalize poorly to underrepresented users. We identify user representations as an underexplored bottleneck in user simulation. User representations are natural-language descriptions that condition simulators to reproduce a particular user's behavior. A representation's quality becomes apparent in hindsight, through the behavior it induces in a simulator. To evaluate and optimize user representations, we formulate a conversation reconstruction task: given an original conversation, we generate a representation, condition a simulator on it, simulate a conversation, and then measure fidelity by comparing user behavior in the original and simulated conversations. This comparison also yields hindsight feedback, which identifies where the representation falls short and how it can be improved. We introduce hindsight distillation, an online training procedure that distills feedback-conditioned improvements into the representation generator, enabling it to generate improved representations directly from the original conversation. We present UserRepGen-4B, a representation generator trained with hindsight distillation. Across 700 conversations spanning seven datasets, hindsight feedback helps improve user representations at test time, and UserRepGen-4B generates representations that enable more faithful simulation than those generated by GPT-5.5. Beyond conversation reconstruction, learned representations improve simulation fidelity for unseen tasks, better preserve the population-level diversity of real users, and better distinguish users across conversations.
Shuhaib Mehri, Philippe Laban, Michel Galley +3
University of Illinois Urbana-Champaign · Microsoft Research