Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers
Organizations: Graduate School of Data Science, Seoul National University
Abstract
As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged: use help-seeker simulators to interact with supporter chatbots. However, current simulators suffer from two critical limitations: (1) they fail to capture the behavioral diversity of real-world seekers, often portraying them as overly cooperative, and (2) they lack the controllability required to simulate specific seeker profiles. To address these challenges, we present a controllable seeker simulator driven by nine psychological and linguistic features that underpin seeker behavior. Using authentic Reddit conversations, we train our model via a Mixture-of-Experts (MoE) architecture, which effectively differentiates diverse seeker behaviors into specialized parameter subspaces, thereby enhancing fine-grained controllability. Our simulator achieves superior profile adherence and behavioral diversity compared to existing approaches. Furthermore, evaluating 7 prominent supporter models with our system uncovers previously obscured performance degradations. These findings underscore the utility of our framework in providing a more faithful and stress-tested evaluation for emotional support chatbots.
Figures & tables
| Simulator | Mean | Std | Min | Max |
| GPT-4.1-mini | 0.301 | 0.131 | 0.160 | 0.580 |
| Llama-3-8B-Instruct | 0.259 | 0.148 | 0.110 | 0.580 |
| Qwen-2.5-14B-Instruct | 0.284 | 0.095 | 0.150 | 0.470 |
| GPT-5 | 0.319 | 0.216 | 0.150 | 0.840 |
| DeepSeek-V3.2 | 0.431 | 0.218 | 0.180 | 0.910 |
| SFT | 0.515 | 0.160 | 0.360 | 0.760 |
| Comparison | Linguistic Naturalness | Role Authenticity | Psychological Plausibility |
|---|---|---|---|
| Ours vs. Eeyore | 68.9 | 66.7 | 71.1 |
| Ours vs. ESC-Judge | 68.9 | 72.2 | 62.2 |
| Ours vs. ESC-Role | 80.0 | 67.8 | 67.8 |
| Metric | Spearman | ||
| Identification | -1.167 | -0.733 | 0.454 |
| Comforting | -0.900 | -0.833 | 0.377 |
| Suggestions | -1.583 | -1.067 | 0.744 |
| Exp. Sharing | -0.417 | -1.067 | 0.402 |
| Informativeness | -1.917 | -1.467 | 0.673 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Feature | IAA | Human–LLM |
| (Percent Agr.) | (Accuracy) | |
| Main coping strategy | 0.442 | 0.729 |
| Utterance style | 0.528 | 0.852 |
| Resistance level | 0.694 | 0.902 |
| Engagement level | 0.597 | 0.854 |
| Self-disclosure level | 0.564 | 0.827 |
| Category | Configuration |
| Base Model | Meta-Llama-3-8B-Instruct |
| LoRA / | 16 / 16 |
| LoRA Target | All Linear Layers |
| LoRA Dropout | 0.05 |
| Learning Rate | |
| Epochs | 1 |
| Metric | 1–2 | 1–3 | 2–3 | All |
| Linguistic Naturalness | 0.811 | 0.433 | 0.411 | 0.367 |
| Role Authenticity | 0.744 | 0.411 | 0.389 | 0.311 |
| Psychological Plausibility | 0.800 | 0.300 | 0.311 | 0.267 |
| Seeker Simulator | TSNE Hull Area | PCA Hull Area |
| ClientCAST | 2954.562 | 0.312 |
| ESC-Judge | 1878.463 | 0.334 |
| ESC-Role | 3093.516 | 0.378 |
| Eeyore | 3057.489 | 0.441 |
| Ours | 3378.139 | 0.659 |
| Metric | Pearson | Spearman |
| Identification | 0.650 | 0.628 |
| Comforting | 0.589 | 0.546 |
| Suggestions | 0.730 | 0.761 |
| Exp. Sharing | 0.636 | 0.541 |
| Informativeness | 0.807 | 0.794 |
| Metric | Quadratic |
| Identification | 0.595 |
| Comforting | 0.345 |
| Suggestions | 0.658 |
| Exp. Sharing | 0.821 |
| Informativeness | 0.774 |
| Feature | LLM Tagging Prompt |
| Main Coping Strategy | Task. Evaluate the client’s main coping strategy, defined as the single most dominant strategy the client uses across the entire dialogue to manage stress, based only on the client’s utterances. Instructions. (1) Read the entire dialogue. Read the client’s main problem for context only. (2) Base the evaluation only on the client’s utterances. (3) Assign a single coping strategy label that best represents the client’s dominant and consistent approach. Labels. problem_focused : concrete actions or cognitive efforts to address the stressor; emotion_processing : emotional expression or processing without problem solving; avoidant : deliberate avoidance of the problem or emotions; maladaptive_behavior : current self-destructive coping behavior. |
| Engagement Level | Task. Evaluate the client’s overall engagement level throughout the entire dialogue, reflecting how active, serious, and honest the client is in the interaction. Instructions. (1) Read the entire dialogue. Read the client’s main problem for context only. (2) Base the evaluation only on the client’s utterances. (3) Assign a single engagement level representing the client’s overall mode of participation. Labels. high : active and reflective participation; medium : inconsistent or biased engagement; low : minimal or disengaged participation. |
| Resistance Level | Task. Evaluate the overall intensity and consistency of resistance behaviors exhibited by the client across the entire dialogue. Instructions. (1) Read the entire dialogue. Read the client’s main problem for context only. (2) Base the evaluation only on the client’s utterances. (3) Assign a single resistance level reflecting the client’s dominant behavioral pattern. Labels. high : frequent and pervasive resistance; medium : situational resistance triggered by specific topics or interventions; low : generally cooperative with little or no resistance. |
| Utterance Style | Task. Evaluate the client’s dominant utterance style used throughout the dialogue, focusing on how the client speaks rather than what is said. Instructions. (1) Read the entire dialogue. Read the client’s main problem for context only. (2) Base the evaluation only on the client’s utterances. (3) Assign a single utterance style that best characterizes the client’s speaking pattern. Labels. plain : direct, clear, and to-the-point communication without excessive emotion or avoidance; upset : expression of frustration, anger, or strong resistance in a confrontational or dismissive tone; verbose : excessively long and detailed utterances that disrupt conversational flow; reserved : minimal, vague, or evasive responses requiring repeated prompting; tangent : deviation from the main topic into unrelated content; pleasing : a persistent pattern of excessive agreement or problem minimization toward the counselor. |
| Feature | LLM Tagging Prompt |
| Seeker Reaction | Task. Evaluate how the seeker reacts to the counselor’s immediately preceding utterance, focusing only on the seeker’s response rather than the seeker’s general emotional state or situation. Instructions. (1) Read the client’s main problem for context only. (2) Read the counselor utterance and the immediately following seeker utterance. (3) Base the evaluation only on the seeker’s utterance. (4) Assign a single reaction label that best represents the seeker’s response to the counselor. Labels. positive : openness, cooperation, acknowledgment, agreement, appreciation, or constructive engagement with the counselor’s message; neutral : vague, minimal, or non-committal responses without clear acceptance or rejection; negative : rejection, resistance, defensiveness, deflection, or pushback toward the counselor’s message. |
| Self-Disclosure Level | Task. Evaluate the depth of personal information or emotional vulnerability revealed in the seeker’s utterance in response to the counselor’s immediately preceding message. Instructions. (1) Read the client’s main problem for context only. (2) Read the counselor utterance and the immediately following seeker utterance. (3) Base the evaluation only on the seeker’s utterance. (4) Assign a single self-disclosure level reflecting the depth of revealed information. Labels. 1 (orientation) : surface-level, socially normative information without emotional content or personal meaning; 2 (exploratory affective exchange) : personal facts or light emotions without deep vulnerability; 3 (affective exchange) : emotionally meaningful experiences, fears, or vulnerabilities with personal significance; 4 (stable exchange) : identity-level beliefs, core assumptions, or existential conclusions that generalize beyond specific events. |
| System Prompt | You are a summarization assistant. |
| User Prompt | You are an evaluator. You will be provided with a first-person Reddit post written by a seeker. Your task is to write a concise summary that captures the seeker’s situation and main problem. Please follow these steps: 1. Read the post carefully and extract the context (what happened) and the core concern or emotional struggle. 2. Write the summary in the third person (do not use “I”, “my”, etc). 3. The summary must begin with “problem:” 4. Keep the summary to 1–2 sentences. 5. Do not include any information that is not clearly stated in the post. 6. Do not include any names or proper nouns. You may mention age or gender only when it is explicitly mentioned in the post, and never guess or infer it. [Post] {OP text} [Output format] problem: [summary in third person, 1–2 sentences, capturing the situation and the seeker’s emotional or psychological struggle] |
| Model | Coping | Utterance | Engagement | Verbosity | Self-Disclosure | Resistance | Profanity | Turns |
| GPT-4.1-mini | 0.33 | 0.36 | 0.27 | 0.28 | 0.19 | 0.24 | 0.58 | 0.16 |
| Llama-3-8B-Instruct | 0.29 | 0.24 | 0.27 | 0.11 | 0.14 | 0.29 | 0.58 | 0.15 |
| Qwen-2.5-14B-Instruct | 0.25 | 0.32 | 0.29 | 0.33 | 0.22 | 0.24 | 0.47 | 0.15 |
| GPT-5 | 0.23 | 0.33 | 0.27 | 0.23 | 0.25 | 0.25 | 0.84 | 0.15 |
| DeepSeek-V3.2 | 0.39 | 0.40 | 0.41 | 0.52 | 0.27 | 0.37 | 0.91 | 0.18 |
| SFT | 0.38 | 0.36 | 0.53 | 0.66 | 0.40 | 0.37 | 0.66 | 0.56 |
| Metric | Evaluation Instruction |
| Linguistic Naturalness | Definition: How much the client’s utterances sound like a real person in emotional support conversations, rather than language that appears overly refined or AI-generated. Instruction: Assess whether the client’s language resembles natural, spontaneous human speech in emotional support contexts. |
| Role Authenticity | Definition: How well the speaker stays in the role of a person seeking help, rather than sounding like a counselor, evaluator, or detached observer. Instruction: Evaluate whether the client speaks from a first-person, emotionally grounded perspective, without excessive self-analysis or professional terminology. |
| Psychological Plausibility | Definition: Whether the client’s emotional reactions and changes over the course of the dialogue feel realistic for a human in a difficult emotional situation. Instruction: Examine whether emotional shifts occur gradually and proportionally in response to the conversation, rather than resolving abruptly. |
| Comparison | Linguistic Naturalness | Role Authenticity | Psychological Plausibility | ||||||
| Win | Loss | Tie | Win | Loss | Tie | Win | Loss | Tie | |
| Ours vs. Eeyore | 62 | 19 | 9 | 60 | 20 | 10 | 64 | 13 | 13 |
| Ours vs. ESC-Judge | 62 | 18 | 10 | 65 | 19 | 6 | 56 | 14 | 20 |
| Ours vs. ESC-Role | 72 | 9 | 9 | 61 | 16 | 13 | 61 | 12 | 17 |
| Simulator | Lexical Diversity | Semantic Diversity | Sentiment Diversity | ||||
| Distinct-2 | Self-BLEU | TokenRep | APD | ACS | SentMean | SentStd | |
| ClientCAST | 0.3031 | 0.8634 | 0.5329 | 0.5832 | 0.4168 | 0.9569 | 0.2619 |
| ESC-Judge | 0.3577 | 0.8389 | 0.4953 | 0.5543 | 0.4457 | 0.9150 | 0.3287 |
| ESC-Role | 0.4373 | 0.7882 | 0.3559 | 0.7012 | 0.2988 | 0.8948 | 0.3575 |
| Eeyore | 0.5201 | 0.7482 | 0.2687 | 0.6767 | 0.3233 | 0.6087 | 0.6454 |
| Ours | 0.5520 | 0.6901 | 0.2186 | 0.7433 | 0.2567 | -0.0555 | 0.7351 |
| Supporter Model | Seeker Simulator | ES-Skills | General-Skills | ||||||||
| Iden. | Comf. | Sugg. | Expe. | Info. | Cons. | Role. | Expr. | Huma. | Over. | ||
| GPT-5-mini | ClientCAST | 4.983 | 4.990 | 3.913 | 2.097 | 3.563 | 5.000 | 5.000 | 5.000 | 5.000 | 5.000 |
| ESC-Judge | 4.980 | 4.977 | 4.070 | 2.473 | 3.853 | 5.000 | 5.000 | 5.000 | 5.000 | 5.000 | |
| ESC-Role | 4.987 | 4.977 | 3.530 | 2.033 | 3.163 | 4.990 | 4.990 | 4.993 | 4.993 | 4.990 | |
| Eeyore | 4.907 | 4.940 | 3.567 | 2.043 | 3.503 | 5.000 | 5.000 | 4.997 | 4.997 | 4.997 | |
| Ours | 4.410 | 4.477 | 2.820 | 1.367 | 2.393 | 4.933 | 4.963 | 4.860 | 4.830 | 4.853 | |
User Simulation
User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.
See all 143 papers