Comparing Human Role-Players and LLM-Simulated Clients in Online Counselling Training: An Analysis of Counselling Patterns

LLM-based client simulation provides a scalable approach to novice counselor training, counseling-dialogue synthesis, and interactive evaluation of automated counseling systems. However, existing approaches are limited by insufficient profile diversity, weak behavioral grounding, and the lack of open Chinese-language resources for simulated counseling clients. We propose PsyCLIENT, a framework that conditions simulated-client responses on client profiles, dialogue histories, and conversational trajectories specifying target behaviors and content constraints at each client turn. We also construct PsyCLIENT-CP, a dataset of 120 Chinese client profiles spanning 60 counseling topics. Evaluations involving 24 professional counselors show that PsyCLIENT receives higher ratings of perceived authenticity and training utility than the comparison methods. In a separate source-identification study, PsyCLIENT dialogues were more frequently misclassified as human-client interactions than those generated by the baselines. These results suggest that conversational trajectory modeling can transform static client profiles into behavior-guided, dynamically unfolding counseling interactions.

Large language models (LLMs) are increasingly being used as ad hoc therapists. While prior research has found that LLMs outperform human counselors in generating single-turn empathetic responses, fewer studies have compared their behaviors across multi-turn sessions. In this study, we compare the session-level behaviors of human peer counselors with those of an LLM, both trained on the same manual to deliver multi-turn, single-session Cognitive Behavioral Therapy (CBT). Our three-phase, mixed-methods study involved: (a) an 18-month ethnography of a peer support platform, where seven counselors iteratively refined CBT prompts through 110 self-counseling sessions and 60 weekly focus groups; (b) a novel session generation method that allows direct, controlled comparison of human and LLM counselors under matched conditions---client responses were drawn from publicly available human-led CBT sessions while counselor responses were generated by a CBT-prompted LLM; and (c) expert evaluations conducted by three licensed clinical psychologists. Through data triangulation, our results show a trade-off. Human peer counselors use relational techniques to interpret subtle cues, adapt CBT to users' values and cultural contexts, and use strategies such as small talk and contextually relevant self-disclosure to build rapport and guide the session, but often at the expense of session structure and therapeutic focus. LLM counselors, on the other hand, demonstrate greater methodological adherence to CBT techniques, but struggle to sustain turn-taking, frequently fail to distinguish between clinically important and trivial content, and are more prone to lecturing and imposing solutions. LLM counselors also tend to produce ``deceptive empathy'', excessively anthropomorphic responses that can inflate user expectations of genuine human care.

Large language models (LLMs) may support counseling training, yet evidence from Japanese-language interactions and automated quality ratings remains limited. We examined 18 fixed Japanese-language counseling transcripts generated through artificial intelligence (AI)-to-AI interactions under three counselor conditions: GPT-minimal (GPT-4-turbo with a minimal role instruction), GPT-SMDP (GPT-4-turbo with the Structured Multi-step Dialogue Prompt [SMDP]), and Claude-SMDP (Claude-3-Opus with SMDP). Fifteen counseling experts rated transcripts on four adapted global scales from the Motivational Interviewing Treatment Integrity coding manual and an overall-quality item; three newer LLMs independently rated the same transcripts in three iterations. In this fixed stimulus set, SMDP-condition dialogues received higher expert ratings for cultivating change talk, partnership, empathy, and overall quality than GPT-minimal dialogues; the two SMDP counselor models did not differ. LLM ratings were reproducible but generally more lenient than expert-reference ratings, particularly for softening sustain talk and overall quality. Simulated-client naturalness was below the scale midpoint. These findings provide an expert-referenced benchmark for Japanese-language AI counseling simulations and show that reproducible LLM ratings should not be treated as calibrated counseling-quality evidence without expert validation. This study does not test clinical effectiveness or human-client outcomes.