To address the critical scarcity of high-quality, publicly available counseling dialogue datasets, we created Multilingual KokoroChat by translating KokoroChat, a large-scale manually authored Japanese counseling corpus, into both English and Chinese. A key challenge in this process is that the optimal model for translation varies by input, making it impossible for any single model to consistently guarantee the highest quality. In a sensitive domain like counseling, where the highest possible translation fidelity is essential, relying on a single LLM is therefore insufficient. To overcome this challenge, we developed and employed a novel multi-LLM ensemble method. Our approach first generates diverse hypotheses from multiple distinct LLMs. A single LLM then produces a high-quality translation based on an analysis of the respective strengths and weaknesses of all presented hypotheses. The quality of ``Multilingual KokoroChat'' was rigorously validated through human preference studies. These evaluations confirmed that the translations produced by our ensemble method were preferred from any individual state-of-the-art LLM. This strong preference confirms the superior quality of our method's outputs. The Multilingual KokoroChat is available at https://github.com/UEC-InabaLab/MultilingualKokoroChat.
Figures & tables
Dataset
Method
Open-source
Language
Dialogues
Avg. utterances
HealMe Xiao et al. (2024)
LLM
✓
En
1,300
6.0
ESD-CoT Zhang et al. (2024)
LLM
✓
En
1,708
23.4
C actus Lee et al. (2024)
LLM
✓
En
31,577
31.5
SMILECHAT Qiu et al. (2024)
LLM
✓
Zh
55,165
33.2
A ug ESC Zheng et al. (2023)
LLM
✓
En
65,000
26.7
Anno-MI Wu et al. (2022)
Human
✓
En
133
72.9
Table 1: Comparison of psychological counseling datasets categorized by construction method: LLM-based (top), human-collected (middle), and hybrid (bottom). We further build and release a multilingual version based on the high-quality human-collected dataset KokoroChat . Note that the slight reduction in the number of dialogues for the multilingual versions is due to the exclusion of content that triggered the LLM’s safety filters.
– This is Japanese text counseling data from role-playing sessions where counselors acted as both counselor and client.
– Each line is separated by a colon (’:’). The left side indicates the role name, and the right side is the utterance.
# Translation Instructions
– As a professional translator, translate this data into {target language} .
– For the translation, please use polite and natural expressions that are appropriate for a counseling context.
Table 2: Prompt for Hypothesis Generation
# Data Description
– This data includes Japanese text counseling data and its { target language } translation candidates.
– Japanese text counseling data was collected through role-playing between counselors acting as counselor and client.
# Input Data Format
– The input is a list of dictionary objects.
## Keys of each dictionary
Table 3: Prompt for Integration and Refinement
System
MetricX↓
XCOMET↑
GPT
3.409 ∗
0.921 ∗
Gemini
3.356 ∗
0.927 ∗
Grok
3.369 ∗
0.927
Proposed
3.345
0.926
Table 4: Automatic Evaluation Results for Japanese-to-English Translation. ∗ indicates a statistically significant performance difference p<0.01 compared to the proposed method.
System
MetricX↓
XCOMET↑
GPT
2.547 ∗
0.859 ∗
Gemini
2.483 ∗
0.870 ∗
Qwen
2.476 ∗
0.871
Proposed
2.445
0.871
Table 5: Automatic Evaluation Results for Japanese-to-Chinese Translation. ∗ indicates a statistically significant performance difference p<0.01 compared to the proposed method.
Figure 2: Human Evaluation Results for Japanese-to-English Translation
Figure 3: Human Evaluation Results for Japanese-to-Chinese Translation
Figure 4: Comparison of Japanese-to-English translation where the proposed method was judged inferior to Grok. This judge resulted from a contextual mismatch inherent in the randomized experimental design: the use of "pathetic" for internal consistency conflicted with the term "uncool" in the context randomly assigned to the evaluators.
Figure 5: Japanese-to-Chinese translation example 1. This demonstrates the analysis and synthesis process that produced a final translation highly preferred by human evaluators over any of the three single-LLM hypotheses.
Figure 6: Japanese-to-Chinese translation example 2. This demonstrates the analysis and synthesis process that produced a translation judged inferior by human evaluators to Qwen’s hypothesis due to being slightly unnatural as a conversational text. The refiner LLM rejected Qwen’s hypothesis for semantic deviation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: A case where contextually natural translation is penalized by utterance-level evaluation. The translation "It is" captures the conversational flow but be judged as semantically incomplete by reference-free metrics that do not account for the preceding context.
Figure 8: A case where contextually natural translation is penalized by utterance-level evaluation. While the translation elegantly resolves a sentence split, utterance-by-utterance evaluation incorrectly penelize them.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Existing translation models are typically trained on sentence-level and formal text, limiting their ability to capture everyday conversational dialogue phenomena such as informality, speaker interaction, and discourse coherence. Most existing Indic translation resources and evaluation benchmarks focus on sentence-level or formal text, making it difficult to assess translation quality of the dialogue phenomena. In this work, we introduce BaatCheet, a multilingual dialogue corpus named after the Hindi term for conversation or chitchat, containing approximately 49,000 dialogues for dialogue translation across five translation directions. We fine-tune five open-source LLMs across seven training data configurations and find that fine-tuning yields substantial gains over zero- and few-shot baselines. To comprehensively evaluate dialogue translation quality, we employ multiple evaluation strategies, including automatic metrics, LLM-as-judge, and human assessments using an SQM-guided Direct Assessment (DA) Protocol.
Priyanka Dasari, Yuvrajsinh D. Bodana, Vandan Mujadia +3
Language Technologies Research Centre, IIIT Hyderabad, India
Large language models (LLMs) may support counseling training, yet evidence from Japanese-language interactions and automated quality ratings remains limited. We examined 18 fixed Japanese-language counseling transcripts generated through artificial intelligence (AI)-to-AI interactions under three counselor conditions: GPT-minimal (GPT-4-turbo with a minimal role instruction), GPT-SMDP (GPT-4-turbo with the Structured Multi-step Dialogue Prompt [SMDP]), and Claude-SMDP (Claude-3-Opus with SMDP). Fifteen counseling experts rated transcripts on four adapted global scales from the Motivational Interviewing Treatment Integrity coding manual and an overall-quality item; three newer LLMs independently rated the same transcripts in three iterations. In this fixed stimulus set, SMDP-condition dialogues received higher expert ratings for cultivating change talk, partnership, empathy, and overall quality than GPT-minimal dialogues; the two SMDP counselor models did not differ. LLM ratings were reproducible but generally more lenient than expert-reference ratings, particularly for softening sustain talk and overall quality. Simulated-client naturalness was below the scale midpoint. These findings provide an expert-referenced benchmark for Japanese-language AI counseling simulations and show that reproducible LLM ratings should not be treated as calibrated counseling-quality evidence without expert validation. This study does not test clinical effectiveness or human-client outcomes.
Japan National Institute of Occupational Safety and Health, Kanagawa, Japan · Kaze To Taiyo, Tokyo, Japan · Saga Occupational Health Association, Saga, Japan +5
Large Language Models (LLMs) have transformed conversational AI, yet high-quality multilingual code-mixed dialogue resources remain scarce, particularly for Indic languages where speakers naturally alternate between English and their native language in both native-script and Romanized forms. We present IndicTalk, one of the largest multilingual Indic code-mixed conversational corpora, comprising over 13,28,604 event-grounded multi-turn conversations across 18 language varieties covering 9 Indic languages. The corpus is generated through a fully automated pipeline that combines real-world news grounding, persona-conditioned dialogue generation using multilingual LLMs, and automatic quality validation. Extensive linguistic, automatic, and human evaluations demonstrate that IndicTalk produces fluent, coherent, and naturally code-mixed conversations across both script variants. We will release IndicTalk to support the development and evaluation of multilingual conversational AI for underrepresented Indic languages. The dataset is available at: https://huggingface.co/datasets/LingoIITGN/IndicTalk .