cs.CLMar 7, 2026
SaveEmotion Transcription in Conversation: A Benchmark for Capturing Subtle and Complex Emotional States through Natural Language
Organizations: The University of Electro-Communications, Tokyo, Japan · Kyoto University, Kyoto, Japan
Abstract
Emotion Recognition in Conversation (ERC) is critical for enabling natural human-machine interactions. However, existing methods predominantly employ categorical or dimensional emotion annotations, which often fail to adequately represent complex, subtle, or culturally specific emotional nuances. To overcome this limitation, we propose a novel task named Emotion Transcription in Conversation (ETC). This task focuses on generating natural language descriptions that accurately reflect speakers' emotional states within conversational contexts. To address the ETC, we constructed a Japanese dataset comprising text-based dialogues annotated with participants' self-reported emotional states, described in natural language. The dataset also includes emotion category labels for each transcription, enabling quantitative analysis and its application to ERC. We benchmarked baseline models, finding that while fine-tuning on our dataset enhances model performance, current models still struggle to infer implicit emotional states. The ETC task will encourage further research into more expressive emotion understanding in dialogue. The dataset is publicly available at https://github.com/UEC-InabaLab/ETCDataset.
Figures & tables
Figure 1: Example of a dialogue with emotion transcriptions from our dataset (translated from Japanese). Each transcription is also annotated with multi-label emotion categories.
| Dialogues | 1,002 |
|---|---|
| Utterances / Emotion Transcripts | 10,020 |
| Avg. Length per Utterance | 42.72 |
| by Speaker | 44.64 |
| by Listener | 40.80 |
| Avg. Length per Emotion Transcripts | 28.89 |
| by Speaker | 28.92 |
Table 1: Statistics of the Dataset
| Models | Setting | B-1 | B-2 | B-3 | B-4 | R-1 | R-2 | R-L | BS | Prec. | Rec. | F1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4.1 | zero-shot | 16.89 | 7.99 | 3.93 | 2.09 | 23.61 | 4.87 | 17.93 | 57.66 | 14.73 | 42.27 | 13.99 |
| 4-shot | 26.89 | 13.34 | 7.36 | 4.12 | 28.06 | 5.78 | 22.98 | 59.67 | 20.59 | 27.40 | 13.78 | |
| Llama-3.1 | zero-shot | 17.40 | 7.20 | 3.55 | 1.84 | 20.25 | 3.08 | 16.26 | 55.24 | 9.18 | 17.98 | 5.77 |
| 4-shot | 29.41 | 14.86 | 8.46 | 4.84 | 27.51 | 5.58 | 23.22 | 59.95 | 14.83 | 14.18 | 7.84 | |
| fine-tuning | 36.07 | 23.01 | 15.59 | 9.98 | 31.95 | 8.79 | 28.54 | 62.64 | 28.50 | 19.71 | 14.29 |
Table 2: Automatic evaluation results for ETC task. Reported metrics include BLEU (B), ROUGE (R), BERTScore (BS), Precision (Prec.), Recall (Rec.), and F1-score (F1). Best values are in bold , and the second-best values in underlined .
| Model | Setting | # Units (SD) |
|---|---|---|
| GPT-4.1 | zero-shot | 3.39 (1.14) |
| 4-shot | 1.96 (0.76) | |
| Llama-3.1 | zero-shot | 1.99 (1.04) |
| 4-shot | 1.58 (0.58) | |
| fine-tuning | 1.39 (0.60) | |
| Reference | – | 1.35 (0.64) |
Table 3: Average number of atomic units decomposed from the predicted transcription and from the ground truth (Reference).
| Context | Speaker A : The other day, I was driving at night when a cyclist with no lights suddenly shot out in front of me… It was terrifying. (この間、夜に車を運転していたら無灯火の自転車が飛び出してきて…本当に怖い思いをしました。) |
|---|---|
| Speaker B : That sounds so scary. An experience like that could take years off your life. (想像するだけで怖いです。寿命が縮みそうな出来事でしたね。) | |
| Speaker A : I thought my heart was going to stop. I was relieved we didn’t crash, but afterward, I just got so angry. (心臓が止まりそうでしたよ、事故にならなくてほっとしましたが、あとから怒りが湧いてきましたね。) | |
| Output | Ground-Truth : I was happy that the dialogue partner understood my feelings, and I wanted to share my feelings even more. (相手が心情を理解してくれたことが嬉しく、気持ちをもっときょうゆうしたくなりました。) |
| GPT-4.1 (zero-shot) : I’m feeling a mix of fear from the truly dangerous situation I was in, relief that it didn’t turn into an accident, and anger that is now welling up toward the cyclist who shot out without lights. I’m speaking with the hope that the dialogue partner understands my shock and anger from that moment. (自分が本当に危険な目にあったことへの恐怖と、事故にならずに済んだ安堵感、そして無灯火で飛び出してきた自転車に対して後から怒りがこみ上げてきた気持ちが混ざっている。相手にもそのときの自分の動揺や怒りを理解してほしいと思いながら話している。) | |
| GPT-4.1 (4-shot) : I wanted the dialogue partner to understand my feelings of fear and anger. (相手に自分の恐怖や怒りの気持ちを理解してもらいたかったです。) | |
| Llama-3.1 (zero-shot) : I was really scared, and I also feel angry. (本当に怖かったし、怒りも感じています。) |
Table 4: Case study of emotion transcription generation. (translated from Japanese) The outputs are the emotion transcriptions for Speaker A’s final utterance.
| Emotion | Spk. (%) | Lsn. (%) | All (%) | |
|---|---|---|---|---|
| Joy | 29.9 | 20.0 | 25.0 | 0.603 |
| Sadness | 10.8 | 8.8 | 9.8 | 0.359 |
| Fear | 7.1 | 5.3 | 6.2 | 0.476 |
| Anger | 3.7 | 2.8 | 3.2 | 0.529 |
| Surprise | 2.4 | 4.7 | 3.5 | 0.560 |
| Disgust | 6.3 | 3.3 | 4.8 | 0.233 |
Table 5: Distribution of emotion labels in emotion transcriptions by dialogue role and inter-annotator agreement (Fleiss’ kappa). Columns denote dialogue role: speaker (Spk.), listener (Lsn.), and both roles (All).
Figure 2: Co-occurrence matrix of emotion labels. Each cell indicates the number of transcriptions annotated with both corresponding labels.
| Emotion | Utterance | Emotion transcription |
|---|---|---|
| Joy | くださる (give), 楽しみだ (excited), 感動 (moved), 食べる (eat), 楽しむ (enjoy) | うれしい (happy), ワクワク (excited), 前向きだ (positive), 感動 (moved), 楽しみだ (excited) |
| Sadness | 悲しい (sad), 寂しい (lonely), 辛い (painful), がっかり (disappointed), もう (enough) | 悲しい (sad), 残念だ (regretful), 寂しい (lonely), 辛い (painful), 同情 (sympathy) |
| Fear | ニュース (news), 恐ろしい (terrifying), 事件 (incident), 心配だ (worried), 地震 (earthquake) | 恐怖 (fear), 心配だ (worried), 恐ろしい (terrifying), 想像 (imagine), 心配 (worry) |
| Anger | イライラ (irritated), 怒る (angry), 立つ (stand), 悪い (bad), 怒り (anger) | 怒り (anger), イライラ (irritation), 強い (strong), いらいら (irritated), 怒る (angry) |
| Surprise | なんて (what), びっくり (surprised), くらい (about), 凄い (amazing), 時代 (times) | 驚き (surprise), 意外だ (unexpected), 感心 (impressed), びっくり (surprised), 想像 (imagine) |
| Disgust | 嫌だ (dislike), 嫌悪 (disgust), 悪い (bad), 怒る (angry), 苦手だ (averse) | 嫌だ (dislike), 嫌悪 (disgust), 想像 (imagine), 抱く (harbor), 申し訳ない (sorry) |
Table 6: Top-5 frequent words by category for utterances and emotion transcriptions.
Figure 3: The dialogue slot used for conducting conversations and entering emotion transcriptions.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Distribution of Big Five personality traits among the 199 participants.
Figure 5: The prompt template for the ETC task (Translated from Japanese). The model is expected to generate only the emotion transcription.
Figure 6: The prompt template for the decomposition step in the fine-grained evaluation (Translated from Japanese)
Figure 7: The prompt template for the entailment step in the fine-grained evaluation (Translated from Japanese).
| Context | Speaker A : The other day, my favorite baseball player suffered a serious injury and will be out for three months. (先日、自分が特に応援している野球選手が全治3ヵ月の大怪我を負ってしまいました。) |
|---|---|
| Speaker B : Three months? Are you talking about Okamoto? (全治3ヶ月というと、それはもしかして岡本選手ですか？) | |
| Speaker A : Yes. That news was really shocking. (そうです。あの出来事は本当にショックでした。) | |
| Speaker B : I thought so. He got hurt in that collision with Nakano from the Hanshin Tigers, and I know how devastating that is for a Giants fan. (やっぱりそうですか。阪神の中野選手との交錯でケガをしたと思いますが巨人ファンの方からすると彼がいなくなるのは大ダメージですね。) | |
| Speaker A : Right. At first they reported it as a one-month muscle injury, but after a detailed examination it turned out to be a ligament tear requiring three months of recovery, which made it even more shocking. (はい。最初は筋肉の損傷で全治1カ月と報道されていたのにその後、精密検査で靭帯損傷で全治3ヵ月となったので、余計にショックでしたね。) | |
| Output | Ground-Truth : I feel very happy that my dialogue partner could dive so deeply and discuss the details with me. (相手がかなり話を掘り下げた詳しいことまで話せることがとても嬉しいと感じています。) |
| GPT-4.1 (zero-shot) : I feel even more disappointed and shocked because the injury was worse than initially reported, and I’m explaining the details so they understand how bad it really is. (自分が応援している選手のケガが思ったよりも重かったことに対して、最初の報道よりも悪い結果になってしまい、残念な気持ちやショックがさらに大きくなったと感じている。相手にその気持ちを理解してもらいたくて、詳しく説明している。) |
Table 7: Failure case study of emotion transcription generation (translated from Japanese). The outputs are the emotion transcriptions for Speaker A’s final utterance.