Multi-Turn Dialogue Evaluation

Overview

This field studies how well AI systems handle changing goals, accumulated context, memory, tools, and user behavior over a conversation, where one mistake can shape later turns. Researchers build interactive benchmarks and user simulations for settings such as retrieval, medical consultation, task completion, emotional support, and spoken or multimodal dialogue, often comparing results with matched single-turn tests. Evaluation methods include human ratings and LLM judges, alongside turn- and session-level rubrics, execution checks, state tracking, and measures of error propagation and interaction cost. A central challenge is making evaluations realistic and diagnostically useful while ensuring that scores predict grounded outcomes: fluent or satisfying dialogue can still be wrong, unsafe, or ineffective.

Latest papers

CardsList

225 papers26 papers in the last four weeks, down 38% on the four weeks before. 0.2% of all new papers.

  1. EgoLink: Egocentric Language-Vision Interactive Network Knowledge Challenge Summary

    Nov 16, 2026Renhe Sun, Yueying Feng, Bohan Yu +12Multimodal Agent EvaluationEgocentric Video Understanding

  2. Talk to Act: A Synthetic Multi-Turn Dialogue Benchmark for Human Intent Extraction and Action Planning in Robotics

    Nov 12, 2026Deepu Krishnareddy, Mohammad Haziq Khan, Michael Danner +1Human-Robot InteractionLarge Language Model-Based Robot Planning

  3. The Neighbour's Label: Opposite Subgroup Effects of Same-Dialogue Exposure in Next-Act Evaluation

    Nov 11, 2026Ali Amini, Masoumeh Chapariniya, Rytis Maskeliunas +1Data LeakageMulti-Turn Dialogue Evaluation

  4. ToolTurns: Benchmarking LLM Agents in Multi-Turn Dialogue

    Nov 11, 2026Milad Gholami, Brendan King, Jeffrey FlaniganTool-Use EvaluationLLM Agent Evaluation

  5. Beyond Fluent Responses: Finding User Dissatisfaction in Human-AI Interactions

    Nov 10, 2026Yuka Saito, Ikumi Numaya, Jun Suzuki +1Multi-Turn Dialogue Evaluation

  6. EndoSCOP-V: A Multi-Turn Video Understanding Evaluation Framework for Multimodal Models in Endoscopy Reporting and Clinical Reasoning

    Nov 7, 2026Abhishek Tyagi, Tanmay Jain, Vilas Biradar +15MLLM EvaluationEndoscopy

  7. TurnFlow: Benchmarking User-State Reasoning in Multi-Turn LLM Agents via Scene-Guided User Simulation

    Nov 7, 2026Zhixun Chen, Ming Li, Anjie Liu +3Multi-Turn Language Model ReasoningMulti-Turn Dialogue Evaluation

  8. How Much Conversation History Does an AI Agent Need?

    Nov 6, 2026Danny Kim, Fernando KochLLM Context ManagementConversational Memory

  9. τ-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks

    Nov 4, 2026Kaden Uhlig, Kiarash Mokhtari, Anıl Arslan +2AI Agent BenchmarksLLM Agent Evaluation

  10. CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

    Nov 4, 2026Yuqin Dai, Zhanwei Zhang, Honghui Yang +9Conversational Shopping AssistantsLarge Language Model-Based Recommendation

  11. YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

    Nov 3, 2026Shengyu Wang, Tengyue Nong, Junming Liu +10Multimodal DatasetMulti-Turn Dialogue Evaluation

  12. Do Multi-Turn Conversational Capabilities Carry Over from Text to Speech? A Parallel Multilingual Benchmark

    Nov 3, 2026Jiaming Zhou, Pei Zhang, Shiwan Zhao +3Audio-Language Model EvaluationSpeech Language Models

  13. Can Large Language Models Extend Their Agentic Capabilities to Goal Pursuit in Open-Ended Multi-Turn Interaction?

    Nov 1, 2026Mingyu Chen, Jingkai Lin, Xiaofen Xing +3Tool-Using AgentsMulti-Turn Dialogue Management

  14. DuSE-Bench: Benchmarking Joint Reasoning over Evolving Dialogue and Documents

    Oct 31, 2026Guanjie Cheng, Jianfeng Gao, Yufei Tong +2Evidence-Grounded QAMulti-Turn Dialogue Evaluation

  15. Conversational Interaction for LLM-Based Clinical Text-to-SQL: A User Study with Healthcare Professionals

    Oct 31, 2026Jun-Min Lee, Suyoung Yoo, Gyubok Lee +2Text-to-SQLHuman-Computer Interaction

  16. PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users

    Oct 30, 2026Zheyuan Deng, Taoran Li, Jingwen Zhang +28AI Agent BenchmarksMulti-Turn Dialogue Evaluation

  17. Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation

    Oct 26, 2026Atri Vivek Sharma, Brian Formento, Alessio LomuscioLLM Agent EvaluationUser Simulation

  18. DecisionTraceBench: Tracking Decision-Critical Information in Multi-Party Spoken Conversations

    Oct 26, 2026Wenchao Du, Fangxin Shang, Long Wu +3State TrackingMulti-Turn Dialogue Evaluation

  19. Why Full-Duplex Speech Models Collapse in Long Conversations

    Oct 23, 2026Yeonju Kim, Bella Godiva, Se Jin Park +1Full-Duplex Spoken Dialogue SystemsLong-Context Modeling

  20. SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots

    Oct 21, 2026Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova +3Clinical Language Model EvaluationUser Simulation

  21. Beyond Small Talk: Modeling Disclosure Trajectories in Human-AI Conversation

    Oct 21, 2026Keith Kwok, Wanting Ning, Tej Deep Pala +4Human-AI InteractionMulti-Turn Dialogue Evaluation

  22. DiaLong: Benchmarking Reminiscence in Long-Context Dialogues

    Oct 18, 2026Owen Bianchi, Samer Aslan, Ryunosuke SaitoConversational MemoryLong-Context Language Model Evaluation

  23. SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents

    Oct 16, 2026Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov +1LLM Agent EvaluationMulti-Turn Dialogue Evaluation

  24. MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support

    Oct 15, 2026John Mendonça, Daniel Tschernutter, José Pombal +6Clinical Language Model EvaluationMulti-Turn Dialogue Evaluation