Multi-Turn Dialogue Evaluation

Ask about this topic

Overview

This field studies how well AI systems handle changing goals, accumulated context, memory, tools, and user behavior over a conversation, where one mistake can shape later turns. Researchers build interactive benchmarks and user simulations for settings such as retrieval, medical consultation, task completion, emotional support, and spoken or multimodal dialogue, often comparing results with matched single-turn tests. Evaluation methods include human ratings and LLM judges, alongside turn- and session-level rubrics, execution checks, state tracking, and measures of error propagation and interaction cost. A central challenge is making evaluations realistic and diagnostically useful while ensuring that scores predict grounded outcomes: fluent or satisfying dialogue can still be wrong, unsafe, or ineffective.

Latest papers

CardsList

203 papers37 papers in the last four weeks, level with the four weeks before. 0.2% of all new papers.

  1. AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

    Mar 11, 2026Dimosthenis Athanasiou, Maria Lymperaiou, Giorgos Filandrianos +2Retrieval-Augmented GenerationRAG Evaluation

    National Technical University of Athens

  2. Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce

    Mar 11, 2026Liang Chen, Qi Liu, Wenhuan Lin +1Rubric-Based EvaluationMulti-Turn Dialogue Evaluation

  3. MM-tau-p2^2: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

    Mar 10, 2026Anupam Purwar, Aditya ChoudharyMultimodal Agent EvaluationAgent Evaluation

    Sprinklr AI, Gurugram, India

  4. StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control

    Mar 8, 2026Haishu Zhao, Aokai Hao, Yuan Ge +3Controllable Speech GenerationSpeech Language Models

    NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

  5. MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?

    Mar 6, 2026Lin Yang, Yuancheng Yang, Xu Wang +2Long-Context Language Model EvaluationClinical Language Model Evaluation

    ByteDance

  6. MPCEval: A Benchmark for Multi-Party Conversation Generation

    Mar 5, 2026Minxing Zhang, Yi Yang, Zhuofan Jia +5Multi-Turn Dialogue EvaluationConversational Agents

    Duke University USA · Tanka AI USA

  7. AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments

    Mar 5, 2026Kylie Zhang, Nimra Nadeem, Lucia Zheng +2Large Language Model-Based Role-Play SimulationLanguage Model Generation Evaluation

    Princeton University, Princeton, NJ, USA · Stanford University, Stanford, USA

  8. Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

    Mar 3, 2026Alejandro Breen Herrera, Aayush Sheth, Steven G. Xu +8Multi-Agent LLM SystemsLLM Agent Workflow Optimization

    Metis · DoorDash

  9. Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systems

    Mar 3, 2026Raad Khraishi, Iman Zafar, Katie Myles +1LLM ReliabilityMulti-Turn Dialogue Evaluation

    NatWest AI Research · University College London

  10. Eval4Sim: An Evaluation Framework for Persona Simulation

    Mar 3, 2026Eliseo Bao, Anxo Perez, Javier Parapar +1Large Language Model-Based Role-Play SimulationLarge Language Model-Based Social Simulation

    IRLab, CITIC, Universidade da Coruña A Coruña, Spain · University of Sheffield Sheffield, United Kingdom

  11. Quantifying Conversational Reliability of Large Language Models under Multi-Turn Interaction

    Mar 2, 2026Jiyoon MyungTool Selection for Language Model AgentsLLM Reliability

    Samsung SDS

  12. EMPA: Evaluating Persona-Aligned Empathy as a Process

    Feb 28, 2026Shiya Zhang, Yuhan Zhan, Ruixi Su +4LLM Agent EvaluationMulti-Turn Dialogue Evaluation

    Team Echo, Nature Select · Sun Yat-sen University

  13. MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games

    Feb 27, 2026Jacob Eisenstein, Fantine Huot, Adam Fisch +2LLM EvaluationMulti-Turn Dialogue Evaluation

    Google DeepMind

  14. MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

    Feb 26, 2026Sara Rosenthal, Yannis Katsis, Vraj Shah +3Retrieval-Augmented GenerationRAG Evaluation

    IBM Research, USA

  15. Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems

    Feb 23, 2026Mukul Chhabra, Luigi Medrano, Arush VermaRAG EvaluationMulti-Turn Dialogue Evaluation

    Dell Technologies, Austin, TX, USA

  16. LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces

    Feb 20, 2026Peter Kirgis, Ben Hawriluk, Sherrie Feng +3Language Model Safety EvaluationLLM Safety Benchmarks

    Princeton University · Phillips Exeter Academy · Independent

  17. AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue

    Feb 19, 2026Adib Sakhawat, Fardeen Sadab, Rakin ShahriarPartially Observable Stochastic GamesMulti-Turn Dialogue Evaluation

    Department of Computer Science and Engineering, Islamic University of Technology, Dhaka, Bangladesh

  18. ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models

    Feb 17, 2026Manav Nitin Kapadnis, Lawanya Baghel, Atharva Naik +1Multimodal Model EvaluationAutomated Evaluation

    Language Technologies Institute Carnegie Mellon University

  19. CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue Quality with Conversational Dynamics

    Jan 15, 2026Viet Cuong Nguyen, Nhi Yen Nguyen, Kristin A. Candan +5Large Language Model-Based Role-Play SimulationSynthetic Data Generation for Language Models

    Georgia Institute of Technology · Northwell Health · Columbia University

  20. When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

    Oct 22, 2025Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3Tool-Use EvaluationAI Agent Benchmarks

    AWS AI Labs · University of Pittsburgh

  21. KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues

    Sep 26, 2025Junhao Chen, Yu Huang, Siyuan Li +7Knowledge-Intensive QAFactual Knowledge in Language Models

    Hong Kong University of Science and Technology (Guangzhou) · The University of Osaka · Hong Kong University of Science and Technology

  22. AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs

    Sep 9, 2025Hoang Nguyen, Sidharth Surapaneni, Akshay Kalkunte +9Audio-Language Model EvaluationAudio-Language Models

    ServiceNow · University of Texas at Austin

  23. Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey

    Mar 28, 2025Shengyue Guan, Jindong Wang, Jiang Bian +3LLM Agent EvaluationMulti-Turn Dialogue Evaluation

    Microsoft, China