Multi-Turn Dialogue Evaluation

Overview

This field studies how well AI systems handle changing goals, accumulated context, memory, tools, and user behavior over a conversation, where one mistake can shape later turns. Researchers build interactive benchmarks and user simulations for settings such as retrieval, medical consultation, task completion, emotional support, and spoken or multimodal dialogue, often comparing results with matched single-turn tests. Evaluation methods include human ratings and LLM judges, alongside turn- and session-level rubrics, execution checks, state tracking, and measures of error propagation and interaction cost. A central challenge is making evaluations realistic and diagnostically useful while ensuring that scores predict grounded outcomes: fluent or satisfying dialogue can still be wrong, unsafe, or ineffective.

Latest papers

CardsList

225 papers26 papers in the last four weeks, down 38% on the four weeks before. 0.2% of all new papers.

  1. Eval4Sim: An Evaluation Framework for Persona Simulation

    Mar 3, 2026Eliseo Bao, Anxo Perez, Javier Parapar +1Large Language Model-Based Role-Play SimulationLarge Language Model-Based Social Simulation

  2. EMPA: Evaluating Persona-Aligned Empathy as a Process

    Feb 28, 2026Shiya Zhang, Yuhan Zhan, Ruixi Su +4LLM Agent EvaluationMulti-Turn Dialogue Evaluation

  3. Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems

    Feb 23, 2026Mukul Chhabra, Luigi Medrano, Arush VermaRAG EvaluationMulti-Turn Dialogue Evaluation

  4. LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces

    Feb 20, 2026Peter Kirgis, Ben Hawriluk, Sherrie Feng +3Language Model Safety EvaluationLLM Safety Benchmarks

  5. AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue

    Feb 19, 2026Adib Sakhawat, Fardeen Sadab, Rakin ShahriarPartially Observable Stochastic GamesMulti-Turn Dialogue Evaluation

  6. ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models

    Feb 17, 2026Manav Nitin Kapadnis, Lawanya Baghel, Atharva Naik +1Multimodal Model EvaluationAutomated Evaluation

  7. CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue Quality with Conversational Dynamics

    Jan 15, 2026Viet Cuong Nguyen, Nhi Yen Nguyen, Kristin A. Candan +5Large Language Model-Based Role-Play SimulationSynthetic Data Generation for Language Models

  8. DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation

    Dec 23, 2025Ziyi Zhu, Olivier Tieleman, Caitlin A. Stamatis +5User SimulationAdversarial Training

  9. When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

    Oct 22, 2025Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3Tool-Use EvaluationAI Agent Benchmarks

  10. KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues

    Sep 26, 2025Junhao Chen, Yu Huang, Siyuan Li +7Knowledge-Intensive QAFactual Knowledge in Language Models

  11. AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs

    Sep 9, 2025Hoang Nguyen, Sidharth Surapaneni, Akshay Kalkunte +9Audio-Language Model EvaluationAudio-Language Models