Multi-Turn Dialogue Evaluation

Follow this topic Ask about this topic

Momentum

29 papers in the last four weeks, up 71% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 104

All topics

FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses
Jun 3, 2026Nicholas SabanLLM Evaluation Agent Evaluation
Patronus AI · University of California, Berkeley
MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
May 30, 2026Yue Jiang, Xue Jiang, Lihua Zhang +6Multimodal Large Language Models Multimodal Hallucination
College of Intelligent Robotics and Advanced Manufacturing, Fudan University · Southern University of Science and Technology · TMLR Group, Hong Kong Baptist University +3
Personalized Turn-Level User Conversation Satisfaction Benchmark
May 28, 2026Zhefan Wang, Zhiqiang Guo, Weizhi Ma +3Human Preference Evaluation LLM Personalization
Department of Computer Science and Technology, Tsinghua University, Beijing, China. · Institute for AI Industry Research, Tsinghua University, Beijing, China. · Meituan.
DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents
May 28, 2026Rongsheng Zhang, Jiji Tang, Junnan Ren +6Large Language Model-Based Role-Play Simulation Long-Horizon Agent Evaluation
1Zhejiang University · 2Fuxi AI Lab, NetEase Inc. · 3Xiamen University
RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
May 20, 2026Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh +3LLM Evaluation Pairwise Comparison
Layer 6 AI · University of Toronto
AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence
May 20, 2026Kate M. Lubrano, Faisal Sayed, Ankita Rathod +4LLM Evaluation Emotion Recognition in Conversations
1Pareto, California, US · 2Thoughtful, California, US · University of Pennsylvania, Pennsylvania, US
MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
May 20, 2026Junhao Ruan, Abudukeyumu Abudula, Bei Li +8Conversational Search Synthetic Benchmark Generation
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · 2Meituan Inc., 3NiuTrans Research, Shenyang, China · 4Tsinghua University, Beijing, China
SKG-Eval: Stateful Evaluation of Multi-Turn Dialogue via Incremental Semantic Knowledge Graphs
May 15, 2026Avijit Shil, Suman SamuiLLM Evaluation Factual Consistency Evaluation
Maulana Abul Kalam Azad University of Technology, West Bengal, India · National Institute of Technology Durgapur, West Bengal, India
Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
May 10, 2026Aditya Sinha, Harald Steck, Vito Ostuni +1LLM Evaluation Multi-Turn Dialogue Evaluation
Netflix Inc. · Los Gatos, CA, USA
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
Evaluating Temporal Consistency in Multi-Turn Language Models
Apr 24, 2026Yash Kumar Atri, Steven L. Johnson, Tom HartvigsenLLM Evaluation Temporal Reasoning in Language Models
University of Virginia
JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
Mar 24, 2026Yuhui Wang, Zhixiong Yang, Ming Zhang +10LLM Evaluation Multi-Turn Dialogue Evaluation
1Fudan University
When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
Oct 22, 2025Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3Tool-Use Evaluation AI Agent Benchmarks
AWS AI Labs · University of Pittsburgh
Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue
Date pendingWilliam Hager, Ishika Rathi, Masum Hasan +1LLM Evaluation AI-Generated Text Detection
University of Rochester · Independent Researcher · Stony Brook University

Jun 3, 2026, cs.LGJ/K move · Enter open · S save

FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses

Nicholas Saban

Patronus AI · University of California, Berkeley

LLM Evaluation Agent Evaluation Multi-Turn Dialogue Evaluation Language Model Probing

Open paper AskPDF arXiv

Standard benchmarks report aggregate accuracy, but practitioners need to know which specific capabilities a model lacks. We introduce FailureScope, a behavioral-diagnosis method that clusters evaluation probes by their cross-model pass/fail patterns (leave-one-model-out, LOMO), and show it yields stable, interpretable failure taxonomies across three regimes usually studied separately: single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks. On 2,664 single-turn tasks across 18 models, taxonomy-conditioned sampling reaches Kendall's tau = 0.81 at 50 tasks (versus 0.34 for random selection), and cross-model failure prediction reaches AUC 0.88. The same primitive recovers interpretable clusters on a 363-task multi-turn corpus and on 630 adversarial agent traces, where it exposes a meta-failure mode: a 73-100 percentage-point gap between LLM-judge ASR and real execution. Cluster cohesion remains strong across all three regimes, which we take as evidence that behavioral clustering is a portable diagnosis primitive that generalizes beyond any single benchmark. We release the pipeline, three annotated corpora, and the cross-regime taxonomies.

Momentum

Related topics

Latest papers 104