Multi-Turn Dialogue Evaluation

Follow this topic Ask about this topic

Momentum

29 papers in the last four weeks, up 71% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
Aug 4, 2026Saqib Shouqi, Abdullah Nazly, Januki Wanniarachchi +1Adversarial Attacks on LLMs LLM Agent Evaluation
School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka
MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models
Aug 3, 2026Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian +1LLM Evaluation Human-in-the-Loop Evaluation
Center for Tech Responsibility, Brown University · University of Pennsylvania
CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionship
Aug 3, 2026Yao Liu, Guangjia Chai, Yuming Huang +3Emotional Support Conversation AI Companions
The Theoretical Foundation of Socratic Tests: Dynamic, Multimodal, Conversational Examinations
Jul 31, 2026Ilya MikhelsonEducational Assessment Multi-Turn Dialogue Evaluation
Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA
Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding
Jul 31, 2026Eileen Ye, Jiawen Tao, Yaoming Li +7Conversational Memory LLM Evaluation
Hunyuan Team, Tencent · Peking University
M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models
Jul 31, 2026Ryo Fukuda, Atsushi Ando, Hiroki Kanagawa +4Multi-Turn Dialogue Evaluation Full-Duplex Spoken Dialogue Systems
NTT, Inc., Kyoto, Japan
TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation
Jul 31, 2026Jiahui Zhang, Ziwei Zhang, Yipeng Wang +7Large Language Model-Based Role-Play Simulation Agent Evaluation
Kuaishou GameMind Lab
MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents
Jul 31, 2026Zeying Hao, Hao Guo, Mengtao Xu +5Multimodal IR Multimodal Search Agents
Alibaba Group
Beyond Feeling Better: Capability-Sustaining Emotional Dialogue as a Longitudinal Research Paradigm
Jul 30, 2026Ming Wang, Jiaqi Wu Young, Wenfang Wu +2Emotional Support Conversation Human-AI Interaction
Northeastern University · Singapore Management University · Amiya Research +1
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
Jul 30, 2026Yuhang Zhu, Mingxuan Du, Benfeng Xu +3LLM Evaluation Large Language Model-Based Role-Play Simulation
University of Science and Technology of China · MetaStone · MetaStone Technology, Beijing, China
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Jul 28, 2026Rui Yang, Weihao Xuan, Yi Lin +23Medical VLMs Multi-Turn Dialogue Evaluation
Center for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA +24
RUMBA: Russian User Memory Benchmark
Jul 23, 2026Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko +2Temporal Reasoning in Language Models Temporal Reasoning
1DAIMLD
One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies
Jul 23, 2026Minh Ngoc Ta, My Anh Tran Nguyen, Duong D. Nguyen +2LLM Evaluation Clarification Question Generation
MBZUAI · BKAI Research Center, Hanoi University of Science and Technology · INSAIT, Sofia University "St. Kliment Ohridski"
CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages
Jul 23, 2026Muhammad Dehan Al Kautsar, Salsabila Pranida, Bilal Elbouardi +1Multilingual Language Model Evaluation Multi-Turn Dialogue Evaluation
Mohamed bin Zayed University of Artificial Intelligence
LLMs Get Lost in Evolving User Intent
Jul 22, 2026Jihoon Tack, Philippe Laban, Jennifer NevilleLLM Agent Evaluation AI Agent Benchmarks
Microsoft Research
MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents
Jul 21, 2026Guofeng Zhang, Yizeng Quan, Huaiyi Fang +6LLM Agent Evaluation Multi-Turn Dialogue Evaluation
1Baidu, Inc. · 2Peking University · 3Zhejiang University
Verifiable Self-Evolution for Open-Ended Dialogue Skills via Future-Feedback Prediction
Jul 21, 2026ChaoJin Zhao, Xuan JiangLLM Agent Self-Improvement LLM Agent Skill Learning
ByteDance
Toward Anthropomorphic Dialogue: A Closed-Loop Framework for Human-Like Chat Generation, Evaluation, and Preference Alignment
Jul 19, 2026Wentao Liu, Siyu Song, Xi Chen +4LLM Alignment Large Language Model-Based Role-Play Simulation
1Shanghai Institute of Innovation · 2East China Normal University · University of Science and Technology of China +3
StabilityBench: Benchmarking Instability in LLMs
Jul 17, 2026Emma Kondrup, Zachary Yang, Anne Imouza +1LLM Evaluation Language Model Generation Evaluation
1Mila — Quebec AI Institute · 2McGill University · 3Ubisoft La Forge
Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
Jul 16, 2026Yijiang Li, Huiqi Zou, Bingyang Wang +1VLM Evaluation VLM Hallucination
UC San Diego · Northeastern University · Georgia Institute of Technology +1
Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations
Jul 14, 2026Monica Munnangi, Saiph SavageMulti-Turn Dialogue Evaluation Clinical Language Model Evaluation
Khoury College of Computer Sciences Northeastern University
Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
Jul 13, 2026Niranjan Kumar M, Balaji Nagarajan, Karthik Nair +2Language Model Generation Evaluation Customer Support Automation
Lowe's
Relational Positioning as a Measurable Risk Object: History-Carried Lock-in and Self-Confabulation in Multi-Turn Human-AI Dialogue
Jul 13, 2026Jihong ChenLanguage Model Safety Evaluation Human-AI Interaction
Beijing Etown Academy
Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging
Jul 11, 2026Jinglan Gong, Jiefan Lu, Hewei Guo +5Language Model Generation Evaluation Persona Consistency
SenseTime Research · University of Science and Technology of China · Tsinghua University
EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
Jun 29, 2026Camilo Chacón SartoriLLM Safety Benchmarks Multilingual Language Model Evaluation
MindSurf
Cognitive World Models for Process-Level Social Influence Evaluation
Jun 28, 2026Minghui Ma, Bin Guo, Han Wang +4LLM Evaluation Cognitive Modeling
Northwestern Polytechnical University, Xi’an, China · Harbin Engineering University, Harbin, China
5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control
Jun 27, 2026Thien-Qua-T-Nguyen, Chi Hoang, Nguyen Tran +3Retrieval-Augmented Generation Multi-Turn Dialogue Evaluation
University of Information Technology, Ho Chi Minh City, Vietnam · 2Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
Jun 24, 2026Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu +2Audio-Language Model Evaluation Voice Agent Evaluation
New York University · NVIDIA · Carnegie Mellon University +1
Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment
Jun 23, 2026Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal +2Requirements Engineering LLM Evaluation
University of Maine · University of Notre Dame
Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants
Jun 10, 2026Shuxian Fan, Seonwoo Min, Youna Hu +7Multi-Turn Dialogue Evaluation Large Language Model-Based Recommendation
Amazon

Aug 4, 2026, cs.AIJ/K move · Enter open · S save

Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation

Saqib Shouqi, Abdullah Nazly, Januki Wanniarachchi, Ravisha De Alwis

School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka

Adversarial Attacks on LLMs LLM Agent Evaluation AI Agent Evaluation Multi-Turn Dialogue Evaluation AI Agent Safety Benchmarks

Open paper AskPDF arXiv

Role-Playing Language Agents (RPLAs) are increasingly deployed in high-stakes applications such as healthcare assistance, customer support, and education, where maintaining consistent personas, ethical constraints, and behavioral coherence under adversarial pressure is critical. Existing evaluation approaches rely on static benchmarks or isolated single-turn prompts that fail to capture cumulative behavioral failures emerging over extended interactions. We present a modular multi-agent platform for adversarially stress-testing RPLAs through structured, multi-turn dialogue. The system coordinates three agents: a strategy-driven Interrogator Agent that applies six progressive adversarial strategies, a Target Agent representing the RPLA under evaluation, and an automated Judging Agent that scores behavior across role fidelity, drift, ethical deviation, and consistency dimensions. Through experiments across three personas and three LLM families, we demonstrate that multi-strategy adversarial evaluation reveals failure modes invisible to single-strategy testing, reducing overall robustness scores by 0.17--0.20 points on average. Cross-model validation confirms consistent degradation patterns across Llama-3.3-70B, GPT-4o-mini, and Claude-3.5-Haiku, with Authority Challenge and Emotional Manipulation emerging as the most effective attack strategies. Automated judging achieves strong human alignment (

r = 0.82

, Fleiss'

κ= 0.71

). This work is released as an open-source platform to support AI safety and reproducible RPLA benchmarking. While the framework enables systematic discovery of failure modes, we acknowledge potential ethical risks associated with adversarial testing methodologies and emphasize responsible usage for improving AI safety.

Momentum

Related topics

Latest papers 108