Agent Evaluation

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 229% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 97

All topics

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects
Mar 4, 2026Ji-Lun Peng, Yun-Nung ChenPersonality Modeling in Language Models OOD Generalization
National Taiwan University, Taipei, Taiwan · Academia Sinica, Taipei, Taiwan
MOSAIC: A Universal Agent-Level Interface for Cross-Paradigm Agent Mixing and Human-AI Collaboration
Mar 1, 2026Abdulhamid M. Mousa, Jinhui Pang, Rakhmonberdi Khajiev +5AI Agent Evaluation Agent Evaluation
School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China · Mathematics Department, Faculty of Science, Ain Shams University, Cairo, Egypt +1
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Feb 11, 2026Wayne Chi, Yixiong Fang, Arnav Yayavaram +8Software Engineering Agents Agent Evaluation
Carnegie Mellon University, Pittsburgh, PA, USA · Princeton University, Princeton, NJ, USA
DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems
Jan 20, 2026Maojun Sun, Yifei Xie, Yue Wu +5LLM Agent Evaluation Agent Evaluation
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
Jul 10, 2025Weihao Tan, Changjiu Jiang, Yu Duan +5Game-Playing Agents Agent Evaluation
Nanyang Technological University, Singapore · Harbin Institute of Technology, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China +1
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
Date pendingTara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent Evaluation Spoken Dialogue Systems
ServiceNow AI Research
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Date pendingLin Shi (Audrey), Haowei Lin (Audrey), Zixuan Zhu (Audrey) +123LLM Agent Evaluation AI Agent Evaluation

Mar 4, 2026, cs.CLJ/K move · Enter open · S save

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

Ji-Lun Peng, Yun-Nung Chen

National Taiwan University, Taipei, Taiwan · Academia Sinica, Taipei, Taiwan

Personality Modeling in Language Models OOD Generalization Agent Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs). However, current evaluation frameworks rely heavily on well-known fictional characters, raising a critical concern: models may be leveraging their internal training memory of these characters rather than demonstrating role-playing capabilities. This reliance often leads to significant performance degradation when RPAs encounter unseen or out-of-distribution personas. To address this, we propose a more rigorous evaluation protocol designed to decouple role-playing proficiency from character recognition. Our experiments across multiple benchmarks demonstrate that anonymizing characters degrades performance, confirming that name exposure provides implicit cues that mask a model's true capability. To mitigate this, we investigate diverse personality augmentation as a method to enhance role fidelity in anonymous settings. We systematically analyze the impact of various personality-description methods on agent behavior and consistency. Our results show that incorporating personality information consistently improves RPA performance. This work establishes a more equitable evaluation standard and validates a scalable, personality-enhanced framework for constructing robust RPAs.

Momentum

Related topics

Latest papers 97