LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

Know It, Act on It: Investigating Memory Utilization in LLM Personalization
Jul 31, 2026Zhaoxin Feng, Jianfei Ma, Emmanuele ChersoniLLM Personalization LLM Agent Memory
The Hong Kong Polytechnic University
MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraft
Jul 31, 2026Jianxin Gao, Beini Hu, Runze Li +6Game-Playing Agents LLM Agent Evaluation
China Agricultural University, Beijing, China · Beijing Normal University, Beijing, China · Jilin University, Changchun, China +2
Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs
Jul 31, 2026Ruoxi Zhao, Maziar RaissiAI Coding Agents LLM Agent Evaluation
University of California, Riverside
MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
Jul 31, 2026Qiming Shi, Yulong Tao, Linbo Jin +10LLM Agent Evaluation Long-Horizon Agent Evaluation
2State Key Lab of CAD&CG, Zhejiang University · 1Alibaba Group · School of Software and Microelectronics, Peking University +2
Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
Jul 30, 2026Shimaa Ahmed, Yiwei Cai, Mohsen Minaei +1LLM Agent Evaluation AI Agent Benchmarks
Visa Research
Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?
Jul 30, 2026Xiaonan Xu, Wenjing WuAutomated Software Testing Automated Program Repair
College of Computing, Georgia Institute of Technology, Atlanta, GA, 30332, USA. · Department of Computer Science, University of Colorado Boulder, Boulder, CO, 80309, USA.
Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
Jul 30, 2026Harsh Raj, Vipul Gupta, Anas Mahmoud +4Agent Failure Analysis LLM Agent Evaluation
Scale AI
Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game
Jul 30, 2026Niklas Bauer, Lars Benedikt Kaesberg, Akiko Aizawa +3Multi-Agent LLM Systems Social Deduction Games
University of Göttingen, Germany · National Institute of Informatics, Japan · University of Tokyo, Japan
ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
Jul 30, 2026Xingjian Wu, Xuhang Zhu, Xingchen Liu +6AI Agent Reliability LLM Agent Evaluation
Meituan
DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
Jul 30, 2026Debin Meng, Jiaming Yang, Zefang Zong +4LLM Agent Evaluation Automated Evaluation
Tencent Shenzhen, China · Xidian University Xi’an, China · Tencent Beijing, China
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
Jul 30, 2026Yuhang Zhu, Mingxuan Du, Benfeng Xu +3LLM Evaluation Large Language Model-Based Role-Play Simulation
University of Science and Technology of China · MetaStone · MetaStone Technology, Beijing, China
Evaluating Agentic Bioinformatics through Function, Evidence, and Validation
Jul 30, 2026Phuc Pham, Truong-Son HyAI Agents for Scientific Discovery Data Provenance
Department of Computer Science, The University of Alabama at Birmingham, USA
Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents
Jul 29, 2026Jinwei Hu, Yi Qi, Xinmiao Huang +3LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science and Informatics, University of Liverpool, UK · Department of Computer Science, University of Leeds, UK · Department of Computer Science, Mohamed bin Zayed University of Artificial Intelligence, UAE
VAmoS Bench: Voice Agent Simulation Bench
Jul 29, 2026Joshua Meyer, Sahar Shayegan, Ritiz Tambi +5Voice Agent Evaluation Customer Support Automation
Veris AI
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
Jul 29, 2026Jingbo Zhou, Yusai Zhao, Qi Bao +12Computer-Use Agent Benchmarks Computer-Use Agents
Agent Frontier Team, Large Model Frontier Research Department, Baidu Inc., China.
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
Jul 29, 2026Jinhu Qi, Wentao Zhang, Siu Man Ng +4Tool-Use Evaluation LLM Agent Evaluation
The Chinese University of Hong Kong Hong Kong, China · Macao Polytechnic University Macau, China
Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models
Jul 29, 2026Ashish Prajapati, Om MohiteMulti-Agent LLM Systems LLM Self-Refinement
Independent Researcher Mumbai, India
SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
Jul 29, 2026Lehan Wang, Boli Chen, Ruixue Ding +7LLM Agent Evaluation Cybersecurity
Tongyi Lab, Alibaba Group · Alibaba Cloud Computing, Alibaba Group · The Hong Kong University of Science and Technology
When LLM Agents Negotiate: Private Information and Dynamic Bargaining in Supply Chains
Jul 29, 2026Chen Liang, Fasheng XuMulti-Agent Negotiation Automated Negotiation
School of Business, University of Connecticut
Exploring Structures in Physics Problems: Can AI Agents Discover Statistical Mechanical Mappings?
Jul 29, 2026Wanyu Zhao, Wanbing ZhaoAI Agents for Scientific Discovery LLM Agent Evaluation
Department of Physics and Astronomy Rice University
RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
Jul 28, 2026Fanqing Meng, Lingxiao Du, Qiguang Chen +4LLM Agent Evaluation LLM Agent Self-Improvement
1Evolvent AI · 2National University of Singapore
Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
Jul 28, 2026Prakhar KhatriAI Coding Agents LLM Agent Evaluation
Independent Researcher
Everyone is unique: Towards Behaviorally Heterogeneous Negotiation Dialogue Systems for Debt Collection
Jul 28, 2026Yuhang Yang, Kai Tang, Chao Ye +4Automated Negotiation LLM Agent Evaluation
1State Key Laboratory of Blockchain and Data Security, Zhejiang University · 2Ant Group
CogEEGAgent: Toward Autonomous Cognitive EEG Analysis with Grounded Execution and Selection-Aware Verification
Jul 27, 2026Dengzhe Hou, Lingyu Jiang, Fangzhou Lin +1Electroencephalography Selective Inference
1Graduate School of Information Sciences, Tohoku University · 2Unprecedented-scale Data Analytics Center, Tohoku University · 3Texas A&M University +1
Delegation Intelligence in Deep Search: A Controllable Framework for Disentangled Capability Diagnosis
Jul 26, 2026Xinhao Yao, Yuanzhuo Liu, Changhao Wang +6Deep Research Agents LLM Agent Evaluation
Tencent · Renmin University of China · UCAS +1
AllocBench: Measuring Online Tool Allocation Capability in LLM Agents
Jul 25, 2026Daniel Wang, Andrew XuTool-Augmented Language Model Agents AI Coding Agents
Yale University
SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows
Jul 25, 2026Summer SunLLM Agent Evaluation AI Agent Benchmarks
Shaqiu Community
The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
Jul 24, 2026Darshan Tank, Baran NamaLLM Agent Evaluation LLM Agent Skill Learning
Sentient Labs
AI Assistants Overassist
Jul 23, 2026Verona Teo, Raghav Jain, Tobias Gerstenberg +1Intelligent Tutoring Systems LLM Agent Evaluation
1Stanford University · University of California, San Diego · University of Washington
Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions
Jul 23, 2026Pengyu Zhu, Lijun Li, Longju Yang +2Deep Research Agents RAG Poisoning Attacks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 3Chongqing University of Posts and Telecommunications

Jul 31, 2026, cs.CLJ/K move · Enter open · S save

Know It, Act on It: Investigating Memory Utilization in LLM Personalization

Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni

The Hong Kong Polytechnic University

LLM Personalization LLM Agent Memory LLM Agent Evaluation User Preference Modeling

Open paper AskPDF arXiv

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.

Momentum

Related topics

Latest papers 793