LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
May 28, 2026Yundong Kim, Heyoung YangLLM Evaluation Language Model Generation Evaluation
Applied Agent Research Center, Korea Institute of Science and Technology Information (KISTI), Republic of Korea · Department of Computer Science and Engineering, University of Seoul, Republic of Korea.
FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification
May 28, 2026Silu PandaLLM Evaluation Language Model Calibration
From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals
May 28, 2026Yeyong Yu, Wenya Hu, Xing Wu +1LLM Evaluation Pairwise Preference Evaluation
School of Computer Engineering & Science, Shanghai University, Shanghai, 200444, China. · Center of Materials Informatics and Data Science, Materials Genome2026 Institute, Shanghai University, Shanghai, 200444, China. · Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education, China. +1
When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs
May 28, 2026Shuai Xiao, Su Liu, Weikai Zhou +4LLM Evaluation LLM Prompting
Independent Researchers
Slogans or Stance? A Label-Light Diagnostic for Entrepreneurial-Discourse Measurement on Chinese SOE Speeches
May 27, 2026Ting Gong, Shangquan SunLLM Evaluation
Tsinghua University
UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning
May 27, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Legal NLP
SecondLayer
SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
May 27, 2026Kaihua Qin, Dawn Song, Arthur GervaisLLM Evaluation Differential Testing
University of Warwick · UC Berkeley · University College London
The Routing Plateau: Understanding the Accuracy Limits of LLM Routers
May 27, 2026Yifan Lu, Qiyue Zhang, Shenrun Zhang +4LLM Evaluation Adaptive Model Routing
Rice University · Amazon
Mind Your Tone: Does Tone Alter LLM Performance?
May 27, 2026Om Dobariya, Akhil KumarLLM Evaluation LLM Prompting
Pennsylvania State University University Park, PA, USA
When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis
May 27, 2026Aisha Najera, Alvin Moon, Vedant Srinivasan +1LLM Evaluation Annotator Disagreement
AI Lab, Princeton University · Engineering and Applied Sciences, RAND Corporation · Science, Technology, and International Affairs, Georgetown University
The Trust Paradox: How CS Researchers Engage LLM Leaderboards
May 27, 2026Pouya Sadeghi, Anamaria Crisan, Jimmy LinModel Selection LLM Evaluation
University of Waterloo Waterloo, Canada
PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
May 27, 2026Yangyi Huang, Ruotian Peng, Zeju Qiu +4LLM Evaluation Fine-Tuning
The Chinese University of Hong Kong · Westlake University · MPI for Intelligent Systems
MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems
May 27, 2026Xinle Deng, Ruobin Zhong, Hujin Peng +15LLM Evaluation Memory-Augmented Language Models
Zhejiang University · Alibaba Group
The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic
May 27, 2026Dominika Agnieszka Długosz, Arlindo Oliveira, Natalia Díaz-RodríguezLLM Evaluation Generalized Linear Models
Instituto Superior Técnico & INESC-ID, Universidade de Lisboa, Portugal · Dept. of Computer Science and AI & DaSCI Institute, Universidad de Granada, Spain
Measuring Form and Function in Language Models
May 27, 2026Héctor Javier Vázquez Martínez, Charles YangLLM Evaluation Child Language Acquisition
University of Pennsylvania Department of Linguistics and Computer and Information Science
Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability
May 27, 2026Leizhen Zhang, Shuhan Chen, Sheng ChenLLM Evaluation Constraint Satisfaction
University of Louisiana at Lafayette, Lafayette, USA · East China Normal University, Shanghai, China
MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation
May 27, 2026Xiaoyu Dong, Zhi Li, Xiao-Ming WuLLM Evaluation Text-to-CAD Generation
1The Hong Kong Polytechnic University · 2Curvature Flow Co., Limited
HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs
May 27, 2026Yansong Ning, Mianpeng Liu, Jingwen Ye +2LLM Evaluation Adaptive Inference
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · AIPD, Tencent
Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach
May 27, 2026Nicolás Benjamín Ocampo, Agnes Paullate Nyiranziza, Davide CeolinLLM Evaluation Pairwise Ranking
1Centrum Wiskunde & Informatica, The Netherlands · 2Vrije Universiteit Amsterdam, The Netherlands
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
May 27, 2026Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach +6Legal NLP LLM Evaluation
Technical University of Munich (TUM) · Ludwig Maximilian University of Munich (LMU) · University of Konstanz +1
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
May 27, 2026Quanen Sun, Changxin Tian, Ke Shi +6LLM Evaluation Language Model Scaling Laws
Ant Group
ATLAS: All-round Testing of Long-context Abilities across Scales
May 27, 2026Deli Huang, Cunguang Wang, Hongyin Tang +15LLM Evaluation Benchmark Construction
Meituan · Fudan University
PetroBench: A Benchmark for Large Language Models in Petroleum Engineering
May 27, 2026Xiang Wang, Tingting Zhang, Sen Wang +4LLM Evaluation Question Answering
School of Petroleum and Natural Gas Engineering, Changzhou University, Changzhou 213164, China · China University of Petroleum (East China), Qingdao, Shandong 266580, China
The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates
May 27, 2026Shaobo Wang, Guo Chen, Ziyue Wang +5LLM Evaluation Language Model Decoding
Shanghai Jiao Tong University · Qwen Team, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations
May 27, 2026Sachin KumarLLM Evaluation Language Model Safety Evaluation
LexisNexis, USA
Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm
May 27, 2026Yuming, Huang, Yao Liu +3LLM Evaluation Benchmark Validity
Rapheal · Cylingo team
SuiChat-CN: Benchmarking Contextual Suicide Risk Assessment in Chinese Group Chats
May 27, 2026Xiangyu Wang, Zhiwei Yu, Chengze Du +3Social Media Analysis LLM Evaluation
University of Chinese Academy of Sciences · 2Tsinghua University · 3Beijing University of Posts and Telecommunications +1
FinBoardBench: Benchmarking Dynamic Wealth Management and Strategic Financial Reasoning of LLMs via Board Game Simulations
May 27, 2026Xuesi Hu, Peng Wang, Jinpeng Miao +7LLM Evaluation Finance
School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China · School of Economics, Anhui University, Anhui, China · SKLPlanets, Macau University of Science and Technology, Macau, China +2
PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management
May 27, 2026Yuxuan Zhao, Sijia Chen, Ningxin SuLLM Evaluation Quantitative Finance
1Yantai Research Institute of Harbin Engineering University · 2The Hong Kong University of Science and Technology (Guangzhou)
ChildEval: When large language models meet children's personalities
May 27, 2026Yanyan Luo, Xue Han, Chunxu Zhao +5LLM Evaluation LLM Personalization
JIUTIAN Research, China Mobile, Beijing, China

May 28, 2026, cs.AIJ/K move · Enter open · S save

TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation

Yundong Kim, Heyoung Yang

Applied Agent Research Center, Korea Institute of Science and Technology Information (KISTI), Republic of Korea · Department of Computer Science and Engineering, University of Seoul, Republic of Korea.

LLM Evaluation Language Model Generation Evaluation CoT Evaluation Structured Reasoning LLM Reasoning Reasoning Trace Analysis

Open paper AskPDF arXiv

Evaluating open-ended outputs from large language models (LLMs) remains challenging due to the absence of ground truth. Existing metrics rely on final-answer accuracy or surface-level statistics, leaving the reasoning process itself unexamined. We introduce TRACE (Toulmin-based Reasoning Assessment through Constructive Elements), a metric that analyzes Chain-of-Thought (CoT) reasoning processes. Rather than judging outcomes, TRACE inspects how arguments are constructed by integrating Toulmin's argumentation theory with Flavell's metacognitive framework to assess reasoning structure. Experiments on 26.3K QA samples across 7 reasoning models show strong correlation with benchmark accuracy (r=0.74). Furthermore, TRACE is effective as a reinforcement learning reward signal, outperforming accuracy-only baselines. Together, these results indicate that logically sound reasoning leads to higher-quality answers. TRACE thus serves as a complementary metric for evaluating open-ended outputs. Code is available at https://github.com/hyyangkisti/trace.

Momentum

Related topics

Latest papers 1,643