LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,610

All topics

Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
Oct 8, 2026Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan +2Automated Peer Review LLM Evaluation
Sakana AI · National University of Singapore
Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models
Oct 7, 2026Daniel Robert Kling Alexander, Catherine Louise KlingLLM Evaluation
University of Michigan School of Information · Tisch University Professor, SC Johnson College of Business, Cornell University
Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation
Oct 7, 2026Yibei Guo, Rui LiuAdaptive Computation LLM Evaluation
College of Aeronautics and Engineering, Kent State University
Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts
Oct 7, 2026Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao +2Sentiment Analysis Social Media Analysis
Old Dominion University, Norfolk, Virginia, USA
LLM Persuasion Is in the Eye of the Evaluation
Oct 7, 2026Kamile Dementaviciute, Julija Vaitonyte, Tijl De BiePersuasion in Language Models LLM Evaluation
Ghent University · Tilburg University, Department of Computational Cognitive Science · ISM University of Management and Economics
Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2LLM Inference ML Reproducibility
Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1
InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain
Oct 7, 2026Linqi Zhang, Chong Qi, Yan Cheng +4AI-Assisted Decision Making LLM Evaluation
School of Computer Science and Technology, Fudan University · School of Integrated Circuits, Nanjing University · School of Economics, Fudan University
Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
Oct 7, 2026Zhifan Sun, Sebastian Gombert, Jannik Lossjew +6Automated Grading Educational Assessment
DIPF | Leibniz Institute for Research and Information in Education · IPN | Leibniz Institute for Science and Mathematics Education · Umeå University +1
How Do LLMs Change Predictions Under Negation?
Oct 7, 2026Jongwook Yoon, Jongwon Lim, Sungjib Lim +2LLM Reliability LLM Evaluation
Graduate School of Data Science, Seoul National University · Department of Computer Science and Engineering, Seoul National University
RELATE: An Evaluation Framework for measuring Relational Orientation of Large Language Models
Oct 7, 2026Shivam Shukla, Jihye Kim, Shubham Gaur +2LLM Evaluation Emotional Support Conversation
University of California, Santa Cruz · Stanford University
MARS: Malware Analysis with Rule-Based Scoring of LLM Claims
Oct 7, 2026Hyeongjun ChoiMalware Classification LLMs for Cybersecurity
78ResearchLab, 8, Yangpyeong-ro 25-gil, Yeongdeungpo-gu, Seoul, 07207, Republic of Korea
Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science
Oct 7, 2026Benjamin Wilcox, Dawei Gao, Pradeeban Kathiravelu +3LLM Abstention LLM Evaluation
University of North Texas, Denton, TX · University of Alaska Anchorage, Anchorage, AK
The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules
Oct 7, 2026Litao Hu, Yutong TangLLM Self-Refinement Language Model Self-Improvement
Meta · Microsoft
Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions
Oct 6, 2026Wenqi Li, Bin Liu, Mindi Ruan +3LLM Evaluation LLM-as-a-Judge
University at Albany, State University of New York · West Virginia University
sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak
Oct 6, 2026Marek Šuppa, Ivan Vykopal, Andrej Ridzik +9LLM Evaluation Multilingual Language Model Evaluation
Comenius University in Bratislava, Slovakia · Cisco Systems · Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia +2
Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue
Oct 6, 2026Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5LLM Evaluation LLM-Assisted Annotation
College of Computing and Information Science, Cornell University, Ithaca, USA. · College of Connected Computing, Vanderbilt University, Nashville, USA. · Third Space Learning, Swindon, UK.
Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation
Oct 6, 2026Shu-Kai Hsieh, Da-Chen LianMultilingual Language Model Evaluation LLM Evaluation
National Taiwan University
Memory Depth and Reconstructed Context Width: A Controlled Evaluation of Hierarchical Retrieval
Oct 6, 2026Michael AndreevConversational Memory LLM Evaluation
Independent Researcher
Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices
Oct 6, 2026Jinhyeok Kim, Hye-Young JungLLM Evaluation Multiple-Choice Question Answering
Applied Artificial Intelligence Hanyang University, Republic of Korea · Mathematical Data Science Hanyang University, Republic of Korea
The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation
Oct 6, 2026Jorma Valjakka, Juhani Kivimäki, Juha Mylläri +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science University of Helsinki
Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective
Oct 6, 2026Zizhuo Zhang, Xiong Peng, Jingwei Sun +3Contextual Faithfulness LLM Evaluation
TMLR Group, Department of Computer Science, Hong Kong Baptist University · Huawei Noah’s Ark Lab.
OTel: Open Telco AI Datasets, Benchmarks, and Models
Oct 6, 2026Farbod Tavakkoli, Gregory Diamos, Kenneth Church +15Language Model Post-Training LLM Reranking
AT&T Chief Data Office · RelationalAI · Northeastern University +9
Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects
Oct 6, 2026Tianxi Li, Jie DingLLM Evaluation LLM-as-a-Judge
School of Statistics, University of Minnesota
Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods
Oct 6, 2026Haotian Yang, Huikang Jiang, Yucheng Wu +4LLM Evaluation Language Model Steering
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Columbia University +2
HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior
Oct 6, 2026Jin Huang, Diego Ferreras Garrucho, Yutong Xie +4LLM Evaluation LLM Fine-Tuning
University of Michigan, Ann Arbor · London School of Economics · MobLab Inc +1
What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training
Oct 5, 2026Subham Rath, Raj Dandekar, Rajat Dandekar +1LLM Evaluation Language Model Generation Evaluation
Accenture Strategy & Consulting · Vizuara AI Labs (vizuara.com) · Vizuara AI Labs (vizuara.ai)
Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints
Oct 5, 2026Shai Feldman, Yaniv RomanoLLM Evaluation LLM Safety Evaluation
Department of Computer Science, Technion IIT, Israel · Department of Electrical and Computer Engineering, Technion IIT, Israel
CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks
Oct 5, 2026Gowthamkumar NandakishoreContrastive Learning LLM Evaluation
Independent researcher
TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts
Oct 5, 2026Oliver Jaffe, Dane SherburnLLM Evaluation AI Agent Benchmarks
P-Zero Research
ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring
Oct 5, 2026Sait Furkan TekeLLM Evaluation Confidence Estimation in Language Models
ufak AI
JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications
Oct 5, 2026Steven Denney, Matthew DiGiuseppeLLM Evaluation Computational Social Science
Leiden Institute for Area Studies, Leiden University. · Institute of Political Science, Leiden University.
FREA: A Multi-Source Expert Benchmark for Reaction Feasibility Verification
Oct 5, 2026Botao Yu, Bo Zhou, Daniel Adu-Ampratwum +7LLM Evaluation AI-Assisted Scientific Research
Department of Computer Science and Engineering, The Ohio State University · Department of Pharmaceutical Sciences, University of Illinois Chicago · College of Pharmacy, The Ohio State University +3
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Oct 5, 2026Hashmath Shaik, Gnaneswar Villuri, Alex DoboliLLM Evaluation LLM-as-a-Judge
Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Oct 5, 2026Yiqi Liu, Joseph James, Yang Wang +3LLM Evaluation LLM-as-a-Judge
The University of Manchester · The University of Sheffield · University of Pittsburgh +1
Anatomy of LLM Sycophancy: What a Flip Rate Hides
Oct 5, 2026Haonan HuangLLM Evaluation LLM Sycophancy
Princeton University
GraphDecide: Benchmarking System One Models on Graph Tasks
Oct 5, 2026Xianliang Yang, Yapu Zhang, Li ZhaoLLM Evaluation LLM Reasoning with Graphs
Microsoft · Beijing University of Technology
Breaking Bureaucracy: Evaluating open-source LLMs for legal document review
Oct 5, 2026Farrukh Baratov, Niki van Stein, Suzan VerberneZero-Shot Learning Legal NLP
Leiden University Leiden, the Netherlands
MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge
Oct 5, 2026Abdul Basit, Muhammad Abdullah Hanif, Muhammad ShafiqueLLM Evaluation Synthetic Benchmark Generation
eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE
Knowing the Rules, Applying the Rules: Evaluating Language Models on Traditional Chinese Bazi
Oct 5, 2026Jiulin Li, Ping HuangLLM Evaluation Cultural Knowledge in Language Models
Beijing Liuyi Guanhua Technology Co., Ltd. · State Key Laboratory of General Artificial Intelligence, BIGAI
Dataset Signatures in Human-LLM Interactions and User Modeling
Oct 4, 2026Joseph Suh, Serina ChangLLM Evaluation User Modeling
University of California, Berkeley
What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects
Oct 4, 2026Bowen Xu, Boyu ChenLLM Evaluation Benchmark Validity
Department of Biology, Stanford University, Stanford, CA, USA · Institute of Health Informatics, University College London, London, UK
GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation
Oct 4, 2026Murad Hossen, Tasneem Selim, Gurur Gamgam +22LLM Evaluation Graph Neural Networks
University of Houston, Texas,USA · BASIRA Lab, Department of Computing, Imperial College London, United Kingdom · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt +23
VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation
Oct 4, 2026Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav +2LLM Evaluation Code Generation
IBM Research San Jose, CA, USA
MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader
Oct 4, 2026Neeraj YadavLLM Evaluation Memory-Augmented Language Models
Called It Inc.
When Verifiable Counts Depend on Wording: Auditing Wording Robustness in Instruction Following
Oct 4, 2026Qishi Zhan, Seoyeon Jang, Zihan Dong +3LLM Evaluation Instruction Following
Marquette University · University of California, San Diego · Georgia Tech +3
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
Benchmarking candidate coverage and rejection policy transfer in typed decision models
Oct 2, 2026Jiawen Lu, Tongtong WuLLM Evaluation Selective Prediction
Monash University
Where LLMs Fail with Visualization DSLs
Oct 1, 2026Chang Han, Andrew McNutt, Katherine IsaacsData Visualization LLM Evaluation
University of Utah
A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings
Oct 1, 2026Sahil KadadekarPrototype-Based Classification LLM Evaluation
New York University
Which LLM to pick? Online Active Model Selection for Large Language Models
Oct 1, 2026Alessandro Turrin, Patrik Okanovic, Torsten Hoefler +1Model Selection LLM Evaluation
TU Delft · ETH Zurich
MCRI: A Four-Dimensional Framework for Analyzing and Evaluating Agent Skills
Oct 1, 2026Zongrui Yang, Li Xintong, Runchen Xu +4LLM Evaluation Agent Evaluation
When Does a Second Model Help? Cross-Model Review in LLM Verification
Oct 1, 2026Tae-Eun SongLLM Evaluation Language Model Error Detection
Daejeon Jungang Cheonggua Co., Ltd.
Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs
Oct 1, 2026Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein +3Prompt Sensitivity LLM Evaluation
Technion – Israel Institute of Technology
What Wins a Vote? Formatting, Length, and Lexical Diversity in the French Compar:IA LLM Arena
Oct 1, 2026Simonas Zilinskas, Maayeesha Farzana, Christophe BenaventLLM Evaluation
Compar:IA · Université Paris Dauphine-PSL
Evaluating the Robustness of Japanese LLMs to IME-Related and Typographical Errors
Oct 1, 2026Ryota Mibayashi, Hiroaki OhshimaLLM Evaluation Language Modeling
Kobe University, Hyogo, Japan · University of Hyogo, Hyogo, Japan
How Much Can Language Models Gain from Test-Time Computation?
Oct 1, 2026Bangji Yang, Jingyuan Li, Jiajun Fan +8LLM Evaluation Test-Time Scaling
University of Illinois at Urbana-Champaign · University of Washington · Tsinghua University
Evaluating LLM-Generated Preference Distributions
Oct 1, 2026Fan Huang, Minsuk Kim, C. Tyler Diggans +1LLM Evaluation Synthetic Data Generation
Indiana University Bloomington Bloomington, IN, USA · Massachusetts Institute of Technology Cambridge, MA, USA · Root Dynamix LLC Sewickley, PA, USA
A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models
Oct 1, 2026Yingzhu Zhao, Vlad Pandelea, Han Yuan +4LLM Evaluation Financial QA
Global Decision Science, American Express · Singapore Decision Science Center of Excellence, American Express, 1 Marina Boulevard, 018989, Singapore.
VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence
Sep 30, 2026Sachin GuptaLLM Evaluation LLM Grounding
Independent Researcher San Jose, California, USA
How Divergence Becomes Decision Flips in Compressed Language Models
Sep 30, 2026Beatriz Almeida FelicioLLM Evaluation LLM Compression
Instituto de Informática (INF) Universidade Federal de Goiás (UFG)

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku+1

Sakana AI · National University of Singapore

Automated Peer Review LLM Evaluation Language Model Error Detection

Open paper AskPDF arXiv

The rapid growth of scientific publishing has strained peer review, particularly in machine learning, raising concerns about declining review quality and increasing reviewer workload. Large language models (LLMs) have been proposed as automated review assistants, yet their evaluation has focused largely on imitating human-written reviews rather than supporting the core functions of peer review. Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task. We present a scalable benchmark that evaluates review systems' ability to identify logical contradictions, constructed through synthetic insertion of errors into conference papers, yielding unambiguous evaluation targets and enabling systematic comparison. We further propose a Multi-Layered Review (MLR) framework that prioritizes detailed manuscript comprehension before review generation, aligning more closely with human reviewing practices while improving token efficiency. Across evaluations, our approach demonstrates strong alignment with human review scores, achieves high error detection performance, and provides complementary perspectives on reviewer focus. These improvements can be attributed to both the choice of the underlying LLM and the design of our system. At the same time, we corroborate persistent vulnerabilities to adversarial manipulation, underscoring the need for robustness in automated review systems. Our findings highlight the importance of rigorous, error-focused evaluation to guide responsible deployment of LLM-based tools in peer review and other critical scientific workflows.

Momentum

Related topics

Latest papers 1,610