LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions
Aug 31, 2026Ahmed El Kady, Aravind Narayanan, Rehana Noorani +2LLM Evaluation Energy-Efficient ML
Vector Institute · Independent researcher · University of Calgary
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Aug 31, 2026Qiyao Yan, Chenpeng Wang, Liangming PanLLM Evaluation Language Model Calibration
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · YiXin-AILab, YIXIN, Beijing, China +1
Evaluating and Improving LLM Self-Modeling
Aug 31, 2026Siqi Zeng, Andre N. Assis, Rowan WangLLM Evaluation Language Model Introspection
University of Illinois · Constellation · Anthropic
CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text
Aug 31, 2026Mohammad Reza Modarres, Armin Tourajmehr, Yadollah Yaghoobzadeh +1Creativity Assessment LLM Evaluation
School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Tehran Institute for Advanced Studies, Khatam University, Iran · Cardiff University, United Kingdom
Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training
Aug 31, 2026Lukas Borggren, Jenny Kunz, Marco KuhlmannContinual Learning for LLMs LLM Evaluation
Linköping University · Bonnier News
Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle
Aug 31, 2026Dennis Gross, Helge SpiekerLLM Evaluation Faithfulness of Language Model Explanations
Institut für Kommunikations- und Prüfungsforschung gGmbH · Simula Research Laboratory, Oslo, Norway
Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
Aug 31, 2026Ante Kapetanovic, Tomislav Duricic, Andro Mercep +1LLM Evaluation LLM Reranking
Infobip · Split, Croatia · Zagreb, Croatia
ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions
Aug 31, 2026Guangxiang Zhao, Qilong Shi, Xusen Xiao +13LLM Evaluation Scientific Reasoning
1Qiyuan Tech · 2Tsinghua University · 3The University of Hong Kong +1
DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark
Aug 31, 2026Jayanta Sadhu, Sayem Shahad, Kenneth MarinoLLM Evaluation Belief Updating in LLMs
University of Utah · Bangladesh University of Engineering and Technology
Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation
Aug 31, 2026Jinyoung Kim, Muhammad Khalifa, Lajanugen Logeswaran +4RL for Language Models LLM Evaluation
University of Michigan · LG AI Research · University of Illinois at Chicago
Beyond Ranking Accuracy: Evaluating LLM-Cited Feature Rationales for Next Basket Repurchase Recommendation
Aug 31, 2026Yanan Cao, Anay Dombe, Murali Mohana Krishna Dandu +5Sequential Recommendation LLM Evaluation
Walmart Global Tech, Sunnyvale, CA, United States
Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark
Aug 31, 2026Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa +3Multilingual Language Model Evaluation LLM Evaluation
North South University · BRAC University
Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs
Aug 31, 2026Ramya Keerthy Thatikonda, Wray Buntine, Ehsan ShareghiLogical Reasoning LLM Evaluation
Department of Data Science & AI, Monash University · ∃Department of Computer Science, University College London
Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation
Aug 31, 2026Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado PortoLanguage Model Pretraining LLM Evaluation
Laboratório Nacional de Computação Científica (LNCC), Petrópolis, Brazil
Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation
Aug 30, 2026Jhen-Ke LinLLM Evaluation Benchmark Design
National Yang Ming Chiao Tung University
Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models
Aug 30, 2026Hermione Warr, Harry Anthony, Lilli J Freischem +3LLM Evaluation PET/CT Imaging
Department of Engineering Science, University of Oxford, UK · Department of Physics, University of Oxford, UK · Department of Oncology, University of Oxford, UK +1
Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models
Aug 30, 2026Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad +1Multilingual Language Model Evaluation LLM Evaluation
1Perle
Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators
Aug 30, 2026Armaan Singh, Ryan Trinh Le, Jasmine Kaur +5LLM Evaluation LLM Interpretability
1Stanford University · 2York University · 3Facebook
Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting
Aug 30, 2026Bo ChenLLM Evaluation Language Model Fingerprinting
Institute of Computing Technology, Chinese Academy of Sciences
Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation
Aug 30, 2026Nishant Balepur, Paiheng Xu, Wei Ai +3LLM Evaluation Multiple-Choice Question Answering
University of Maryland · New York University · Nanyang Technological University
Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments
Aug 30, 2026Lin Chen, Yitong Chen, Yong LiLLM Evaluation Belief Updating in LLMs
1Network Science Institute, Northeastern University · Department of Physics, Northeastern University · Department of Electronic Engineering, BNRist, Tsinghua University
Evaluating the Capabilities of LLMs for Persuasive Dialogue
Aug 30, 2026Jordan Robinson, Angus R. Williams, Katie Atkinson +1LLM Evaluation Multi-Agent Debate
University of Liverpool · 2The Alan Turing Institute · University of Leeds
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Aug 25, 2026Xiulin Yang, Ethan Gotlieb Wilcox, Catherine ArnettMultilingual Language Model Evaluation LLM Evaluation
Georgetown University · EleutherAI
Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding
Aug 25, 2026Yujing Chang, Thinh Pham, Van-Phat Thai +4LLM Safety Benchmarks LLM Evaluation
1ATMRI, Nanyang Technological University (NTU), Singapore · Centre of AI Research, VinUniversity, Vietnam · School of Management, Indian Institute of Technology Mandi, India
LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Tasks
Aug 24, 2026Xiao Zhang, Qumeng Sun, Jiahao Li +3Agent Memory LLM Evaluation
Infinite Evolution Lab, EvoMap
ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation
Aug 23, 2026Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. AgichteinLLM Evaluation Rubric-Based Evaluation
Department of Computer Science Emory University Atlanta, USA · School of Computer Science University of Waterloo Waterloo, Canada
FlavourBench: Executable Culinary Reward Maps for Language Model Evaluation and Post-Training
Aug 20, 2026Josef Chen, Erim HayretciLLM Evaluation Supervised Fine-Tuning
Independent Researcher · Imperial College London
FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models
Aug 20, 2026Dingzirui Wang, Xuanliang Zhang, Keyan Xu +2LLM Evaluation Automated Theorem Proving
Harbin Institute of Technology
Decomposing Wrong-Consensus Agreement in LLM Self-Consistency
Aug 19, 2026Lizhuo Zhang, Mengmeng Tang, Chenfeng Long +2LLM Evaluation LLM Reliability
College of Information and Intelligence, Hunan Agricultural University, Changsha 410128, China · Yuelushan Laboratory, Changsha 410128, China · School of Computer Science and Technology, Changsha University of Science and Technology, Changsha 410114, China +1
The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
Aug 18, 2026Emma Yanyang Kong, JJ Tan, Ishan Gupta +8LLM Evaluation LLM-as-a-Judge
Netflix · Los Gatos, California, USA

Aug 31, 2026, cs.LGJ/K move · Enter open · S save

Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions

Ahmed El Kady, Aravind Narayanan, Rehana Noorani, Yani Ioannou+1

Vector Institute · Independent researcher · University of Calgary

LLM Evaluation Energy-Efficient ML Benchmark Auditing Language Model Bias Evaluation

Open paper AskPDF arXiv

Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after the evaluation itself is made cheaper. We stress-test conclusion robustness in responsible-AI benchmarking by evaluating three dense and mixture-of-experts models on BBQ and BBQ-V under seven conditions spanning batching, quantization, benchmark reduction, and their combinations. Rather than treating preserved aggregate accuracy as sufficient, we compare accuracy, bias severity and prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy against a full-benchmark BF16 baseline. Larger batching keeps accuracy within 0.35 percentage points of baseline and produces comparatively small subgroup changes, while reducing energy in five of six model--dataset settings. INT8 largely preserves quality but uses 1.79--4.26

\times

baseline energy. INT4 causes larger, model- and context-dependent changes. Reduced benchmarks provide the most consistent savings, but very small subsets are substantially more sensitive to which items are retained. Efficient evaluation should therefore be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. Our project website is https://vectorinstitute.github.io/sustainable-rai-evaluation/ and the code is available at https://github.com/VectorInstitute/sustainable-rai-evaluation.

Momentum

Related topics

Latest papers 1,631