LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity
Jun 5, 2026Zifan Lyu, Chahine Nejma, Tobias Wegel +2Model Selection Multi-Armed Bandits
1ETH Zurich · 2Centrale Sup´elec · 3ENS de Cachan +1
Sycophantic Praise: Evaluating Excessive Praise in Language Models
Jun 5, 2026Daniel Vennemeyer, Phan Anh Duong, Meryl Ye +2LLM Evaluation LLM Sycophancy
University of Cincinnati · Carnegie Mellon University · Texas A&M University
The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs
Jun 5, 2026Yang Zhang, Xiao Fei, Amr Mohamed +6Multilingual Language Model Evaluation LLM Evaluation
Ecole Polytechnique · MBZUAI · ENS-PSL +1
Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese
Jun 5, 2026Xing Yue, Yongliang Shen, Weiming LuLLM Evaluation
Zhejiang University
Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models
Jun 5, 2026Dewi Gould, Francis Rhys Ward, Anders Cairns Woodruff +18LLM Evaluation AI Safety Evaluation
Redwood Research · Astra Fellows Program · Aether Research +10
Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling
Jun 5, 2026Xing Yue, Linjuan Wu, Daoxin Zhang +2Reward Modeling LLM Evaluation
Zhejiang University · Xiaohongshu Inc.
MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights
Jun 5, 2026Yilun Liu, Miao Zhang, Shimin Tao +9Multilingual Language Model Evaluation LLM Evaluation
Huawei, China
Adversarial Robustness of Activation Steering in Large Language Models
Jun 5, 2026Kien Le, Thai LeLLM Evaluation Language Model Steering
Independent Researcher · Indiana University
OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios
Jun 5, 2026Xinyi Li, Zhen Fang, Yongxin Deng +12LLM Evaluation Hallucination Detection
University of Technology Sydney · University of Bristol · University of Wisconsin–Madison +2
From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing
Jun 5, 2026Guannan Lai, Haoran Hu, Long Chen +2LLM Evaluation LLM Routing
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong University of Science and Technology +1
ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning
Jun 5, 2026Vladislav Smirnov, Chieu Nguyen, Sergey Senichev +14LLM Evaluation Test-Time Scaling
1MBZUAI · 7Independent Researcher · 2ETH Zürich +4
Evidence-Grounded Ensemble Diagnosis of 802.11 Packet Captures: A Multi-Stage Pipeline with Deterministic Reliability Scoring
Jun 5, 2026Jerome Henry, Swadhin Pradhan, Miroslav PopovicLLM Evaluation Evidence-Grounded Reasoning
Cisco Systems
Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses
Jun 5, 2026Indu Panigrahi, Tal AugustLLM Evaluation Language Model Generation Evaluation
Siebel School of Computing and Data Science University of Illinois Urbana-Champaign
When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding
Jun 4, 2026Zixian He, Bharath Raahul Murugesan, Patrick Brandt +1LLM Evaluation LLM-Assisted Annotation
Independent Researcher · Illinois Institute of Technology · The University of Texas at Dallas
Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions
Jun 4, 2026Haizhou XiaLLM Evaluation LLM Grounding
University of Western Ontario, Canada
Evaluating LLM Usage for Efficient and Explainable Numerical and Classified Implicit Sentiment Analysis of Product Desirability
Jun 4, 2026Sherri Weitl-Harms, John HastingsLLM Evaluation Sentiment Analysis
Creighton University, 2500 California Plz, Omaha, NE, 68178, United States · The Beacom College of Computer & Cyber Sciences, Dakota State University, 820 N Washington Ave., Madison, SD, 57042, United States
RECAP: Regression Evaluation for Continual Adaptation of Prompts
Jun 4, 2026Harsh Deshpande, Kushal Chawla, Sangwoo Cho +2Continual Learning for LLMs LLM Evaluation
Capital One
UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
Jun 4, 2026Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4LLM Evaluation Language Model Generation Evaluation
University of British Columbia · 2Independent Researcher
Re-Centering Humans in LLM Personalization
Jun 4, 2026Lechen Zhang, Jiarui Liu, Tal AugustHuman Preference Evaluation LLM Evaluation
†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University
Benchmark Everything Everywhere All at Once
Jun 4, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5LLM Evaluation Benchmark Design
MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen +3
A Komi-Yazva--Russian Parallel Corpus and Evaluation Protocol for Zero- and Few-Shot LLM Translation
Jun 4, 2026Petr ParshakovLLM Evaluation Low-Resource MT
HSE University, Perm, Russia · School of Management SKOLKOVO, Moscow, Russia
Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness
Jun 4, 2026Victor De Marez, Luna De Bruyne, Walter DaelemansLLM Evaluation LLM Sycophancy
Centre for Computational Linguistics, Psycholinguistics and Sociolinguistics University of Antwerp Antwerp, Belgium
FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays
Jun 4, 2026Yijun Liu, Yifan Song, John Gallagher +2LLM Evaluation
University of Illinois Urbana-Champaign Urbana, IL, USA
Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs
Jun 4, 2026Giovanni Dettori, Matteo Boffa, Danilo Giordano +2Long-Context Retrieval LLM Evaluation
Department of Computer Science Politecnico di Torino Torino, IT 10129 · Department of Computer Science University of Turin Torino, IT 10124
RedditPersona: A Modular Framework for Community-Conditioned LLM Adaptation from Reddit
Jun 4, 2026Amirhossein Ghaffari, Ali Goodarzi, Huong Nguyen +3LLM Evaluation LLM Fine-Tuning
Future Computing Group University of Oulu Oulu, Finland · Centre for Applied Computing University of Oulu Oulu, Finland
Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
Jun 4, 2026Karolina Drożdż, Micha HeilbronLLM Evaluation Language Model Scaling Laws
IDEAS Research Institute; Warsaw, Poland · Max Planck Institute for Psycholinguistics; Nijmegen, Netherlands · University of Amsterdam, Amsterdam Brain and Cognition; Amsterdam, Netherlands
Analysis of the Neglect-Zero Effect in Large Language Models
Jun 4, 2026Jin Tanaka, Daiki Matsuoka, Ryoma Kumon +1Logical Reasoning LLM Evaluation
The University of Tokyo · RIKEN · Tohoku University
Benchmarks in Leipzig
Jun 4, 2026Andrei Balakin, Miklós Bóna, Marie-Charlotte Brandenburg +45Mathematical Reasoning Benchmarks LLM Evaluation
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4LLM Evaluation LLM-as-a-Judge
1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 3Shanghai Innovation Institute, Shanghai 200231, China
Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation
Jun 4, 2026Arslan Bisharat, Brian Ortiz, Eric Spencer +5LLM Evaluation Formal Verification
Department of Computer Science, Loyola University Chicago, Chicago, IL 60660, USA

Jun 5, 2026, cs.LGJ/K move · Enter open · S save

Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity

Zifan Lyu, Chahine Nejma, Tobias Wegel, Fanny Yang+1

1ETH Zurich · 2Centrale Sup´elec · 3ENS de Cachan+1

Model Selection Multi-Armed Bandits LLM Evaluation Language Model Generation Evaluation Best-Arm Identification

Open paper AskPDF arXiv

Large Language Models are typically benchmarked by evaluating every model on every test query. For practitioners seeking the best model to deploy, this is often wasteful: if a model clearly performs worse than others, there is no need to precisely estimate its performance. Best-arm identification algorithms can be naturally applied to drastically reduce costs by adaptively allocating evaluation budget. Further, language models often respond similarly to the same prompt-a property previous work has tried to leverage with mixed success. We propose Synchronized Successive Rejects (SySRs), augmenting the classical Successive Rejects algorithm with paired comparisons. Unlike prior attempts to leverage model similarity in best-model identification, our approach is hyperparameter-free and enjoys performance guarantees that improve with the degree of similarity between evaluated models. Empirically, our method outperforms all baselines in terms of average error rate across 15 standard benchmarks, and in terms of worst-case budget for reliably identifying the best model.

Momentum

Related topics

Latest papers 1,643