LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Jun 11, 2026Bora Kargi, David SalinasLLM Evaluation Pairwise Comparison
ELLIS Institute Tübingen · OpenEuroLLM
SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection
Jun 11, 2026Fuqiang Niu, Bowen ZhangLLM Evaluation Stance Detection
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China
MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Multilingual Language Model Evaluation LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.
Authority, Truth, and Citation Bias: A Large-Scale Multi-Domain Benchmark for Studying Epistemic Susceptibility in Large Language Models
Jun 11, 2026Aryan Khurana, Aravind Ramana RN, Dhruv KumarLLM Evaluation Hallucination in Language Models
BITS Pilani, Pilani Campus, Rajasthan, India.
The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems
Jun 11, 2026Jiaqi Luo, Jiarun Dai, Zhile Chen +8LLM Evaluation AI Agent Security Benchmarks
Fudan University · Shanghai Artificial Intelligence Laboratory · Concordia AI +1
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Jun 11, 2026Pierre Beckmann, Marco Valentino, Andre FreitasLLM Evaluation Scientific Reasoning
Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK +2
CalBrief: A Pilot Diagnostic Benchmark for Evidence-Calibrated Scientific Briefing with Large Language Models
Jun 11, 2026Yu Fu, Yongqi Kang, Yong ZhaoLLM Evaluation AI-Assisted Scientific Research
Sichuan University
GENIE: A Fine-Grained Measure for Novelty
Jun 11, 2026Ramya Namuduri, Manya Wadhwa, Anshun Asher Zheng +2LLM Evaluation Language Model Generation Evaluation
The University of Texas at Austin · New York University
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
Jun 10, 2026Alan Cooney, David Africa, Geoffrey IrvingLLM Evaluation Deception in Language Models
AI Security Institute
Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generation
Jun 10, 2026Zahra Habibzadeh, Paria Khoshtab, Amir Mesbah +1LLM Evaluation LLM Grounding
Tehran Institute for Advanced Studies, Khatam University, Iran
Measuring Semantic Progress in Multi-turn Dialogue via Information Gain
Jun 10, 2026Paul He, Shiva Kasiviswanathan, Dominik JanzingLLM Evaluation Multi-Turn Dialogue Evaluation
NTU Singapore · Amazon Research, Tübingen, Germany · Amazon
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Jun 10, 2026Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7LLM Evaluation LLM Reliability
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4
A Controlled Study of Decoding-Time Truthfulness Methods on Instruction-Tuned LLMs
Jun 10, 2026Ao SunLLM Evaluation Language Model Decoding
Independent Researcher
Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation
Jun 10, 2026Selen Erkan, Bastian Boll, Kristian Kersting +2LLM Evaluation Instruction Following
Aleph Alpha Research Lab1141 · TU Darmstadt Hessian.AI Lab1141
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Jun 10, 2026Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff +4Creativity Assessment LLM Evaluation
1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University +1
Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment
Jun 10, 2026Yijie Deng, He Zhu, Wen Wang +3LLM Evaluation Urban Planning
Behavioral and Spatial AI Lab, Tongji University · Behavioral and Spatial AI Lab, Peking University · College of Architecture and Urban Planning, Tongji University
Are LLMs Bad at Moral Reasoning?
Jun 10, 2026Menghang Zhu, Seth LazarLLM Evaluation Moral Reasoning in Language Models
School of Philosophy (Political Philosophy) Renmin University of China · School of Government and Policy Johns Hopkins University
GraphInfer-Bench: Benchmarking LLM's Inference Capability on Graphs
Jun 10, 2026Zhuoyi Peng, Jingzhou Jiang, Hanlin Gu +2LLM Evaluation Graph Neural Networks
The Hong Kong University of Science and Technology · Webank
HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification
Jun 9, 2026Maohua Nie, Jiang Zhu, Jingqun Zhang +5Formal Hardware Verification LLM Evaluation
University of Washington · Independent Researcher
MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation
Jun 9, 2026Yukuan Zhang, Mengxin Zheng, Qian LouSecure Multi-Party Computation LLM Evaluation
University of Central Florida
Schützen: Evaluating LLM Safety in Bulgarian and German Contexts
Jun 9, 2026Kiril Georgiev, Yuxia Wang, Dimitar Iliyanov Dimitrov +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI
Flaws in the LLM Automation Narrative
Jun 9, 2026George Perrett, Javae Elliott, Jennifer Hill +1LLM Evaluation Code Generation
New York University
PhantomBench: Benchmarking the Non-existential Threat of Language Models
Jun 9, 2026Haeji Jung, Hila GonenLLM Evaluation Hallucination in Language Models
University of British Columbia · Canada CIFAR AI Chair, Amii
Janus: A Benchmark for Goal-Conditioned Information Distortion in LLMs
Jun 9, 2026Polydoros Giannouris, Mohsinul Kabir, Sophia AnaniadouLLM Evaluation Deception in Language Models
Department of Computer Science, National Center for Text Mining, The University of Manchester
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval
Jun 9, 2026João Maria Janeiro, Mathurin Videau, Andrea Caciolai +3LLM Evaluation Multiple-Choice Question Answering
FAIR at Meta · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Criteo AI Lab, Paris, France
Benchmarking Knowledge Editing using Logical Rules
Jun 9, 2026Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera +1Knowledge Editing LLM Evaluation
Data Science Group, Heinz Nixdorf Institute, Paderborn University
KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
Jun 9, 2026Sanghee Park, Geewook Kim, Kee-Eung KimVLM Evaluation Mathematical Reasoning Benchmarks
NAVER Cloud AI · KAIST AI
From Context-Aware to Conflict-Aware: Generalizing Contrastive Decoding for Knowledge Conflict in LLMs
Jun 9, 2026Runze Jiang, Taiqiang Wu, Yan Wang +2LLM Evaluation Knowledge Conflicts in Language Models
1Peking University · 2Alibaba Group · 3The University of Hong Kong
Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations
Jun 9, 2026Aniket Anand, Yiwei Hou, Daniel Fields +4LLM Evaluation Cybersecurity
University of Chicago · USA · University of California, Berkeley +1
RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning
Jun 8, 2026Yiteng Mao, Kenan Xu, Yijia Lyu +3Mathematical Reasoning Benchmarks LLM Evaluation
University of Wisconsin–Madison · East China Normal University · New York University +2

Jun 11, 2026, cs.LGJ/K move · Enter open · S save

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

Bora Kargi, David Salinas

ELLIS Institute Tübingen · OpenEuroLLM

LLM Evaluation Pairwise Comparison LLM-as-a-Judge Pairwise Preference Evaluation Pairwise Ranking Language Model Calibration

Open paper AskPDF arXiv

Evaluating new large language models typically requires costly human annotation campaigns at scale. LLM-as-a-judge offers a cheaper alternative, but judge scores carry systematic errors - such as position bias, self-preference, or intransitivity - that can strongly miscalibrate the resulting rankings. We quantify the resulting judge-human disagreement at two complementary levels. At the local level, we estimate per-battle uncertainty from the judge's own score differences by propagating calibrated win probabilities rather than hard labels into the Bradley-Terry procedure. This alone provides a drastic improvement to Elo estimation accuracy, bringing LLM-derived ratings within 17.9 Elo MAE of human-derived ones when averaged over 55 held-out models on LMArena. At the global level, we apply split conformal prediction to the residual gap between LLM-derived and human-derived Elo ratings across held-out models, producing prediction intervals with distribution-free marginal coverage guarantees that account for irreducible LLM-human disagreement. Together, these two layers yield a low-cost evaluation tool that provides developers with calibrated Elo estimates and honest uncertainty bounds, without access to large-scale human annotations. To facilitate reproducibility, we release our code at https://github.com/kargibora/SoftElo .

Momentum

Related topics

Latest papers 1,643