LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Predicting Inference-Time Scaling Gains from Labeled Validation-Set Output Statistics
Jun 2, 2026Luyang Zhang, Jingyan LiLLM Evaluation Inference-Time Scaling
Carnegie Mellon University · Johns Hopkins University
Linguistic Productivity in Large Language Models: Models Coerce, but do not Preempt
Jun 1, 2026Claire Bonial, Claire Benet Post, Laura Michaelis +1LLM Evaluation Language Modeling
Georgetown University · University of Colorado Boulder · University of Bath
Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models
Jun 1, 2026Simone Caldarella, Davide Talon, Rahaf Aljundi +2LLM Evaluation Overthinking in Language Models
University of Trento · 3Fondazione Bruno Kessler · 2Toyota Motor Europe
Translating Classical Poetry into Modern Prose
Jun 1, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation Machine Translation
Computational Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
Acceptance-Test-Driven Evaluation Protocols for Business-Centric LLM Systems
Jun 1, 2026Eric LiangLLM Evaluation Language Model Safety Evaluation
Oracle
Greener Than Humans? Environmental Attitudes in Large Language Models
Jun 1, 2026Stefanie Kunkel, Tilman Hartwig, Marcus Voss +2LLM Evaluation Language Model Robustness
Research Institute for Sustainability at GFZ Centre for Geosciences, Berliner Straße 130, 14467 Potsdam, Germany · German Environment Agency, Wörlitzer Platz 1, 06844 Dessau-Roßlau, Germany · Birds on Mars, Marienstraße 10, 10117 Berlin, Germany
LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability
Jun 1, 2026Aastha Sapkota, M. G. Sarwar MurshedLLM Evaluation LLM-as-a-Judge
Computer Science Department University of Wisconsin - Green Bay, Green Bay, WI, USA
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Jun 1, 2026Nahyun Lee, Dongkeun Yoon, Guijin Son +12Computer-Use Agent Benchmarks LLM Evaluation
Chung-Ang University1 · KAIST2 · Seoul National University3 +3
PlanarBench: Evaluating LLM Spatial Reasoning via Planar Graph Drawing
Jun 1, 2026Oleksandr Nikitin, Anna KravchenkoSpatial Reasoning Benchmarks LLM Evaluation
CARTE: A Benchmark for Mapping Language Model Knowledge Across France
Jun 1, 2026Sarah Almeida Carneiro, Christos Xypolopoulos, Xiao Fei +2LLM Evaluation Factual Knowledge in Language Models
École Polytechnique, Paris · École Polytechnique, Institut Polytechnique de Paris, France · École Polytechnique, Paris, NTUA +3
What to Format and How: A Benchmark and Workflow Approach for Document Formatting
Jun 1, 2026Shihao Rao, Liang Li, Jiapeng Liu +6LLM Evaluation Document Layout Analysis
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs
Jun 1, 2026Yangfan Ye, Xiaocheng Feng, Jialong Tang +5LLM Evaluation Cultural Bias in Language Models
Harbin Institute of Technology · Harvard University · The University of Hong Kong
Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
Jun 1, 2026Yujia Tong, Yuxi Wang, Yunyang Wan +3LLM Quantization LLM Evaluation
Wuhan University of Technology · Nanyang Technological University
Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses
Jun 1, 2026Sugyeong Eo, Heuiseok LimLLM Evaluation Pragmatic Reasoning in Language Models
Department of Software, Yonsei University Mirae Campus, Republic of Korea · Department of Computer Science and Engineering, Korea University, Republic of Korea
Argument Collapse: LLMs Flatten Long-Form Public Debate
Jun 1, 2026Yekyung Kim, Yapei Chang, Chau Minh Pham +1LLM Evaluation Computational Argumentation
University of Maryland, College Park
Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation
Jun 1, 2026Junjie Chen, Yuxi Dong, Haitao Li +5LLM Evaluation LLM-as-a-Judge
Department of Computer Science and Technology, Tsinghua University · University of Science and Technology Beijing
Question Type, Cognitive Load, and CEFR Alignment: Evaluating LLM-Generated EFL Grammar Drill Exercises
Jun 1, 2026Steve Woollaston, Brendan Flanagan, Yuko Toyokawa +1LLM Evaluation
Kyoto University, Japan · Ritsumeikan University, Japan
Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment
May 31, 2026Hamidreza Hasani Balyani, Seyed Pouyan Mousavi Davoudi, Alireza Amiri-Margavi +2LLM Evaluation LLM Alignment
Amazon Lab126, HW Tech Org. Sunnyvale, CA, USA · Independent Researcher in AI and Statistics Tehran, Iran · Computational Modeling and Simulation University of Pittsburgh Pittsburgh, PA, USA +1
Before and After Temperature: A Distributional View of Creative LLM Generation
May 31, 2026V. S. Raghu Parupudi, Harsha Ponnada, Aditi Kaushal +3Open-Ended Generation Creativity Assessment
University of California, San Diego · Indian Institute of Technology, Kharagpur · Delhi Technological University +3
Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs
May 31, 2026Denica Kjorvezir, Marko Djukanović, Ana Gjorgjevikj +2LLM Evaluation
1Computer Systems Department, Jožef Stefan Institute, Ljubljana, Slovenia · 2Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · Center for Astrophysics and Cosmology, University of Nova Gorica, Nova Gorica, Slovenia
Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware
May 31, 2026Sagar Bhetwal, Rajan Bastakoti, Nirajan Acharya +2LLM Evaluation LLM Inference
Department of Computer Science, University of the Cumberlands, Kentucky, United States · Department of Computer Science, DePaul University, Chicago, IL, United States · Youngstown State University, Youngstown, Ohio, United States +1
The Case for Model Science: Verify, Explore, Steer, Refine
May 31, 2026Przemyslaw Biecek, Luca Longo, Jianlong Zhou +3LLM Evaluation LLM Interpretability
Center for Credible AI · University of Warsaw · Warsaw University of Technology +7
Diagnosing LLM Arbitration Behavior over Pre-evidence Epistemic States in RAG-based Fact-Checking
May 31, 2026Yuxi Sun, Wenbo Shang, Wei Gao +2Retrieval-Augmented Generation LLM Evaluation
Hong Kong Baptist University · Singapore Management University
Child-directed speech facilitates production, not comprehension, in BabyLMs
May 31, 2026Bastian Bunzeck, Sina ZarrießLanguage Model Pretraining LLM Evaluation
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
May 30, 2026Yongxi Zhou, Lai Yun Choi, Jiaxi Wen +1LLM Evaluation Language Model Generation Evaluation
Northeastern University, Massachusetts, USA · University of Southern California, California, USA
IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs
May 30, 2026F. Carichon, S. Sharma, M. Girard +2Creativity Assessment LLM Evaluation
McGill University, Montreal, Canada · Mila, Montreal, Canada · Concordia University, Montreal, Canada
RWGBench: Evaluating Scholarly Positioning in Related Work Generation
May 30, 2026Anzhe Xie, Weihang Su, Jiaxin Mao +4LLM Evaluation Automated Evaluation
Tsinghua University Beijing, China · Renmin University of China Beijing, China
Finite Certificates for In-Context Determinacy and a Threshold Theory of Emergence in Language Models
May 30, 2026Faruk Alpay, Hamdi AlakkadLLM Evaluation In-Context Learning
Department of Computer Engineering Bahcesehir University, Istanbul, Turkey · Department of Artificial Intelligence Engineering Bahcesehir University, Istanbul, Turkey
On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
May 30, 2026Etienne Casanova, Rafal Kocielnik, R. Michael AlvarezLLM Evaluation Zero-Shot Text Classification
California Institute of Technology, Pasadena, CA, USA.
ProtStructQA: A Denotation Threshold in Protein Structural Reasoning
May 30, 2026Aravind Mandiga, Guoming Li, Jin Lu +3LLM Evaluation Scientific QA
University of Georgia

Jun 2, 2026, cs.CLJ/K move · Enter open · S save

Predicting Inference-Time Scaling Gains from Labeled Validation-Set Output Statistics

Luyang Zhang, Jingyan Li

Carnegie Mellon University · Johns Hopkins University

LLM Evaluation Inference-Time Scaling Best-of-N Sampling

Open paper AskPDF arXiv

Best-of-

N

inference scaling (drawing

N

candidate answers from a language model and returning the one a reward model ranks highest) improves accuracy by an amount that varies across models, but predicting that amount in advance currently requires running the procedure end-to-end. Prior work links cheap statistics of a model's sampled outputs and validation-set correctness (how often samples agree, how diverse they are, how confident the model is, and where correct samples appear) to model behavior, but does not isolate which of these form a stable, compact predictor of best-of-

N

gain. We fit ridge predictors on features computed from a single labeled validation-set sampling pass, use bootstrap-Lasso as a stability analysis of the candidate feature set, and give a concentration analysis with an explicit linear-approximation residual. Across three base-model families, six post-training methods, and math and reasoning task domains, the stability analysis identifies a strict three-feature core spanning prompt-level agreement spread, label-assisted first-correct-sample position, and completion-length variance; a compact ridge predictor built from this core plus an entropy add-on reaches Spearman

ρ= 0.90

with actual best-of-

N

gain under a reward-model verifier. The intended use is labeled validation-set screening of candidate configurations before paying the full reward-model scoring cost.

Momentum

Related topics

Latest papers 1,643