LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

$Φ$ -Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?
Sep 9, 2026Leilei Ding, Shumin Wang, Yuting Huang +10LLM Evaluation Large Language Model-Guided Optimization
University of Science and Technology of China · StepFun · Peking University +3
RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases
Sep 9, 2026Yingqian Wu, Jingcong Liang, Siyuan Wang +4LLM Evaluation
Fudan University · The Chinese University of Hong Kong · University of Oxford +1
MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes
Sep 9, 2026Remco HendriksLLM Evaluation Intelligent Transportation Systems
Continker
Grounded Evaluation and Repair for NL-to-PDDL Problem Generation
Sep 9, 2026Joana Rosa, Pedro Santos, Valdemar Oliveira +3LLM Evaluation LLM Planning
INESC INOV, Lisbon, Portugal · Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal · Motamineral Minerais Industriais S.A. +1
Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format
Sep 9, 2026Touchapon Kraisingkorn, Krittin Pachtrachai, Wachiravit ModecruaLLM Evaluation Language Model Calibration
Amity Research and Application Center (ARAC), Amity Group
What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores
Sep 8, 2026Dana Paquin, Riddhiman JainLLM Evaluation LLM Auditing
Stanford University
Do LLMs Make More Mistakes If They Do Not Believe the Input Data?
Sep 8, 2026Peter Kochelka, Aleš Manuel Papáček, Vojtěch Dvořák +1LLM Evaluation Knowledge Conflicts in Language Models
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Prague, Czech Republic
Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
Sep 8, 2026Leyuan Tang, Kangda Wei, Tianyu Jiang +1LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Engineering, Texas A&M University · Department of Computer Science, University of Cincinnati
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
Sep 8, 2026Jaewoo Lim, Sungbok Shin, Sanghyun HongLLM Evaluation LLM Interpretability
Oregon State University, Corvallis USA · Sogang University, Seoul, South Korea
Evaluation of Contextual Understanding in Large Language Models
Sep 8, 2026Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe +4LLM Evaluation LLM Grounding
Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, Coral Gables, Florida, USA.
Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation
Sep 8, 2026Xingyu Deng, Mingzi Cao, Nikolaos Aletras +2LLM Evaluation LLM Grounding
University of Sheffield · Sheffield, UK
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
Sep 8, 2026Jennifer Wang, Joachim Baumann, Daniel E. Ho +1LLM Evaluation Language Model Generation Evaluation
Stanford University
Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Sep 8, 2026Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah +1LLM Evaluation LLM Auditing
Qatar Computing Research Institute, HBKU
A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
Sep 8, 2026Maysam Khatib, Moysis Symeonides, Demetris Trihinas +2LLM Inference Efficiency LLM Evaluation
University of Cyprus, Nicosia, Cyprus · University of Nicosia, Nicosia, Cyprus
CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows
Sep 7, 2026Yonghong Zhang, Ricardo Correia, Isabel M. Parra +1LLM Evaluation Causal Effect Estimation
Department of Finance Universidad Autónoma de Madrid Madrid, Spain · Spanish National Research Council (CSIC) Madrid, Spain
Do Large Language Models Know What They Don't Know II? A Fully Behavioral, Non-Cognitive Measure of Epistemic Honesty
Sep 7, 2026Ali Şenol, H. Russell Bernard, Huan LiuLLM Evaluation LLM Honesty
Department of Computer Engineering, Tarsus University, Tarsus 33400, Mersin, Türkiye · School of Computing and Augmented Intelligence (SCAI), Arizona State University (ASU), Tempe, AZ 85281, USA · Institute for Social Science Research and School of Human Evolution and Social Change, Arizona State University (ASU), Tempe, AZ 85287, USA
How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement
Sep 7, 2026Riyadh Alnasser, Yusuf Mücahit Çetinkaya, Sumin Zhao +1LLM Evaluation LLM Sycophancy
University of Edinburgh · Middle East Technical University
ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making
Sep 7, 2026Jun Xiang, Zhijie Bao, Rong Hu +3LLM Evaluation Clinical Decision-Making
School of Data Science, Fudan University, China · 2Shanghai Innovation Institute, China · Department of Obstetrics, Obstetrics & Gynecology Hospital of Fudan University, Shanghai Key Lab of Reproduction and Development, Shanghai Key Lab of Female Reproductive Endocrine Related Diseases, Shanghai, China +2
FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
Sep 7, 2026Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo +10LLM Evaluation LLM Reliability
Department of Computer Science, University of Oxford · Oxford Internet Institute, University of Oxford · Thomson Reuters +7
Human-like moral judgments conceal divergent motive attributions in large language models
Sep 7, 2026Xiaoyan Wu, Jean-Claude DreherLLM Evaluation Moral Reasoning in Language Models
Department of Adult Psychiatry and Psychotherapy, University of Zurich, Zurich, Switzerland · Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Lyon, France
LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances
Sep 7, 2026Vamsi Krishna Kodavali, Rituraj SinghLLM Evaluation Instruction Following
Samsung
Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach
Sep 7, 2026Janek Groß, Elena Zentgraf, Jens HeidrichLLM Evaluation
University of Applied Sciences Mainz, Mainz, Rhineland-Palatinate, Germany
FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity
Sep 7, 2026Tyrone White, Yuki AraseLLM Evaluation Language Modeling
School of Computing, Institute of Science Tokyo Tokyo, Japan
PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us
Sep 5, 2026Gal Sapir, Alon Diament, Adva Wolf +7LLM Evaluation Benchmark Design
Pheno.AI, Tel Aviv, Israel · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel · Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot, Israel +1
Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models
Sep 4, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation LLM Reliability
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3
Can Large Language Models Anticipate Behavioral Responses to Social Policies? A Case of Pension Enrollment Prediction among China's Flexible Workers
Sep 4, 2026Yumiao Li, Peixin Liu, Donglin Di +2LLM Evaluation Policy Evaluation
School of Economics and Management, Tsinghua University, Beijing, China · Actuarial Science and Risk Management, University of Illinois Urbana-Champaign, Urbana, IL, USA · School of Computing, Harbin Institute of Technology, Harbin, China
SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding
Sep 4, 2026Shenxi Wu, Yuhong Liu, Haosong Zhang +8Document Understanding LLM Evaluation
The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory · New York University +5
PerfReasoning: How Well Do LLMs Reason on Hardware Performance?
Sep 3, 2026Dan Zhao, Karthikeyan Sankaralingam, Christos Kozyrakis +1LLM Evaluation
NVIDIA
Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
Sep 3, 2026Haoyaun Zhu, Jie ZhangLLM Evaluation LLM-as-a-Judge
School of Electronic and Electrical Engineering, University of Sheffield, Sheffield S10 2TN, UK · R&D Department, Ranplan Wireless Network Design Ltd., Cambridge CB23 3UY, UK · R&D Department, Cambridge AI+ Ltd., Cambridge CB23 3UY, UK
Investigating the Ability of Large Language Models to Analyze Recipes for Diabetes
Sep 3, 2026Revathy Venkataramanan, Aditya Luthra, Venkatesan Nadimuthu +1LLM Evaluation Healthcare
AI Institute, University of South Carolina 1112 Greene Street, Columbia, SC 29201 USA

Sep 9, 2026, cs.CLJ/K move · Enter open · S save

$Φ$ -Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan+9

University of Science and Technology of China · StepFun · Peking University+3

LLM Evaluation Large Language Model-Guided Optimization LLM Code Generation

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present

Φ

-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories,

Φ

-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.

Momentum

Related topics

Latest papers 1,631