LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models
Jun 25, 2026Henry Shaowu Yuchi, Michal Kucer, Benjamin H. Sims +2LLM Evaluation Scientific QA
Los Alamos National Laboratory
The Capability Frontier: Benchmarks Miss 82% of Model Performance
Jun 25, 2026Bradley Fowler, Ryan Smith, Daniel Thi Graviet +8LLM Evaluation Language Model Generation Evaluation
1Martian · 3ThoughtWorks · University of Oxford
Towards Evaluation of Implicit Software World Models in Coding LLMs
Jun 25, 2026Egor Bogomolov, Yaroslav ZharovLLM Evaluation World Model Evaluation
1JetBrains Research
Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform
Jun 25, 2026Manar Alsaid, Chimdumebi Nebolisa, Faris AbbasLLM Evaluation Automated Program Repair
Department of Computer Science and Information Systems, East Texas A&M University, 2600 W. Neal Street, Commerce, 75428, Texas, United States · Department of Dual Enrollment and P-16 Programs, Texas Woman’s University, 3042026 Administration Drive, Denton, 76204, Texas, United States
Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication
Jun 25, 2026Jerome Marston, Tino Kreutzer, Salomé Garnier +3LLM Evaluation Human-in-the-Loop Annotation
Kobo Inc · McGill University · Harvard University
An Empirical Study of LLM-Generated Specifications for VeriFast
Jun 25, 2026Wen Fan, Minh Tran, Sanya Dod +5LLM Evaluation Formal Verification
Purdue University West Lafayette, Indiana, USA · University of Michigan Ann Arbor, Michigan, USA · Meta Menlo Park, California, USA
DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
Jun 24, 2026Aaron J. Li, Hao Huang, Youngmin Park +6LLM Evaluation Language Model Generation Evaluation
University of California, Berkeley · Arena · University of California, Los Angeles
Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data
Jun 24, 2026Kylie AnglinLLM Evaluation Confidence Region Estimation
Department of Educational Psychology, Neag School of Education, University of Connecticut
InvestPhilBench: A Multi-Layer Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy
Jun 24, 2026Mingguang Chen, Bo QuLLM Evaluation Finance
University of California, Riverside (UCR). · 2Illinois Institute of Technology (IIT).
Manipulation Is Task-Dependent: A Multi-Axis, Multi-Environment Evaluation of Frontier LLMs
Jun 24, 2026Adeeb Zaman, Erik Nordby, Fred HeidingLLM Evaluation Language Model Safety Evaluation
Cambridge Boston Alignment Initiative (CBAI), Cambridge, US · The AI and Cybersecurity Institute (TAICI), Cambridge, US
Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval
Jun 24, 2026Leonardo Venuta, Francesco Tosoni, Paolo FerraginaLLM Evaluation Semantic Search
Sant’Anna School of Advanced Studies, Pisa, Italy
Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
Jun 24, 2026Hiroki TambaLLM Evaluation LLM-as-a-Judge
Tamba Research Academy
Evaluating LLMs on Real-World Software Performance Optimization
Jun 24, 2026Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim +1Software Engineering LLM Evaluation
Siemens AG · Technical University of Munich Munich, Germany · Technical University of Munich Heilbronn, Germany
How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring
Jun 24, 2026Yang GaoLLM Evaluation LLM-as-a-Judge
Veyon Solutions
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Jun 24, 2026Abrar Alotaibi, Raed Mughus, Moataz AhmedMultilingual Language Model Evaluation LLM Evaluation
Information and Computer Science Department, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia. · College of Computer Science and Information Technology, Imam Abdulrahman Bin Faisal University, Dammam, 31441, Saudi Arabia. · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia.
Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
Jun 24, 2026Alex KwonLLM Evaluation Memory-Augmented Language Models
Independent Researcher
Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment
Jun 23, 2026Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal +2Requirements Engineering LLM Evaluation
University of Maine · University of Notre Dame
LLM Performance on a Real, Double-Marked GCSE Benchmark
Jun 23, 2026Malachy Fox, Kavi Samra, Paul JungLLM Evaluation Educational Assessment
Medly AI
On the Stability of Prompt Ranking in Large Language Model Evaluation
Jun 23, 2026Shaoshuai Du, Penghao Liang, Yixian Shen +3LLM Evaluation LLM Prompting
University of Amsterdam, Amsterdam, Netherlands · Northeastern University, Boston, MA, USA · University of California San Diego, La Jolla, CA, USA +1
Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
Jun 23, 2026Guruprakash J, Krithika L. BLLM Evaluation LLM Alignment
SCOPE, VIT-AP University, Amaravathi, Andhra Pradesh, 522241, India · SCORE, VIT, Katpadi, Tamil Nadu, 632014, India
CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Jun 23, 2026Morayo Danielle Adeyemi, Ryan A. Rossi, Franck DernoncourtLLM Evaluation LLM Compression
Independent · Adobe Research
You Don't Need to Run Every Eval
Jun 22, 2026Yuchen Zeng, Dimitris PapailiopoulosLLM Evaluation Matrix Completion
Microsoft Research, AI Frontiers
TriggerBench: Investigating Prospective Memory for Large Language Models
Jun 22, 2026Tianhua Zhang, Xinjiang Wang, Qianxi Zhang +6LLM Evaluation Long-Context Language Model Evaluation
The Chinese University of Hong Kong · Microsoft Research Asia
HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs
Jun 22, 2026Yucheng Wu, Jundong Xu, Mingzhen Ju +4Logical Reasoning LLM Evaluation
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · School of Computing, National University of Singapore +3
When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysis
Jun 22, 2026Elroy Stav, Dvir Berlowitz, Maayan Orner +1LLM Self-Correction LLM Evaluation
Department of Computer Science and Artificial Intelligence, Bar-Ilan University · Independent Researcher
Who Owns the AI Recommendation? A Multi-Industry Empirical Map of Brand Category Ownership Across Large Language Models
Jun 22, 2026Dmitrij ŻatuchinLLM Evaluation LLM Auditing
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.
StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs
Jun 22, 2026Youxin Zhu, Yixuan Ding, Peng Lai +3LLM Evaluation AI Agent Benchmarks
Southern University of Science and Technology · The University of Hong Kong · Alibaba Group +1
When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
Jun 22, 2026Yujun Zhou, Christopher M. AckermanLLM Evaluation LLM Alignment
MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
Jun 22, 2026Devleena Das, Rajeev Patwari, Vikram Kumar Bukka +3LLM Evaluation
Advanced Micro Devices (AMD)
Measuring Behavior Portability in Large Language Models
Jun 22, 2026Tianjia Dong, Nadav Kunievsky, James A. EvansLLM Evaluation LLM Decision-Making
Knowledge Lab University of Chicago

Jun 25, 2026, cs.CLJ/K move · Enter open · S save

NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models

Henry Shaowu Yuchi, Michal Kucer, Benjamin H. Sims, Selma Peterson+1

Los Alamos National Laboratory

LLM Evaluation Scientific QA Synthetic Benchmark Generation

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge. In nuclear engineering, problem solving often requires not only factual knowledge but also quantitative reasoning and conceptual understanding. To address the need for systematic evaluation in this domain, we introduce NuclearQAv2, a benchmark for assessing LLMs on nuclear engineering knowledge. The benchmark comprises approximately 1,240 question-answer pairs spanning three categories: boolean, numeric, and verbal. NuclearQAv2 is constructed using a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora. By leveraging structured prompting for both automated question generation and response evaluation, the proposed framework enables scalable benchmark construction and evaluation. We evaluate a diverse set of LLMs using NuclearQAv2 and observe substantial performance differences across task types. While the models generally perform well on factual questions, quantitative reasoning and conceptual understanding remain considerably more challenging. These results highlight the importance of multi-faceted evaluation frameworks and establish NuclearQAv2 as a scalable benchmark for assessing LLM capabilities in technical domains.

Momentum

Related topics

Latest papers 1,643