LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Assessing AI in Introductory Physics Problem Solving
Jul 15, 2026Amir Bralin, N. Sanjay RebelloLLM Evaluation Multimodal Large Language Models
Texas Tech University, Lubbock, TX · Purdue University, West Lafayette, IN
Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Jul 15, 2026Xiao Ye, Jacob Dineen, Evan Zhu +3LLM Evaluation Forecasting Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
Pezego-HITL: A policy-grounded large language model architecture for agricultural extension in Ghana
Jul 15, 2026Shunbao Li, Zhipeng Yuan, Amoako Ofori +5Retrieval-Augmented Generation LLM Evaluation
School of Computer Science, University of Sheffield, Regent Court (CS), 211 Portobello, Sheffield, S1 4DP, United Kingdom · Institute for Environment and Sanitation Studies, College of Basic and Applied Sciences, University of Ghana, P. O. Box LG 209, Legon, Accra, Ghana · Mutus Tech Ltd, 54 St. James Street, Liverpool, L1 0AB, England
Rating the Raters: Rasch Measurement Theory for LLM Evaluation
Jul 14, 2026Pratik S. Sachdeva, Nathan BoudolLLM Evaluation LLM-as-a-Judge
University of California, Berkeley · Grenoble INP
The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
Jul 14, 2026Yanzhe Zhang, Sanmi Koyejo, Diyi YangLLM Evaluation Neural Network Robustness
Georgia Tech · Stanford University
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
Jul 14, 2026Hanhua Hong, Yizhi Li, Jiaoyan Chen +4LLM Evaluation Rubric-Based Evaluation
The University of Manchester · IQuest Research · University of Information Technology, VNU +2
The One-Word Census: Answer-Choice Conformity Across 44 Language Models
Jul 14, 2026Tapan ParikhLLM Evaluation Language Model Generation Evaluation
Cornell Tech
Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative
Jul 14, 2026Koen OostermeijerLLM Evaluation Proper Scoring Rules
Aleph Alpha Research
Epistemic Stance Flexibility Probing: Measuring Prompt-Conditioned Register Shift in Large Language Models
Jul 14, 2026Binwen Liu, Yilin RenLLM Evaluation Pragmatic Reasoning in Language Models
Institute of Artificial Intelligence and Robotics Xi’an Jiaotong University Xi’an, China
LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos
Jul 14, 2026Julius Steiglechner, Lucas Mahler, Gabriele LohmannLLM Evaluation LLM Reasoning
Dept. of Biomedical Magnetic Resonance, University Hospital Tübingen, Tübingen, Germany · Max-Planck-Institute for Biological Cybernetics, Magnetic Resonance Center, Tübingen, Germany
Extractable Memorization From First Principles
Jul 14, 2026A. Feder Cooper, Marika Swanberg, Jamie Hayes +5LLM Evaluation Memorization in Language Models
Yale, AVERI, Stanford · Google Research · Google DeepMind +2
The Model Knows Your Project, Not You: Measuring Recognition in LLMs with NameRank
Jul 14, 2026Bojie Li, Noah ShiLLM Evaluation Factual Knowledge in Language Models
Pine AI · University of Washington
From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings
Jul 14, 2026Zahra AnvariLLM Evaluation Document Information Extraction
Independent AI Researcher.
Comparing Semantic Navigation in Humans and Large Language Models using Natural Language Processing
Jul 13, 2026Gabriel Paris-Colombo, Rodrigo M. Cabral-Carvalho, Felipe D. Toro-HernándezLLM Evaluation
Center for Mathematics, Computing and Cognition, Federal University of ABC
Can LLMs Perform Technical Comprehension of Computer Architecture Papers?
Jul 13, 2026Nishant Aggarwal, Aishwarya Lekshmi Chithra, Ayushi Dubal +8Document Understanding LLM Evaluation
University of Wisconsin–Madison and NVIDIA Research.
Beyond Sally-Anne: Evaluating Theory of Mind in LLMs using Epistemic Schelling Points
Jul 13, 2026Roberta Rocca, Sami Boukortt, Geoff Keeling +1LLM Evaluation Theory of Mind
1Google
What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities
Jul 13, 2026Sukai Huang, Chenyuan Zhang, Fucai Ke +4LLM Evaluation LLM Planning
Faculty of Information Technology, Monash University
Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists
Jul 13, 2026Chuhan Shi, Xiaoquan Ren, Sicheng Song +3AI for Science LLM Evaluation
Southeast University · East China Normal University · Hong Kong University of Science and Technology
ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers
Jul 13, 2026Saba Imran, Debanjum Singh SolankyLLM Evaluation Citation Verification
Khoj Inc.
From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth
Jul 13, 2026Aleh ManchuliantsauLLM Evaluation Probabilistic Forecasting
Independent Researcher
Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflows
Jul 12, 2026Keyur GabaniLLM Evaluation Financial Fraud Detection
The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese
Jul 12, 2026Siyuan Song, Zhiheng Qian, Yunhao Zhang +11Language Model Pretraining LLM Evaluation
Princeton University · Shanghai Jiao Tong University · Chinese Academy of Sciences +5
CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps
Jul 12, 2026JungMin Yun, JuneHyoung Kwon, YoungBin KimMulti-Hop QA LLM Evaluation
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness
Jul 11, 2026Hong-In Won, Jinseok Jang, Hyoseop KimLLM Evaluation Moral Reasoning in Language Models
KITECH, Incheon, Republic of Korea · National Research Council of Science and Technology (NST), Sejong, Republic of Korea · KITECH, Daegu, Republic of Korea
Mirror Horizon: Viable Path Entropy as a Measure of Bounded Reflection
Jul 11, 2026Tiantian ZhangLLM Evaluation
Crystal · Department of Computer Science, Columbia University
Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
Jul 10, 2026Renuka Oladri, Mohan Vamsi Varadaraju Priya, Jerry WuLLM Quantization LLM Evaluation
Science Academy University of Maryland College Park, MD, USA · Electrical and Computer Engineering University of Maryland College Park, MD, USA
Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations
Jul 10, 2026Nada Zine, Tristan Coignion, Vincenzo Stoico +3LLM Inference Efficiency LLM Evaluation
Univ. Lille, CNRS, Inria · Univ. Bordeaux, CNRS, LaBRI · Vrije Universiteit Amsterdam +1
NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision
Jul 9, 2026Berkay AnahtarciLLM Evaluation Partial Identification
Department of Mathematical Engineering, Özyeğin University, Istanbul, Türkiye
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
Jul 9, 2026Baha Rababah, Cuneyt Gurcan Akcora, Carson K. LeungLLM Evaluation Post-Training Quantization
Department of Computer Science, University of Manitoba, Canada · AI Initiative, University of Central Florida, USA

Jul 15, 2026, physics.ed-phJ/K move · Enter open · S save

Assessing AI in Introductory Physics Problem Solving

Amir Bralin, N. Sanjay Rebello

Texas Tech University, Lubbock, TX · Purdue University, West Lafayette, IN

LLM Evaluation Multimodal Large Language Models Large Reasoning Models

Open paper AskPDF arXiv

Reasoning or inference-scaling models are the new generation of Large Language Models (LLMs) capable of complex problem solving. To investigate their problem-solving capability in physics, we evaluated model o4-mini by OpenAI on solving traditional, end-of-chapter problems from Halliday and Resnick's "Fundamentals of Physics," spanning core topics in the undergraduate physics curriculum. Performance was analyzed across modality and problem difficulty. The model solved the problems with overall accuracy of about 90%, but performance depended strongly on representation: accuracy was much higher on text-only problems (96%) than on problems requiring coordinated interpretation of text and images (79%). Accuracy also declined significantly as the problem difficulty increased from low to medium to high. These results show that state-of-the-art LLMs can solve much of the standard introductory physics problems, but that their performance remains uneven and constrained by problem modality and problem difficulty.

Momentum

Related topics

Latest papers 1,643