LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts
May 3, 2026Garvin KruthofKnowledge Editing LLM Evaluation
Technical University of Munich
What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
May 3, 2026Ranit Karmakar, Jayita ChatterjeeLLM Evaluation Language Model Calibration
Harvard University, Boston, MA 02115 · Boston, MA 02134
Conventional Commit Classification using Large Language Models and Prompt Engineering
May 3, 2026H. M. Sazzad Quadir, Sakib Al Hasan, Md. Nurul Ahad TawhidSoftware Engineering LLM Evaluation
Institute of Information Technology University of Dhaka Dhaka, Bangladesh
Language models fail at extended rule following
May 3, 2026Tianxiang Dai, Jonathan FanLLM Evaluation LLM Reliability
Department of Electrical Engineering, Stanford University, Stanford, CA 94305.
StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models
May 3, 2026Yongrui Chen, Yangyang Ma, Xiaoying Huang +4LLM Evaluation Benchmark Design
Southeast University · Hangzhou Dianzi University · Zhejiang University +1
Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models
May 3, 2026Kotaro Furuya, Takahito TanimuraTransformer Interpretability LLM Evaluation
Research and Development Group, Hitachi, Ltd. 1-280 Higashi-Koigakubo, Kokubunji-shi, Tokyo 185-8601, Japan
Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring
May 3, 2026Dekun YangPrompt Sensitivity LLM Evaluation
Zhejiang University
RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
May 3, 2026Yangyang Zhou, Yi-Chen LiReward Modeling LLM Evaluation
Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Molecular Representations for Large Language Models
May 3, 2026Nicholas T. Runcie, Fergus Imrie, Charlotte M. DeaneLLM Evaluation Large Language Model-Guided Scientific Discovery
Department of Statistics University of Oxford Oxford, UK
Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
May 2, 2026Deep Pankajbhai MehtaPrompt Sensitivity LLM Evaluation
Adobe Inc.
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks
May 2, 2026Sunil Kumar Maurya, Xin LiuLLM Evaluation LLM Reasoning with Graphs
University of Tokyo Tokyo, Japan · AIST Tokyo, Japan
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
May 2, 2026Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer +32LLM Evaluation Healthcare
Sophont · MedARC · Prime Intellect
Structural Ranking of the Cognitive Plausibility of Computational Models of Analogy and Metaphors with the Minimal Cognitive Grid
May 2, 2026Alessio Donvito, Antonio LietoLLM Evaluation Computational Cognitive Modeling
University of Bari, Italy · University of Salerno, CIIT Lab / ICAR-CNR, Italy
On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility
May 2, 2026Zhitao He, Haolin Yang, Rui Min +2LLM Evaluation Long-Form Document Generation
Hong Kong University of Science and Technology
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
May 2, 2026Jikai Jin, Vasilis SyrgkanisLLM Evaluation Off-Policy Evaluation
Psychologically Potent, Computationally Invisible: LLMs Generate Social-Comparison-Eliciting Posts They Fail to Detect
May 1, 2026Hua Zhao, Jiapei Gu, Michelle Mingyue GuSocial Media Analysis LLM Evaluation
Department of English Language Education, The Education University of Hong Kong · Analytics/Assessment Research Centre, The Education University of Hong Kong
Can AI Debias the News? LLM Interventions Improve Cross-Partisan Receptivity but LLMs Overestimate Their Own Effectiveness
May 1, 2026Faisal Feroz, Jonas R. KunstLLM Evaluation Human Behavior Simulation
Department of Experimental Psychology, University of Oxford · Uehiro Oxford Institute, University of Oxford · Psychology Programme, School of Social Sciences, Nanyang Technological University +1
When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
May 1, 2026Sailesh Panda, Pritam Kadasi, Abhishek Upperwal +1LLM Evaluation Instruction Following
Indian Institute of Technology Gandhinagar · Soket AI
Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?
May 1, 2026Lennard C. Froma, Tom Kouwenhoven, Maaike H. T. de Boer +2LLM Evaluation Language Model Scaling
Leiden University - The Netherlands · TNO - The Netherlands · TU Delft - The Netherlands
Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
May 1, 2026Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger +4Mathematical Reasoning Benchmarks LLM Evaluation
ETH Zurich · INSAIT · Sofia University "St. Kliment Ohridski"
Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference
May 1, 2026Yuxuan Gao, Megan Wang, Yi Ling YuLLM Evaluation Energy-Efficient ML
University of Pennsylvania · OpenMesh AI · Columbia University
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
Apr 30, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +3LLM Evaluation Automated Grading
DIPF | Leibniz Institute for Research and Information in Education · Faculty of Computer Science, Goethe University Frankfurt · 3Chemnitz University of Technology
Investigating LLM's Problem Solving Capability -- a Study on Statics Questions
Apr 30, 2026Tanner Culleton, Hung-Fu ChangLLM Evaluation Physical Reasoning
1R. B, Annis School of Engineering, University of Indianapolis, Indianapolis, IN, 46227, USA
TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering
Apr 30, 2026An-Yang Ji, Jun-Peng Jiang, De-Chuan Zhan +1Table QA LLM Evaluation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
Apr 30, 2026Garvin KruthofLLM Evaluation LLM Alignment
Technical University of Munich
Taming the Centaur(s) with LAPITHS: a framework for a theoretically grounded interpretation of AI performances
Apr 30, 2026Matteo Da Pelo, Alessio Donvito, Claudio Frongia +2LLM Evaluation Computational Cognitive Modeling
University of Cagliari, Italy · University of Bari Aldo Moro, Italy · University of Salerno, CIIT Lab, Italy
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Apr 30, 2026Sihong Wu, Owen Jiang, Yilun Zhao +5LLM Evaluation Scholarly Peer Review
Yale University · New York University · TCS Research
Test Before You Deploy: Governing Updates in the LLM Supply Chain
Apr 30, 2026Mohd Sameen Chishti, Damilare Peter Oyinloye, Jingyue LiLLM Evaluation LLM Auditing
Department of Computer Science NTNU Trondheim, Norway
RuC: HDL-Agnostic Rule Completion Benchmark Generation
Apr 30, 2026Arnau Ayguadé Domingo, Miquel Alberti-Binimelis, Cristian Gutierrez-Gomez +5LLM Evaluation LLM Prompting
Barcelona Supercomputing Center · Universitat Polit`ecnica de Catalunya

May 3, 2026, cs.CLJ/K move · Enter open · S save

EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts

Garvin Kruthof

Technical University of Munich

Knowledge Editing LLM Evaluation Factual Consistency Evaluation Language Model Editing

Open paper AskPDF arXiv

Local factual edits in scientific manuscripts often create non-local revision obligations. If a dataset changes from 215 to 80 documents, claims such as 'medium-scale' or 'a few hundred items' may also become stale, even though they do not repeat the edited number. In an audit of recent arXiv cs.CL benchmark and dataset papers, we find fact-dependent qualitative claims in 37.2% of papers, suggesting that this dependency pattern is common in the target genre. We introduce EditPropBench, a benchmark for measuring whether LLM editors propagate factual edits through dependent manuscript claims. Each item contains an ML/NLP-style synthetic manuscript, a targeted edit, and a controlled fact graph with sentence-level labels for direct targets, required downstream updates, and unrelated text that should remain unchanged. We summarize cascade success with Edit-Ripple Adherence (ERA), the fraction of required downstream updates correctly revised, and validate the metric with adversarial probes and stress-test variants. On the hardest cases, where dependent claims use implicit or free-form wording rather than repeating the edited value, five LLM editing systems span ERA 0.148-0.705. Even the strongest misses roughly 30% of required cascade updates. This advantage persists in a mixed evaluation that includes easy cases solvable by deterministic substitution. EditPropBench shows that current LLM editors can repair many implicit consequences of factual edits, but reliable scientific revision still requires cascade-aware checking.

Momentum

Related topics

Latest papers 1,643