LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Reversing Arrows in Large Language Models
Aug 4, 2026Sefika Efeoglu, Adrian PaschkeKG Construction LLM Evaluation
Institute of Computer Science, Department of Mathematics and Computer Science, Freie Universität Berlin, 14195 Berlin, Germany · Data Analytic Center (DANA), Fraunhofer Institute FOKUS, 10589 Berlin, Germany
AI World Cup 2026: Benchmarking Large Language Models for End-to-End Football Tournament Prediction
Aug 4, 2026Jonaid Shianifar, Iias FaiudLLM Evaluation Forecasting Benchmarks
AI World Cup Project
Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces
Aug 4, 2026Francesca Carlon, Vincent Ginis, Andres AlgabaLLM Evaluation Overthinking in Language Models
Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussels, Belgium · imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussels, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA
MMLongBench-Doc-V2: A Corrected-Annotation, Semantics-Aware Revision of MMLongBench-Doc
Aug 4, 2026Mingtian ZhangLLM Evaluation Long-Context Language Model Evaluation
PageIndex
Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inference
Aug 4, 2026Shahrukh Mohiuddin, Chalamalasetti Kranti, Sherzod Hakimov +1LLM Evaluation Language Model Probing
Computational Linguistics, Department of Linguistics University of Potsdam, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks
Aug 4, 2026Ine Gevers, Walter DaelemansLLM Evaluation Reasoning Evaluation
CLiPS, University of Antwerp
Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform
Aug 4, 2026Xavier Wrenn, Radoslav Raykov, Aleksandar Angelov +3LLM Evaluation Workflow Automation
IBM
On the Diversity of Analogy Making in Large Language Models
Aug 4, 2026Yuanhao Shen, Daniel Xavier de Sousa, Caio César Sifuentes Barcelos +2LLM Evaluation Text Generation
Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute, Queen’s University, Canada · Instituto Federal de Goiás, Anápolis, Brazil · National Research Council Canada.
Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains
Aug 4, 2026Yanchao Li, Wanhao Liu, Jiaqing Xie +4LLM Evaluation LLM Auditing
Nanjing University · Shanghai Artificial Intelligence Laboratory · North University of China
Beyond Accuracy: A Multidimensional Evaluation of Statistical Reasoning in Large Language Models
Aug 4, 2026Monnie McGee, Mateo Langston Smith, Julian CabreraLLM Evaluation
Department of Statistics and Data Science Southern Methodist University · Research Technology Services Southern Methodist University
On the missing benchmarks layer and a potential solution
Aug 4, 2026Francis F Daniel, Mauro Ibañez, Francis Perelman +1LLM Evaluation Benchmark Auditing
1SURUS · 2Independent
Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores
Aug 4, 2026Zeyu Zhang, Bradly C. StadieLLM Evaluation LLM Auditing
Department of Statistics and Data Science Northwestern University
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Aug 4, 2026Bhavin Jawade, Cameron R. WolfeLLM Evaluation LLM-as-a-Judge
Netflix
Mapping the City Through the Lens of Language Models
Aug 4, 2026Wanqi Liu, Rong Zhao, Zhizhou Sha +2LLM Evaluation
Centre for Advanced Spatial Analysis (CASA), University College London, London, UK · Department of Computer Science, The University of Texas at Austin, Austin, TX, USA · Department of Civil and Transportation Engineering, South China University of Technology, Guangzhou, China +1
Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
Aug 3, 2026Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1LLM Evaluation Psychometric Validation
École Polytechnique, Institut Polytechnique de Paris, France · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates
onepot-Bench 0: towards lab-aware in silico chemistry benchmarks
Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety Benchmarks LLM Evaluation
Onepot AI, Inc.
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Aug 3, 2026Nan Chen, Zhouhao Yang, Soufiane HayouZero-Shot Learning LLM Evaluation
Department of Applied Mathematics and Statistics Johns Hopkins University
Why Large Language Models Fail at Tabular Prediction
Aug 3, 2026Marta Garnelo, Wojciech M. CzarneckiLLM Evaluation Tabular Classification
Fundamental Technologies
MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models
Aug 3, 2026Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian +1LLM Evaluation Human-in-the-Loop Evaluation
Center for Tech Responsibility, Brown University · University of Pennsylvania
From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution
Aug 3, 2026Can Wang, Haoran Chen, Haowen Gao +3LLM Evaluation Benchmark Design
Shandong Key Laboratory of Digital Service Computing Technology and Systems · Alibaba Token Hub · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences +1
How Much Does a Reasoning Summary Reveal? An Observability Ladder for Large Language Models
Aug 3, 2026Andres Algaba, Francesca Carlon, Lynn Delcon +3LLM Evaluation LLM Interpretability
1Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussel, Belgium · 2imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussel, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA
Conformity Mitigations in Large Language Models Lie on a Single Resistance-Receptivity Frontier
Aug 3, 2026Zafar Hussain, Kristoffer NielboLLM Evaluation Normative Conformity in Language Models
Aarhus University
Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict?
Aug 3, 2026Kazuki Nakayashiki, Keisuke WatanabeLLM Evaluation Language Model Ensembles
Glasp Inc.
Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation
Aug 3, 2026Fengxian Ji, Yuke Li, Jingpu Yang +8LLM Evaluation LLM-as-a-Judge
Wuhan University · Northeastern · Zhongguancun Academy +3
Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal standard
Aug 3, 2026Hector Zenil, Luan OzelimLLM Evaluation Bayesian Inference
Oxford Immune Algorithmics, Oxford University Innovation & London Institute for Healthcare Engineering, U.K · Department of Biomedical Computing, School of Biomedical Engineering and Imaging Sciences & King’s Institute for AI, King’s College London, U.K
Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models
Aug 2, 2026Fali Wang, Ali Al-Lawati, Iliyas Bektas +5LLM Evaluation LLM Reasoning with Graphs
The Pennsylvania State University, University Park, PA, USA · Rensselaer Polytechnic Institute, Troy, NY, US
When Policies Change Probabilities: Modular Decision-Making for LLM Code Review
Aug 2, 2026Rasvik Kudum, Max Corbett, Hitansh Paliwal +3LLM Evaluation LLM Decision-Making
Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+
Aug 2, 2026Sherzod Hakimov, Karl Osswald, Jelle Psurek +3Multilingual Language Model Evaluation LLM Evaluation
Computational Linguistics, Department of Linguistics University of Potsdam, Germany · 1German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Models
Aug 2, 2026Xiaocui Yang, Xican Tan, Shoujie Chen +3LLM Evaluation LLM Grounding
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Institute of Intelligent Computing, University of Electronic Science and Technology of China, Chengdu, China · School of Civil and Commercial Law, Southwest University of Political Science and Law, Chongqing, China +1
Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation
Aug 2, 2026Francis Luis Santos Vargas, Rodrigo Brandão Mansilha, Diego KreutzLLM Evaluation Code Generation
AI Horizon Labs, PPGES, Universidade Federal do Pampa (UNIPAMPA), Alegrete, Brazil

Aug 4, 2026, cs.AIJ/K move · Enter open · S save

Reversing Arrows in Large Language Models

Sefika Efeoglu, Adrian Paschke

Institute of Computer Science, Department of Mathematics and Computer Science, Freie Universität Berlin, 14195 Berlin, Germany · Data Analytic Center (DANA), Fraunhofer Institute FOKUS, 10589 Berlin, Germany

KG Construction LLM Evaluation Relational Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) have achieved strong performance on text-to-knowledge graph generation and related tasks. Nevertheless, it is still unclear whether they accurately model the direction-dependent semantics of inverse relations, in which reversing the order of the arguments alters the meaning of a relation (e.g., \textit{mother} versus \textit{child}). To the best of our knowledge, this work presents the first systematic study of inverse relation directionality in LLMs, using a benchmark consisting of 5,457 instances spanning 27 distinct inverse relation labels. We evaluate five open-source LLMs under a multiple-choice prompting framework and further examine the influence of relation descriptions and entity representations by substituting the original entities with synthetic and masked entities. Our findings reveal systematic asymmetries in inverse relation classification across LLMs, indicate that relation descriptions do not consistently improve performance, and show that model performance can be sensitive to variations in entity representations.

Momentum

Related topics

Latest papers 1,643