LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax
Sep 24, 2026Zhenyan Lu, He Wang, Xiaohui HuangLLM Evaluation Language Model Decoding
College of International Studies, National University of Defense Technology, Nanjing, China
SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search
Sep 24, 2026Zhongxin Huang, Songyang Li, Renzhe Zhou +5LLM Evaluation Automated Evaluation
Peking University Beijing, China · Kuaishou Technology Beijing, China · Kuaishou Technology Hangzhou, China +2
JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
Sep 24, 2026Delip Rao, Chris Callison-BurchLLM Inference Efficiency LLM Evaluation
University of Pennsylvania
Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?
Sep 24, 2026Alexandru Stefan Stoica, Traian Rebedea, Marian Cristian MihaescuLLM Evaluation Automated Program Repair
University Politehnica of Bucharest Bucharest, Romania · University of Craiova Craiova, Romania
Likelihood Ranking doesn't Scale Like Prompting in LLMs
Sep 24, 2026Alessandro Bondielli, Lucia Passaro, Davide Bacciu +1LLM Evaluation Language Model Generation Evaluation
CoLingLab, Department of Philology, Literature and Linguistics, University of Pisa · Department of Computer Science, University of Pisa
From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring
Sep 24, 2026Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan +1LLM Evaluation LLM Information Extraction
Reliable Intelligence Team, VTT Technical Research Centre of Finland Ltd., 02150 Espoo, Finland
Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms
Sep 24, 2026Rong Wang, Kun Sun, Yadong GuoLLM Evaluation Pragmatic Reasoning in Language Models
University of Tübingen · Tongji University
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
Sep 23, 2026Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband +3LLM Evaluation Software Engineering Benchmarks
York University Toronto, Canada · The University of Texas at Dallas Texas, USA
Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?
Sep 23, 2026AbdulRahman A. Morsy, Aya ZiriklyLLM Evaluation Arabic NLP
Department of Computer Science, School of Engineering and Applied Sciences, George Washington University, Washington DC, United States · Department of Computer Science, School of Engineering and Applied Sciences, George Washington University, Washington, DC, United States · Center for Speech and Language Processing, Whiting School of Engineering, Johns Hopkins University, Baltimore MD, United States +1
How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?
Sep 23, 2026Chen Yang, Xianyang Zhang, Jun ChenLLM Evaluation Selective Inference
Department of Statistics, Texas A&M University · Division of Computational Biology, Department of Quantitative Health Sciences, Mayo Clinic
Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing
Sep 23, 2026Norah Almousa, Shayan Peyghambari Oskoui, Raquel Coelho +3LLM Evaluation LLM Alignment
University of Pittsburgh
Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
Sep 23, 2026Imtiaz Ul Hassan, Öykü Akbulut, Onur Kaya +4Multilingual Language Model Evaluation LLM Evaluation
Edge Hill University, United Kingdom · Ermetal Otomotiv ve E¸sya San. Tic. A.¸S., Türkiye
Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding
Sep 23, 2026Fan Zhang, Yankai Chen, Zhuohan Xie +11LLM Evaluation Efficient Language Model Inference
The University of Tokyo · MBZUAI · McGill University +8
When Context Misleads: In-context Learning with Jurisdiction in Large Language Models
Sep 23, 2026Pei-lin Li, Qingle Liu, Junyang Feng +4LLM Evaluation In-Context Learning
Department of Computer Science and Technology, Tsinghua University · School of Integrated Circuits, Huazhong University of Science and Technology
Speculative Evaluation of Stochastic LLMs
Sep 23, 2026Qianli Shen, Xiang Li, Ruomeng Ding +3LLM Evaluation Language Model Generation Evaluation
Alibaba Group · National University of Singapore · University of North Carolina at Chapel Hill
Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Sep 23, 2026Kaifeng Tan, Yudong Li, Linlin ShenLLM Evaluation Data Contamination in Language Models
Shenzhen University · Tsinghua University
What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs
Sep 22, 2026Han Chen, Yingrui LiLLM Evaluation Evidence Selection
Classifying Interpretive Canons at the Sentence Level: A Benchmark from the German Federal Constitutional Court
Sep 22, 2026Felix RingeLegal NLP LLM Evaluation
Department of Law, Freie Universität Berlin, Berlin, Germany.
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
Sep 22, 2026Lijuan Tang, Yuemeng ZhengLLM Evaluation Tool-Use Evaluation
Northeastern University, Seattle
A retrospective analysis on the use of LLMs to study infant syntax learning
Sep 22, 2026Hélie Bazin, Anouk Barberousse, François YvonLLM Evaluation Language Modeling
SCAI, SND, ISIR · Sorbonne Université, Sorbonne Center for Artificial Intelligence (SCAI) · SND +3
A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation
Sep 22, 2026Lorenzo Zangari, Davide PiccaLLM Evaluation Language Model Generation Evaluation
University of Lausanne Lausanne, Switzerland
Calibration as a First-Class Criterion in LLM Evaluation
Sep 22, 2026Mario Sanz-Guerrero, Katharina von der WenseLLM Evaluation Confidence Estimation in Language Models
Johannes Gutenberg University Mainz, Germany · University of Colorado Boulder, USA
EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions
Sep 22, 2026Jiayi Li, Ziyuan Wang, Daniel Garijo +1LLM Evaluation Ontology Learning
Ontology Engineering Group, Universidad Politécnica de Madrid
Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
Sep 22, 2026Vansh WahiReward Hacking LLM Evaluation
Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices
Sep 22, 2026Qian Xie, Yueli He, Nairen CaoMulti-Armed Bandits LLM Evaluation
Cornell University · Columbia University · New York University +1
SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation
Sep 21, 2026Fatih Deniz, Yazan Boshmaf, Issa KhalilLLM Safety Benchmarks LLM Evaluation
Qatar Computing Research Institute (QCRI), HBKU, Doha, Qatar
The Copy Ceiling: An Input-Exposure Control for Ontology-Grounded Generation over Curated Corpora
Sep 21, 2026John J. O'HareGrounded Text Generation LLM Evaluation
DreamLab AI, The Loom / VisionFlow neurosymbolic stack
Evaluating Decision Models for Text Annotation in Computational Social Science
Sep 21, 2026Hazem Ibrahim, Yasir ZakiLLM Evaluation LLM-Assisted Annotation
Computer Science, New York University Abu Dhabi, Abu Dhabi, UAE
LLJ Cards: Best practices for the Use of LLMs as Judges
Sep 21, 2026Khaoula Chehbouni, Melina Medjdoub, Florian Carichon +2LLM Evaluation LLM-as-a-Judge
McGill University · Mila - Quebec AI Institute · Concordia University

Sep 24, 2026, cs.CLJ/K move · Enter open · S save

Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax

Zhenyan Lu, He Wang, Xiaohui Huang

College of International Studies, National University of Defense Technology, Nanjing, China

LLM Evaluation Language Model Decoding Language Modeling Language Model Probing

Open paper AskPDF arXiv

A language model can fail a syntactic test in two distinct ways: by not encoding the relevant structure, or by encoding it but failing to use it at the output. Behavioral evaluation alone cannot tell these apart. We propose a three-level evaluation framework (behavioral deployment, LM-head readout, and probe recoverability) measured on the same items under the same binary decision. Using a compact trilingual (English, Chinese, German) control-dependency benchmark, we find that probe recoverability exceeds or equals LM-head readout, which in turn exceeds or equals behavioral deployment, across seven models and all three languages in the aggregate. The recoverability surplus is never negative across all 14 (model, task) conditions. The disconnect concentrates in subject-control, where a nearest-noun heuristic gives the wrong answer. The single largest gap (0.653) appears on Qwen3-0.6B Instruct in question answering. The gap persists at Qwen3-14B Instruct. Instruction tuning degrades deployment more than encoding in percentage terms. We rule out option-position bias, late-layer erasure, output-formatting artifacts, and probe-training variance. The pattern is consistent with decoding that favors surface shortcuts, and the behavior-probe gap measures the strength of that preference. Activation patching shows the gap is layer-localized. Under instruction tuning, the LM-head-decoded layer shifts approximately ten layers later than the probe-decoded layer. These findings argue that behavioral evaluation understates what models encode, while probing alone overstates what they deploy.

Momentum

Related topics

Latest papers 1,631