LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

TokEval: A Tokenizer Evaluation Suite
Aug 18, 2026Clara MeisterLLM Evaluation
EPFL
PTXBench: Benchmarking and Adapting LLMs for GPU Kernel Optimization with Architecture-specific PTX
Aug 18, 2026Genghan Zhang, Yixin Dong, Chengze Fan +4LLM Evaluation GPU Kernel Optimization
Stanford University · Carnegie Mellon University · Independent Researcher +1
A Probe Direction Is a Property of Its Prompt
Aug 13, 2026Valentin NoëlPrompt Sensitivity LLM Evaluation
Devoteam
Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation
Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2LLM Evaluation LLM Auditing
School of Statistics and Data Science, Southwestern University of Finance and Economics · School of Statistics and Data Science, Southwestern University of Finance and Economics Chengdu, China
TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems
Aug 13, 2026Shunwen Bai, Ziping Ma, Chaoyang Zhang +4LLM Evaluation Inference-Time Search
Zhejiang University · Ant Group · Central South University
Which LLM Is Your Ideal Companion? Evaluating Emotional Companion Capabilities of LLMs Based on Adult Attachment Theory
Aug 13, 2026Junkai Zhou, Shiting Guan, Zhaoyi ZhangLLM Evaluation Personality Modeling in Language Models
LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation
Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM Evaluation Benchmark Design
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China
The Embedder's Dilemma: LLMs Are Better, but at What Cost?
Aug 13, 2026Adnan El Assadi, Niklas Muennighoff, Jinhyuk LeeLLM Inference Efficiency LLM Evaluation
Harvard University · Stanford University · Independent Researcher
Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
Aug 12, 2026Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar +2LLM Evaluation LLM-as-a-Judge
1Meta Superintelligence Labs · 2FAIR at Meta
Novels generated by language models show compressed formal variation
Aug 12, 2026Mehdy Sedaghat Payam, Justin QuinnLLM Evaluation Long-Form Document Generation
University of Maryland · University of West Bohemia · Charles University
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Aug 12, 2026Jiarui Ma, Jianghan Wang, Yuheng Ma +2LLM Evaluation Benchmark Design
Southern University of Science and Technology School of Microelectronics Shenzhen, China
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLLM Evaluation Language Model Generation Evaluation
Federal University of Santa Catarina (UFSC) Brazil
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
Aug 12, 2026Jinjun Huang, Zhongzhen Wen, Tongtong Xu +3LLM Evaluation GPU Kernel Generation
College of Computer Science and Technology and the State Key Laboratory of Blockchain and Data Security, Zhejiang University · State Key Lab for Novel Software Technology, Nanjing University · Software Engineering Application Technology Laboratory, Huawei +1
Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
Aug 12, 2026Karl Hanna, Chen FengLLM Evaluation Multiple-Choice Question Answering
Queen’s University Belfast
OEIS Open: How many conjectures can language models turn into theorems?
Aug 12, 2026Tom AdamczewskiLLM Evaluation Automated Theorem Proving
Epoch AI
Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction
Aug 12, 2026Mariya I. VasilevaLLM Evaluation Instruction Following
Work done while at Meta Superintelligence Labs.
Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework
Aug 12, 2026Avinash Agarwal, Vridhi JainLLM Evaluation
Unique Identification Authority of India, New Delhi, India
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
Aug 12, 2026Shailja Thakur, Sungeun An, Chad DeLuca +1Prompt Sensitivity LLM Evaluation
IBM Research India · IBM Research Almaden
TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs
Aug 11, 2026Valentin Rodionov, Shamil AssylbekovLLM Evaluation LLM Reliability
Intellicat Cleveland, OH 44106
Mapping and Measuring the Behavioral Evolution of Large Language Models
Aug 11, 2026Dong Qiao, Chris Ding, Jicong FanLLM Evaluation Representation Geometry in Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
Chemically Meaningful Textualization Enables Explainable Validation of Metal-Organic Frameworks by Large Language Models
Aug 11, 2026Guobin Zhao, Xiao-Yan LiLLM Evaluation LLM Interpretability
Department of Chemistry, National University of Singapore, 3 Science Drive 3, Singapore 117543, Singapore
SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information
Aug 11, 2026Junjie Ye, Zhuohui Sheng, Shaofan Liu +12LLM Evaluation AI Agent Benchmarks
1Fudan University · 2Tencent Hunyuan Team
Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases
Aug 11, 2026Davood Wadi, Mohsen Ghodrat, Matthew PhilpLLM Evaluation Cultural Bias in Language Models
McGill University · University Canada West · Marketing Management +1
From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models
Aug 11, 2026Si'an Xie, Jiaxun Liu, Biao Yang +4LLM Evaluation Multi-Hop Reasoning
Beijing University of Posts and Telecommunications · Peking University · Kuaishou Technology
How Robust Are LLMs to Vietnamese Dialects?
Aug 11, 2026Minh Tran, Trinh Chau, Thanh-Nhan Le +4LLM Evaluation Language Model Robustness
University of Science, VNU-HCM · VNU University of Engineering and Technology · New York University Abu Dhabi +4
Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation
Aug 11, 2026Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie +1LLM Evaluation LLM-as-a-Judge
The University of Queensland · Brisbane, Australia
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt Sensitivity LLM Evaluation
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA
Toward Human Rights Benchmarking for LLMs: A Pilot Methodology
Aug 10, 2026Savannah Thais, Wm. Matthew Kennedy, Abhigyan Acherjee +3LLM Evaluation Moral Reasoning in Language Models
Hunter College, New York, USA · Oxford Internet Institute, University of Oxford, UK · King’s College London, UK +2
The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discourse
Aug 10, 2026Maurice FlechtnerLLM Evaluation Multi-Agent Debate
University of Zurich, Zurich, Switzerland · ETH Zurich, Zurich, Switzerland
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
Aug 10, 2026Laurens Samson, Iva Gornishka, Gossa Lô +2Multilingual Language Model Evaluation LLM Evaluation
City of Amsterdam · Socially Intelligent Artificial Systems Group, University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg

Aug 18, 2026, cs.CLJ/K move · Enter open · S save

TokEval: A Tokenizer Evaluation Suite

Clara Meister

EPFL

LLM Evaluation

Open paper AskPDF arXiv

Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers' training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting models on bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman rho up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree.

Momentum

Related topics

Latest papers 1,631