LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
Aug 5, 2026Yinghui He, Ling Yang, Jiarui Liu +6LLM Evaluation Reasoning Evaluation
Princeton University · Carnegie Mellon University · University of Toronto +3
Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?
Aug 5, 2026Réemi Andrieu, Damien SileoLogical Reasoning LLM Evaluation
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France
Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems
Aug 5, 2026Isaiah AndrewsLLM Evaluation Reference-Free Evaluation
Department of Economics, Massachusetts Institute of Technology, and NBER
SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models
Aug 5, 2026Sihan Hu, Lyuhan Huang, Youjin Deng +1LLM Evaluation Scientific Code Generation
Hefei National Laboratory College of Mechanical and Electrical Engineering University of Science and Technology of China · Hefei National Laboratory Institute of Theoretical Physics University of Science and Technology of China Chinese Academy of Sciences Hefei 230026, China · Chinese Academy of Sciences Beijing 100190, China
LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs
Aug 5, 2026Jiahao Zhang, Yongzhi Tong, Zelin Fu +4LLM Evaluation LLM Personalization
1Shenzhen University of Advanced Technology · ♣Work done at Ant Group · 2Ant Group +1
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Aug 5, 2026Elena Merdjanovska, Omar Zaidan, Andreas RückléLLM Evaluation Selective Prediction
Humboldt-Universität zu Berlin Science of Intelligence · Amazon
RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
Aug 5, 2026Yuexi Yang, Alyssa Wu, Ji Luo +4Software Engineering LLM Evaluation
Zhejiang University · Hunyuan, Tencent · Ningbo Global Innovation Center, Zhejiang University
What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend
Aug 5, 2026Shahed Masoudian, Passant Shafaei, Monorama Swain +1LLM Evaluation Language Model Generation Evaluation
Johannes Kepler University Linz
IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)
Aug 5, 2026Shahd Gaben, Heba Sbahi, Samer Rashwani +5LLM Evaluation Benchmark Design
Hamad Bin Khalifa University, Doha, Qatar. · Nazarbayev University, Astana, Kazakhstan.
Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark
Aug 5, 2026Enrico Mensa, Lorenzo Zane, Calogero Jerik Scozzaro +3Figurative Language Understanding LLM Evaluation
Department of Computer Science, University of Turin, Turin, Italy · Independent Researcher
Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
Aug 5, 2026Ian B. de Haan, Peter van der Putten, Max van DuijnLLM Evaluation Theory of Mind
LIACS, Leiden University, The Netherlands
Agreement Before Diversity: Verification-First Complementarity for Heterogeneous Language-Model Coordination
Aug 5, 2026Ruitong Li, Binjie Guo, Aisheng Mo +3LLM Evaluation LLM Answer Verification
1The University of Hong Kong · Zhejiang University · 3Independent Researcher
When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Models
Aug 5, 2026Byoungjae Min, Kennedy Edemacu, Sae-Hong Cho +3LLM Evaluation LLM Grounding
Department of Computer Science, Sangmyung University, Seoul, Republic of Korea · College of Staten Island, The City University of New York, New York, NY, USA · School of Computer Engineering, Hansung University, Seoul, Republic of Korea +2
The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
Aug 5, 2026Yushi Sun, Yanjie Zhang, Rui ShengLLM Evaluation LLM Personalization
LIGHTSPEED, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong, China · Tencent LIGHTSPEED
STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation
Aug 5, 2026Bhiman Kumar Baghel, Anna Chrabaszcz, Tessa Warren +3LLM Evaluation Controllable Text Generation
University of Pittsburgh, Pittsburgh, PA, USA · University of Wisconsin–Madison, Madison, WI, USA
EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks
Aug 5, 2026Pau Arnal, Khaled Denfir, Danylo Smahliuk +2Open-Ended Generation LLM Evaluation
Sovrano AI
The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
Aug 5, 2026Alicia Guerra, Yibo HuLLM Evaluation LLM-as-a-Judge
Illinois Institute of Technology
FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation
Aug 5, 2026Yinghao Tang, Tan Zhenwei, Yiyao Wang +4Financial Services LLM Evaluation
State Key Laboratory of CAD&CG, Zhejiang University · Ant Group
The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale
Aug 5, 2026Mingguang Chen, Bo Qu, Licheng WangLLM Self-Correction LLM Evaluation
DeepGrounding · AlphaAvatar
Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluation
Aug 5, 2026Bo Xue, Zhi Hong, Jiayi Li +3Multi-Armed Bandits LLM Evaluation
City University of Hong Kong · South China University of Technology · Zhejiang University
Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary
Aug 4, 2026S. Ashwin Hebbar, Peiyao Sheng, Sewoong Oh +1LLM Evaluation Tool-Use Evaluation
Princeton University · Sentient Labs · University of Washington
From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models
Aug 4, 2026Fusheng LuoLLM Evaluation Quantitative Finance
Johns Hopkins University Baltimore, Maryland, USA
Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap
Aug 4, 2026Ankit Goyal, Jaideep RayMultilingual Language Model Evaluation LLM Evaluation
BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding
Aug 4, 2026Yangxuan Zhou, Sha Zhao, Yuning Chen +4Electroencephalography LLM Evaluation
State Key Laboratory of Brain-machine Intelligence, Zhejiang University · College of Computer Science and Technology, Zhejiang University · MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University
WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament
Aug 4, 2026Zhenran Wang, Zhonghan Bian, Jinsong Li +1LLM Evaluation Forecasting Benchmarks
Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
Aug 4, 2026Mohsen Hariri, Weicong Chen, Nahal Shahini +11LLM Evaluation Test-Time Scaling
Case Western Reserve University
VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs
Aug 4, 2026Andrei Chetvergov, Alexander Evseev, Timofei Sivoraksha +4LLM Evaluation Language Model Bias Evaluation
Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
Aug 4, 2026Shunfan Zheng, Dongsheng Shi, Yue Li +3LLM Evaluation Text-to-SQL
1East China Normal University · 2Hasso Plattner Institute/University of Potsdam
How Closely Do LLM Reviews Align with Human Peer Review?
Aug 4, 2026Abraham Camelo-Guerrero, Jairo Diaz-RodriguezLLM Evaluation Human-AI Agreement
School of Information Technology York University Toronto, Ontario M3J 1P3 · Department of Mathematics and Statistics York University Toronto, Ontario M3J 1P3
AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality
Aug 4, 2026Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber +2LLM Evaluation Scholarly Peer Review
Social Computing Research Group Technical University of Munich Boltzmannstraße 3, 85748, Garching, Germany

Aug 5, 2026, cs.CLJ/K move · Enter open · S save

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

Yinghui He, Ling Yang, Jiarui Liu, Yongjin Yang+5

Princeton University · Carnegie Mellon University · University of Toronto+3

LLM Evaluation Reasoning Evaluation RL for Language Model Reasoning

Open paper AskPDF arXiv

Long-horizon reasoning in recent LLMs demands that the model switch between distinct skills inside a reasoning chain, such as first doing a math derivation, then using the result to plan a schedule. We call such problems cross-skill long-horizon tasks: multi-step tasks whose steps require different reasoning skills and depend on earlier outputs. Existing benchmarks often evaluate individual skills, lacking a principled way to measure how well a model switches between skills. We address this gap from both the evaluation and training sides. We introduce Skill Entropy, a measure of the difficulty of switching from one skill to another. We then propose Skill^2-Bench, a benchmark of cross-skill long-horizon tasks built over 558 skills across 9 verifiable and open-ended domains. Each task is assigned a task-level skill-entropy score and grouped into three difficulty levels. Evaluating 8 frontier and 4 open-source models on Skill^2-Bench reveals a skill-switching gap: accuracy decreases on higher-entropy tasks. We then turn skill entropy from a benchmark scale into a training signal. We propose Skill-Entropy RL, an RL framework where the model predicts not only the answer at each step but also the skill used to produce it. The reward combines step-level correctness with a skill-entropy reward that measures the alignment between the model-predicted skill sequence and the gold skill sequence. On Qwen3-4B-Instruct and Qwen3-1.7B, Skill-Entropy RL improves the Skill^2-Bench score from 34.4% to 68.4% and from 14.6% to 40.1%, respectively, outperforming competitive baselines. The same pipeline can be applied to off-the-shelf training data such as OpenR1-Math, indicating that skill entropy is a reusable training signal. Code available at: https://github.com/Gen-Verse/Skill-Entropy-RL

Momentum

Related topics

Latest papers 1,643