LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
Mar 30, 2026Han Wang, Yifan Sun, Brian Ko +8CoT Faithfulness LLM Evaluation
University of Illinois Urbana-Champaign · University of Washington · University of California San Diego
Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP
Mar 30, 2026Urja Khurana, Michiel van der Meer, Enrico Liscio +2LLM Evaluation
Department of Intelligent Systems, Delft University of Technology · Leiden Institute of Advanced Computer Science (LIACS), Leiden University · Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam
Learning to Predict Future-Aligned Research Proposals with Language Models
Mar 28, 2026Heng Wang, Pengcheng Jiang, Jiashuo Sun +4LLM Evaluation Research Idea Generation
University of Illinois Urbana-Champaign
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Mar 27, 2026Juan Gabriel Kostelec, Qinghai GuoLLM Inference Efficiency LLM Evaluation
Huawei Switzerland · ACS Lab, Huawei Technologies
In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts
Mar 26, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation Knowledge Conflicts in Language Models
Institute for Continuum and Material Mechanics, Technical University of Hamburg, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, GermanyJun · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 215022026 Geesthacht, Germany +2
JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
Mar 24, 2026Yuhui Wang, Zhixiong Yang, Ming Zhang +10LLM Evaluation Multi-Turn Dialogue Evaluation
1Fudan University
PROMPT2BOX: Uncovering Entailment Structure among LLM Prompts
Mar 22, 2026Neeladri Bhuiya, Shib Sankar Dasgupta, Andrew McCallum +1LLM Evaluation Representation Learning
CICS, University of Massachusetts Amherst, USA · Amazon AWS AI
Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable
Mar 20, 2026Rounak Saha, Gurusha Juneja, Dayita Chaudhuri +3LLM Evaluation AI-Generated Text Detection
♠Indian Institute of Science · ♢University of California, Santa Barbara · ♡Carnegie Mellon University
When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal
Mar 20, 2026Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien +5Audio-Language Model Evaluation LLM Evaluation
National Taiwan University, Taiwan
ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs
Mar 19, 2026Wanjia Zhao, Ludwig Schmidt, Yejin Choi +3LLM Evaluation AI Agent Benchmarks
Stanford University · Microsoft Research
Toward Measuring Structural Drift in LLM Communication Loops
Mar 18, 2026Wael Hafez, Amir Nazeri, Chenan WeiLLM Evaluation
Semarx Research LLC Alexandria, VA, USA
Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models
Mar 17, 2026Mohamed Adel, Bashar Alhafni, Nizar HabashLLM Evaluation Arabic NLP
Computational Approaches to Modeling Language Lab 1New York University Abu Dhabi · 2Mohamed bin Zayed University of Artificial Intelligence
SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era
Mar 17, 2026Han Jang, Junhyeok Lee, Kyu Sung ChoiLLM Evaluation Scientific Communication
Seoul National University · Seoul National University Hospital · Seoul National University College of Medicine
Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessions
Mar 12, 2026Tae-Eun SongLLM Self-Correction LLM Evaluation
Daejeon Jungang Cheonggua Co., Ltd.
Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions
Mar 4, 2026Qianyun Guo, Yibo Li, Yue Liu +1LLM Evaluation LLM Personalization
National University of Singapore, Singapore.
CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents
Mar 4, 2026Martin Kostelník, Michal Hradiš, Martin DočekalDocument Understanding Historical Document Analysis
Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic
Verify to Amplify: Improving Reasoning via Learned Chain-of-Thought Verification
Mar 3, 2026Maria-Florina Balcan, Avrim Blum, Kiriaki Fragkia +2LLM Evaluation LLM Answer Verification
Carnegie Mellon University · Toyota Technological Institute at Chicago · Northwestern University
A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities
Mar 3, 2026Faiz Ghifari Haznitrama, Faeyza Rishad Ardi, Alice OhLLM Evaluation
School of Computing, KAIST, Daejeon, South Korea
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Feb 28, 2026Zeyu Chen, Huanjin Yao, Ziwang Zhao +1LLM Evaluation LLM-as-a-Judge
HEARTS: Benchmarking LLM Reasoning on Health Time Series
Feb 25, 2026Sirui Li, Shuhan Xiao, Mihir Joshi +4LLM Evaluation Time Series Reasoning
University of California, Los Angeles, 2Google Research · 2Google Research
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Feb 24, 2026Santiago Gonzalez, Alireza Amiri Bavandpour, Peter Ye +48Mathematical Reasoning Benchmarks LLM Evaluation
Disentangling Geometry, Performance, and Training in Language Models
Feb 24, 2026Atharva Kulkarni, Jacob Mitchell Springer, Arjun Subramonian +1LLM Evaluation Neural Representation Geometry
University of Southern California · 2Carnegie Mellon Univer · University of California, Los Angeles.
CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
Feb 23, 2026Yuzhe Wang, Yaochen Zhu, Jundong LiLLM Evaluation Causal Reasoning in Language Models
University of Virginia Charlottesville, Virginia, USA
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Feb 20, 2026Baris Arat, Emre SeferLLM Evaluation LLM Reranking
Ozyegin University Istanbul, Turkey
VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
Feb 20, 2026Yutong Xin, Qiaochu Chen, Greg Durrett +1LLM Evaluation Formal Verification
The University of Texas at Austin · New York University
Capabilities Ain't All You Need: Measuring Propensities in AI
Feb 20, 2026Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11LLM Evaluation Item Response Theory
Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands +5
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34LLM Evaluation Benchmark Design
ETH Zurich · ETH AI Center · Stanford University +23
SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
Feb 16, 2026Srivatsa Kundurthy, Clara Na, Michael Handley +5LLM Evaluation Language Model Generation Evaluation
Longitude Labs Inc., New York, NY · Cornell University, Ithaca, NY · Carnegie Mellon University, Pittsburgh, PA +1
On Calibration of Large Language Models: From Response To Capability
Feb 14, 2026Sin-Han Yang, Cheng-Kuang Wu, Chieh-Yen Lin +3LLM Evaluation Confidence Estimation in Language Models
Appier AI Research · National Taiwan University
Investigating Learner-Aware Design of LLM-Generated Educational Feedback
Feb 12, 2026Momoka Furuhashi, Kouta Nakayama, Noboru Kawai +3LLM Evaluation Educational Technology
Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · Osaka Kyoiku University +2

Mar 30, 2026, cs.AIJ/K move · Enter open · S save

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

Han Wang, Yifan Sun, Brian Ko, Mann Talati+7

University of Illinois Urbana-Champaign · University of Washington · University of California San Diego

CoT Faithfulness LLM Evaluation CoT Monitoring

Open paper AskPDF arXiv

Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the actual reasons (i.e., decision-critical factors) driving the model's behavior, leading to the reduced CoT monitorability problem. This limits the use of CoTs for reliable oversight. However, a comprehensive and fully open-source benchmark for thoroughly evaluating CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two prompting stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments show that CoT monitorability is a conditional property affected by the evaluated LLM, monitor LLM, and task characteristics. Across these factors, monitorability is higher when decision-critical factors shape the intermediate reasoning process, rather than merely influencing the final answer. Under stress-test prompting, most evaluated LLMs can intentionally reduce monitorability, mainly on tasks where decision-critical factors are not structurally required by the reasoning process. Overall, MonitorBench provides a basis for further research on AI control, reasoning faithfulness, stress-test monitorability, and monitoring scaffords. The code is available at https://github.com/ASTRAL-Group/MonitorBench.

Momentum

Related topics

Latest papers 1,631