LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation
Sep 21, 2026Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang +3VLM Evaluation LLM Evaluation
CSU-JPG, Central South University · National University of Singapore
OSCAR: Order-aware Scoring and Calibration for AI Rankings
Sep 21, 2026You Liu, Yue Liu, Quanchao Lu +1LLM Evaluation LLM-as-a-Judge
From Tables to Quantified Statements: Evaluating LLM Inference Generation through Executable Verification
Sep 21, 2026Mai Mohamed Eida, Gunjan Anand, Ayush Singh +1Table QA LLM Evaluation
University of Illinois Urbana Champaign
Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model
Sep 21, 2026Simiao Ren, Kidus Zewde, Xingyu Shen +6LLM Inference Efficiency LLM Evaluation
Scam.ai (Reality Inc.)
Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap
Sep 20, 2026Akash ChavanConstrained Decoding LLM Evaluation
Financial Language Models as Applied Artificial Intelligence Systems for News-Based Trading under Market Frictions
Sep 20, 2026Kemal KirtacLLM Evaluation Financial Sentiment Analysis
Department of Computer Science, University College London, 66–72 Gower Street, London WC1E 6EA, United Kingdom · Department of Computer Science, University of Warwick, 6 Lord Bhattacharyya Way, Coventry CV4 7EZ, United Kingdom
Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Tasks
Sep 20, 2026Jie Ying, Zhefan Wang, Zihong Chen +11LLM Evaluation On-Policy Distillation
Shanghai Artificial Intelligence Laboratory · Yazhouwan National Laboratory · Shanghai Innovation Institute +2
Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM Evaluation Text Summarization
ServiceNow Canada
Chinese Competitive Debating Dataset and Benchmark
Sep 18, 2026Zongrui Yang, Haoyuan Li, Zhongsheng Wang +5LLM Evaluation
University of Auckland, New Zealand · South China Agricultural University, China · Sanming University, China
Intrinsic Sequence-Likelihood Confidence in Retrieval-Dominated Extractive QA: Two Pre-Specified Negatives, and What They Do and Do Not Attribute
Sep 17, 2026Gunwoo Lee, Changmin Sung, Sang-Hwan Gwak +3LLM Evaluation Selective Prediction
Large-scale AI Research Center, Korea Institute of Science and Technology Information (KISTI), 245 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea. · Department of Applied AI, University of Science and Technology (UST), 217 Gajeong-ro, Yuseong-gu, Daejeon, 34113, Republic of Korea.
Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models
Sep 17, 2026Ha Van Dau, Thanh Tung Khuat, Nguyen Thanh DungLLM Evaluation Recurrent Transformers
NuverxAI - AI & Creative Innovation Company Limited · Ho Chi Minh City University of Technology (HCMUT)
KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms
Sep 17, 2026Soha Lee, Soojin Lee, Heesung Yang +8LLM Evaluation Language Modeling
School of Computer Science and Engineering, Kyungpook National University · International Exchange Department, Kyungpook National University · Dept. of Korean Language and Literature Education, Kyungpook National University +4
PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces
Sep 17, 2026Pyrros Koussios, Benjamin Jäger, John Hua Yao +3LLM Evaluation Temporal Reasoning in Language Models
ETH Zürich · Stanford University
Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Sep 17, 2026Veronika Batzdorfer, Carlo Romano Marcello Alessandro SantagiustinaLLM Evaluation LLM-Assisted Annotation
KIT · Department of Sociology and Computational Sociology, Karlsruhe Institute of Technology (KIT), Douglasstr. 24, 76133 Karlsruhe, Germany · ALMAnaCH, médialab, Sciences Po +2
F $^{2}$ DR: A Fine-Grained Full-Pipeline Reward Framework for DeepSearch Workflows
Sep 17, 2026Bojian Xiong, Wentao Ding, Yujing Lu +11Reward Modeling LLM Evaluation
Tianjin University · TJUNLP Lab, Tianjin University, Tianjin, China · Baidu Inc. +1
A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems
Sep 17, 2026Shaina Raza, Ahmed Y. Radwan, Imran Liaquat +1LLM Evaluation AI Agent Evaluation
Vector Institute for Artificial Intelligence, Toronto, ON M5G 0C6, Canada
A Benchmark Framework for Screening Automation in Systematic Reviews
Sep 16, 2026Gauransh Kumar, Luciano Marchezan, Guillaume Genois +2LLM Evaluation Evidence Selection
Université de Montréal Montreal, Canada
Playing log(N)-Questions over Wikipedia Abstracts: How Per-Round Errors Compound Under Information Asymmetry
Sep 16, 2026Peter PotashLLM Evaluation Pragmatic Reasoning in Language Models
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Reward Hacking LLM Evaluation
Core contributors · Equal senior contribution
Which LLM is Best for Translating Natural Language Goals to PDDL
Sep 16, 2026Tomas Balyo, Lukas Chrpa, G. Michael YoungbloodLLM Evaluation LLM Planning
Filuta AI, Inc., 1606 Headway Cir STE 9145, Austin, TX 78754, USA
Cultural Competence in Context: A Large Language Model Passes the Turing Test in Finland
Sep 16, 2026Otto Segersven, Pentti HenttonenMultilingual Language Model Evaluation LLM Evaluation
Department of Computer Science, University of Helsinki · Faculty of Social Sciences, University of Tampere
GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models
Sep 16, 2026Alexander Didenko, Anna Shabanova, Vladislav Zapylikhin +2LLM Evaluation Cross-Cultural Language Model Evaluation
Artificial Intelligence Laboratory, SKOLKOVO School of Management, Moscow · Artificial Intelligence Laboratory, University of Tyumen · Institute of Business Studies, Russian Presidential Academy of National Economy and Public Administration, Moscow
Made in Hungary: Comments on the performance of generative language models
Sep 16, 2026Mátyás Osváth, Enikő Héja, Noémi Ligeti-NagyLLM Evaluation Language Model Generation Evaluation
Institute for Language Technologies and Applied Linguistics ELTE Research Centre for Linguistics Budapest, Hungary
Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagement
Sep 16, 2026Xinglang Zhang, Yuanmeng Xiang, Yunyao Zhang +3LLM Evaluation Preference Alignment
Huazhong University of Science and Technology
BENCHCOMPASS: From Scores to Signals for Training and Harness Decisions in Payment-Domain LLMs
Sep 16, 2026Sijie Dong, Wei Ren, Xuanwei Hu +9Financial Services LLM Evaluation
Ant International
A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
Sep 16, 2026Jerry KaplanLLM Quantization LLM Evaluation
Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University
The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?
Sep 15, 2026Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3LLM Inference Efficiency LLM Quantization
Vizuara
Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
Sep 15, 2026Adam Zachary Wasserman, David BeaucheminLLM Evaluation Cross-Lingual Representation Alignment
1Open Honest Foundation · 2Group for Research in Artificial Intelligence of Laval University (GRAIL), Université Laval, Québec, Canada
Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
Sep 15, 2026Vanessa Lama, Sanjay Das, Emily Herron +5LLM Evaluation Scientific QA
Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA · The Ohio State University, Columbus, Ohio, USA · Max-Planck-Institut für Astronomie, Königstuhl 17, D-69117 Heidelberg, Germany +1
Can LLMs Follow the Pulse of a Crisis? Evaluating Crisis Sentiment in Bangladesh's July Uprising
Sep 15, 2026Md. Samiul Alim, Mahir Shahriar Tamim, Tanvir Ahmed Khan +4LLM Evaluation Sentiment Analysis
North South University, Dhaka, Bangladesh · Charles Sturt University, Australia

Sep 21, 2026, cs.CVJ/K move · Enter open · S save

ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation

Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang, Jiahao Tang+2

CSU-JPG, Central South University · National University of Singapore

VLM Evaluation LLM Evaluation LLM-as-a-Judge Code Generation LLM Reliability Vision-Language Model-as-a-Judge

Open paper AskPDF arXiv

Building strong chart-to-code systems increasingly relies on reinforcement learning, whose effectiveness depends critically on the quality of the reward signal. Large Multimodal Models (LMMs) play a natural critical role in jointly assessing chart visual appearance and task requirements. They are therefore increasingly used as visual critics and reward models, yet their reliability as judges remains largely unexplored. To this end, we introduce ChartJudgeBench, a diagnostic vision-language benchmark for assessing LMM judges in chart-to-code workflows. It includes 1,003 Chart Perception Alignment (CPA) instances for pairwise chart comparison and 650 Chart Reasoning Judgment (CRJ) instances for binary Accept/Reject verification in Chart Reproduction and Chart Editing. Together, these tasks emulate the core judging decisions required in agentic refinement and RL-based chart optimization. Our evaluation of strong LMMs reveals four systematic limitations: (i) positional bias in pairwise comparison, (ii) a strong tendency to overpredict Accept, (iii) difficulty in matching visual styles and aesthetics, and (iv) an unexpected leniency bias in RL-trained models. These findings show that current LMM judges require explicit reliability validation before being used as critics or reward models in chart-to-code optimization. The code and data are available on ChartJudgeBench.

Momentum

Related topics

Latest papers 1,631