LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraints
May 9, 2026Jiaxiang Geng, Yiyi Lu, Lunyu Zhao +3LLM Evaluation LLM Fine-Tuning
Duke Kunshan University · Flower Labs · University of Cambridge
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
May 9, 2026Devin Yasith De Silva, Dhaval Patel, Christodoulos Constantinides +7Logical Reasoning LLM Evaluation
2Rensselaer Polytechnic Institute · 1IBM · 3Independent Contributor
Evaluating Developmental Cognition Capabilities of LLMs
May 8, 2026Xiao Xiao, Hayoun Noh, Mar Gonzalez-FrancoLLM Evaluation
De Vinci Research Center, France · University of Oxford, UK · Google, USA
Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation
May 8, 2026Adib Sakhawat, Tahsin Islam, Takia Farhin +3LLM Evaluation Language Model Robustness
Systems and Software Lab (SSL) Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh
CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs
May 8, 2026Shiyang Li, Zijian Zhang, Guangyan Sun +5LLM Evaluation GPU Kernel Optimization
University of Minnesota · Northeastern University
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
May 8, 2026I. F. Atasoy, B. Mutlu, E. A. Sezer +1LLM Evaluation LLM-Assisted Annotation
Department of Computer Engineering, Hacettepe University, Ankara, 06800, Türkiye · Department of Computer Engineering, Ankara University, Ankara, 06830, Türkiye · Zephlen AI and Information Technologies Inc., Kocaeli, 41400, Türkiye
CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging
May 8, 2026Shiyang Li, Haoyang Chen, Mattia Fazzini +1LLM Evaluation Automated Program Repair
University of Minnesota
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
May 8, 2026Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6Legal NLP LLM Evaluation
Maritaca AI Campinas, São Paulo, Brazil · Jusbrasil Salvador, Bahia, Brazil
A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
May 8, 2026Zhanliang Wang, Jiancong Xiao, Ruochen Jin +3LLM Evaluation Confidence Estimation in Language Models
University of Pennsylvania, Philadelphia, PA · Dartmouth College, Hanover, NH
Sanity Checks for Long-Form Hallucination Detection
May 8, 2026Geigh Zollicoffer, Minh Vu, Hongli Zhan +2LLM Evaluation Hallucination Detection
Los Alamos National Laboratory · The University of Texas at Austin · University of British Columbia
Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need?
May 8, 2026Jane Paik KimLLM Evaluation LLM-as-a-Judge
Department of Psychiatry and Behavioral Sciences Stanford University Stanford, CA 94304
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
May 8, 2026Hexuan Deng, Xiaopeng Ke, Yichen Li +6LLM Evaluation Automated Peer Review
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Zhongguancun Academy, Beijing, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China +2
Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
May 8, 2026Sree Bhattacharyya, Samarth Khanna, Leona Chen +3LLM Evaluation Confidence Estimation in Language Models
Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.
Tracing Uncertainty in Language Model "Reasoning"
May 8, 2026Nils Grünefeld, Bertram Højer, Philipp Mondorf +5LLM Evaluation CoT Reasoning
Data Science Section, IT University of Copenhagen · Pioneer Centre for Artificial Intelligence · MaiNLP, Center for Information and Language Processing, LMU Munich +2
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
May 8, 2026Andrea Sassella, Andrea Chizzola, Tommaso Bianchi +2LLM Evaluation Mixture-of-Experts Language Models
2DEIB, Politecnico di Milano · 1AIRIC, Politecnico di Milano
FactoryBench: Evaluating Industrial Machine Understanding
May 8, 2026Yanis Merzouki, Coral Izquierdo, Matei Ignuta-Ciuncanu +8Manufacturing LLM Evaluation
1ETH Zurich · 2Forgis · 3UC3M +4
Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation
May 8, 2026Abigail Victoria Gurin Schleifer, Moriah Ariely, Beata Beigman Klebanov +2LLM Evaluation AI in Education
Weizmann Institute of Science, Rehovot, Israel · ETS, Princeton, USA
Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction
May 8, 2026Adnan Labib, Qiao Wang, Yixuan Huang +1LLM Evaluation Grammatical Error Correction
King’s College London, London, United Kingdom · Hosei University, Tokyo, Japan · Waseda University, Tokyo, Japan +1
Post-training makes large language models less human-like
May 8, 2026Marcel Binz, Elif Akata, Abdullah Almaatouq +76LLM Evaluation LLM Alignment
Position: The Term "Machine Unlearning" Is Overused in LLMs
May 8, 2026Sangyeon Yoon, Yeachan Jun, Albert NoLLM Evaluation Machine Unlearning
Department of Artificial Intelligence, Yonsei University, Seoul, Korea.
Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study
May 8, 2026Moaath Alshaikh, Tasneem Alshaher, Ricardo Vieira +7Software Engineering LLM Evaluation
Federal University of Bahia · State University of Feira de Santana · Federal University of Sergipe +1
Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
May 8, 2026Saloni Garg, Amit SagtaniLLM Evaluation LLM-as-a-Judge
Independent Researcher San Francisco, California, USA · San Francisco State University San Francisco, California, USA
From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs
May 8, 2026Hanmeng Liu, Shichao Weng, Xiulai Liu +3Logical Reasoning LLM Evaluation
Hainan University, Haikou, China · Fudan University, Shanghai, China
FAME: Forecasting Academic Impact via Continuous-Time Manifold Evolution
May 8, 2026Jianrong Ding, Jianyuan Zhong, Zhengyan Shi +1AI for Science LLM Evaluation
Department of Computer Science and Engineering The Chinese University of Hong Kong
The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
May 8, 2026Zekai Tong, Ruiyao Xu, Aryan Shrivastava +2LLM Evaluation LLM Reliability
University of Chicago · 2Northwestern University
MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries
May 8, 2026Zixuan Xie, Xinyu Liu, Shangtong ZhangLLM Evaluation LLM Agent Evaluation
University of Virginia
The Position Curse: LLMs Struggle to Locate the Last Few Items in a List
May 8, 2026Zhanqi Zhang, Hua-Dong Xiong, Robert C. Wilson +3LLM Evaluation LLM Fine-Tuning
UC San Diego · Georgia Tech · New York University
SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
May 8, 2026Tianyu Wang, Nianjun ZhouComputational Literary Studies LLM Evaluation
Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center +1
NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models
May 8, 2026George Boateng, Naafi Ibrahim, Samuel John +8Mathematical Reasoning Benchmarks LLM Evaluation
ETH Zurich, Switzerland · Kwame AI Inc., U.S. · Ashesi University, Ghana +1
An Interpretable and Scalable Framework for Evaluating Large Language Models
May 7, 2026Xinhao Qu, Qiang Heng, Hao Zeng +1LLM Evaluation Item Response Theory
Department of Statistics, University of California, Riverside · School of Statistics and Data Science, Southeast University · Department of Statistics and Data Science, Southern University of Science and Technology

May 9, 2026, cs.CLJ/K move · Enter open · S save

EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraints

Jiaxiang Geng, Yiyi Lu, Lunyu Zhao, Yan Gao+2

Duke Kunshan University · Flower Labs · University of Cambridge

LLM Evaluation LLM Fine-Tuning Edge ML Federated Fine-Tuning Federated Learning

Open paper AskPDF arXiv

Federated fine-tuning offers a promising paradigm for adapting large language models (LLMs) on edge devices by leveraging the rich, diverse, and continuously generated data from smartphones and IoT devices without compromising user data privacy. Such edge-side adaptation can improve model personalization, robustness, and responsiveness to local contexts. However, the practical feasibility of federated LLM fine-tuning on real edge devices remains unclear, as most existing work focuses on cross-silo or simulation-based settings, overlooking the resource and runtime constraints that determine whether a method is deployable on real edge systems. We present EdgeFlowerTune, a deployment-oriented benchmark for federated LLM fine-tuning under realistic edge-system constraints. EdgeFlowerTune jointly evaluates model quality and system costs, including communication, wall-clock latency, memory usage, energy consumption, and robustness to dynamic edge conditions. To compare methods in terms of effectiveness, efficiency, and robustness, EdgeFlowerTune introduces three complementary protocols: Quality-under-Budget, Cost-to-Target, and Robustness. We instantiate EdgeFlowerTune as a real-device platform built on Flower and MobileFineTuner, spanning commercial Android smartphones and NVIDIA edge development boards. Our benchmark results show that accuracy-only evaluation can lead to misleading conclusions: methods with similar final quality may differ substantially in deployability once realistic system constraints are considered. EdgeFlowerTune provides a reproducible benchmark for system-aware evaluation of federated LLM fine-tuning at the edge.

Momentum

Related topics

Latest papers 1,643