Model Evaluation

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 167% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 62

All topics

Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair
Oct 6, 2026Anvi Kalpesh Shah, Umamaheswara Sharma BAutomated Program Repair Code Quality
National Institute of Technology, Calicut
The Price of Correlated Tests: How Strict Should a Model Release Gate Be?
Oct 1, 2026Marco PollanenRelease Gating
Department of Mathematics & Statistics Trent University Peterborough, ON, Canada
A First Glance at Jev for Network Traffic Classification: Accuracy, Processing Time, and Cost
Sep 30, 2026Shenghe Xu, Lifan MeiModel Evaluation Cloud
Amazon.com, Inc. · Xi’an Jiaotong-Liverpool University
Reliability Testing of Medical Model Performance under Distributed Deployment
Sep 29, 2026Yifei Wang, Xiaohan Zhang, Youtao Ding +4Model Evaluation Pre-Deployment Safety Assessments
Shanghai Jiao Tong University, China · Beihang University, China · Nanyang Technological University, Singapore +1
Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks
Sep 28, 2026Manya Singh, Arjun PakrashiAmbiguity Multi-Label Classification
School of Computer Science, University College Dublin, Ireland
A Visual Classification Dataset and Model Evaluation for Historical Manuscript Illustrations
Sep 27, 2026Yoav Evron, Michal Bar-Asher Siegal, Michael FireVision Datasets Historical Manuscripts
Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Be’er Sheva, Israel · The Goldstein-Goren Department of Jewish Thought, Ben-Gurion University of the Negev, Be’er Sheva, Israel
Evaluating Explanation Methods by the Predictors They Induce
Sep 17, 2026Jacob Selbæk, Hugo L. HammerShapley Additive Explanations Explainable AI Methods
Department of Computer Science, Oslo Metropolitan University, Oslo, Norway · Department of Holistic Systems, SimulaMet, Oslo, Norway · Department of Plastic and Reconstructive Surgery, Oslo University Hospital, Oslo, Norway
The widening evaluation gap in medical large language model research 2023 to 2026
Sep 11, 2026Raad Bin Tareaf, Murad Al-Rajab, Samia LoucifSystematic Review Large Language Model Evaluation
XU Exponential University of Applied Sciences, Data Science and Artificial Intelligence Cluster, Marlene-Dietrich-Allee 12B, 14482 Potsdam, Germany · College of Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates · College of Technological Innovation, Zayed University, Abu Dhabi, United Arab Emirates
Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation
Sep 7, 2026Keren Artiaga, Sabyasachi Kamila, Haithem Afli +2Gloss-Free Sign Language Translation Sign Language Translation
ADAPT Centre, Munster Technological University, Cork, Ireland · Manipal Institute of Technology Bengaluru, MAHE, Manipal, India · Nimbus Research Centre, Munster Technological University, Cork, Ireland +1
Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity
Sep 1, 2026Sinjini Banerjee, Tim Marrinan, Anand D. SarwateEnsemble Model Evaluation
Department of Electrical and Computer Engineering, Rutgers University · Pacific Northwest National Lab
Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation
Aug 30, 2026Jhen-Ke LinModel Evaluation Multi-Dimensional Evaluation
National Yang Ming Chiao Tung University
Incremental Evaluation and Training in Relational Deep Learning
Aug 13, 2026Jakub Peleška, Gustav ŠírRelational Deep Learning Retraining
Czech Technical University in Prague
Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction
Aug 5, 2026Quinn Ledingham, Zhengsen Xu, Yimin Zhu +6Wildfire Prediction Rainfall
Department of Geomatics Engineering, University of Calgary, Calgary, AB, Canada
Dynamically Allocating Evaluation Effort for Model Ranking
Aug 4, 2026Vilém Zouhar, Julia Kreutzer, Alon Lavie +4Model Evaluation Evaluation Benchmarks
ETH Zurich · Cohere Labs · Carnegie Mellon University +3
Object Detection for Autonomous Driving in Chinese Rural Scenes: An Experimental Study on Real-Synthetic Data Mixing and Model Evaluation
Jul 29, 2026Danning Zhu, Ziyan Lin, Jing WuNaturalistic Driving Data Object Detection
Wuhan University of Technology Wuhan, China
BayesAME: Bayesian Active Model Evaluation
Jul 29, 2026Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet +2Geometry-Aware Uncertainty Coresets Model Selection
Imperial College London (work conducted while at Google DeepMind) · Google DeepMind
Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods
Jul 29, 2026Yearn Tan Yin Tze, Charles GrelloisModel Evaluation Benchmark Datasets
School of Computer Science, University of Sheffield, U.K.
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
Jul 20, 2026Zhaokai Wang, Tianlin Gui, Jiayuan Rao +3Large Language Model Forecasting Model Evaluation
Shanghai Jiao Tong University · Shanghai, China · Nanjing University +4
Can We Trust Item Response Theory for AI Evaluation?
Jul 16, 2026Han Jiang, Sunbeom Kwon, Jinwen Luo +2Artificial Intelligence Benchmarks Item Response Theory
1Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles
Genre Bias or Aesthetic Perception? Identifying and Mitigating Shortcut Learning in Music Evaluation
Jul 15, 2026Yizhou Zhang, Wangjin Zhou, Yi Zhao +3Aesthetics Model Evaluation
Graduate School of Informatics, Kyoto University, Japan · WXG, Tencent, China
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Jul 9, 2026Ofir Arviv, Kristjan Greenewald, Yotam Perlitz +3Model Evaluation Benchmark Saturation
IBM Research
CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion
Jul 6, 2026Adam Fisch, Daniel Deutsch, Joshua Maynez +5Model Evaluation Multi-Dimensional Evaluation
Google DeepMind · Google Research
HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
Jun 29, 2026Xinrui Ruan, Zhenyu Zhao, Waverly Wei +4Model Evaluation Human Annotations
Division of Biostatistics, University of California, Berkeley · Roblox Corporation · Department of Data Sciences and Operations, University of Southern California +2
Certification of Machine Learning Models via Directional Sharpness
Jun 23, 2026Gefei Tan, Adria Gascon, Sarah Meiklejohn +1Certification Sharpness-Aware Minimization
Northwestern University & Google · Google · University College London
You Don't Need to Run Every Eval
Jun 22, 2026Yuchen Zeng, Dimitris PapailiopoulosModel Evaluation Frontier Models
Microsoft Research, AI Frontiers
MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery
Jun 17, 2026Hongxuan Liu, Roman Bushuiev, Ivy Lightheart +12Tandem Mass Spectra Mle-Bench Lite
1Massachusetts Institute of Technology · Institute of Organic Chemistry and Biochemistry of the Czech Academy of Sciences · 3Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague +2
Model Stealing Through the Lens of Model Multiplicity
Jun 13, 2026Eliott Baltz, Satoshi Hara, Ulrich AïvodjiSurrogates Model Evaluation
§ÉTS, Mila : Montréal · ‡The University of Electro-Communications : Tokyo
AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages
Jun 10, 2026Happy Buzaaba, Cheikh Mouhamadou Bamba Dione, David Ifeoluwa Adelani +15African Languages Syntactic Structure
Princeton University · Laboratory for Artificial Intelligence, Princeton University · Gaston Berger University +20
When Do Local Score Models Extrapolate Across Size? A Diagnostic Theory and Benchmark
Jun 8, 2026Wenjie XiModel Size Extrapolation
Department of Physics and HK Institute of Quantum Science & Technology The University of Hong Kong, Pokfulam Road, Hong Kong, China
Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation
Jun 7, 2026Bitya Neuhof, Yuval BenjaminiModel Evaluation Leaderboard
Department of Statistics and Data Science The Hebrew University of Jerusalem
Hybrid CNN-LSTM Framework for Intelligent Cyber Attack Detection and Prevention in U.S. Critical Digital Infrastructure: A Comparative Machine Learning Evaluation on CSE-CIC-IDS2018
Jun 4, 2026Md. Iqbal Hossan, Md. Serajul Kabir Chowdhury Rubel, Md. Arifur Rahman +1Threat Detection Intrusion Detection
Department of Computer Science, Maharishi International University, Fairfield, Iowa, USA. · Department of Information Studies, Trine University, Angola, Indiana, USA. · Department of Business Information Systems, Central Michigan University, Mount Pleasant, Michigan, USA.
GENEB: Why Genomic Models Are Hard to Compare
Jun 3, 2026Daria Ledneva, Mikhail Nuridinov, Denis KuznetsovGenome-Wide Association Study Model Evaluation
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia · Moscow State Institute of Steel and Alloys, Moscow, Russia
TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation
May 29, 2026Yeil Jeong, Youngjin Yoo, Jiyoung Bae +5Model Evaluation
Indiana University Bloomington · Pai Chai University · Korea University +4
Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialogues
May 28, 2026Zhangqi Duan, Shuyan Huang, Alexander Scarlatos +3User Simulation Tutors
University of Massachusetts Amherst · 2Eedi
A Full-Pipeline Framework for Evaluating Membership Inference Attacks in Machine Learning
May 28, 2026Ding Chen, Xinwen Cheng, Xuyang Zhong +3Membership Inference Attacks Model Evaluation
City University of Hong Kong · Shanghai Jiao Tong University · denotes the correspondence author.
Semi-Supervised Hypothesis Testing by Betting on Predictions
May 27, 2026Yaniv Tenzer, Elad Tolochinsky, Yaniv RomanoUnlabeled Data Two-Sample Testing
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology
The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates
May 27, 2026Shaobo Wang, Guo Chen, Ziyue Wang +5Pretraining Decoding
Shanghai Jiao Tong University · Qwen Team, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
RULER: Representation-Level Verification of Machine Unlearning
May 26, 2026Georgina Cosma, Axel FinkeMachine Unlearning Representation-Level Misalignment
Department of Computer Science, Loughborough University, UK · School of Mathematics, Statistics and Physics, Newcastle University, UK
Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild
May 22, 2026Zhimin Zhao, Zehao Wang, Abdul Ali Bangash +2Model Evaluation Engineering
Software Analysis and Intelligence Lab (SAIL), School of Computing, Queen’s University, Canada · Concordia University, Canada · Lahore University of Management Sciences (LUMS), Pakistan
Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning
May 22, 2026Trinh Pham, Viet Huynh, Hongzhi Yin +2Model Evaluation Unlabeled Data
Griffith University · Edith Cowan University · The University of Queensland
An Objective Performance Evaluation of the LSTM Networks in Time Series Classification
May 19, 2026Sooraj Sunil, Balakumar BalasingamTime-Series Classification Cnn-Lstm
Electrical and Computer Engineering University of Windsor Windsor, ON, Canada
Training ML Models with Predictable Failures
May 14, 2026Will Schwarzer, Scott NiekumEarly Failure Prediction Pre-Deployment Safety Assessments
University of Massachusetts Amherst
NodeSynth: Socially Aligned Synthetic Data for AI Evaluation
May 14, 2026Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5Synthetic Data Generative Artificial Intelligence
Google Research · Google · Google Deepmind
The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
May 13, 2026Adil AminFrontier Models Leaderboard
ZEHEN Labs
A Transfer Learning Evaluation of Deep Neural Networks for Image Classification
May 12, 2026Nermeen Abou Baker, Nico Zengeler, Uwe HandmannTransfer Learning Pretraining
Computer Science Institute, Ruhr West University of Applied Sciences, 46236 Bottrop, Germany
Slum Detection and Density Mapping with AlphaEarth Foundations: A Representation Learning Evaluation Across 12 Global Cities
May 11, 2026Shuyang Hou, Ziqi Liu, Haoyue Jiao +7Geospatial Foundation Models Urban Environments
State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan, China
FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning
May 8, 2026Fabian Stricker, Jose A. Peregrina, David Bermbach +1Federated Learning Model Evaluation
Institute of Data-Centric Software Systems (IDSS), Hochschule Karlsruhe, Karlsruhe, Germany · Scalable Software Systems Research Group, TU Berlin, Berlin, Germany
Query-efficient model evaluation using cached responses
May 8, 2026Hayden Helm, Ben Johnson, Carey PriebeModel Evaluation Responses
Helivan · Jataware · Johns Hopkins University
Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions
May 7, 2026Chengjie Wang, Jingzheng Wu, Xiang Ling +2Vulnerable Code Code Quality
Intelligent Software Research Center, Institute of Software, Chinese Academy of Sciences, Beijing, China · Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China
Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols
May 6, 2026Jaewook Kim, Hyeoncheol KimModel Evaluation Traces
Korea University, Seoul, Republic of Korea
Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone
May 6, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1Model Evaluation Artificial Intelligence Alignment
Department of Computer Science, University of Oxford
Sentiment Analysis of Mobile Legends App Reviews Using Machine Learning and LSTM-Based Deep Learning Models
May 2, 2026Vira Putri Maharani, Kharisa Harvanny, Daris Samudra +3Sentiment Term Frequency-Inverse Document Frequency
Department of Data Science Institut Teknologi Sumatera Lampung Selatan, Indonesia
Measuring the Sensitivity of Classification Models with the Error Sensitivity Profile
Apr 28, 2026Andrea MaurinoFeature Importance Model Evaluation
Università degli studi di Milano Bicocca Dipartimento di Informatica, Sistemistica e Comunicazione Viale Sarca 336,20126 Milano, Italy
PLMGH: What Matters in PLM-GNN Hybrids for Code Classification and Vulnerability Detection
Apr 28, 2026Mohamed Taoufik Kaouthar El Idrissi, Edward Zulkoski, Mohammad HamdaqaGnn-Based Detectors Graph Neural Networks
Polytechnique Montréal Montréal, Canada · Quantstamp San Francisco, United States
SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization
Apr 21, 2026Bo-Jyun Wang, Ying-Jia Lin, Hung-Yu KaoSummarization Large Language Model Reranking
Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Artificial Intelligence, Chang Gung University · Artificial Intelligence Research Center, Chang Gung University +1
FLARE: Task-agnostic embedding model evaluation through a normalization process
Apr 19, 2026Jingzhou Jiang, Yixuan Tang, Yi Yang +1Model Evaluation Latent Flow
The Hong Kong University of Science and Technology
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Apr 18, 2026Tyler H. Merves, Michael H. Conaway, Joseph M. Escobar +2Attacker Large Language Model Large Language Model Agents
Department of Cybersecurity · Department of Information Sciences and Technology
DEEP-GAP: Deep-learning Evaluation of Execution Parallelism in GPU Architectural Performance
Apr 16, 2026Kathiravan PalaniappanNvidia Efficient Inference
Independent Researcher (University of Colorado Colorado Springs Alumni)
The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models
Oct 27, 2025Timo Freiesleben, Sebastian ZezulkaModel Evaluation Epistemic State Replication
LMU Munich, MCMP & MCML · University of Tübingen
Perspective of Software Engineering Researchers on Machine Learning Practices Regarding Research, Review, and Education
Nov 28, 2024Anamaria Mojica-Hanke, David Nader Palacio, Denys Poshyvanyk +2Software Engineering Model Evaluation
University of Passau, Passau, Germany. · Microsoft, Redmond, WA, USA · William & Mary, Williamsburg, USA +1

Oct 6, 2026, cs.SEJ/K move · Enter open · S save

Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair

Anvi Kalpesh Shah, Umamaheswara Sharma B

National Institute of Technology, Calicut

Automated Program Repair Code Quality Model Evaluation

Open paper AskPDF arXiv

Automated Program Repair (APR) with language models is usually evaluated by whether a generated patch passes the test suite, which can hide differences in maintainability, security, and computational cost. We propose a Weighted Quality Index (QI), inspired by the ISO/IEC 25010 software quality model, that combines functional correctness, maintainability, security, and generation efficiency under configurable weighting schemes. We evaluate three dense Qwen2.5-Coder models (3B, 7B, 14B) and the 16B-parameter DeepSeek-Coder-V2-Lite Mixture-of-Experts (MoE) model (2.4B active parameters) on 40 QuixBugs and 90 Defects4J bugs, all run locally on identical hardware to control for infrastructure effects. Model rankings change with the weighting scheme, showing that single-metric evaluation can hide trade-offs. The MoE model shows almost no statistically significant difference in correctness from the 7B and 14B dense models (McNemar's exact test) while using 3-6 times fewer active parameters, whereas correctness increases significantly across the three dense scales. These results suggest that active parameter count can be a more informative lens than total parameter count for sparse code models.

Momentum

Related topics

Latest papers 62