Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection

Xuanyan Liu; Ignacio Cabrera Martin; Marcello Trovati; Xiaolong Xu; Nikolaos Polatidis

cs.LGApr 15, 2026

Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection

Authors: Xuanyan Liu, Ignacio Cabrera Martin, Marcello Trovati, Xiaolong Xu, Nikolaos Polatidis

Organizations: Jiangsu Key Laboratory of Big Data Security and Intelligent Processing, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China. · School of Cyber Science and Engineering, Southeast University, Nanjing, 211189, China. · University of Brighton, United Kingdom. · University of Lancashire, United Kingdom.

Abstract

The evaluation of supervised machine learning models is a critical stage in the development of reliable predictive systems. Despite the widespread availability of machine learning libraries and automated workflows, model assessment is often reduced to the reporting of a small set of aggregate metrics, which can lead to misleading conclusions about real-world performance. This paper examines the principles, challenges, and practical considerations involved in evaluating supervised learning algorithms across classification and regression tasks. In particular, it discusses how evaluation outcomes are influenced by dataset characteristics, validation design, class imbalance, asymmetric error costs, and the choice of performance metrics. Through a series of controlled experimental scenarios using diverse benchmark datasets, the study highlights common pitfalls such as the accuracy paradox, data leakage, inappropriate metric selection, and overreliance on scalar summary measures. The paper also compares alternative validation strategies and emphasizes the importance of aligning model evaluation with the intended operational objective of the task. By presenting evaluation as a decision-oriented and context-dependent process, this work provides a structured foundation for selecting metrics and validation protocols that support statistically sound, robust, and trustworthy supervised machine learning systems.

Figures & tables

ID	Dataset	Domain	Instances	Features	Classes
A	Default of Credit Card	Finance	30000	23	2
B	Bank Marketing	Business	45211	16	2
C	Breast Cancer Diagnostic	Medical	569	30	2
D	Chronic Kidney Disease	Medical	400	24	2
E	Dry Bean	Agriculture	13611	16	7
F	Covertype	Ecology	581012	54	7

Table 1: Summary of classification datasets

ID	Dataset	Domain	Instances	Features	Target type
K	California Housing	Real Estate	20640	8	Continuous
L	Power Consumption	Energy	2075259	9	Continuous
M	Parkinson Telemonitoring	Medical	5875	19	Continuous
N	Seoul Bike Sharing	Transportation	8760	13	Count data
O	Diabetes Progression	Medical	442	10	Continuous

Table 2: Summary of regression datasets

Figure 1: Comparison of evaluation behavior on two imbalanced classification datasets.

Figure 2: Threshold trade-offs in medical classification, emphasizing the interaction between Precision and Recall.

Figure 3: Comparison of Micro F1 and Macro F1 across multiclass datasets with different class distributions.

Figure 4: Contrast between Accuracy and Log Loss, showing how calibration can alter model ranking.

Figure 5: Comparison of MAE and RMSE under heavy-tailed regression errors.

Figure 6: Residual analysis illustrating why global goodness-of-fit measures should be complemented by error diagnostics.

Figure 7: Illustration of the instability of percentage-based error metrics near low baseline values.

Evaluation Sovereignty in Metadata-Driven Classification: A Multi-Track Framework for Weakly Supervised Information Systems
Jun 11, 2026Raymond VasquezWeakly Supervised Learning
Lawrence Livermore National Laboratory, Livermore, CA, USA
No evaluation without fair representation : Impact of label and selection bias on the evaluation, performance and mitigation of classification models
Mar 10, 2026Magali Legast, Toon Calders, François FoussAlgorithmic Fairness Dataset Bias
Université catholique de Louvain, Institute of Information and Communication Technologies, Electronics and Applied Mathematics, Louvain-la-Neuve, Belgium · Universiteit Antwerpen, Adrem Data Lab, Antwerpen, Belgium · Université catholique de Louvain, Louvain Research Institute in Management and Organizations, Institute of Information and Communication Technologies, Electronics and Applied Mathematics, Mons, Belgium
Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild
May 22, 2026Zhimin Zhao, Zehao Wang, Abdul Ali Bangash +2Software Engineering Automated Evaluation
Software Analysis and Intelligence Lab (SAIL), School of Computing, Queen’s University, Canada · Concordia University, Canada · Lahore University of Management Sciences (LUMS), Pakistan

Jun 11, 2026J/K move · Enter open · S save

Evaluation Sovereignty in Metadata-Driven Classification: A Multi-Track Framework for Weakly Supervised Information Systems

Raymond Vasquez

Lawrence Livermore National Laboratory, Livermore, CA, USA

Weakly Supervised Learning

Open paper AskPDF arXiv

Evaluation in machine learning is typically treated as a neutral measurement process. However, in operational information systems, evaluation outcomes are often conditioned by the processes used to generate labels. This paper does not seek to improve classification performance. Instead, it examines the validity of performance measurement under differing label-authority regimes. This issue is particularly relevant in large-scale metadata-driven systems, where labels are often incomplete, inconsistent, or weakly supervised. We introduce evaluation sovereignty, defined as the degree to which performance metrics are independent of label authority and supervision regime, and propose a multi-track evaluation framework that systematically varies training and evaluation label sources. Using hierarchical multi-label classification on large-scale scientific metadata, we demonstrate that models exhibiting strong performance under operational ("silver") evaluation degrade substantially under independent ("gold") evaluation, particularly for fine-grained classification. For example, Micro-F1 decreases from approximately 0.54 to 0.03. Notably, ranking-based metrics remain above baseline, revealing a divergence between latent model signal and classification validity. These findings suggest that commonly reported performance metrics may reflect alignment with labeling processes rather than true predictive capability. We therefore reconceptualize evaluation validity as a system-level property shaped by label governance and provide a practical methodology for auditing intelligent systems operating under weak supervision.

Abstract

Explore similar work

Evaluation Sovereignty in Metadata-Driven Classification: A Multi-Track Framework for Weakly Supervised Information Systems

No evaluation without fair representation : Impact of label and selection bias on the evaluation, performance and mitigation of classification models

Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild