Benchmarking Classical Machine Learning Models for Shading-Fault Detection in Rooftop Photovoltaic Systems Using Real Sensor Data
Organizations: School of Computing and Engineering Sciences, Strathmore University, Nairobi, Kenya
Abstract
Most published comparisons of machine-learning classifiers for photovoltaic (PV) fault detection are validated on simulated faults. We compare seven classical and ensemble-tree classifiers on the Réunion dataset, a 4 kW domestic rooftop plant on which shading faults were physically staged and logged. Readings within a staged episode are near-duplicates, so every train/test split is grouped by episode. A 20,800-row subsample that retains all 17 fault episodes was evaluated with three repeats of five-fold group-aware nested cross-validation, with synthetic minority oversampling (SMOTE) applied inside the training folds. Random Forest and XGBoost gave the highest mean accuracy (0.838 and 0.831) and Matthews correlation coefficient (MCC; 0.526 and 0.522). After Holm correction across the 21 pairwise Wilcoxon tests, only Random Forest's accuracy advantage over k-nearest neighbours and the multi-layer perceptron remained significant. Logistic Regression and the Decision Tree (accuracy 0.820) could be separated neither from the ensembles nor from the weakest models. SMOTE lowered binary MCC for six of the seven classifiers. Errors were concentrated in whole episodes: the Random Forest almost entirely missed four of the 17 fault episodes, including one of two uniform-shading and two of five intermittent-shading episodes. A post-hoc diagnostic shows that the no-fault rows were sampled over a wider range of times of day and output levels than the fault rows. Clock time alone reaches an MCC of 0.44, and PV-side voltage alone 0.72. The reported scores therefore reflect operating-condition differences as well as shading signatures. We release the pipeline and the diagnostics, and identify matched no-fault sampling as the step needed before the classifier ranking can be generalised.
Figures & tables
| Study | Data and task | Difference in this work |
|---|---|---|
| Eldeghady et al. (2024) | Simulated PV array faults; several supervised classifiers; accuracy-focused | Real staged faults; six metrics; corrected paired tests |
| Khandeparkar et al. (2025) | Simulated fault signatures, four types; five models incl. RF and XGBoost | Same ensemble baselines on real data with grouped splits |
| Mouleloued et al. (2025) | Seven-class fault dataset; five models incl. RF | MCC and AUC-ROC alongside accuracy; significance testing |
| Toche Tchio et al. (2024) | Low-cost-sensor data incl. partial shading; Extra Trees vs. six classifiers | Episode-grouped CV, repeated paired tests, SHAP |
| Utama et al. (2023) | Comparison of explanation methods for PV faults | SHAP across seven classifiers on a normalised scale |
| Kappler et al. (2024) | Real installation; LSTM + physical model; forecasting | Discrete fault classification; multi-model comparison |
| Code | Description | Rows | Share (%) | Episodes |
|---|---|---|---|---|
| 0.0 | No fault | 2,904,284 | 97.17 | 18 |
| 1.0 | Uniform shading (entire string obscured) | 6,318 | 0.21 | 2 |
| 2.1 | Constant partial shading, 1 module | 2,148 | 0.07 | 1 |
| 2.2 | Constant partial shading, 2 modules | 2,460 | 0.08 | 1 |
| 2.3 | Constant partial shading, 3 modules | 3,287 | 0.11 | 1 |
| 3.1 | Constant partial shading, 1/3 module | 2,717 | 0.09 | 3 |
| Model | Search space |
|---|---|
| Logistic Regression | ; L2 penalty (all 6 evaluated) |
| kNN | ; weights {uniform, distance}; |
| Decision Tree | max depth ; min samples per leaf ; criterion {Gini, entropy} |
| SVM (RBF) | ; {scale, auto, , , } |
| MLP | hidden layers {(16), (32), (64), (16,8), (32,16)}; ; initial learning rate ; early stopping |
| Random Forest | trees ; max depth ; min samples per leaf ; features per split |
| Model | Modal configuration |
|---|---|
| Logistic Regression | ; L2 |
| kNN | ; ; uniform weights |
| Decision Tree | max depth 3; min samples per leaf 1; Gini |
| SVM (RBF) | ; |
| MLP | one hidden layer of 64 units; ; initial learning rate |
| Random Forest | 300 trees; max depth 8; min samples per leaf 10; features per split |
| Component | Version |
|---|---|
| Python | 3 (author’s workstation) |
| pandas | 2.0 |
| NumPy | 1.24 |
| scikit-learn | 1.3 |
| imbalanced-learn | 0.11 |
| SciPy (Wilcoxon testing) | 1.10 |
| Model | Accuracy | Precision | Recall | F1 | MCC | AUC-ROC |
|---|---|---|---|---|---|---|
| Logistic Regression | .820 .124 | .776 .269 | .766 .275 | .741 .231 | .483 .378 | .893 .170 |
| kNN | .772 .137 | .707 .293 | .689 .273 | .663 .238 | .388 .342 | .822 .172 |
| Decision Tree | .820 .175 | .806 .282 | .773 .276 | .764 .252 | .500 .420 | .861 .198 |
| SVM (RBF) | .800 .130 | .784 .272 | .667 .266 | .690 .237 | .435 .354 | .846 .179 |
| MLP | .757 .196 | .712 .314 | .702 .299 | .670 .261 | .387 .416 | .820 .230 |
| Random Forest | .838 .156 | .844 .267 | .744 .297 | .759 .256 | .526 .416 | .859 .260 |
| Model | Accuracy | Precision | Recall | F1 | MCC | AUC-ROC |
|---|---|---|---|---|---|---|
| Logistic Regression | .830 .149 | .824 .265 | .742 .289 | .751 .248 | .511 .406 | .901 .170 |
| kNN | .792 .146 | .763 .278 | .693 .288 | .687 .239 | .439 .362 | .849 .177 |
| Decision Tree | .823 .189 | .818 .290 | .778 .278 | .773 .261 | .516 .442 | .878 .206 |
| SVM (RBF) | .805 .141 | .819 .268 | .657 .261 | .700 .234 | .456 .363 | .841 .175 |
| MLP | .788 .180 | .763 .294 | .722 .291 | .709 .252 | .451 .404 | .854 .220 |
| Random Forest | .839 .157 | .852 .263 | .739 .296 | .761 .255 | .531 .416 | .861 .253 |
| Model | TN | FP | FN | TP |
|---|---|---|---|---|
| Logistic Regression | 36,778 | 5,222 | 4,471 | 15,929 |
| kNN | 35,138 | 6,862 | 6,019 | 14,381 |
| Decision Tree | 37,415 | 4,585 | 4,454 | 15,946 |
| SVM (RBF) | 37,546 | 4,454 | 6,420 | 13,980 |
| MLP | 34,871 | 7,129 | 5,914 | 14,486 |
| Random Forest | 39,199 | 2,801 | 5,026 | 15,374 |
| kNN | DT | SVM | MLP | RF | XGB | |
|---|---|---|---|---|---|---|
| LR | .055 | .599 | .055 | .277 | .720 | .847 |
| kNN | .151 | .188 | 1.000 | .001 ( =.93) | .009 ( =.79) | |
| DT | .303 | .095 | .679 | .975 | ||
| SVM | .421 | .121 | .121 | |||
| MLP | .001 ( =1.0) | .010 ( =.73) | ||||
| RF | .421 |
| With SMOTE | Without SMOTE | |||||||
| Accuracy | MCC | Accuracy | MCC | |||||
| Model | bin. | tier | bin. | tier | bin. | tier | bin. | tier |
| Logistic Regression | .820 | .650 | .483 | .339 | .830 | .758 | .511 | .483 |
| kNN | .772 | .664 | .388 | .328 | .792 | .699 | .439 | .354 |
| Decision Tree | .820 | .619 | .500 | .338 | .823 | .763 | .516 | .527 |
| SVM (RBF) | .800 | .728 | .435 | .428 | .805 | .744 | .456 | .442 |
| Configuration | Features | Accuracy | MCC | AUC-ROC |
|---|---|---|---|---|
| A: raw channels | 7 | .821 .171 | .504 .423 | .870 .201 |
| B: rolling and difference features (no ) | 18 | .743 .092 | .366 .295 | .860 .223 |
| C: all features | 29 | .839 .156 | .528 .417 | .858 .260 |
| Code | Description | Pooled rows | Recall | Episodes detected |
|---|---|---|---|---|
| 0.0 | No fault (specificity) | 42,000 | 0.935 | – |
| 1.0 | Uniform shading | 2,400 | 0.501 | 1 / 2 |
| 2.1 | Constant partial, 1 module | 1,200 | 0.999 | 1 / 1 |
| 2.2 | Constant partial, 2 modules | 1,200 | 1.000 | 1 / 1 |
| 2.3 | Constant partial, 3 modules | 1,200 | 0.007 | 0 / 1 |
| 3.1 | Constant partial, 1/3 module | 3,600 | 0.947 | 3 / 3 |
| Inputs | Accuracy | MCC | AUC-ROC |
|---|---|---|---|
| Clock hour only | .732 .114 | .443 .260 | .792 |
| only | .616 .130 | .196 .269 | .735 |
| only | .880 .075 | .724 .158 | .913 |
| , | .822 .091 | .592 .189 | .821 |
| Seven raw channels | .851 .115 | .685 .217 | .876 |
| Six raw channels (no ) | .888 .084 | .744 .185 | .922 |
Fault Detection
Fault detection identifies abnormal behavior in machines, sensors, infrastructure, software, and autonomous systems before it causes harm or disruption.
See all 115 papers