Effective IT change management is important for businesses that depend on software and services, particularly in highly regulated sectors such as finance, where operational reliability, auditability, and explainability are essential. A significant portion of IT incidents are caused by changes, making it important to identify high-risk changes before deployment. This study presents a predictive incident risk scoring approach at a large international bank. The approach supports engineers during the assessment and planning phases of change deployments by predicting the potential of inducing incidents. To satisfy regulatory constraints, we built the model with auditability and explainability in mind, applying SHAP values to provide feature-level insights and ensure decisions are traceable and transparent. Using a one-year real-world dataset, we compare the existing rule-based process with three machine learning models: HGBC, LightGBM, and XGBoost. LightGBM achieved the best performance, particularly when enriched with aggregated team metrics that capture organisational context. Our results show that data-driven, interpretable models can outperform rule-based approaches while meeting compliance needs, enabling proactive risk mitigation and more reliable IT operations.
Figures & tables
LightGBM
Metric
Base
HGBC
XGB
Without
With
Threshold
60
99
98
17
29
Precision
0.04
0.02
0.05
0.02
0.04
wR
0.56
0.86
0.89
0.91
0.93
w F2
0.88
0.88
0.90
0.92
0.93
AUC
0.55
0.61
0.71
0.67
0.60
Table 1. Performance of Baseline, ML Models, and LightGBM with and without Additional Features
Figure 1. Feature Importance of LightGBM. Horizontal bar chart showing the top 15 features ranked by SHAP values for the LightGBM model. Higher values indicate a greater importance of a feature to the model’s predictions across the evaluation set. The plot highlights the relative influence of textual, configuration-related, and temporal features in identifying incident-inducing changes (RQ2).
Figure 2. Sliding Window of Baseline vs. LightGBM Performance.
Figure 3. Feature Importance of LightGBM with Additional Features. Horizontal bar chart showing the top 15 features ranked by SHAP values for the LightGBM model extended with additional features. Higher values indicate a greater importance of a feature to the model’s predictions across the evaluation set. The figure illustrates how textual, organisational, configuration-related, and operational metrics contribute to identifying incident-inducing changes (RQ3).
Figure 4. Top 10 SHAP Values for Two Example Changes.