cs.AIAug 21, 2026

SENTRY: Deterministic, Intelligent Risk Assessment for IT Change Management

Authors: Daniel Arulpragasam, Christer Henrysson, Ella Ly, Deepika Anbalagan, Leo Feng

Organizations: Royal Bank of Canada · RBC Borealis

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Technology change management in large financial institutions depends on risk assessments that are accurate, consistent, and auditable. In practice, many institutions still rely on self-reported questionnaires. Those questionnaires are subjective, easy to game, and poor at separating routine changes from the ones that later trigger major incidents. This paper presents SENTRY, a risk assessment platform that replaces questionnaire-based scoring with a deterministic machine learning pipeline built from gradient-boosted decision trees (XGBoost) and hybrid retrieval-augmented generation (RAG). The system combines structured operational metadata, application dependency graphs, and historical incident records with a hybrid semantic and lexical search over historical change requests. The retrieval step captures the risk signal in unstructured change request text, then compresses that signal into a single scalar feature before model inference. That design keeps the model deterministic and preserves per-prediction explainability via SHAP values. Evaluated on enterprise-scale change data, SENTRY achieves a ROC AUC of 0.87 and 85% overall accuracy, and it detects high-risk changes at roughly 3.25 times the rate of the existing process. We close by examining the architectural trade-offs behind this design and what they imply for the use of machine learning in regulated change management.

Figures & tables

Figure 1: High-level architecture of the SENTRY risk assessment platform.
Figure 2: Hybrid risk score based on changes and the respective incident they caused. α=0.4\alpha=0.4 β=0.6\beta=0.6 .
 Predicted LowPredicted High/Med
Actual Low15018
Actual High/Med1322
Table 1: Confusion matrix on the held-out test set ( n=203n=203 ).
Method% of Incident Causing Changes Caught
Questionnaire-based20%
SENTRY (XGBoost + Hybrid RAG)63%
Table 2: SENTRY vs. the questionnaire-based baseline on high-risk detection.
Figure 3: ROC curve on the held-out test set (AUC = 0.87).
ClassPrecisionRecallF1Support
Low risk0.920.890.91168
High/Medium risk0.550.630.5935
Table 3: SENTRY per-class precision, recall, and F1 on the held-out test set.
RankFeatureGain (%)
1type_standard15.3
2crown_jewel5.4
3inherent_risk_score5.0
4hybrid_search_score4.9
5num_deps4.8
6data_classification4.7
Table 4: Top-8 features by gain-based importance (best Optuna trial). The hybrid search score ranks 4th, confirming that unstructured text adds signal beyond structured metadata alone.
More in this topic

AI Risk Management

AI risk management studies how to identify, assess, reduce, and monitor harms from AI across development, deployment, and wider social systems.

See all 108 papers

Explore similar work

CardsList

  1. Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment

    Apr 15, 2026Eileen Kapel, Jan Lennartz, Luis Cruz +2

    ING Bank Amsterdam, The Netherlands · ING Bank Brussels, Belgium · Delft University of Technology Delft, The Netherlands

  2. ERP-RiskBench: Leakage-Safe Ensemble Learning for Financial Risk

    Mar 3, 2026Sanjay MishraStacked Ensemble LearningFinancial Fraud Detection

    IEEE Member Raleigh, NC, USA

  3. A Preliminary Study on Explaining Risk of Code Changes using LLM-Based Prediction Models

    Jul 2, 2026Yalin Liu, Kosay Jabre, Rui Abreu +10Automated Code Review

    Meta Platforms, Inc. · Concordia University