FMMO: Detecting the Divergence Between Local Attribution and Global Drift

Muhammad Rehman Zafar; Ali El-Sharif; Naimul Khan

cs.LGSep 5, 2026

FMMO: Detecting the Divergence Between Local Attribution and Global Drift

Authors: Muhammad Rehman Zafar, Ali El-Sharif, Naimul Khan

Organizations: Toronto Metropolitan University, Toronto, Ontario, Canada · St. Clair College, Windsor, Ontario, Canada

Abstract

Post-deployment drift poses a critical risk to algorithmic accountability, particularly when ground truth labels are delayed and performance degradation becomes a "silent failure". While Explainable AI (XAI) is often relied upon to audit these shifts, we demonstrate that popular local attribution methods (e.g., TreeSHAP) can exhibit misleading stability even as model reliability collapses. In this paper, we propose a Framework for Model Monitoring and Observability (FMMO) designed to expose the divergence between local explanation stability and global distribution shifts. Using benchmark, synthetic, and real-world datasets, we show that local XAI methods fail to flag drift-induced disparate impact, specifically where False Positive Rates spike for protected groups while feature attributions remain unchanged. By integrating global surrogate models with model utilization measurements, FMMO mitigates this fairness blind spot, ensuring that stakeholders can detect discriminatory deterioration that standard local XAI tools overlook.

Figures & tables

Panel of Figure 1 — Figure 1: FMMO reports for COMPAS dataset when Gaussian noise was added.

Figure 2: High-level block diagram of the proposed Framework for Model Monitoring and Observability (FMMO). It consists of three stages which includes 1) the ML pipeline, 2) the XAI pipeline, and 3) the ML model monitoring pipeline.

Dataset	ML Model	Accuracy	ROC AUC
CES – NA	RF	$89.85$	$85.90$
CES – NA	XGB	$90$	$85.55$
CES	RF	$91.30$	$87.00$
CES	XGB	$90.88$	$85.43$
COMPAS	RF	$65.60$	$70.80$
COMPAS	XGB	$67.77$	$73.93$

Table 1: Baseline performance metrics on the test dataset

Dataset	Observations	Features	Type
COMPAS	$7,214$	$53$	Binary
GC	$1,000$	$30$	Binary
CC	$2,215$	$128$	Binary
CES	$16,789$	$26$	Binary
Synthetic	$25,000$	$20$	Binary

Table 2: Summary of Datasets

Dataset	Drift	Group	Settings	FPR	Accuracy
COMPAS	High	Protected	Baseline	$37.84$	$63.93$
		Protected	Drift	70.39	59.78
		Privileged	Baseline	$10.17$	$67.98$
		Privileged	Drift	36.17	$63.63$
	Low	Protected	Baseline	$37.84$	$63.93$
		Protected	Drift	44.50	63.06

Table 3: Performance Metrics for COMPAS Dataset using Random Forest (RF)

Dataset	Noise	ML Model	ROC AUC	TreeSHAP						DT		Alert
				$k=3$		$k=5$		$k=10$		$k=5$
				$\mathcal{J}$	$\alpha$	$\mathcal{J}$	$\alpha$	$\mathcal{J}$	$\alpha$	$\mathcal{J}$	$\alpha$
CES - NA	None	RF	84.54	$100$	$100$	$100$	$100$	$100$	80	66.67	20	TRUE
CES - NA	None	XGB	82.90	$100$	33.33	66.67	40	$100$	20	66.67	60	TRUE
CES	Gaussian	RF	$86.99$	$100$	$100$	$100$	$100$	$100$	80	25	20	TRUE
	Gaussian	XGB	$85.28$	$100$	$100$	$100$	$100$	$100$	$100$	66.67	40	TRUE

Table 4: Comparative summary of ROC AUC, explanation stability and features rank agreement under covariate drift.

Panel of Figure 3 — Figure 3: FMMO reports for CES dataset when Gaussian noise was added

Panel of Figure 4 — Figure 4: FMMO reports for the COMPAS dataset when Gaussian noise was added

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 424 papers

Does Explainability Survive Data Drift?
Oct 4, 2026Samuel OzechiExplainability Evaluation
A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification
Aug 24, 2026Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao +2Explainability Evaluation Black-Box Model Auditing
Doctoral School Modeling–Computer Science, University of Fianarantsoa, Madagascar
Fairness of Explanations in Artificial Intelligence (AI): A Unifying Framework, Axioms, and Future Direction toward Responsible AI
May 11, 2026Gideon Popoola, John SheppardExplainable Artificial Intelligence Algorithmic Auditing
Montana State University, Bozeman, Montana, USA

Oct 4, 2026J/K move · Enter open · S save

Does Explainability Survive Data Drift?

Samuel Ozechi

Explainability Evaluation

Open paper AskPDF arXiv

Model performance monitoring is a standard practice in machine learning deployments. Detection performance is tracked continuously, and model decay is expected as the relationship between the feature and target variables degrades, a phenomenon known as concept drift. Explanation fidelity, however, is rarely monitored with the same discipline, even in domains such as financial systems, healthcare, and other regulated environments where explanations are required for governance purposes. This paper investigates whether explanations can decay under data drift, even when the feature-target relationship remains stable, and whether explanations produced before drift occurs remain faithful to the decisions of the model that replaces them. Using the IEEE-CIS Transaction Fraud Detection dataset, we find statistically significant covariate shift but no statistically significant evidence of concept drift under the implemented conditional-drift tests, thereby providing an empirical setting in which input distributional change can be studied separately from detectable changes in the feature-target relationship. Local explanations are generated with ExIFFI and evaluated at three levels: path validity, structural behaviour, and fidelity under controlled intervention. Results show that while prior explanations retain substantial decision relevance to a retrained model, they are consistently less faithful than newly generated explanations, with no evidence of a systematically widening gap across the evaluated windows. The study shows that explanation fidelity requires its own monitoring, that structural stability of explanations does not guarantee functional fidelity, and that explanations should be treated as artifacts tied to the model that produced them.

Abstract

Algorithmic Fairness

Explore similar work

Does Explainability Survive Data Drift?

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification

Fairness of Explanations in Artificial Intelligence (AI): A Unifying Framework, Axioms, and Future Direction toward Responsible AI