Does the Explanation Drift Before the Guarantee? Label-Free Early Warning on Evolving Graphs

Tomasz Szandała; Damian Raczkowski

cs.AIOct 12, 2026

Does the Explanation Drift Before the Guarantee? Label-Free Early Warning on Evolving Graphs

Authors: Tomasz Szandała, Damian Raczkowski

Organizations: Faculty of Information and Communication Technology, Wroclaw University of Science and Technology · independent researcher

Abstract

Deployed models can fail silently as the world they describe evolves. Attribution drift has served as an early warning in tabular data; we ask whether, in graph streams, a model's own explanations can forewarn the loss of nominal conformal coverage. Doing so requires repairing the naive drift statistic twice. A vanishing edge loses mask mass regardless of whether the model relied on it, so we condition the importance on edge presence. Evolving graphs also break edge identity: a relabelling that changes nothing shifts the identity-based statistic by 0.69 on synthetic graphs and 0.83 on real ego-networks, whereas aggregating over structural roles brings both to 0.0 and stays well-defined when the support changes. We test six pre-registered regimes on thirty seeds each. A two-channel monitor, tracking conditional mask mass and its distribution over roles, precedes coverage collapse by 8 to 14.5 steps across four gradual regimes (node addition, removal, replacement, and edge thinning), all surviving Holm correction. A sudden knockout yields the expected zero lead, and a harmless change triggers no alarm. Leads track the length of the transitional phase (Spearman 0.79 over 150 runs). On real transaction ego-networks, the untuned mass channel forewarns staged drift by 27 steps and is the only channel, attributional or structural, that never alarms on a stationary stream. The main limitation is that the mass channel is blind to covariate drift, which breaks the guarantee in every run; the distributional channel detects it and also rescues the rewiring regime, so we report the two channels separately rather than as a union.

Figures & tables

Figure 1: The two repairs on four edges. (a) When an edge thins, the raw mask loses mass whether or not the model relied on it; dividing by presence isolates the model’s dependence per unit of presence. (b) A relabelling (or a sampled instance) changes what coordinate $(0,1)$ names ; the role key—the unordered pair of endpoint degree bins—names the same structural object regardless.

Regime	Stage-one mechanism	Support	Predicted lead
thinning	$A$ ’s density decays in place	fixed	$+$
knockout	both motifs collapse at once	fixed	$0$ (no transitional phase)
add	$A$ ’s edges rewired through new nodes $C$	grows	$+$
remove	$A$ ’s nodes retired entirely	shrinks	$+$
mix	removal of $A$ plus non-predictive $C$ traffic	both	$+$
null	$A,B$ untouched; background-level traffic on $C$	grows	silent

Table 1: Six drift regimes with outcomes registered before the campaign, plus the covariate control that measures the predicted blind spot. A further pre-registered prediction: the current-frequency denominator underperforms under remove . Stage one occupies drift progress

s\in[0,0.4]

; stage two (

s\in[0.6,1]

) degrades the survivor

B

Regime	Pred.	$L$ (med.)	IQR	frac $L>0$	Holm $p$	$\Psi$	Verdict
thinning	$+$	$+9.0$	$[-3.2,\,21.0]$	$0.70$	$0.005$	$0.92$	confirmed
knockout	$0$	$-1.0$	$[-2.0,\,-0.2]$	$0.03$	-	$0.35$	confirmed (control 1)
add	$+$	$-14.0$	$[-20.8,\,-6.0]$	$0.07$	$1.000$	$0.66$	failed; § 4.2
remove	$+$	$+13.0$	$[5.5,\,30.0]$	$0.93$	$<10^{-4}$	$0.92$	confirmed
mix	$+$	$+8.0$	$[-2.0,\,22.8]$	$0.70$	$0.002$	$0.91$	confirmed
null	silent	-	-	$0/30$ alarms	-	$-0.05$	confirmed (control 2)

Table 2: Mass channel

D_{\text{role}}

against coverage collapse; one-sided Wilcoxon tests of

L>0

, Holm-corrected over the four regimes predicting a lead;

30

seeds per regime; detection rate

1.00

in every harmful regime. Under covariate the guarantee collapses in

30/30

runs while the signal stays silent in all

30

; on null , coverage itself wobbled in

7/30

runs, the signal in none.

Figure 2: Per-regime leads against coverage collapse (left) and attribution–coverage coupling $\Psi$ (right), $30$ seeds per regime.

Alarms on the controls, by channel:
Regime	$L_{\text{union}}$ (median)	IQR	frac $L>0$	Holm $p$
thinning	$+14.5$	$[5.5,\,23.5]$	$0.87$	$<10^{-4}$
knockout	$0.0$	$[-1.0,\,0.0]$	$0.13$	-
add	$+8.5$	$[0.5,\,25.2]$	$0.73$	$0.0005$
remove	$+13.0$	$[5.5,\,30.0]$	$0.93$	$<10^{-4}$
mix	$+8.0$	$[-2.0,\,22.8]$	$0.70$	$0.0007$

Table 3: Two-channel monitor: all four transitional regimes carry leads that survive a Holm correction (

p\leq 0.001

throughout), and the knockout control still yields none. The mass channel never false-alarms on either control but wakes late on add ; the union catches add and inherits the distributional channel’s sensitivity to background density.

Figure 3: Injected staged drift on Elliptic ego-networks, one repetition. Top: the harm clock (stratified error); bottom: the untuned mass channel $D_{\text{role}}$ . Stage one retires hub nodes, stage two makes illicit instances mimic licit ones; the attribution alarm fires in stage one, $27$ steps before the harm onset.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: All $150$ harmful runs ( $30$ seeds per regime): the two-channel lead against the length of the coverage transition. The dotted diagonal is the trivial upper bound $L\leq t_{\text{cov}}-t_{\text{onset}}$ ; Spearman $0.70$ over the five medians and $0.79$ ( $95\%$ CI $[0.73,0.84]$ ) over the $150$ individual harmful runs; within each regime separately it ranges from $0.66$ to $0.95$ .

Channel	$\lambda$	pooled $L$	alarms on covariate	thinning	add
$D_{\text{role}}$ (mass)	$10$	$+3.0$	$0.00$	$+14$	$-10$
$D_{\text{role}}^{L1}$ (distribution)	$40$	$-0.5$	$0.93$	$+4$	$+4$
union	$40$	$+5.0$	$0.93$	$+10$	$+4$
degree MMD	$80$	$+8.0$	$1.00$	$+9$	$+22$
embedding MMD	$56$	$0.0$	$0.97$	$+13$	$-13$
conditioned identity focus	$24$	$+10.5$	$0.00$	$+22$	$+25$

Appendix

Table 4: Each channel at the smallest threshold with zero false alarms on the

30

harmless null runs; pooled median lead over the

150

harmful runs, then the alarm rate on the

30

covariate runs, where coverage collapses through background density while the model’s reasoning is untouched. A high value there is a correct detection of harm; the attribution channels’

0.00

is the blind spot of § 4.1 , and what they offer in exchange is that their alarm is specific to a change in the model’s reasoning. Leads for two individual regimes are shown for orientation; predictive entropy, the raw identity focus and the full threshold grid are in Figure 5 , Appendix 4 . The identity channel leads well on this fixed universe and is equally specific – but a relabelling that changes nothing moves it by

0.69

(§ 4.1 ) and it is undefined under instance sampling (§ 4.5 ). Our claim is not dominance on fixed graphs; it is validity on evolving ones.

Figure 5: Full threshold sweep for every channel. In both panels the horizontal axis is the false-alarm rate on the $30$ harmless null runs as the detector threshold $\lambda$ varies. Left: pooled median lead over the $150$ harmful runs. Right: alarm rate on the $30$ covariate runs, where coverage collapses through background density while the model’s reasoning is untouched. The structural baselines detect that harm at every threshold; the attribution channels do not at any threshold, which is the blind spot measured in Table 2 . Table 4 reads off one point per curve, the smallest $\lambda$ with zero false alarms on null .

Figure 6: Median lead per regime against graph size ( $10$ seeds per regime per size).

Figure 7: Empirical coverage under marginal calibration at the natural $\approx 90{:}10$ composition, overall and on the illicit class; median over $15$ repetitions, boxes show the interquartile range. The minority deficit is present on the calibration window itself, widens over the monitored window, and becomes total after the shutdown, while the marginal curve rises above its own calibration level.

Distribution Shift Detection

Distribution shift detection identifies when data or model behavior changes between reference and deployment settings.

See all 63 papers

Does Explainability Survive Data Drift?
Oct 4, 2026Samuel OzechiExplainability Evaluation
Amortised Post-Hoc Explanation with Exact Preservation for Dynamic Graph Anomaly Detectors
Aug 16, 2026Iyad Assaad Nekka, Hamida Seba, Walid Khaled Hidouci +1GNN Explainability Interpretable Anomaly Detection
LCSI Laboratory, National higher School of Computer Science (ESI), Algiers, Algeria · Université Claude Bernard Lyon, Lyon, France
Counterfactual Explanations Under Concept Drift
May 17, 2026Marcin Kostrzewa, Jerzy Stefanowski, Maciej ZiębaCounterfactual Explanations Concept Drift
Wrocław University of Science and Technology · Pozna´n University of Technology · Tooploox

Oct 4, 2026J/K move · Enter open · S save

Does Explainability Survive Data Drift?

Samuel Ozechi

Explainability Evaluation

Open paper AskPDF arXiv

Model performance monitoring is a standard practice in machine learning deployments. Detection performance is tracked continuously, and model decay is expected as the relationship between the feature and target variables degrades, a phenomenon known as concept drift. Explanation fidelity, however, is rarely monitored with the same discipline, even in domains such as financial systems, healthcare, and other regulated environments where explanations are required for governance purposes. This paper investigates whether explanations can decay under data drift, even when the feature-target relationship remains stable, and whether explanations produced before drift occurs remain faithful to the decisions of the model that replaces them. Using the IEEE-CIS Transaction Fraud Detection dataset, we find statistically significant covariate shift but no statistically significant evidence of concept drift under the implemented conditional-drift tests, thereby providing an empirical setting in which input distributional change can be studied separately from detectable changes in the feature-target relationship. Local explanations are generated with ExIFFI and evaluated at three levels: path validity, structural behaviour, and fidelity under controlled intervention. Results show that while prior explanations retain substantial decision relevance to a retrained model, they are consistently less faithful than newly generated explanations, with no evidence of a systematically widening gap across the evaluated windows. The study shows that explanation fidelity requires its own monitoring, that structural stability of explanations does not guarantee functional fidelity, and that explanations should be treated as artifacts tied to the model that produced them.

Abstract

Distribution Shift Detection

Explore similar work

Does Explainability Survive Data Drift?

Amortised Post-Hoc Explanation with Exact Preservation for Dynamic Graph Anomaly Detectors

Counterfactual Explanations Under Concept Drift