Protocol-Aware Auditing of Domain-Blind Feature Attribution in Network Intrusion Detection
Organizations: Baylor University, Waco, TX, USA · Marist University, Poughkeepsie, NY, USA
Abstract
Feature-attribution methods can query rule-violating network states through perturbations, coalitions, references, paths, and ablations, making explanations and their evaluation sensitive to model behavior outside the operational domain. We formalize this failure mode as domain blindness and prove valid-query invariance: when models agree on the required oracle information within a declared rule-valid set, a procedure confined to that set returns the same attribution. We instantiate one protocol-aware query-control principle in LIME, SHAP, Integrated Gradients, and DeepLIFT and audit paired configurations across four NIDS datasets and three classifier families. In six settings where standard LIME reports perfect top-feature repeatability, PA-LIME reveals substantial variability, with a mean of 0.575. Under FGSM followed by protocol repair, PA-DeepLIFT and PA-IG achieve the largest attribution-similarity curve areas, 0.452 and 0.413, while mean within-PA rank agreement rises to 0.366 from 0.217. PA-XAI therefore turns structured domain assumptions into explicit, testable controls for explanation auditing; support modeling, raw-flow reconstruction, and adaptive explanation attacks provide complementary validation layers.
Figures & tables
| Method | Standard choice | PA query control | Controlled property | Scope of control |
|---|---|---|---|---|
| LIME | Gaussian neighborhood | Sample protocol; draw empirical discrete marginals; perturb continuous coordinates; apply ; fit the same surrogate | Repaired neighborhood rows satisfy implemented checks | Joint-support and reconstruction tests are separate validation stages |
| SHAP | Background and masked completion | Use benign backgrounds; repair KernelSHAP completions; pass the background to TreeSHAP/DeepSHAP | KernelSHAP evaluates repaired coalition completions | Tree and gradient backends use the background without repair |
| IG | Zero reference and direct interpolation in the study comparator | Select a threshold-near training reference; integrate continuous coordinates with discrete groups fixed; add ordered endpoint differences | Avoids fractional discrete interpolation and preserves the stated sum identity | Discrete endpoint control alone does not certify full-path containment |
| DeepLIFT | Zero reference in the study comparator | Use the threshold-near training reference with Rescale endpoint differences | Uses an observed reference without interpolated discrete inputs | An observed reference does not certify every internal query |
| Data | ||||
|---|---|---|---|---|
| N | 0.262 | 0.226 | 0.163 | 0.366 |
| C17 | 0.256 | 0.168 | 0.244 | 0.378 |
| U | 0.309 | 0.322 | 0.231 | 0.414 |
| C18 | 0.256 | 0.213 | 0.230 | 0.306 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Procedure | Specified algorithmic logic | Additional replication detail |
|---|---|---|
| PA-LIME | Protocol draw; protocol-filtered discrete marginals; Gaussian continuous perturbation; ordered repair; weighted surrogate | Neighborhood count; Gaussian scales; kernel and sparsity settings; dataset-specific rule map |
| Kernel PA-SHAP | Benign background; coalition completion; ordered repair before scoring | Background count; coalition count; mask grouping; repaired-coordinate policy |
| Tree/Deep PA-SHAP | Benign background passed to the native backend | Backend marginalization mode and internal query-state diagnostics |
| PA-IG | Threshold-near class reference; continuous segment; ordered discrete endpoint differences; conditional summation identity | Candidate-class convention; ties; protocol matching; discrete order and group-to-column allocation; quadrature rule |
| PA-DeepLIFT | Threshold-near reference; Rescale endpoint differences | Candidate-class convention; ties; protocol matching; backend/operator coverage and residual analysis |
| Dataset | Role | Additional detail for exact replication |
|---|---|---|
| NSL-KDD | Four-module benchmark | Release identifier, class mapping, split indices, encoded dimensions, and preprocessing-fit scope |
| CIC-IDS-2017 | Four-module benchmark | Release identifier, class mapping, split indices, encoded dimensions, and preprocessing-fit scope |
| UNSW-NB15 | Four-module benchmark | Release identifier, class mapping, split indices, encoded dimensions, and preprocessing-fit scope |
| CSE-CIC-IDS2018 | Four-module benchmark; domain-reference-set analysis | Release identifier, class mapping, split indices, encoded dimensions, preprocessing-fit scope, and reference-set definitions |
| Classifier family | Explanation-quality methods | Additional role |
|---|---|---|
| DNN | SHAP, LIME, IG, DeepLIFT (Std/PA) | Attacked classifier; downstream feature-reduction classifier |
| Random Forest | SHAP, LIME (Std/PA) | Distinct pruning proxy; downstream classifier |
| XGBoost | SHAP, LIME (Std/PA) | Primary explanation-quality classifier |
| 1D-CNN | – | Downstream feature-reduction classifier |
| SVM | – | Downstream feature-reduction classifier |
| Component | Reported configuration | Additional replication detail |
|---|---|---|
| Data preparation | Four datasets; median imputation; one-hot encoding; min–max scaling | Release IDs, split indices and seeds, label maps, encoded schemas, transform-fit scope |
| Models and explainers | Model/explainer families and PA procedure logic | Architectures, hyperparameters, seeds, checkpoints, backend map |
| Module sampling | Agreement/interactions and reduction calibration/proxy counts | Module-1/2 sample indices and repeat schedules |
| Attacks and reduction | Budgets, attack-then-repair order, proxy RF, stopping rule, downstream families, baselines | Optimizer details, repair-distance audit, exact matched subsets |
| Systems | Prototype latency | Hardware, software versions, timing harness, code, environment, logs |
| Faithfulness | Repeatability | ||||||
| Model | Method | Std | PA | Std | PA | ||
| DNN | SHAP | 0.63 | 0.62 | -0.01 | 1.00 | 1.00 | 0.00 |
| DNN | LIME | 0.28 | -0.07 | -0.35 | 0.63 | 0.46 | -0.17 |
| DNN | IG | 0.57 | 0.43 | -0.14 | 1.00 | 1.00 | 0.00 |
| DNN | DeepLIFT | 0.51 | 0.54 | +0.03 | 1.00 | 1.00 | 0.00 |
| RF | SHAP | 0.79 | 0.80 | +0.01 | 1.00 | 1.00 | 0.00 |
| Model | Method | Standard | PA |
|---|---|---|---|
| DNN | SHAP | 65.10 | 828.8 |
| DNN | LIME | 92.20 | 1690.6 |
| DNN | IG | 0.70 | 54.6 |
| DNN | DeepLIFT | 0.01 | 6.7 |
| RF | SHAP | 13.90 | 248.5 |
| RF | LIME | 127.30 | 1738.0 |
| NSL-KDD | CIC-IDS-2017 | UNSW-NB15 | CSE | |||||
| Std | PA | Std | PA | Std | PA | Std | PA | |
| Faithfulness | ||||||||
| DNN/LIME | -0.05 | 0.20 | 0.50 | -0.05 | 0.19 | -0.14 | 0.49 | -0.30 |
| RF/LIME | 0.18 | 0.66 | 0.31 | 0.39 | 0.66 | 0.56 | 0.36 | 0.36 |
| XGB/LIME | 0.14 | 0.38 | 0.21 | 0.29 | 0.25 | 0.10 | 0.31 | -0.07 |
| Repeatability | ||||||||
| Attack | Regime | |||||
|---|---|---|---|---|---|---|
| FGSM | Unrestricted | 20.1 | 46.9 | 63.7 | 74.9 | 76.1 |
| FGSM | After repair | 18.3 | 45.9 | 51.5 | 67.2 | 75.3 |
| PGD | Unrestricted | 21.1 | 54.9 | 74.6 | – | – |
| PGD | After repair | 18.5 | 49.1 | 58.8 | – | – |
| Attack | Unrestricted | After repair | Relative change |
|---|---|---|---|
| FGSM | 64.09 | 57.86 | |
| PGD | 52.86 | 44.99 |
| Method | nAUC | |||||
|---|---|---|---|---|---|---|
| SHAP | 0.695 | 0.365 | 0.285 | 0.106 | 0.038 | 0.221 |
| LIME | 0.043 | 0.160 | 0.160 | 0.152 | 0.185 | 0.153 |
| IG | 0.706 | 0.374 | 0.316 | 0.080 | -0.025 | 0.212 |
| DeepLIFT | 0.204 | 0.021 | -0.069 | -0.053 | -0.025 | -0.023 |
| PA-SHAP | 0.698 | 0.261 | 0.166 | -0.024 | -0.004 | 0.123 |
| PA-LIME | 0.666 | 0.175 | 0.098 | 0.058 | 0.077 | 0.132 |
| Method | Top-1 | Top-3 | Mean rank |
|---|---|---|---|
| LIME | 0.064 | 0.200 | 73.0 |
| SHAP | 0.000 | 0.000 | 123.0 |
| PA-LIME | 0.152 | 0.384 | 41.2 |
| PA-SHAP | 0.038 | 0.122 | 87.5 |
| Dataset | Wilc. (%) | ||||||
|---|---|---|---|---|---|---|---|
| NSL-KDD | 0.226 | 0.300 | 84.8 | 0.163 | 0.366 | ||
| CIC-IDS-2017 | 0.168 | 0.240 | 89.4 | 0.244 | 0.378 | ||
| UNSW-NB15 | 0.322 | 0.328 | 84.8 | 0.231 | 0.414 | ||
| CSE-CIC-IDS2018 | 0.213 | 0.262 | 84.8 | 0.230 | 0.306 |
| Dataset | Pair | Comparison context | |
|---|---|---|---|
| NSL-KDD | DNN-LIME vs. DNN-SHAP | Fixed DNN; explainer changes | |
| NSL-KDD | PA-DeepLIFT vs. PA-IG | DNN-only; shared reference rule | |
| CIC-IDS-2017 | PA-DeepLIFT vs. PA-IG | DNN-only; shared reference rule | |
| CSE-CIC-IDS2018 | PA-DeepLIFT vs. PA-IG | DNN-only; shared reference rule | |
| CSE-CIC-IDS2018 | DNN-PA-LIME vs. RF-LIME | Model and generation mode change |
| Dataset | DNN stat. | RF stat. | DNN/RF | LIME | |
|---|---|---|---|---|---|
| NSL-KDD | 2.10 | 1.31 | 1.60 | -0.535 | |
| CIC-IDS-2017 | 3.06 | 1.95 | 1.57 | 0.415 | |
| UNSW-NB15 | 1.50 | 1.47 | 1.02 | 0.483 | |
| CSE-CIC-IDS2018 | 3.30 | 2.24 | 1.47 | 0.280 |
| Attack | Model | Method | RRA@15 | Record type |
|---|---|---|---|---|
| Bot | DNN | DeepLIFT | 0.589 | attack-specific leader |
| FTP-BruteForce | DNN | DeepLIFT | 0.404 | attack-specific leader |
| DoS GoldenEye | DNN | PA-IG | 0.495 | attack-specific leader |
| DoS Hulk | DNN | PA-IG | 0.464 | attack-specific leader |
| Infiltration | DNN | PA-IG | 0.471 | attack-specific leader |
| SSH-Bruteforce | RF | PA-SHAP | 0.631 | attack-specific leader |
| Result | Calculation | Interpretation |
|---|---|---|
| Six-case PA-LIME mean | Paired stability summary | |
| Stability difference | Change from standard perfect overlap | |
| Faithfulness sign changes | comparisons; DNN | Qualitative evaluator changes |
| Within-mode agreement | PA , standard , difference | Descriptive coherence gap |
| FGSM evasion nAUC | ; | Repair-pipeline effect |
| PGD evasion nAUC | ; | Repair-pipeline effect |
Feature Attribution
Feature attribution estimates which input features or internal components contribute to a model’s predictions.
See all 247 papers