cs.AINov 11, 2026

A Reliability Study of Explainable AI for Freight Mode Classification

Authors: Ali Alawi Balfaqih, Mansur M. Arief

Organizations: AI Verification and Validation (AI-VNV) Lab, King Fahd University of Petroleum & Minerals (KFUPM), Dhahran, Saudi Arabia · Department of Industrial and Systems Engineering, IRC for Smart Mobility and Logistics (IRC-SML), and AI-VNV Lab, KFUPM, Saudi Arabia

Abstract

Machine learning models are increasingly used to support freight transportation decisions, but their predictions are rarely accompanied by evidence that the associated explanations can be trusted. In this study we investigate explainable machine learning for classifying shipments as Air/Air Charter or Truck/Ocean using the PEPFAR Supply Chain Management System Delivery History dataset, with 5,964 shipment records and 47 input features after cleaning, where freight cost and pack price are excluded as near-proxies of the transportation mode. We compare three Multi-Layer Perceptron (MLP) architectures, Logistic Regression, Random Forest, and gradient boosting with imbalance-aware metrics and paired significance tests across one hundred random replications. We then evaluate the explanations themselves, namely the stability of SHAP feature rankings across retraining and across models, the faithfulness of SHAP and LIME attributions under a feature-deletion test, and the geographic structure of the destination effects, which we map for all 38 destination countries. The results suggest that the tree ensembles significantly outperform the neural and linear models (86.8% against 83.3-85.0% mean testing accuracy) while their leading features remain the most stable across retraining. Removing the five features ranked most important by SHAP or LIME changes the predicted probabilities roughly four times as much as removing randomly chosen features. The destination effects form a coherent geographic pattern that simple port access does not explain. However, models with similar accuracy disagree substantially on which features matter most, and thus explanation stability and faithfulness should be assessed before such explanations are used to support freight transport decisions.

Figures & tables

Explore similar work

CardsList
  1. Evaluating Explanation Methods by the Predictors They Induce

    Sep 17, 2026Jacob Selbæk, Hugo L. HammerFeature AttributionSHAP Feature Attribution