Interpretable and Explainable Surrogate Modeling for Simulations: A State-of-the-Art Survey and Perspectives on Explainable AI for Decision-Making
Authors: Pramudita Satria Palar, Paul Saves, Muhammad Daffa Robani, Nicolas Verstaevel, Moncef Garouani, Julien Aligon, Koji Shimoyama, Joseph Morlier, Benoit Gaudou
Organizations: Faculty of Mechanical and Aerospace Engineering, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung, 15414, West Java, Indonesia. · IRIT, Université Toulouse Capitole, Toulouse, 31000,France. · AwanTunai, Urban Suites, 3 Hullet Road, Singapore, 229158, Singapore, Singapore. · Department of Mechanical Engineering, Kyushu University, 744 Motooka, Fukuoka, 819-0395, Japan. · Université de Toulouse, ISAE-SUPAERO, 10 avenue Marc Pellegrin, Toulouse, Occitanie, France.
The simulation of complex systems increasingly relies on sophisticated but fundamentally opaque computational black-box simulators. Surrogate models play a central role in reducing the computational cost of complex systems simulations across a wide range of scientific and engineering domains. Notwithstanding, they inevitably inherit and often exacerbate this black-box nature, obscuring how input variables drive physical responses. Conversely, Explainable Artificial Intelligence (XAI) offers powerful tools to unpack these models. Yet, XAI methods struggle with engineering-specific constraints, such as highly correlated inputs, dynamical systems, and rigorous reliability requirements. Consequently, surrogate modeling and XAI have largely evolved as distinct fields of research, despite their strong complementarity. To reconnect these approaches, this state-of-the-art survey provides a structured perspective that maps existing XAI techniques onto the various stages of surrogate modeling workflows for design and exploration. To ground this synthesis, we draw upon illustrative applications across both equation-based simulations and agent-based modeling. We survey a broad spectrum of techniques, highlighting their strengths for revealing interactions and supporting human comprehension. Finally, we identify pressing open challenges, including the explainability of dynamical systems and the handling of mixed-variable systems, and propose a research agenda to make explainability a core, embedded element of simulation-driven workflows from model construction through decision-making. By transforming opaque emulators into explainable tools, this agenda empowers practitioners to move beyond accelerating simulations to extracting actionable insights from complex system behaviors.
Figures & tables
Figure 1: Workflow for explainable surrogate models for complex systems simulations co-design.
Figure 2: Workflow of the survey: The progression from foundational principles to methodological tools, their practical deployment, and open research challenges.
Figure 3: Publication trends, in log-scale, from Scopus (2014–2025) based on keyword searches within “Article Title, Abstract, and Keywords”. ML and SM, respectively, stand for Machine Learning and Surrogate Model.
Criterion
Advantages (PROs)
Limitations (CONs)
Example of Surrogate Models
Prediction Accuracy
Captures underlying trends and relationships; enables reliable knowledge extraction
Risk of overfitting or underfitting; may require large datasets or careful hyperparameter tuning
Regular GP, random forest, TabPFN, deep ensemble
Smoothness
Facilitates physical consistency and stable post-hoc explanations
May fail to capture sharp discontinuities; some models yield stepwise responses
Systematic exploration of Agent-Based Models (ABMs) is challenged by the curse of dimensionality and their inherent stochasticity. We present a multi-stage pipeline integrating the systematic design of experiments with machine learning surrogates. Using a predator-prey case study, our methodology proceeds in two steps. First, an automated model-based screening identifies dominant variables, assesses outcome variability, and segments the parameter space. Second, we train Machine Learning models to map the remaining nonlinear interaction effects. This approach automates the discovery of unstable regions where system outcomes are highly dependent on nonlinear interactions between many variables. Thus, this work provides modelers with a rigorous, hands-off framework for sensitivity analysis and policy testing, even when dealing with high-dimensional stochastic simulators.
Paul Saves, Matthieu Mastio, Nicolas Verstaevel +1
IRIT, Université Toulouse Capitole, Toulouse, France
The demand for Explainable AI (XAI) has triggered an explosion of methods, producing a landscape so fragmented that we now rely on surveys of surveys. Yet, fundamental challenges persist: conflicting metrics, failed sanity checks, and unresolved debates over robustness and fairness. The only consensus on how to achieve explainability is a lack of one. This has led many to point to the absence of a ground truth for defining ``the'' correct explanation as the main culprit. This position paper posits that the persistent discord in XAI arises not from an absent ground truth but from a ground truth that exists, albeit as an elusive and challenging target: the causal model that governs the relevant system. By reframing XAI queries about data, models, or decisions as causal inquiries, we prove the necessity and sufficiency of causal models for XAI. We contend that without this causal grounding, XAI remains unmoored. Ultimately, we encourage the community to converge around advanced concept and causal discovery to escape this entrenched uncertainty.
Amir-Hossein Karimi
Department of Electrical & Computer Engineering, University of Waterloo & Vector Institute, Canada
This paper surveys uncertainty-aware explainable artificial intelligence (UAXAI), examining how uncertainty is incorporated into explanatory pipelines and how such methods are evaluated. Across the literature, three recurring approaches to uncertainty quantification emerge (Bayesian, Monte Carlo, and Conformal methods), alongside distinct strategies for integrating uncertainty into explanations: assessing trustworthiness, constraining models or explanations, and explicitly communicating uncertainty. Evaluation practices remain fragmented and largely model centered, with limited attention to users and inconsistent reporting of reliability properties (e.g., calibration, coverage, explanation stability). Recent work leans towards calibration, distribution free techniques and recognizes explainer variability as a central concern. We argue that progress in UAXAI requires unified evaluation principles that link uncertainty propagation, robustness, and human decision-making, and highlight counterfactual and calibration approaches as promising avenues for aligning interpretability with reliability.
Helena Löfström, Tuwe Löfström, Anders Hjort +1
Jönköping International Business School, Gjuterigatan 5, 553 18 Jönköping, Sweden · Jönköping School of Engineering, Gjuterigatan 5, 553 18 Jönköping, Sweden · Eiendomsverdi AS, Nedre Slottsgate 8, 0157 Oslo, Norway +2