Interpretable Deep Reinforcement Learning for Element-level Bridge Life-cycle Optimization

Seyyed Amirhossein Moayyedi; David Y. Yang

cs.AIApr 2, 2026

Interpretable Deep Reinforcement Learning for Element-level Bridge Life-cycle Optimization

Authors: Seyyed Amirhossein Moayyedi, David Y. Yang

Organizations: Department of Civil and Environmental Engineering, Portland State University, Portland, OR 97201

Abstract

The new Specifications for the National Bridge Inventory (SNBI), in effect from 2022, emphasize the use of element-level condition states (CS) for risk-based bridge management. Instead of a general component rating, element-level condition data use an array of relative CS quantities (i.e., CS proportions) to represent the condition of a bridge. Although this greatly increases the granularity of bridge condition data, it introduces challenges to set up optimal life-cycle policies due to the expanded state space from one single categorical integer to four-dimensional probability arrays. This study proposes a new interpretable reinforcement learning (RL) approach to seek optimal life-cycle policies based on element-level state representations. Compared to existing RL methods, the proposed algorithm yields life-cycle policies in the form of oblique decision trees with reasonable amounts of nodes and depth, making them directly understandable and auditable by humans and easily implementable into current bridge management systems. To achieve near-optimal policies, the proposed approach introduces three major improvements to existing RL methods: (a) the use of differentiable soft tree models as actor function approximators, (b) a temperature annealing process during training, and (c) regularization paired with pruning rules to limit policy complexity. Collectively, these improvements can yield interpretable life-cycle policies in the form of deterministic oblique decision trees. The benefits and trade-offs from these techniques are demonstrated in both supervised and reinforcement learning settings. The resulting framework is illustrated in a life-cycle optimization problem for steel girder bridges.

Figures & tables

Figure 1: Tree-based interpretable decision models: (a) standard decision tree and (b) oblique decision tree.

Figure 2: A simple soft tree with one internal node for three-class classification.

Figure 3: Temperature-controlled sigmoid function.

Figure 6: Example of pruning frozen tree: (a) original frozen tree and (b) oblique decision tree after pruning.

Figure 7: Implementation of a differentiable soft tree model (depth = 3): (a) neural network layers, (b) rearrangement to soft tree structure, and (c) output of class logits.

Figure 8: General architecture for supervised classification with soft tree models.

Figure 9: PPO architecture for reinforcement learning with soft tree based actors

Figure 10: Labeled synthetic data for supervised classification experiments

Model	Training accuracy (%)	Validation accuracy (%)	Test accuracy (%)
Standard tree	57.78	55.90	57.10
Oblique tree	60.52	58.90	59.75
Neural network	98.50	98.25	97.80
Soft tree	90.53	90.35	90.35

Table 1: Model performance of different classification models.

Temperature and annealing	Soft tree score (%)	Frozen tree score (%)	Relative difference
Fixed at 100	46.90	27.05	42.32%
Fixed at 1	90.35	83.10	8.02%
Fixed at 0.01	77.55	76.05	1.93%
Anneal: 100 to 0.01	80.70	80.65	0.06%
Anneal: 1 to 0.01	91.85	91.80	0.05%
Note : “Frozen” tree refers to the full oblique tree converted from the soft tree baseline prior to pruning.

Table 2: Test accuracy of soft and frozen trees associated with different temperatures and temperature annealing schedules.

Figure 11: Effects of $L_{1}$ regularization on (a) number of pruned nodes and (b) model performance

Figure 12: Effects of pruning threshold on (a) model performance and (b) pruned nodes.

Figure 13: CS evolution under two policies: (a) do-nothing; (b) conduct rehabilitation ( $a=3$ ) if CS4 percentage is greater than 5%, do nothing otherwise.

CS	InfoBridge Data		Fitted Dirichlet
	Mean (%)	StD (%)	Mean (%)	StD (%)
CS1	63.5	37.5	42.7	42.6
CS2	33.3	35.4	31.8	40.1
CS3	3.2	8.9	14.3	30.1
CS4	$\sim$ 0	$\sim$ 0	11.2	27.2
Note : Fitted Dirichlet is parameterized by theta vector of $[0.1496,0.1114,0.0500,0.0393]^{\intercal}$

Table 3: Mean and StD of CS proportions

Actor model	Hyperparameter	Value
Neural network	Network layout	Two fully connected hidden layers, each with 64 neurons and an expoential linear unit (ELU) as the activation function
Soft tree	Maximum depth	11
	$L_{1}$ regularization scaler	$0.01$
	Initial temperature	1
	Temperature update	0.9550 (from $T=1$ to $T=0.01$ in 100 stages)
	Update frequency	1 batch per stage

Table 4: Hyperparameters for different actor models.

Category	Hyperparameter	Value
Critic model	Network layout	Three fully connected hidden layers, each with 32 neurons and an ELU as the activation function.
PPO loss	Clipping epsilon	0.01
	Entropy coefficient	0.05
	Critic coefficient	0.5
General advantage	Lambda	0.95
Training	Total batches	100

Table 5: Hyperparameters for critic model and PPO training (common to both types of actors).

Figure 14: Learning curves associated with both types of actor models.

Actor model	Average LCC (monetary unit)	StD (monetary unit)
Neural network	1540.66	640.41
Soft tree	1577.03	696.20

Table 6: Validation results (in monetary units) associated with neural network and soft tree actors.

$L_{1}$ scaler	Average LCC (monetary unit)	StD (monetary unit)
0.1	3754.33	432.58
0.01 (baseline)	1577.03	696.20
0.001	1582.21	712.49

Table 7: Model performance of soft tree policies trained with different

L_{1}

regularization scalers.

Figure 15: RL-derived oblique decision tree after pruning.

Figure 16: RL-derived oblique decision tree integrated with agency rules.

Policy	Average LCC (monetary unit)	StD (monetary unit)
Oblique decision tree (RL)	1590.86	740.31
Oblique decision tree (RL with ad hoc rule)	1560.96	672.09
Condition-based policy (DP)	2133.42	1178.30
Reliability-based policy (GA)	1758.91	918.04

Table 8: Model performance with interpretable life-cycle policies.

Explore similar work

COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenance
Mar 8, 2026Dennis GrossInterpretability in RL Safe RL
Interpretable reinforcement learning with decision-tree pruning
Aug 7, 2026Mark Leon Ringer, Michel TokicInterpretability in RL
Faculty of Mathematics, Informatics and Statistics, Ludwig-Maximilians-University Munich, Munich, Germany · Siemens AG, Data & Artificial Intelligence, Otto-Hahn-Ring 6, 81739 Munich, Germany
Cross-fitted Proximal Learning for Model-Based Reinforcement Learning
Apr 6, 2026Nishanth Venkatesh, Andreas A. MalikopoulosReinforcement Learning Model-Based RL
Department of Systems Engineering, Cornell University, Ithaca, NY, USA · Applied Mathematics, Systems Engineering, Mechanical Engineering, Electrical & Computer Engineering, and School of Civil & Environmental Engineering, Cornell University, Ithaca, NY, USA

Mar 8, 2026J/K move · Enter open · S save

COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenance

Dennis Gross

Interpretability in RL Safe RL Formal Verification

Open paper AskPDF arXiv

Aging bridge networks require proactive, verifiable, and interpretable maintenance strategies, yet reinforcement learning (RL) policies trained solely on reward signals provide no formal safety guarantees and remain opaque to infrastructure managers. We demonstrate COOL-MC as a tool for verifying and explaining RL policies for multi-bridge network maintenance, building on a single-bridge Markov decision process (MDP) from the literature and extending it to a parallel network of three heterogeneous bridges with a shared periodic budget constraint, encoded in the PRISM modeling language. We train an RL agent on this MDP and apply probabilistic model checking and explainability methods to the induced discrete-time Markov chain (DTMC) that arises from the interaction between the learned policy and the underlying MDP. Probabilistic model checking reveals that the trained policy has a safety-violation probability of 3.5% over the planning horizon, being slightly above the theoretical minimum of 0% and indicating the suboptimality of the learned policy, noting that these results are based on artificially constructed transition probabilities and deterioration rates rather than real-world data, so absolute performance figures should be interpreted with caution. The explainability analysis further reveals, for instance, a systematic bias in the trained policy toward the state of bridge 1 over the remaining bridges in the network. These results demonstrate COOL-MC's ability to provide formal, interpretable, and practical analysis of RL maintenance policies.

Abstract

Explore similar work

COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenance

Interpretable reinforcement learning with decision-tree pruning

Cross-fitted Proximal Learning for Model-Based Reinforcement Learning