The new Specifications for the National Bridge Inventory (SNBI), in effect from 2022, emphasize the use of element-level condition states (CS) for risk-based bridge management. Instead of a general component rating, element-level condition data use an array of relative CS quantities (i.e., CS proportions) to represent the condition of a bridge. Although this greatly increases the granularity of bridge condition data, it introduces challenges to set up optimal life-cycle policies due to the expanded state space from one single categorical integer to four-dimensional probability arrays. This study proposes a new interpretable reinforcement learning (RL) approach to seek optimal life-cycle policies based on element-level state representations. Compared to existing RL methods, the proposed algorithm yields life-cycle policies in the form of oblique decision trees with reasonable amounts of nodes and depth, making them directly understandable and auditable by humans and easily implementable into current bridge management systems. To achieve near-optimal policies, the proposed approach introduces three major improvements to existing RL methods: (a) the use of differentiable soft tree models as actor function approximators, (b) a temperature annealing process during training, and (c) regularization paired with pruning rules to limit policy complexity. Collectively, these improvements can yield interpretable life-cycle policies in the form of deterministic oblique decision trees. The benefits and trade-offs from these techniques are demonstrated in both supervised and reinforcement learning settings. The resulting framework is illustrated in a life-cycle optimization problem for steel girder bridges.
Figures & tables
Figure 1: Tree-based interpretable decision models: (a) standard decision tree and (b) oblique decision tree.
Figure 2: A simple soft tree with one internal node for three-class classification.
Figure 6: Example of pruning frozen tree: (a) original frozen tree and (b) oblique decision tree after pruning.
Figure 7: Implementation of a differentiable soft tree model (depth = 3): (a) neural network layers, (b) rearrangement to soft tree structure, and (c) output of class logits.
Figure 8: General architecture for supervised classification with soft tree models.
Figure 9: PPO architecture for reinforcement learning with soft tree based actors
Figure 10: Labeled synthetic data for supervised classification experiments
Model
Training accuracy (%)
Validation accuracy (%)
Test accuracy (%)
Standard tree
57.78
55.90
57.10
Oblique tree
60.52
58.90
59.75
Neural network
98.50
98.25
97.80
Soft tree
90.53
90.35
90.35
Table 1: Model performance of different classification models.
Temperature and annealing
Soft tree score (%)
Frozen tree score (%)
Relative difference
Fixed at 100
46.90
27.05
42.32%
Fixed at 1
90.35
83.10
8.02%
Fixed at 0.01
77.55
76.05
1.93%
Anneal: 100 to 0.01
80.70
80.65
0.06%
Anneal: 1 to 0.01
91.85
91.80
0.05%
Note : “Frozen” tree refers to the full oblique tree converted from the soft tree baseline prior to pruning.
Table 2: Test accuracy of soft and frozen trees associated with different temperatures and temperature annealing schedules.
Figure 11: Effects of L1 regularization on (a) number of pruned nodes and (b) model performance
Figure 12: Effects of pruning threshold on (a) model performance and (b) pruned nodes.
Figure 13: CS evolution under two policies: (a) do-nothing; (b) conduct rehabilitation ( a=3 ) if CS4 percentage is greater than 5%, do nothing otherwise.
CS
InfoBridge Data
Fitted Dirichlet
Mean (%)
StD (%)
Mean (%)
StD (%)
CS1
63.5
37.5
42.7
42.6
CS2
33.3
35.4
31.8
40.1
CS3
3.2
8.9
14.3
30.1
CS4
∼ 0
∼ 0
11.2
27.2
Note : Fitted Dirichlet is parameterized by theta vector of [0.1496,0.1114,0.0500,0.0393]⊺
Table 3: Mean and StD of CS proportions
Actor model
Hyperparameter
Value
Neural network
Network layout
Two fully connected hidden layers, each with 64 neurons and an expoential linear unit (ELU) as the activation function
Soft tree
Maximum depth
11
L1 regularization scaler
0.01
Initial temperature
1
Temperature update
0.9550 (from T=1 to T=0.01 in 100 stages)
Update frequency
1 batch per stage
Table 4: Hyperparameters for different actor models.
Category
Hyperparameter
Value
Critic model
Network layout
Three fully connected hidden layers, each with 32 neurons and an ELU as the activation function.
PPO loss
Clipping epsilon
0.01
Entropy coefficient
0.05
Critic coefficient
0.5
General advantage
Lambda
0.95
Training
Total batches
100
Table 5: Hyperparameters for critic model and PPO training (common to both types of actors).
Figure 14: Learning curves associated with both types of actor models.
Actor model
Average LCC (monetary unit)
StD (monetary unit)
Neural network
1540.66
640.41
Soft tree
1577.03
696.20
Table 6: Validation results (in monetary units) associated with neural network and soft tree actors.
L1 scaler
Average LCC (monetary unit)
StD (monetary unit)
0.1
3754.33
432.58
0.01 (baseline)
1577.03
696.20
0.001
1582.21
712.49
Table 7: Model performance of soft tree policies trained with different L1 regularization scalers.
Figure 15: RL-derived oblique decision tree after pruning.
Figure 16: RL-derived oblique decision tree integrated with agency rules.
Policy
Average LCC (monetary unit)
StD (monetary unit)
Oblique decision tree (RL)
1590.86
740.31
Oblique decision tree (RL with ad hoc rule)
1560.96
672.09
Condition-based policy (DP)
2133.42
1178.30
Reliability-based policy (GA)
1758.91
918.04
Table 8: Model performance with interpretable life-cycle policies.
Aging bridge networks require proactive, verifiable, and interpretable maintenance strategies, yet reinforcement learning (RL) policies trained solely on reward signals provide no formal safety guarantees and remain opaque to infrastructure managers. We demonstrate COOL-MC as a tool for verifying and explaining RL policies for multi-bridge network maintenance, building on a single-bridge Markov decision process (MDP) from the literature and extending it to a parallel network of three heterogeneous bridges with a shared periodic budget constraint, encoded in the PRISM modeling language. We train an RL agent on this MDP and apply probabilistic model checking and explainability methods to the induced discrete-time Markov chain (DTMC) that arises from the interaction between the learned policy and the underlying MDP. Probabilistic model checking reveals that the trained policy has a safety-violation probability of 3.5% over the planning horizon, being slightly above the theoretical minimum of 0% and indicating the suboptimality of the learned policy, noting that these results are based on artificially constructed transition probabilities and deterioration rates rather than real-world data, so absolute performance figures should be interpreted with caution. The explainability analysis further reveals, for instance, a systematic bias in the trained policy toward the state of bridge 1 over the remaining bridges in the network. These results demonstrate COOL-MC's ability to provide formal, interpretable, and practical analysis of RL maintenance policies.
Reinforcement learning policies are difficult to inspect, but interpreting them is a prerequisite for trustworthiness. Converting a trained policy into explicit decision-tree rules improves transparency and the resulting artifacts often remain too complex for human understanding. We present a pruning process that simplifies such rule-based policies while preserving task performance and making edits to the policy auditable. The process defines a small set of structural and usage-aware operators and evaluates candidate edits by re-executing the policy to measure return and interpretability proxies. This exposes an transformation process from complex to compact policy structures. We investigate this approach on classic control and MuJoCo benchmarks, where pruning traces reveal consistent interpretability improvements while maintaining high performance.
Mark Leon Ringer, Michel Tokic
Faculty of Mathematics, Informatics and Statistics, Ludwig-Maximilians-University Munich, Munich, Germany · Siemens AG, Data & Artificial Intelligence, Otto-Hahn-Ring 6, 81739 Munich, Germany
Model-based reinforcement learning is attractive for sequential decision-making because it explicitly estimates reward and transition models and then supports planning through simulated rollouts. In offline settings with hidden confounding, however, models learned directly from observational data may be biased. This challenge is especially pronounced in partially observable systems, where latent factors may jointly affect actions, rewards, and future observations. Recent work has shown that policy evaluation in such confounded partially observable Markov decision processes (POMDPs) can be reduced to estimating reward-emission and observation-transition bridge functions satisfying conditional moment restrictions (CMRs). In this paper, we study the statistical estimation of these bridge functions. We formulate bridge learning as a CMR problem with nuisance objects given by a conditional mean embedding and a conditional density. We then develop a K-fold cross-fitted extension of the existing two-stage bridge estimator. The proposed procedure preserves the original bridge-based identification strategy while using the available data more efficiently than a single sample split. We also derive an oracle-comparator bound for the cross-fitted estimator and decompose the resulting error into a Stage I term induced by nuisance estimation and a Stage II term induced by empirical averaging.
Nishanth Venkatesh, Andreas A. Malikopoulos
Department of Systems Engineering, Cornell University, Ithaca, NY, USA · Applied Mathematics, Systems Engineering, Mechanical Engineering, Electrical & Computer Engineering, and School of Civil & Environmental Engineering, Cornell University, Ithaca, NY, USA