Deployed reinforcement learning agents often face safety requirements that are specified only after training: new hazard maps, revised risk thresholds, or behavioral alignment constraints. We study zero-update deployment-time adaptation in which a fixed library of risk-neutral source policies must be reused under a newly specified reward-risk tradeoff. We propose TRAM (Test-time Risk Adaptation with Mixture of Agents), a source-scored composition rule that evaluates each source under the target reward and an occupancy-based deployment risk, then selects actions using risk-adjusted source scores. Unlike training-time risk-sensitive methods tied to a fixed surrogate such as return variance, TRAM supports spatial barrier exposure, divergence to a reference behavior, and local volatility risks specified at test time. We make the surrogate nature of the method explicit: TRAM is not claimed to solve the full occupancy-control problem of the stitched policy, but it admits a measurable source-hull mismatch term connecting source-scored risk to realized risk. Experiments in gridworlds, MuJoCo Reacher, Safety-Gymnasium, and an inference-time LLM alignment setting show that TRAM improves deployment risk while preserving reward and requires no parameter updates at test time.
Figures & tables
Figure 1: TRAM decouples source training from deployment-time risk specification. (a) Before deployment, a library of risk-neutral source policies is trained once and frozen; the shaded band around each route marks the states that policy visits. (b) At deployment, a new hazard S is specified. At each state, TRAM scores every source and acts with the best one, so the executed path is stitched from the source routes in (a); keeping π1 past state 2 (dashed) would enter the hazard. (c) A source’s score is its value maxaQTπj(s,a) (full bar) minus a fixed per-source risk penalty (hatched), Eq. 4.3 . At state 2, π1 has the highest value but π2 the highest score, so the penalty changes the choice. TRAM selects the maximum by Eq. 4.4 and re-scores at the next state; all quantities are schematic, and no parameters are updated.
Method
Risk- aware
General Risk
Transfer
Task Struct.
Test- time
Risk-sensitive RL
✓
✗
✗
✗
✗
Zero-shot RL
✗
✗
✓
✗
✓
Dual RL
✓
✓
✗
✗
✗
Risk-aware Adaptation
✓
✗
✓
✗
✗
TRAM (ours)
✓
✓
✓
✓
✓
Table 1: Comparison with representative families. Columns: risk sensitivity, support for risk types beyond variance, cross-task transfer, use of shared task structure, and little/no test-time compute.
Figure 2: Motivating limitations of variance-based risk-aware transfer. (a) A risk-aware source is optimal for its training-time (variance) risk but specialized to it; when the deployment risk differs, it can miss the target-optimal route, while risk-neutral training preserves diverse reusable behaviors. (b) Trajectory-return variance can miss deployment-time risks: stochastic per-step rewards, spatial danger regions, and deployment-specific hazards are not reliably captured by return variance alone.
Figure 3: Gridworld mechanism validation. (a) Spatial hazards and per-step volatility expose trajectory-variance limitations; TRAM succeeds through occupancy-based risk modeling while baselines fail. (b) TRAM exploits high-reward paths when risk is absent and becomes conservative when risk appears, while RaSF remains uniformly conservative due to training-time risk aversion. Small grids show source policies; in (b), red sources are trained risk-aware (used by RaSF) and black sources risk-neutral (used by SF and TRAM). G is the return; bold marks the best value in each column.
Figure 4: Continuous-control deployment-time risk results. (a) In Reacher, a new target goal and barrier are specified after source training; TRAM substantially reduces failures relative to risk-neutral SF transfer, with a modest increase in final distance to the goal. (b) In Safety-Gymnasium, TRAM directly scores the requested deployment barrier risk and achieves the lowest barrier-hit probability while matching the lowest normalized CVaR among valid methods.
Figure 5: LLM alignment on Berkeley Nectar. The left panel summarizes the zero-update evaluation protocol using fixed source models, a target reward model, and a safe reference. The right panel shows that risk-free transfer maximizes the target reward-model score, while GPT-4 pairwise preferences favor TRAM overall.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Trained on deployment task
Success ↑
Barrier-hit prob. ↓
Constrained PPO
yes ( 750,000 steps)
0.997
0.240
TRAM, c=6.4×104
no
0.630
0.360
TRAM, c=1.28×105
no
0.366
0.290
TRAM, c=106
no
0.102
0.086
Appendix
Table 2: Constraint-aware baseline. Safety-Gymnasium, held-out goal (0.5,0.5) , barrier radius 0.18 , 300 evaluation episodes, common evaluator. TRAM rows are means over three evaluation seeds (Table 3 ); PPO is one training run. Best value in each column in bold.
c
Return ↑
Success ↑
Barrier-hit prob. ↓
6.4×104
0.518±0.028
0.630±0.030
0.360±0.020
1.28×105
0.268±0.062
0.366±0.031
0.290±0.018
106
0.244±0.023
0.102±0.002
0.086±0.010
Appendix
Table 3: Sensitivity to c at the main configuration. Barrier radius 0.18 ; mean ± standard deviation over three evaluation seeds, 300 episodes each.
Barrier-hit probability ↓
Success ↑
c / radius
0.03
0.09
0.18
0.24
0.30
0.03
0.09
0.18
0.24
0.30
3.2×104
0.247
0.303
0.470
0.590
0.867
0.963
0.740
0.757
0.783
0.893
6.4×104
0.117
0.173
0.340
0.457
0.563
0.780
0.607
0.617
0.633
0.647
1.28×105
0.073
0.177
0.270
0.403
0.450
0.597
0.363
0.330
0.437
0.433
2.56×105
0.067
0.203
0.327
0.423
0.553
0.350
0.307
0.273
0.277
0.390
106
0.040
0.057
0.117
0.153
0.190
0.227
0.213
0.120
0.120
0.087
Appendix
Table 4: Sensitivity to c across barrier radii. Barrier-hit probability (left, lower is better) and success (right, higher is better) for one fixed source library; one 300-episode evaluation per cell.
Single-source selection
Dynamic selection
c
Return
Success
Hit prob.
c
Return
Success
Hit prob.
0
2.278
1.000
0.907
104
0.724±0.025
0.879±0.028
0.411±0.078
1.0
1.202
0.627
0.497
1.2×104
0.667±0.033
0.814±0.038
0.289±0.029
1.5
0.762
0.460
0.293
1.6×104
0.550±0.038
0.720±0.019
0.216±0.006
2.0
0.280
0.130
0.033
Appendix
Table 5: Log-barrier risk at barrier radius 0.09. Left: single-source selection, one 300-episode evaluation per row. Right: dynamic per-state selection, mean ± 95% confidence half-width over three seeds.
Rollouts
Wall-clock time (s)
TV to 200-rollout estimate
50
47.0[46.7,47.3]
0.0128[0.0061,0.0195]
100
94.0[93.3,94.6]
0.0021[0,0.0045]†
200
188.0[186.7,189.4]
reference
Appendix
Table 6: Occupancy-estimation cost for one source. Mean and 95% Student- t interval over five evaluation seeds. TV is the total-variation distance to the 200-rollout estimate of the same seed.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.