Safe in-context reinforcement learning (Safe ICRL) asks a policy with fixed parameters to improve on an unseen constrained task from its interaction history while aiming to keep its cost within a budget. Prior work studied Safe ICRL on five benchmarks in three families: a grid world with a hidden goal and obstacles (DarkRoom), continuous navigation where goals and hazards stay invisible until reached (PointGoal, CarGoal), and locomotion with a hidden velocity limit (HalfCheetah, Ant). Safe Algorithm Distillation (Safe AD), trained on the learning histories of a safe RL algorithm, has shown adaptation failures in prior evaluations. For example, in the locomotion tasks its return stays below zero, and it rarely reaches the goal in CarGoal. We revisit two parts of Safe AD. First, we distill a budget-aware teacher with two phases. It searches for the hidden task, exploring more when more budget remains. After its first reward or cost, an expert policy exploits the task within the remaining budget. We collect the teacher's histories under fixed or dynamic budgets. Dynamic budgets increase the occupancy entropy of the return and budget inputs in every environment, and at test time they improve return and lower cost in DarkRoom and HalfCheetah. Second, we propose A-Lag, a test-time adjustment that reweights the distilled policy's actions by their predicted cost, with an adaptive Lagrange multiplier that is updated from the costs observed. In our experiments, A-Lag had lower mean episode cost than Safe AD in all five environments, with lower mean final return in four.
Figures & tables
Figure 1: Models trained on dynamic-budget histories (Dynamic) and on fixed-budget histories (Fixed), on the test tasks. Each curve shows the mean episode return or cost over three seeds, with a shaded band of ±2 standard errors. With dynamic budgets, A-Lag has a lower cost in four of the five environments, all but PointGoal, and Safe AD has a higher return in all five, compared with fixed budgets.
Figure 2: In-context adaptation on the test tasks. Each curve shows the mean episode return or cost over three seeds, with a shaded band of ±2 standard errors. A-Lag starts every deployment at λ=0 , and λ is adjusted during test time by Eq. ( 9 ). In DarkRoom, PointGoal and CarGoal the return of A-Lag and Safe AD rises over the episodes while their cost falls. In HalfCheetah and Ant their cost drops sharply after the first episode, and in HalfCheetah the return falls with it. A-Lag’s Ant return also declines across episodes. A-Lag has a lower cost than Safe AD in DarkRoom, CarGoal and Ant, at a lower return. The mean costs are also slightly lower in PointGoal, at a slightly lower return, and in HalfCheetah, where it ends with a higher return. CDT has a higher cost than both in the navigation and velocity tasks and does not find the goal in DarkRoom.
Figure 3: Violation rate and excess cost max(0,Gc−B0) over the episodes of a deployment on the test tasks. Each curve is the mean over three seeds, with a shaded band of ±2 standard errors. Averaged over the episodes, A-Lag has the lowest excess cost in DarkRoom, PointGoal, CarGoal and HalfCheetah and is close to Safe AD in Ant. In most later DarkRoom episodes CDT, which does not find the goal, is lower.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Example training and evaluation layouts, adapted from Moeini et al. (2026) . Training layouts are center-oriented, whereas evaluation layouts are edge-oriented. We use this shift for the DarkRoom, PointGoal and CarGoal test tasks.
Chosen q
Episodes over budget
Environment
( q≈0.5 level)
q≈0.5
q=1
DarkRoom (random-action probability 0.3)
0.49
5.3%
5.0%
PointGoal
0.55
25.8%
11.5%
CarGoal
0.43
33.4%
12.7%
HalfCheetah
0.44
25.6%
12.3%
Ant
0.51
12.5%
7.3%
Appendix
Table 2: Chosen quality of the weaker expert and the share of teacher episodes over budget in the dynamic-budget histories.
Environment
Switching event
Search
Expert of quality q
DarkRoom
first reward, at the goal
tries new moves, more often when more budget is left. Never repeats a move that cost
takes the shortest known path without cost. For q<1 , mixes in random actions at a fixed rate
PointGoal, CarGoal
first reward, at the goal
drives through a grid of points over the arena. Goes less far around known hazards when more budget is left
drives to the goal position estimated from the history, using a PPO controller. For q<1 , the controller is saved earlier in training
HalfCheetah, Ant
first cost, from running too fast
runs faster every 25 steps while the budget allows another try
runs at the fastest speed whose known cost fits the budget, using a PPO controller. For q<1 , the controller is saved earlier in training
Appendix
Table 3: The teacher and its data. Top: the environment-specific parts of the teacher. Everything else is shared by all five environments. Bottom: teacher data per environment, over both budget regimes and both quality levels.
Parameter
DarkRoom
PointGoal/CarGoal
Velocity tasks
Hidden Dim
256
256
256
Token Embedding Dim
64
128
128
Num Layers
6
6
6
Num Heads
4
8
8
Seq Len
100
500
400
Attention Dropout
0.5
0.5
0.1
Appendix
Table 4: Training settings. Top: the sequence models (Safe AD, A-Lag and CDT). Num Layers is L in Section 4.1 . Seq Len counts environment steps, not token positions. Safe AD and A-Lag use four positions per step plus two constant prefix positions. Token Embedding Dim is the width of the per-modality encoders before they are projected to the Hidden Dim. The navigation tasks use four gradient accumulation steps, giving an effective batch size of 32. Action-input dropout sets each component of an action input in the history to zero with the given probability during training. The action targets are unchanged. Bottom: the gradient-based safe meta-RL baselines, SafeMeta and MAML with a cost penalty, as in the preliminary study. The navigation tasks are PointGoal and CarGoal, and the velocity tasks are HalfCheetah and Ant.
Figure 5: Occupancy of the (RTG, CTG) plane in the teacher data, fixed budgets (top) and dynamic budgets (bottom), both quality levels. Color is the share of steps on a logarithmic scale. In PointGoal and CarGoal the RTG is 0 or 1, so it is drawn as two columns. Under a fixed budget the remaining budget starts at Bmax and falls in steps of the incurred cost, which leaves gaps between the CTG values that occur.
Figure 6: Fixed multipliers on the training tasks (the reported A-Lag models, averaged over three seeds). Top: final return relative to λ=0 , with a band of ±2 standard errors of the paired difference. Bottom: violation rate with a band of ±2 standard errors. The wide gray line marks λmax . Larger fixed multipliers lower the return in every environment, and the violation rate in DarkRoom, HalfCheetah and Ant. The violation rate is nonmonotonic in PointGoal and CarGoal and can increase as the multiplier grows, including within the selected range in CarGoal. A-Lag uses this sweep only to set λmax and tunes λ within [0,λmax] during the test deployment, from the costs it incurs.
Figure 7: A-Lag’s mean multiplier per episode on the training tasks, as a fraction of λmax , averaged over three seeds.
Figure 8: Final return and mean episode cost on the test tasks against the share of q=1 histories (16 histories per task), averaged over three seeds, with bands of ±2 standard errors. The main models and the violation rates are in Table 5 .
Environment
Metric
A-Lag, fixed λ
A-Lag
DarkRoom ( λ=0.3 )
Return
0.831
0.832
Cost
3.47
3.45
Violation
0.190
0.180
Excess cost
0.46
0.39
PointGoal ( λ=1 )
Return
0.150
0.164
Cost
10.17
9.69
Appendix
Table 7: A-Lag with its adaptive multiplier against A-Lag with a fixed multiplier, on the test tasks, with the same trained weights. The fixed value, given next to each environment, was chosen on training tasks by a fixed rule: the value with the lowest violation rate whose return is not clearly below that at λ=0 (Appendix E ). Both columns are means over three seeds. Bold marks the better of the two in each row: the higher return and the lower cost, violation rate and excess cost. In CarGoal no tested value met the rule, so we use the one whose return came closest, λ=0.3 .
A-Lag − Safe AD
A-Lag − A-Lag, λ=0
Environment
Metric
Seed 1
Seed 2
Seed 3
Seed 1
Seed 2
Seed 3
DarkRoom
Return
−0.057
−0.047
−0.080
−0.059
−0.021
−0.053
Cost
−0.57
+0.09
−0.73
−0.17
−0.16
−0.24
Violation
−0.064
−0.011
−0.073
−0.025
−0.026
−0.036
PointGoal
Return
+0.021
−0.027
−0.018
−0.008
+0.032
+0.001
Cost
−0.12
−1.79
+0.87
−1.16
−0.63
+0.27
Appendix
Table 8: Per-seed differences on the test tasks: A-Lag against Safe AD, and A-Lag against its own weights with the multiplier held at zero. Each entry is the first model minus the second, averaged over the test tasks and deployments of one training seed.
Safe AD, dynamic − fixed
A-Lag, dynamic − fixed
Environment
Metric
Seed 1
Seed 2
Seed 3
Seed 1
Seed 2
Seed 3
DarkRoom
Return
+0.043
+0.147
+0.311
+0.085
+0.134
+0.152
Cost
−0.78
−0.91
−0.74
−1.19
+0.16
−1.49
Violation
−0.080
−0.104
−0.071
−0.128
−0.012
−0.152
PointGoal
Return
+0.081
+0.078
+0.059
+0.105
+0.093
+0.018
Cost
+0.53
+0.77
+1.23
+2.08
+2.12
+0.97
Appendix
Table 9: Per-seed differences on the test tasks between models trained on dynamic-budget and on fixed-budget histories. Each entry is the first model minus the second, averaged over the test tasks and deployments of one training seed.
Figure 9: Safe AD trained on the teacher data (Teacher Data) and the Safe AD baseline of Moeini et al. (2026) , trained on PPO-Lagrangian learning histories (PPO-Lag Data).
Safe in-context reinforcement learning (ICRL) adapts online from interaction history without test-time parameter updates while controlling episode cost under a safety budget. Under out-of-distribution (OOD) deployment shifts, pretraining-only safe ICRL can give poor reward-safety tradeoffs because the remaining budget affects behavior only through frozen policy conditioning, not an explicit action-level check against predicted future cost. We propose a latent Q-Barrier shield that learns a context representation, latent dynamics, and an ensemble cost critic before deployment. Without parameter updates, the shield infers context from history and filters or softly reweights candidate actions using the remaining budget and predicted future cost. We prove a conditional one-step barrier-margin bound over realized candidate sets, with explicit Bellman and latent-prediction error terms. Across five safe ICRL benchmarks, the shield improves deployment-time reward-safety tradeoffs over a strong safe-ICRL baseline: after a short context window, it achieves higher return in four of five benchmarks while matching or lowering average episode cost in all five. Code is available at https://github.com/kmj1122/q-barrier.
Safety guarantees are a prerequisite to the deployment of reinforcement learning (RL) agents in safety-critical tasks. Often, deployment environments exhibit non-stationary dynamics or are subject to changing performance goals, requiring updates to the learned policy. This leads to a fundamental challenge: how to update an RL policy while preserving its safety properties on previously encountered tasks? The majority of current approaches either do not provide formal guarantees or verify policy safety only a posteriori. We propose a novel a priori approach to safe policy updates in continual RL by introducing the Rashomon set: a region in policy parameter space certified to meet safety constraints within the demonstration data distribution. We then show that one can provide formal, provable guarantees for arbitrary RL algorithms used to update a policy by projecting their updates onto the Rashomon set. Empirically, we validate this approach across grid-world navigation environments (Frozen Lake and Poisoned Apple) where we guarantee an a priori provably deterministic safety on the source task during downstream adaptation. In contrast, we observe that regularisation-based baselines experience catastrophic forgetting of safety constraints while our approach enables strong adaptation with provable guarantees that safety is preserved.
Maksim Anisimov, Francesco Belardinelli, Matthew Wicker
Imperial College London · Imperial College London, London, United Kingdom
Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents' interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least 9.6× fewer environment samples.