Measuring the Conservatism of Latent Safety Filters for World Models
Organizations: University of California San Diego · San Diego State University
Abstract
A latent safety filter decides from the latent of a world model whether a planned action is safe. Conformal calibration sets the threshold of the filter so that it accepts an unsafe action only with a small probability. However, the filter can still be conservative and refuse many safe actions, depending on the latent. To measure the conservatism that a latent forces on the filter, we group similar latents and actions into cells. A filter that gives one decision per cell can refuse the unsafe transitions of a cell only by refusing the safe ones in it as well. We call the share of safe transitions that lie in such cells the overlap. We prove that the overlap is the minimum share of safe transitions that such a filter has to refuse in order to accept no unsafe transition in the data. Both the overlap and the conformal calibration can be applied to any latent. We use this to compare three existing designs, in which the safety labels train a classifier on top of the world model, the world model itself, or a separate encoder beside it. In a visual navigation task, the separate encoder has the lowest overlap, and at the same conformal level its filter refuses the fewest safe actions and lets the planner reach the most goals of the three.
Figures & tables
| Measured on cells | Calibrated filter, | |||||
|---|---|---|---|---|---|---|
| Latent | Overlap | Scene pairs in one cell | All hazards accepted | Block hazards accepted | Scene pairs separated | Safe actions refused |
| Planner latent | 0.96–0.97 | 1.00 | 0.018 | 0.153 | 0.00 | 0.302 |
| Jointly trained latent | 0.94–0.96 | 0.98 | 0.021 | 0.116 | 0.10 | 0.318 |
| Safety latent | 0.44–0.46 | 0.09 | 0.021 | 0.029 | 0.86 | 0.161 |
| Filter | Backup used | Stopped | Goals reached | Wall contacts | Block contacts |
|---|---|---|---|---|---|
| None | 0.00 | 0.00 | 0.72 | 0.37 | 0.11 |
| Full state (reference) | 0.50 | 0.22 | 0.74 | 0.03 | 0.00 |
| Planner latent | 0.79 | 0.61 | 0.39 | 0.01 | 0.03 |
| Jointly trained latent | 0.81 | 0.68 | 0.32 | 0.00 | 0.01 |
| Safety latent | 0.57 | 0.22 | 0.72 | 0.06 | 0.01 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | PointMaze U, map with seven free cells. One extra wall cell, the block, is placed uniformly in one of them in each episode. |
| State | Ball position and velocity . The full state adds the block position and the position of the ball relative to the block. |
| Observation | Current and previous RGB frames and . |
| Action | Five pushes in , one per simulator step. The classifier reads all ten components, and the action bins use their two-dimensional sum. |
| Data | 3000 episodes of 60 random actions. Episodes 0–1599 for training, 1600–1999 for development (choice of checkpoints and settings), 2000–2599 for calibration and 2600–2999 for testing. The last 1000 were generated after all models and settings were fixed. |
| Backup | Brake for actions. After these actions the ball had zero speed and no later contact in all 2500 rollouts we checked. |
| Labels | An action is a hazard if the ball touches a wall or the block during the five pushes or during the brake actions that follow. 16.8 % of the logged transitions are hazards. All encoders and classifiers were trained on the same earlier labels, which did not yet count a contact during the pushes, and all are evaluated on the labels above. |
| Planner latent | Jointly trained | Safety latent | |
|---|---|---|---|
| Input | Current frame, previous frame and their difference, and . | Same as . | Same as . |
| Encoder | ViT, four blocks, width 192, patch size 8. The class token and pass through an MLP. | Same as . | Four convolutional blocks (32, 64, 128, 128 channels) and a fully connected layer of width 256. |
| Latent | 192 numbers. | 192 numbers. | 20 numbers, 16 image features and . |
| Predictor | Six-layer predictor of the next latent. | Predictor of and a label predictor on the latent and the action, two layers of width 256. | One layer of width 64 on the latent and the action, with a hazard output and a brake-time output. |
| Loss | Next-latent prediction and endpoint inverse dynamics. | Loss of plus cross-entropy on the safety label with weight one. | Cross-entropy on the safety label and a Huber loss on the brake time. |
| Cells | K-means with clusters on 96,000 training transitions per representation, crossed with bins of the net push. Three partition seeds. |
| Overlap | Share of the safe training transitions whose cell holds one of the training hazards. |
| Classifier | MLP with hidden widths 128 and 64 on the standardized representation and the ten action components. |
| Threshold | One hazard with a start in from each of the 598 calibration episodes that hold one, and the 11th smallest score ( ). 200 draws. |
| Safe actions refused | Share of the safe test transitions with a score of at least , mean over the draws. |
| Hazards accepted | Share of the test hazards with a start in that score below , per episode and then averaged. A block hazard is a hazard that would not be one without the block, and for block hazards the share is taken among them. |
| Scene pairs | Cells of the first partition seed and the threshold of the median draw. A pair is in one cell when both of its scenes fall into the same cell. A pair is separated when the filter refuses the action in the hazard scene and accepts it in the safe scene. |
| Representation | |||||||
|---|---|---|---|---|---|---|---|
| Proprioceptive state | 0.840 | 0.780 | 0.692 | 0.556 | 0.365 | 0.199 | 0.082 |
| Full state | 0.600 | 0.496 | 0.377 | 0.259 | 0.163 | 0.092 | 0.050 |
| Planner latent | 0.668 | 0.555 | 0.402 | 0.302 | 0.195 | 0.105 | 0.041 |
| Jointly trained latent | 0.643 | 0.543 | 0.427 | 0.318 | 0.196 | 0.109 | 0.042 |
| Safety latent | 0.280 | 0.247 | 0.205 | 0.161 | 0.114 | 0.072 | 0.044 |
| Planner | CEM on the unchanged world model with : horizon 5, 300 samples, goal given as an encoded image. It executes its five-action plan and plans again when the plan is used up or after a brake. Each action is one push held for the five simulator steps. |
| Filter | Checks every proposed action, and on a refusal the brake is applied for the five simulator steps. On , and it uses the classifier of Table , on the classifier trained with its encoder. Threshold of the median calibration draw. |
| Episodes | Three runs of 50, one seed of the safety encoder per run. Block cell, start and goal are drawn at random from the episode seed, the same for every filter. An episode is counted up to the goal (within a distance of 0.5), the first contact or 60 actions. |
| Reachable goal | Start and goal lie on the same side of the block, in 109 of the 150 episodes. |
| Goals reached, stopped | Shares of the 109 episodes that reach the goal before any contact, or neither the goal nor a contact within 60 actions. |
| Backup used | Share of the 109 episodes in which the filter braked at least once. |
| Contacts | Shares of all 150 episodes that end in a contact with a wall or with the block. |
Safety Filtering
Safety filtering intercepts actions or generated content to prevent harm while preserving as much useful behavior as possible.
See all 132 papers