Vision-language-action policies generalize broadly but provide no execution-time guarantees; classical model-based planners respect kinematic and geometric constraints but generalize poorly. We study whether an action-conditioned Joint-Embedding Predictive Architecture (JEPA) world model can predict, before execution, both task progress and physical risk for candidate action chunks, and whether coupling these predictions to an embodiment-specific model-based safety shield yields a deployable pipeline for heterogeneous robots. We propose a receding-horizon decision pipeline: (1) a proposer produces K candidate action chunks; (2) an action-conditioned JEPA rolls each candidate forward in a frozen-encoder latent space conditioned on an embodiment embedding; (3) calibrated risk and progress heads score each rollout and report uncertainty; (4) a deterministic per-embodiment safety shield filters inadmissible candidates; (5) a fallback ladder handles empty-admissible-set cases. The learned ranking only reorders admissible candidates; enforcement guarantees come from the deterministic shield and fallback ladder. We evaluate with a pre-registered protocol in simulation (LIBERO-Long). In 600-episode configurations the full framework improved success over a shield-only baseline and reduced collision false negatives at matched recall. Deployment-efficiency measurements on target on-robot and edge accelerators are included. Real-robot experiments and an offline reranking significance test remain future work; see the paper for disclosures.
Figures & tables
Figure 1: System overview. One encoder pass per control step; K candidate action chunks from a mixed proposer; K action-conditioned latent rollouts; calibrated risk/progress scoring; and a deterministic, per-embodiment shield that decides admissibility . Learned components (blue) can only reorder candidates within the admissible set; the shield (red) and the fallback ladder are the only sources of enforced constraints (Proposition 1 ). Compute per step is fixed at 1+K network evaluations.
Figure 2: Action-conditioned JEPA. The frozen encoder is applied once to ot ; the predictor Fϕ rolls the latent forward autoregressively under the candidate action chunk, conditioned on the embodiment embedding erob through AdaLN/FiLM. Targets come from an EMA target encoder applied to the observed future with a stop-gradient (Eq. ( 7 )). No pixel decoder is used at any point. Risk and progress heads read the rollout, not the raw observation.
Figure 3: Decision pipeline for one control step. Scoring and shielding are separate stages in a fixed order. The learned score (step 3) produces a total order over candidates; the shield (step 4) produces a subset . Selection is the arg-min of the score restricted to that subset — a low-cost candidate that the shield rejects is never executed, regardless of how confident the model is. The illustrated scores are schematic, not measurements.
Figure 4: What is shared and what is not. Representation and prediction are shared across embodiments and conditioned on erob , following the cross-embodiment transfer evidence in [ 31 ] . Safety is not shared: stopping distance, footprint, joint limits and traversable slope are per-platform specifications, so each embodiment carries its own deterministic shield. Adding a robot means supplying a new erob and re-verifying its safe-stop manoeuvre, not retraining the world model.
Figure 5: Asynchronous edge–cloud execution. The proposer, shield and safe-stop controller run on-robot every control tick; world model rollouts run asynchronously and their ranking is consumed whenever it becomes available. The controller never blocks on the world model. If a ranking is stale beyond Δmax or the link drops, the system degrades to proposer-plus-shield — weaker in task performance but with an unchanged enforced constraint set, by Proposition 1 .
Method
AUROC
F1
FNR ↓
Brier ↓
ECE ↓
Random
0.50 ± 0.02
0.21 ± 0.02
0.88 ± 0.03
0.25 ± 0.01
0.31 ± 0.02
Rule-based
0.63 ± 0.02
0.34 ± 0.03
0.52 ± 0.04
0.19 ± 0.01
0.24 ± 0.02
Single-frame
0.71 ± 0.02
0.43 ± 0.03
0.38 ± 0.04
0.15 ± 0.01
0.17 ± 0.02
DINOv2 feats. a
0.79 ± 0.01
0.55 ± 0.02
0.27 ± 0.03
0.12 ± 0.01
0.13 ± 0.01
V-JEPA feats. b
0.83 ± 0.01
0.61 ± 0.02
0.21 ± 0.02
0.10 ± 0.01
0.11 ± 0.01
No action cond.
0.84 ± 0.01
0.62 ± 0.02
0.20 ± 0.02
0.10 ± 0.01
0.10 ± 0.01
Table 1: Level 2 — predictive and calibration performance of the collision head, measured. LIBERO, mean ± sd over 3 seeds ( {0,1,2} ). FNR is at a fixed recall of 0.90 chosen on validation, so all rows are compared at equal caution.
Figure 6: Reliability diagram and confidence histogram, LIBERO seed 0 ( n=1000 ). Produced by scripts/make_illustrative_figures.py from real per-example Level-2 predictions ( results/level2_libero_seed0.csv ). It shows an over-confident risk head (red) whose ranking is unchanged but whose probabilities are unusable as thresholds, and the same head after temperature scaling [ 15 ] (blue). ECE here (0.072 before, 0.052 after scaling) is broadly consistent with, but not identical to, Table 1 ’s 3-seed-mean calibration-loss ablation (0.14 vs. 0.04) — see Sec. 6 for why the two are related but not the same comparison.
Figure 7: ROC and precision–recall for the collision head, LIBERO seed 0 ( n=1000 ). Same data as Fig. 6 — AUROC here (0.887) is consistent with Table 1 ’s 3-seed mean (0.88). The marked operating point is the fixed-recall threshold at which false-negative rate is reported, so that all methods in Table 1 are compared at equal caution rather than at each method’s own most flattering threshold; the FNR value here (0.097) is not directly comparable to the table because the threshold is chosen and evaluated on the same held-out split rather than separate validation/test splits (see Sec. 6 for why). The precision–recall panel is shown because collisions are rare and ROC alone is optimistic under class imbalance.
Task & safety
Behaviour
ID
Configuration
Success ↑
Collision ↓
Stuck ↓
Interv. ↓
Recovery ↑
Reject. rate
Fallback
B1
Base policy
54%
18%
12%
21%
–
0%
0%
B2
+ rule-based shield
53%
11%
14%
17%
41%
9%
6%
B3
+ model-based shield
55%
4%
17%
15%
63%
16%
11%
B4
+ JEPA reranking (no shield)
59%
9%
10%
16%
48%
0%
0%
B6
Full, DINOv2 predictor
59%
4%
10%
13%
65%
15%
10%
Table 2: Level 4 — closed-loop performance, measured in simulation. LIBERO-Long, 10 tasks; 200 episodes × 3 seeds ( {0,1,2} ) = 600 episodes per configuration; percentages are episode-level rates. This is a simulation result, not a real-robot deployment Note the deliberate inclusion of rejection and fallback columns: without them, a controller that refuses to move would dominate the collision column. Oracle is an offline upper bound, not a method. Significance: with 600 episodes, a 95% Wilson interval around an observed rate of 62% has half-width ≈3.9 points, around 12% has half-width ≈2.6 points, and around 3% has half-width ≈1.4 points. Under these intervals, B5’s success margin over B3 ( +7 points) is significant at the episode level ( p≈0.014 , two-proportion test); B5’s margin over B4 ( +3 points) is not ( p≈0.29 ). We report this as a genuine limitation rather than rounding it away: the central thesis requires B5 to beat both B3 and B4, and on this sample size it beats B3 but not B4 at conventional significance.
Metric
On-robot
Edge-assisted
Encoder latency (median)
78 ms
19 ms
Per-candidate rollout latency
31 ms
7 ms
End-to-end decision latency (p50)
590 ms
165 ms
End-to-end decision latency (p99)
870 ms
310 ms
Candidate throughput (cand. s -1 )
27
97
Peak GPU memory
9.4 GB
11.2 GB
Table 3: Deployment efficiency, measured. On-robot: Jetson AGX Orin 64GB. Edge-assisted: RTX 4090 reached over a gigabit LAN. Frozen encoder ∼ 300 M parameters, predictor ∼ 300 M parameters, 16 frames at 256×256 ( → 2048 tokens), bf16, K=16 candidates, 2 s horizon; single warm accelerator, no batching across requests. For reference, the one comparable published figure we could verify is 16 s per action step for latent planning versus roughly 4 minutes for a video-generation world model [ 5 , Tab. 3] ; that figure is not directly comparable to the rows below because it reports full CEM planning rather than scoring a fixed candidate set.
Recent vision-language-action (VLA) models are promising for general-purpose manipulation, but long-horizon execution remains fragile. Small state-estimation or control errors can lead to irreversible failures (e.g., collisions and object drops). Avoiding these risks requires a proactive safety mechanism capable of anticipating hazards. In this paper, we introduce SafeLoop, a non-invasive external wrapper that adds hazard prediction and rollback-based recovery to a VLA model without changing its parameters. SafeLoop trains a risk predictor from vision and proprioception to output four values: the probability and time-to-hazard for body collisions and for object failures. A lightweight controller then chooses one of three actions based on the predicted risk: continue execution (noop), save a safety checkpoint (record), or retreat in joint space (rollback). Rollback moves the robot back to a recent safe waypoint and queries the base policy again, which may yield an alternative continuation. Across 24 LIBERO tasks (16 random seeds each) and three real-robot tasks (25 rollouts each), SafeLoop achieves a stronger overall safety-success trade-off than alternative methods, reducing hazard cases by roughly 70% while preserving task success and the base-policy control rate. Project code is available at https://github.com/Loule0-0/SafeLoop/tree/release/safeloop.
Zeyu Lou, Tianran Zhang, Xinquan Yue +2
Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Beijing University of Technology, Beijing, China
Physical interactions can create future hazards that are not apparent from the robot's current geometric surroundings. We present a framework termed Predictive Semantic Safety (PSS), which connects visual physical reasoning to backup-based safety filtering. A vision-language model (VLM) predicts physical events and their timing or directly predicts object displacements. An explicit motion model converts event hypotheses into object trajectories. Split conformal prediction calibrates position errors jointly across specified objects, observation times, and future times; geometric shape bounds convert the resulting position regions into predicted object occupancy. PSS evaluates a prescribed backup maneuver against this occupancy and derives input-affine constraints for minimally modifying the nominal input while preserving backup feasibility under the robot dynamics and input limits. MuJoCo experiments with a Unitree Go1 consider falling fixtures, impact-driven support loss, and contact propagation. PSS achieves a safe episode rate of 99.3%, compared with 43.3% for a Backup Control Barrier Function baseline that only uses current obstacle geometry.
Taekyung Kim, Salem Fradi, Yanning Dai +2
Department of Robotics, University of Michigan, Ann Arbor, MI 48109, USA · Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. · Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Switzerland. +2
Vision-Language-Action (VLA) and World-Action Models (WAMs) have demonstrated strong capabilities in general-purpose robotic manipulation, yet their generated actions may violate hard physical constraints and therefore be unsafe or infeasible for deployment. Existing safety approaches either optimize statistical safety objectives without deterministic per-step guarantees or correct unsafe actions only during inference, creating a mismatch between policy training and execution. We introduce ActSafeGuard, a differentiable and training-aligned safeguard layer for flow-matching based policies. ActSafeGuard integrates hard action feasibility into policy learning, not merely treating safety as an inference-time external component. Through an analytical ray-scaling operator design, ActSafeGuard enables boundary-aware gradients to guide the model to naturally learn constrained manifolds. Extensive experiments on multiple standard foundation backbones (π0.5 and Fast-WAM) across various tasks demonstrate that ActSafeGuard consistently achieves a 100% step safety rate while fully preserving or even boosting task success rates, providing a scalable and minimally invasive solution for safe embodied AI deployment.
Jianming Ma, Rongjun Jin, Xiaxi Si +3
1Shanghai Jiao Tong University · 2Shanghai Innovation Institute