cs.ROAug 18, 2026

Calibrated Predictive Safety for Heterogeneous Robots: An Action-Conditioned JEPA Framework with Model-Based Safety Shields

Authors: Kaiming Zhong, Tianhua Liu, Yue Wang

Organizations: Guangdong Bifang Intelligent Control Technology Co., Ltd.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language-action policies generalize broadly but provide no execution-time guarantees; classical model-based planners respect kinematic and geometric constraints but generalize poorly. We study whether an action-conditioned Joint-Embedding Predictive Architecture (JEPA) world model can predict, before execution, both task progress and physical risk for candidate action chunks, and whether coupling these predictions to an embodiment-specific model-based safety shield yields a deployable pipeline for heterogeneous robots. We propose a receding-horizon decision pipeline: (1) a proposer produces K candidate action chunks; (2) an action-conditioned JEPA rolls each candidate forward in a frozen-encoder latent space conditioned on an embodiment embedding; (3) calibrated risk and progress heads score each rollout and report uncertainty; (4) a deterministic per-embodiment safety shield filters inadmissible candidates; (5) a fallback ladder handles empty-admissible-set cases. The learned ranking only reorders admissible candidates; enforcement guarantees come from the deterministic shield and fallback ladder. We evaluate with a pre-registered protocol in simulation (LIBERO-Long). In 600-episode configurations the full framework improved success over a shield-only baseline and reduced collision false negatives at matched recall. Deployment-efficiency measurements on target on-robot and edge accelerators are included. Real-robot experiments and an offline reranking significance test remain future work; see the paper for disclosures.

Figures & tables

Figure 1: System overview. One encoder pass per control step; KK candidate action chunks from a mixed proposer; KK action-conditioned latent rollouts; calibrated risk/progress scoring; and a deterministic, per-embodiment shield that decides admissibility . Learned components (blue) can only reorder candidates within the admissible set; the shield (red) and the fallback ladder are the only sources of enforced constraints (Proposition 1 ). Compute per step is fixed at 1+K1{+}K network evaluations.
Figure 2: Action-conditioned JEPA. The frozen encoder is applied once to oto_{t} ; the predictor FϕF_{\phi} rolls the latent forward autoregressively under the candidate action chunk, conditioned on the embodiment embedding erobe_{\mathrm{rob}} through AdaLN/FiLM. Targets come from an EMA target encoder applied to the observed future with a stop-gradient (Eq. ( 7 )). No pixel decoder is used at any point. Risk and progress heads read the rollout, not the raw observation.
Figure 3: Decision pipeline for one control step. Scoring and shielding are separate stages in a fixed order. The learned score (step 3) produces a total order over candidates; the shield (step 4) produces a subset . Selection is the arg-min of the score restricted to that subset — a low-cost candidate that the shield rejects is never executed, regardless of how confident the model is. The illustrated scores are schematic, not measurements.
Figure 4: What is shared and what is not. Representation and prediction are shared across embodiments and conditioned on erobe_{\mathrm{rob}} , following the cross-embodiment transfer evidence in [ 31 ] . Safety is not shared: stopping distance, footprint, joint limits and traversable slope are per-platform specifications, so each embodiment carries its own deterministic shield. Adding a robot means supplying a new erobe_{\mathrm{rob}} and re-verifying its safe-stop manoeuvre, not retraining the world model.
Figure 5: Asynchronous edge–cloud execution. The proposer, shield and safe-stop controller run on-robot every control tick; world model rollouts run asynchronously and their ranking is consumed whenever it becomes available. The controller never blocks on the world model. If a ranking is stale beyond Δmax⁡\Delta_{\max} or the link drops, the system degrades to proposer-plus-shield — weaker in task performance but with an unchanged enforced constraint set, by Proposition 1 .
MethodAUROCF1FNR ↓\downarrowBrier ↓\downarrowECE ↓\downarrow
Random0.50 ±\pm 0.020.21 ±\pm 0.020.88 ±\pm 0.030.25 ±\pm 0.010.31 ±\pm 0.02
Rule-based0.63 ±\pm 0.020.34 ±\pm 0.030.52 ±\pm 0.040.19 ±\pm 0.010.24 ±\pm 0.02
Single-frame0.71 ±\pm 0.020.43 ±\pm 0.030.38 ±\pm 0.040.15 ±\pm 0.010.17 ±\pm 0.02
DINOv2 feats. a0.79 ±\pm 0.010.55 ±\pm 0.020.27 ±\pm 0.030.12 ±\pm 0.010.13 ±\pm 0.01
V-JEPA feats. b0.83 ±\pm 0.010.61 ±\pm 0.020.21 ±\pm 0.020.10 ±\pm 0.010.11 ±\pm 0.01
No action cond.0.84 ±\pm 0.010.62 ±\pm 0.020.20 ±\pm 0.020.10 ±\pm 0.010.10 ±\pm 0.01
Table 1: Level 2 — predictive and calibration performance of the collision head, measured. LIBERO, mean ±\pm sd over 3 seeds ( {0,1,2}\{0,1,2\} ). FNR is at a fixed recall of 0.900.90 chosen on validation, so all rows are compared at equal caution.
Figure 6: Reliability diagram and confidence histogram, LIBERO seed 0 ( n=1000n{=}1000 ). Produced by scripts/make_illustrative_figures.py from real per-example Level-2 predictions ( results/level2_libero_seed0.csv ). It shows an over-confident risk head (red) whose ranking is unchanged but whose probabilities are unusable as thresholds, and the same head after temperature scaling [ 15 ] (blue). ECE here (0.072 before, 0.052 after scaling) is broadly consistent with, but not identical to, Table 1 ’s 3-seed-mean calibration-loss ablation (0.14 vs. 0.04) — see Sec. 6 for why the two are related but not the same comparison.
Figure 7: ROC and precision–recall for the collision head, LIBERO seed 0 ( n=1000n{=}1000 ). Same data as Fig. 6 — AUROC here (0.887) is consistent with Table 1 ’s 3-seed mean (0.88). The marked operating point is the fixed-recall threshold at which false-negative rate is reported, so that all methods in Table 1 are compared at equal caution rather than at each method’s own most flattering threshold; the FNR value here (0.097) is not directly comparable to the table because the threshold is chosen and evaluated on the same held-out split rather than separate validation/test splits (see Sec. 6 for why). The precision–recall panel is shown because collisions are rare and ROC alone is optimistic under class imbalance.
  Task & safetyBehaviour
IDConfigurationSuccess ↑\uparrowCollision ↓\downarrowStuck ↓\downarrowInterv. ↓\downarrowRecovery ↑\uparrowReject. rateFallback
B1Base policy54%18%12%21%–0%0%
B2+ rule-based shield53%11%14%17%41%9%6%
B3+ model-based shield55%4%17%15%63%16%11%
B4+ JEPA reranking (no shield)59%9%10%16%48%0%0%
B6Full, DINOv2 predictor59%4%10%13%65%15%10%
Table 2: Level 4 — closed-loop performance, measured in simulation. LIBERO-Long, 10 tasks; 200 episodes ×\times 3 seeds ( {0,1,2}\{0,1,2\} ) = 600 episodes per configuration; percentages are episode-level rates. This is a simulation result, not a real-robot deployment Note the deliberate inclusion of rejection and fallback columns: without them, a controller that refuses to move would dominate the collision column. Oracle is an offline upper bound, not a method. Significance: with 600 episodes, a 95% Wilson interval around an observed rate of 62%62\% has half-width ≈ ⁣3.9\approx\!3.9 points, around 12%12\% has half-width ≈ ⁣2.6\approx\!2.6 points, and around 3%3\% has half-width ≈ ⁣1.4\approx\!1.4 points. Under these intervals, B5’s success margin over B3 ( +7+7 points) is significant at the episode level ( p≈0.014p\approx 0.014 , two-proportion test); B5’s margin over B4 ( +3+3 points) is not ( p≈0.29p\approx 0.29 ). We report this as a genuine limitation rather than rounding it away: the central thesis requires B5 to beat both B3 and B4, and on this sample size it beats B3 but not B4 at conventional significance.
MetricOn-robotEdge-assisted
Encoder latency (median)78 ms19 ms
Per-candidate rollout latency31 ms7 ms
End-to-end decision latency (p50)590 ms165 ms
End-to-end decision latency (p99)870 ms310 ms
Candidate throughput (cand. s -1 )2797
Peak GPU memory9.4 GB11.2 GB
Table 3: Deployment efficiency, measured. On-robot: Jetson AGX Orin 64GB. Edge-assisted: RTX 4090 reached over a gigabit LAN. Frozen encoder ∼\sim 300 M parameters, predictor ∼\sim 300 M parameters, 16 frames at 256×256256\times 256 ( →\to 2048 tokens), bf16, K=16K{=}16 candidates, 2 s horizon; single warm accelerator, no batching across requests. For reference, the one comparable published figure we could verify is 16 s per action step for latent planning versus roughly 4 minutes for a video-generation world model [ 5 , Tab. 3] ; that figure is not directly comparable to the rows below because it reports full CEM planning rather than scoring a fixed candidate set.
More in this topic

Robot Safety

Robot safety studies how to prevent harmful physical actions while enabling robots to complete tasks around people and in uncertain environments.

See all 229 papers

Explore similar work

CardsList

  1. SafeLoop: Risk-Aware Rollback for Vision-Language-Action Manipulation

    Sep 22, 2026Zeyu Lou, Tianran Zhang, Xinquan Yue +2Long-Horizon Robotic ManipulationRobot Safety

    Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Beijing University of Technology, Beijing, China

  2. Predictive Semantic Safety: From Visual Physical Reasoning to Safety-Critical Control

    Sep 28, 2026Taekyung Kim, Salem Fradi, Yanning Dai +2Robotic ControlSafety-Critical Control

    Department of Robotics, University of Michigan, Ann Arbor, MI 48109, USA · Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. · Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Switzerland. +2

  3. ActSafeGuard: Differentiable and Training-Aligned Constraint Enforcement for Flow-Matching Policies

    Sep 10, 2026Jianming Ma, Rongjun Jin, Xiaxi Si +3Robotic ControlRobot Foundation Models

    1Shanghai Jiao Tong University · 2Shanghai Innovation Institute