Deployment-Time Reliability of Learned Robot Policies

Christopher Agia

cs.ROMar 12, 2026

Deployment-Time Reliability of Learned Robot Policies

Authors: Christopher Agia

Organizations: THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY

Abstract

Recent advances in learning-based robot manipulation have produced policies with remarkable capabilities. Yet, reliability at deployment remains a fundamental barrier to real-world use, where distribution shift, compounding errors, and complex task dependencies collectively undermine system performance. This dissertation investigates how the reliability of learned robot policies can be improved at deployment time through mechanisms that operate around them. We develop three complementary classes of deployment-time mechanisms. First, we introduce runtime monitoring methods that detect impending failures by identifying inconsistencies in closed-loop policy behavior and deviations in task progress, without requiring failure data or task-specific supervision. Second, we propose a data-centric framework for policy interpretability that traces deployment-time successes and failures to influential training demonstrations using influence functions, enabling principled diagnosis and dataset curation. Third, we address reliable long-horizon task execution by formulating policy coordination as the problem of estimating and maximizing the success probability of behavior sequences, and we extend this formulation to open-ended, language-specified tasks through feasibility-aware task planning. By centering on core challenges of deployment, these contributions advance practical foundations for the reliable, real-world use of learned robot policies. Continued progress on these foundations will be essential for enabling trustworthy and scalable robot autonomy in the future.

Figures & tables

Figure 2.1 : We present Sentinel , a runtime monitor that detects unknown failures of generative robot policies at deployment time. Constructing Sentinel requires only a set of successful policy rollouts and a description of the task, from which it detects diverse failures by monitoring (a) the temporal consistency of action-chunk distributions generated by the policy and (b) the task progress of the robot(s) through video QA with Vision-Language Models. More details can be found on the Sentinel website: https://sites.google.com/stanford.edu/sentinel .

Figure 2.2 : Action sequence prediction overlap during policy rollout.

Figure 2.3 : Overview of Sentinel. The images depict a policy rollout for timesteps $t=1,\ldots,T$ . Temporal Consistency Detector: At each timestep $t$ , the state $s_{t}$ is passed to the generative policy to obtain action distributions $\pi_{t}$ between which statistical distances $\hat{D}_{t}$ are computed to measure temporal consistency. The statistical distances are summed up to the current timestep $T$ (as in Eq. 2.1 ) and thresholded by $\gamma$ to detect policy failure. Vision-Language Model (VLM) Detector: The VLM classifies whether the policy is failing to make progress on its task given a video up to timestep $T$ and a description of the task. Execution stops if either detector raises a warning.

Figure 2.4 : Temporal consistency scores grow faster when the policy fails. Error bars indicate the 5-th and 95-th score quantiles.

Figure 2.5 : Detecting failures in PushT. Left: Our failure detector ( STAC ) which measures the temporal consistency of a generative policy outperforms several families of out-of-distribution detectors. Right: The best performance comes from measuring temporal consistency with statistical distance functions; augmenting baselines with temporal consistency does not always increase their performance.

Figure 2.6 : Detecting task progression failures. Combining VLMs with STAC yields an accurate overall detector ( Sentinel ) for both task progression and erratic failures ( Table 2.1 ). See Section A.3.2 for extended results and analysis.

Figure 3.1 : We present CUPID , a robot data curation method that leverages influence functions to predictively answer counterfactual questions about the effect of each demonstration on downstream policy performance. More details can be found on the CUPID website: https://cupid-curation.github.io .

Figure 3.2 : Data curation with Cupid . Upon training a policy on a set of demonstrations using behavior cloning, we evaluate it online to collect closed-loop rollout trajectories and estimate the policy’s expected return. Cupid ranks demonstration based on their measured influence on this performance estimate and selects the top- $k$ . Thus, curating with Cupid results in a dataset of demonstrations that most strongly influences closed-loop policy success.

Panel of Figure 3 — Figure 3.3 : RoboMimic mixed-quality curation results. Top: Data Quality. Baselines often prioritize demonstration quality (e.g., DemInf [ 109 ] ), but higher demonstration quality does always translate to higher policy success rates. In contrast, Cupid targets demonstrations that most strongly contribute to downstream policy performance. Bottom: Policy Performance. Diffusion policies trained on data curated by Cupid achieve higher success rates than baselines, despite using demonstrations of perceived lower quality. Although combining performance and quality measures ( Cupid-Quality ) yields the best policies on mixed-quality datasets, quality measures can degrade performance in other settings (see Fig. 3.4 ). Results are averaged over 3 random seeds (500 policies trained across settings). Success rates are computed over 50 rollouts from the last 10 checkpoints (500 rollouts total).

Figure 3.6 : Cupid ablation on the number of policy rollouts. Performance influences ( Eq. 3.3 ) converge with $m\in[25,50]$ rollouts on “Lift MH” and “Square MH” (yielding similar quality trends), and $m\in[50,100]$ rollouts on “Transport MH,” validating the practical applicability of Cupid under realistic rollout budgets. Curation is performed on diffusion policies. Results are averaged over 3 random seeds. Errors bars represent standard error.

Figure 3.7 : Data curated for single-task diffusion policies improves

\pi_{0}

[ 25 ] post-training performance. Additional results in Fig. B.8 .

Figure 4.1 : Sequential manipulation tasks often contain geometric dependencies between actions. In this example, the robot needs to use the hook to pull the block into its kinematic workspace so it is close enough to pick up. The top row shows how greedy execution of skills results in the robot picking up the hook in a way that prevents it from reaching the block. We present a method for planning with skills to maximize long-horizon success without the need to train the skills on long-horizon tasks. More details can be found on the STAP website: https://sites.google.com/stanford.edu/stap . © 2023 IEEE.

Figure 4.2 : Planning in a 2D toy domain. The agent needs to get the green block under the brown receptacle with two skills: $\text{Place}()$ and $\text{Push}()$ that operate on the horizontal position $x$ of the green block. Plots (a) and (b) show the Q-functions across $(x,\theta)$ for each skill. $\text{Place}()$ is only trained to get the green block on the ground, so the planner must determine $a=x$ s.t. $\text{Push}()$ is unobstructed. The optimal action maximizes the probability of long-horizon task success ( Eq. 4.3 ), approximated by the product of Q-functions in plot (c). © 2023 IEEE.

Panel of Figure 4 — Figure 4.3 : Left: Training pipeline. We train each skill independently on single-step environments with skill-specific rewards. We use the experience collected by each skill to train (1) a dynamics model and (2) a SCOD [ 239 ] model to predict OOD critic inputs per skill. The benefit of training each skill independently is that we can easily add skills to the library or even mix skill acquisition strategies (e.g. RL, imitation learning, and handcrafted skills). Our planning framework ensures that the skills can be composed to solve any long-horizon task even if the skills were not explicitly trained to perform those tasks. Right: Example evaluation tasks. We evaluate our method on 9 tasks: 3 Hook Reach tasks, where the robot needs to use the hook to bring objects closer, 3 Constrained Packing tasks, where the robot needs to place blocks on the rack, and 3 Rearrangement Push tasks, where the robot needs to remove obstacles to push a target block under the rack. The 9 tasks feature a range of geometric complexities and plan skeleton lengths. © 2023 IEEE.

Figure 4.4 : A small-scale experiment comparing the performance of our method to Oracle planning. The left plot shows the average success rates across two domains ( Hook Reach and Constrained Packing ). The dark bars indicate the ground truth task success, and the light bars indicate sub-goal completion rate, which measures how close the plan was to successfully completing the task. The predicted task success computed from the product of Q-values is indicated by a dotted line. Our method with Policy CEM is able to nearly match the success rate of Oracle while taking 4 orders of magnitude less time, as shown in the plot on the right. © 2023 IEEE.

Figure 4.5 : Planning experiment with 3 domains, each with 3 tasks. Our method with Policy CEM is able to generalize to all of these tasks without ever seeing them during training. On 6 out of the 9 tasks, our method either matches or outperforms DAF-Skills , which is trained directly on the evaluation task. DAF-Gen shows the generalization performance of the DAF-Skills models when evaluated on unseen tasks within the same domain. © 2023 IEEE.

Figure 4.6 : Integration of our planning framework with task planning and UQ to solve TAMP problems. The poor performance of the methods without SCOD in the Constrained Packing problem highlights the importance of UQ when attempting to solve unseen TAMP problems with learned skills. © 2023 IEEE.

Figure 5.1 : To carry out the instruction “get two primary-colored objects onto the rack,” the robot must apply symbolic reasoning over the scene description and language instruction to deduce what skills should be executed to acquire a second primary-colored object, after noticing that a red object is already on the rack (i.e. on(red, rack) ). It must also apply geometric reasoning to ensure that skills are sequenced in a manner that is likely to succeed. Unlike prior work [ 6 , 118 ] that myopically executes skills at the current timestep, Text2Motion constructs sequences of skills and coordinates their geometric dependencies with geometric feasibility planning [ 3 ] . Upon planning the skill sequence Pick(hook) , Pull(blue, hook) , Pick(blue) , Place(blue, rack) , our method computes a grasp position on the hook that enables pulling the blue object into the robot workspace so that it can be successfully picked up in the next step. More details can be found on the Text2Motion website: https://sites.google.com/stanford.edu/text2motion . © 2023 Springer Nature.

Figure 5.2 : shooting and greedy-search planning overview . Both shooting and greedy-search planners use the LLM to predict the set of valid goal states given the user’s natural language instruction and a description of the current state of the environment. These predicted goals are used to decide when the instruction is satisfied and planning can terminate. Left: The shooting strategy uses the LLM to propose full skill sequences first and then runs geometric feasibility planning afterwards. As shown in the experiments, this approach fails when the space of candidate task plans is large but few skill sequences are geometrically feasible. Right: In the greedy-search strategy, the LLM is used to propose $K$ candidate skills with the top LLM scores. The geometric feasibility planner then evaluates the feasibility of each candidate skill, and the one with the highest product of LLM and geometric feasibility scores is selected. The successor state of this skill is predicted by the geometric feasibility planner’s dynamics model. If the successor state does not satisfy any of the predicted goals, then it is given to the LLM to plan the next skill. If a goal is satisfied, then the planner returns the skill sequence for execution. By interleaving LLM task planning with geometric feasibility planning at each planning iteration, greedy-search is able to reliably find feasible plans across the different families of tasks we study in the experiments. © 2023 Springer Nature.

Figure 5.3 : Proposed hybrid planner . After predicting goals for a given instruction, Text2Motion iterates the process: i) invoke shooting to plan full skill sequences, and if no goal-reaching plan is found, ii) take a greedy-search step and check if executing the selected “best” skill would reach the goal. Note that the entire planning process occurs before execution. See Fig. 5.2 for a visualization of the shooting and greedy-search planners. © 2023 Springer Nature.

Figure 5.4 : TableEnv manipulation evaluation task suite . We evaluate the performance of all methods on tasks based on the above manipulation domain. The tasks considered vary in terms of difficulty and each task contains a subset of three properties: being long horizon (Tasks 1, 2, 3, 5, 6), containing lifted goals (Tasks 4, 5, 6), and having partial affordance perception (Tasks 4, 5, 6). During evaluation, we randomize the geometric parameters of each task. © 2023 Springer Nature.

Figure 5.5 : Results on the TableEnv manipulation domain with 10 random seeds for each task. Top: Our method (Text2Motion) significantly outperforms all baselines on tasks involving partial affordance perception (Task 4, 5, 6). For tasks without partial affordance perception, the methods that use geometric feasibility planning (Text2Motion, shooting , greedy-search ) convincingly outperform the methods ( saycan-gs and innermono-gs ) that do not. We note that shooting performs well on the tasks without partial affordance perception as it has the advantage of outputting multiple goal-reaching candidate plans and selecting the one with the highest execution success probability. Bottom: Methods without geometric feasibility planning tend to have high sub-goal completion rates but very low success rates. This divergence arises because it is possible to make progress on tasks without resolving geometric dependencies in the earlier timesteps; however, failure to account for geometric dependencies results in failure of the overall task. © 2023 Springer Nature.

Figure 5.6 : Failure modes of language-based planners on two categories of tasks . In this plot, we analyse the various types of failure modes that occur with Text2Motion, shooting and greedy-search when evaluated on tasks with partial affordance perception (PAP; see Section 5.5.4 for an explanation) and tasks without partial affordance perception (non-PAP). Top : For the PAP tasks, shooting incurs many planning failures because the space of possible plans is large but only few can be feasibly executed. In contrast, greedy-search uses value functions during search to narrow down the space of plans to those that are feasible. Text2Motion relies on greedy-search as a fallback if shooting fails, and thus can also contend with PAP tasks. Bottom : For the non-PAP tasks, shooting outperforms greedy-search . We attribute this difference to shooting ’s ability to output multiple task plans while greedy-search can only output a single plan. Finally, Text2Motion matches the performance of shooting as it also outputs and selects among multiple task plans. © 2023 Springer Nature.

Hybrid planning breakdown	Task 4	Task 5	Task 6
% shooting only	14%	0%	0%
% greedy-search only	0%	0%	0%
% Combination	86%	100%	100%
Avg. greedy-search Steps	1.0	2.6	3.0
Avg. Plan Length	5.0	7.0	7.0

Table 5.1 : Ablation on hybrid planning method. We analyze the usage percentages of both shooting and greedy-search in successful plans found by our hybrid planner (see Fig. 5.3 ). We find that, as tasks increase in difficulty (Task 4, 5, 6), the majority of solutions involve a combination of both planners. This result indicates that shooting-based and search-based planning strategies play complementing roles in the success of Text2Motion. © 2023 Springer Nature.

Figure 5.7 : Ablation on termination method: goal proposition prediction vs stop scoring . We compare the performance of greedy-search using two different plan termination methods: using the LLM to predict goals a priori (ours) and scoring a stop skill [ 6 ] during search. We present results averaged across all six tasks and ten seeds for each variation (120 experiments in total). We find that terminating planning when LLM-predicted goals are satisfied results in a 10% boost in success rate over stop scoring. © 2023 Springer Nature.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Evaluation Domains. We evaluate our failure detection framework across three simulation domains and one real-world domain. These domains provide coverage over different data distributions (e.g., action multimodality, high-dimensional actions) and modes of generative policy failure. For example, generative policies tend to fail erratically in the Close Box domain, but smoothly in the Cover Object domain. An effective failure detector should be performant across multiple domains, which entails coverage over heterogeneous failure modes.

Figure A.2 : Performance variation of STAC subject to different policy prediction and execution horizons in PushT .

Figure A.3 : Performance variation of STAC based on the choice of statistical distance function in PushT .

Figure B.1 : RoboMimic state ablation: Data quality trends under varying number of rollouts. Performance influences ( Eq. 3.3 ) converge around $m\in[25,50]$ rollouts for “Lift MH” and “Square MH” (yielding similar quality trends), but continue to evolve until $m\in[50,100]$ rollouts for “Transport MH.” Curation performed on state-based diffusion policies. Results are averaged over 3 random seeds. Errors bars represent the standard error.

Figure B.2 : RoboMimic image ablation: Data quality trends under varying number of rollouts. Performance influences ( Eq. 3.3 ) converge around $m\in[25,50]$ rollouts for “Lift MH” and “Square MH” (yielding similar quality trends), but continue to evolve until $m\in[50,100]$ rollouts for “Transport MH.” Curation performed on image-based diffusion policies. Results are averaged over 3 random seeds. Errors bars represent the standard error.

Figure B.3 : RoboMimic state data quality results. Curation performed on state-based diffusion policies. Results are averaged over 3 random seeds. Errors bars represent the standard error.

Figure B.4 : RoboMimic image data quality results. Curation performed on image-based diffusion policies. Results are averaged over 3 random seeds. Errors bars represent the standard error.

Panel of Figure 35 — Figure B.5 : Franka diffusion policy – distribution of demonstrations filtered ( $S^{\star}$ in Task 1 ). See Fig. 3.5 for distributions of the corresponding curated datasets used for policy training.

Panel of Figure 36 — Figure B.6 : Franka diffusion policy curated dataset distributions for selection ( Task 2 ). Cupid selects higher-quality demonstrations (Figure-8) and robust strategies (TuckBox), improving policy performance across tasks (see Fig. 3.4 ). While curation heuristics employed by baselines may be effective in some cases (e.g., Cupid-Quality in Figure-8), they can lead to suboptimal selection in others.

Panel of Figure 37 — Figure B.7 : Franka diffusion policy – distribution of demonstrations selected ( $S^{\star}$ in Task 2 ). See Fig. B.6 for distributions of the corresponding curated datasets used for policy training.

Figure B.8 : Data curated for single-task diffusion policies improves

\pi_{0}

[ 25 ] post-training performance. As in Fig. 3.4 , quality measures ( Cupid-Quality ) may degrade performance when higher-quality demonstrations induce brittle strategies at test time (TuckBox), whereas curating based on performance ( Cupid ) is robust across settings.

Task ID	Properties	Instruction
Task 1	LH	How would you pick and place all of the boxes onto the rack?”
Task 2	LH + LG	How would you pick and place the yellow box and blue box onto the table,
		then use the hook to push the cyan box under the rack?”
Task 3	LH + PAP	How would you move three of the boxes to the rack?”
Task 4	LG + PAP	How would you put one box on the rack?”
Task 5	LH + LG + PAP	How would you get two boxes onto the rack?”

Appendix

Robotics

Robotics studies the design and control of machines that perceive, move, and act in physical environments.

See all 747 papers

Mar 6, 2026

Learning-Based Robust Control: Unifying Exploration and Distributional Robustness for Reliable Robotics via Free Energy

A key challenge towards reliable robotic control is devising computational models that can both learn policies and guarantee robustness when deployed in the field. Inspired by the free energy principle in computational neuroscience, to address these challenges, we propose a model for policy computation that jointly learns environment dynamics and rewards, while ensuring robustness to epistemic uncertainties. Expounding a distributionally robust free energy principle, we propose a modification to the maximum diffusion learning framework. After explicitly characterizing robustness of our policies to epistemic uncertainties in both environment and reward, we validate their effectiveness on continuous-control benchmarks, via both simulations and real-world experiments involving manipulation with a Franka Research~3 arm. Across simulation and zero-shot deployment, our approach narrows the sim-to-real gap, and enables repeatable tabletop manipulation without task-specific fine-tuning.

Hozefa Jesawada, Giovanni Russo, Abdalla Swikir +1

Mechanical Engineering Program, New York University Abu Dhabi, Abu Dhabi, UAE. · Department of Information and Electrical Engineering & Applied Mathematics, University of Salerno, Fisciano, Italy. · Mohammed bin Zayed University for Artificial Intelligence, Abu Dhabi, UAE.

Robust Control Distributionally Robust RL Free Energy Principle Model-Based RL Diffusion Models Robotic Manipulation

Ask PDF

Oct 14, 2026

Back on Track: Deployment-time Policy Improvement via World Model Planning

State-of-the-art visuomotor policies are increasingly capable of executing complex tasks, yet can fail during deployment in ways they cannot predict or recover from on their own. Improving their performance and robustness typically requires collecting more data, retraining, or modifying the policy architecture, all of which are time- and resource-intensive in practice. We instead propose an offline-trained monitoring-and-intervention framework that improves a policy during deployment using only black-box query access. From a limited set of labeled on-policy rollouts, we learn a policy-conditioned liveness value that measures the nominal policy's progress toward task completion; we additionally train a compact latent world model using these rollouts together with play data. At deployment, the liveness value continuously monitors execution. When it detects a loss of progress, a short-horizon planner searches the world model for corrective actions using the same value as its planning objective, after which control returns to the nominal policy. We evaluate our method on the LIBERO-Goal task suite and on a Franka Panda manipulator, using the vision-language-action model

π_{0.5}

and a diffusion policy as nominal policies, and demonstrate significant improvements in task success rates, while supporting real-time control. Additional results and videos are available on the project website (https://policy-improvement-wm.pages.dev/).

Colton Crosby, Hao Wang, Mac Schwager +1

The authors are associated with the Aeronautics and Astronautics Department, Stanford University. · The author is associated with the Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California.

Robot Policy Adaptation World Model-Based Planning

Ask PDF

Sep 21, 2026

Learning Beyond What Humans Can Demonstrate

Behavior cloning for robot manipulation relies on expert demonstrations. However, for tasks that require dynamic stability, precise contact timing, or dexterous coordination, human operators may find it hard or even impossible to collect data. We study this infeasible-demonstration regime and propose GLIDE: Guardrails for Learning from Infeasible Demonstrations Efficiently, a framework that infers task-specific failure modes and converts them into executable guardrails for data collection and policy deployment. Given a task description and the conditioning teleoperation code, GLIDE writes guardrails that use system states to filter teleoperation and policy commands, constrain failure-prone actions, and iteratively improve from trajectory feedback. Across three tasks, GLIDE discovers emergent guardrails that go beyond domain-expert hardcoded ones, improving data collection over naive VR teleoperation and domain-expert hardcoded guardrails. After refinement, GLIDE raises data-collection success from 0-10 percent to 70-90 percent across the three tasks. During policy execution, mixed-data guarded policies reach 70 percent, 60 percent, and 60 percent success on Tomato plate transfer, Marker handover and stand, and Wine serving tasks. These results show that GLIDE can support policy learning when direct demonstrations are infeasible. Project website: http://guardrail-policy.github.io/

Yuchen Song, Aditya Mittal, Unnat Jain

University of California, Irvine

Robotic Data Collection Robotic Manipulation Learning from Demonstration Robot Imitation Learning Robot Learning

Ask PDF

Abstract

Robotics

Explore similar work

Learning-Based Robust Control: Unifying Exploration and Distributional Robustness for Reliable Robotics via Free Energy

Back on Track: Deployment-time Policy Improvement via World Model Planning

Learning Beyond What Humans Can Demonstrate