Long-term robot autonomy depends on more than effective action generation: a policy must recognize when its behavior is no longer sufficiently supported and use assistance to improve. We introduce Equilibrium Energy Control (EqC), a framework for introspective lifelong robot learning built around an explicit conditional energy landscape. EqC casts generative action refinement as equilibrium-seeking optimization, while the geometry of the refinement process reveals whether the generated behavior remains consistent with the learned landscape. This competence signal enables Energy-Gated DAgger, where persistent energy-consistency violations trigger expert takeover and the resulting recovery trajectories are aggregated for subsequent policy updates. EqC integrates naturally into both Diffusion Transformer policies and flow-matching-based vision-language-action (VLA) policies. Across simulation and real-world bimanual manipulation, EqC matches or improves control performance and robustness while enabling timely intervention near the policy's competence boundary. A real-world study further shows how these interventions progressively expand the range of situations handled autonomously, providing a practical path from static generative controllers toward self-improving robot systems.
Figures & tables
Table 1: Success rates (%) on LIBERO with the DiT policy and VLABench with VLA policies.
Table 2: VLA policy success rates (%) on RoboTwin 2.0 (Easy / Hard) and LIBERO-Plus.
Table 3: Control and competence-awareness evaluation. (a) Success and latency on three real-world bimanual tasks (20 trials each). (b) Failure detection on pooled LIBERO and LIBERO-Plus rollouts. “EqC+Diff-DAgger” denotes the Diff-DAgger flow-loss detector applied to the EqC backbone; EqC uses path-wise energy consistency.
Figure 3: Real-world failure detection (mean ±1std , n=25 ). (a) EqC cleanly separates success and failure early for timely intervention, whereas (b) Diff-DAgger exhibits heavy overlap.
Figure 4: Real-world lifelong learning evaluation with Energy-Gated DAgger. Top: Quantitative success curves across deployment rounds (averaged over 3 independent runs, with 20 autonomous evaluation trials per checkpoint per run). Bottom: Qualitative rollouts: early drift triggers UΔE alarms (red banners) for expert recovery; fine-tuned policies achieve autonomous execution (Rows 3–4). Across both (a) Hierarchical Block Stacking and (b) Garment Folding , EqC achieves 96.7% and 98.3% mean success rates, respectively, decisively outperforming Diff-DAgger and human intervention.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Coverage
Episodes
LIBERO–DiT
4 suites; 40 tasks
50/task
LIBERO–VLA
4 suites; 40 tasks
50/task
RoboTwin 2.0
5 tasks; Easy and Hard
100/task/setting
VLABench
Tracks 1, 2, 3, 4, and 6; 50 tasks
50/task
LIBERO-Plus
7 perturbation families
10030 total
Appendix
Table 4: Evaluation coverage and rollout budgets.
Seed
Goal
Spatial
Object
Long
Avg.
0
95.6
97.2
98.6
85.4
94.3
1
96.0
97.0
98.4
87.0
94.7
2
95.0
96.0
97.6
85.4
93.5
3
95.8
95.2
97.8
84.2
93.3
Mean ± sample s.d.
95.60±0.43
96.35±0.93
98.10±0.48
85.50±1.15
93.95±0.66
Appendix
Table 5: Detailed EqC success rates (%) across multiple training seeds on LIBERO.
Method
Mean (%)
Sample s.d. (%)
Fixed RF, 20 steps
89.12
0.80
GeCO
92.30
1.02
EqC (ours)
93.95
0.66
Appendix
Table 6: Overall LIBERO success across multiple training seeds.
Method
Goal
Spatial
Object
Long
Avg.
π0 + FAST ( Pertsch et al., 2025 )
88.6
96.4
96.8
60.2
85.5
π0
95.8
96.8
98.8
85.2
94.2
π0 + GeCO
96.4
96.8
97.0
85.4
93.9
π0 + EqC (ours)
94.6
97.6
98.8
86.2
94.3
Appendix
Table 7: LIBERO success rates (%) with the VLA backbone.
Table 8: Detailed composition of the shared 10% calibration split ( 1,203 total rollouts).
Table 9: Sensitivity analysis of detection window size m under two distinct threshold calibration strategies on the 90% test split.
FPR-Constrained Calibration
Youden’s J Index Calibration
m
Diff-DAgger
EqC + Diff
EqC (ours)
Diff-DAgger
EqC + Diff
EqC (ours)
3
33.6
44.5
45.2
47.9
51.8
53.7
4
31.8
42.5
45.3
39.2
53.8
57.4
5
32.1
42.7
44.5
49.4
51.8
55.5
6
30.6
42.6
46.7
45.2
52.2
51.5
7
32.0
42.8
46.6
47.0
50.2
57.1
Appendix
Table 10: Evaluation of early intervention capability (Time Saved %) on the 90% test split. The operational decision thresholds are calibrated using the 10% calibration split under FPR-Constrained and Youden’s J index strategies.
Method
Orig.
Camera
Robot
Lang.
Light
Backgr.
Noise
Layout
W. Avg.
Diff-DAgger ( m=4 )
27.7
50.5
50.1
46.4
30.0
34.0
44.1
47.9
46.1
Diff-DAgger ( m=11 )
26.5
44.2
46.0
39.9
32.5
32.3
37.3
44.9
41.5
Diff-DAgger ( m=13 )
28.9
47.2
48.6
44.1
33.8
34.5
43.1
46.1
44.6
Diff-DAgger ( M=16,m=4 )
27.0
45.0
47.5
44.1
33.8
28.0
39.0
44.0
42.7
Diff-DAgger ( M=16,m=11 )
23.5
41.3
42.4
37.4
28.5
30.2
35.1
40.6
38.4
Diff-DAgger ( M=16,m=13 )
25.9
42.8
43.5
39.0
30.7
30.5
37.1
41.8
39.8
Appendix
Table 12: Breakdown of theoretically optimal inference time saved (%) across LIBERO and LIBERO-Plus suites under representative evaluation setups. Orig. corresponds to the LIBERO tasks, while the remaining seven columns represent environmental perturbation variants in LIBERO-Plus. Diff-DAgger uses M=10 unless otherwise specified.
Method
Mean Latency (ms)
Diff-DAgger ( M=10 )
79.44
Diff-DAgger ( M=16 )
91.43
Diff-DAgger ( M=32 )
122.11
EqC + Diff-DAgger ( M=10 )
171.03
EqC (ours)
108.86
Appendix
Table 13: Comparison of per-replan inference latency (ms) across methods. The latency includes both model action generation and failure-detection signal computation (evaluated at batch size = 1 after JAX warm-up). The optimization step budget K for EqC is set to 10.
Table 14: Ablation analysis comparing single-point energy signals against full path-wise EqC across window sizes m under two calibration strategies on the 90% test split.
Hyperparameter
Value
Base VLA Backbone
Galaxea G0.5
Optimizer
AdamW
Batch Size
16
Peak Learning Rate
8×10−5
Warmup Schedule
Linear warmup (500 steps)
Decay Schedule
Cosine decay ( warmup_constant_cosine )
Appendix
Table 15: Unified training and inference hyperparameters for real-robot experiments.
Method
Time (min)
Overhead
Flower Insertion
Galaxea G0.5
11.58
—
Galaxea G0.5 + EqC (ours)
14.58
+25.9%
Towel Folding
Galaxea G0.5
40.48
—
Galaxea G0.5 + EqC (ours)
49.42
+22.1%
Appendix
Table 16: Wall-clock fine-tuning time across real-robot tasks.
Method
Run
R0 (18)
R1 (24)
R2 (30)
R3 (36)
R4 (42)
R5 (48)
Human Intervention
Run 1
0.0
0.0
5.0
15.0
20.0
50.0
Run 2
0.0
0.0
10.0
35.0
60.0
70.0
Run 3
0.0
0.0
0.0
20.0
45.0
55.0
Mean ± Std
0.0±0.0
0.0±0.0
5.0±5.0
23.3±10.4
41.7±20.2
58.3±10.4
Galaxea G0.5 + Diff-DAgger
Run 1
0.0
0.0
20.0
60.0
70.0
90.0
Run 2
0.0
0.0
10.0
80.0
70.0
70.0
Appendix
Table 17: Detailed progressive success rates (%) across 3 independent lifelong learning runs on the physical Galaxea bimanual workstation (20 autonomous trials per checkpoint per run). Both individual run outcomes and summary statistics ( Mean±Std ) are reported across deployment rounds.
Figure 5: Cross-task adaptation on Hierarchical Cup Stacking . Top (Round 0): Zero-shot deployment causes grasp slippage and drift, promptly caught by a persistent UΔE failure alarm. Bottom (Round 2): After two rounds of Energy-Gated DAgger, the policy achieves fully autonomous execution, cleanly assembling the three-tier cup tower.
Metric / Task
Round 0 (Base)
Round 1 (+6 Cup)
Round 2 (+6 Cup, +6 Block)
Cumulative Demos
48 (Block only)
54 (48 Block, 6 Cup)
66 (54 Block, 12 Cup)
Block Stacking Success
100.0
60.0
100.0
Cup Stacking Success
0.0
50.0
100.0
Dual-Task Average
50.0
55.0
100.0
Appendix
Table 18: Progressive performance (%) across continual learning rounds under task shift.
Methods
Goal
Spatial
Object
Long
Average
EqC (uniform 15 steps)
95.8
95.2
98.4
83.0
93.1
EqC (uniform 20 steps)
95.4
96.2
98.0
87.2
94.2
EqC (Cosine decay)
95.6
97.2
98.6
85.4
94.3
Appendix
Table 19: Success rate (%) of EqC under different optimization step schedules across the LIBERO suites. The highest values in each column are highlighted in bold.
Step k
0
5
6
7
8
9
14
19
32
Mean Gradient Norm ∥∇E∥
49.3
44.4
29.9
12.9
6.0
3.6
1.2
0.7
0.3
E Saturation Ratio (%)
0.00
82.51
95.18
99.22
99.79
99.91
99.99
–
–
Appendix
Table 20: Convergence profile of mean gradient norm ∥∇E∥ and normalized energy saturation ratio across optimization steps k on LIBERO.
Task Suite
Baseline ( K=32 )
gmin=1.0
gmin=3.0
libero_goal
95.6 / 32.0
97.0 / 16.9
97.0 / 11.1
libero_spatial
96.4 / 32.0
95.2 / 17.3
98.0 / 11.3
libero_object
98.1 / 32.0
98.0 / 16.2
97.6 / 10.9
libero_long
85.5 / 32.0
84.6 / 16.8
84.2 / 10.9
Average
94.0 / 32.0
93.7 / 16.8
94.2 / 11.1
Appendix
Table 21: Performance and inference efficiency under adaptive early stopping across LIBERO task suites. Metrics are reported as Success Rate (%) / Average Optimization Steps per action chunk.
Figure 6: Proactive competence-boundary detection of EqC across rigid and deformable manipulation. Persistent UΔE violations trigger timely expert recovery at the earliest onset of execution drift, preserving an ample window for teleoperation before irreversible physical compounding occurs. Top row (Hierarchical Block Stacking): Left: Alarm raised when the block is accidentally pushed away, preventing it from being pushed further; Right: Alarm triggered by angular misalignment. Bottom row (Garment Folding): Left: Alarm triggered when the robot attempts to grasp the right sleeve but fails to secure it; Right: Alarm triggered upon failure to fold the garment to the designated position.
Figure 7: Qualitative failure modes of Galaxea G0.5 + Diff-DAgger across rigid and deformable manipulation. Rows 1–2 evaluate Hierarchical Block Stacking under nominal setups (two independent instances per category), while Row 3 evaluates Garment Folding . Left column (Delayed Detection): The detector remains silent throughout compounding drift, alarming only belatedly after the block arrangement is disrupted or a severe angular tilt has already occurred, missing the teleoperation recovery window. Middle column (Spurious Alarm): During nominal execution—including pristine, fully unfolded garment setups (Row 3)—noisy flow-loss expectation triggers false-positive alarms, needlessly aborting valid rollouts. Right column (Irreversible Failure): The detector triggers only after irrecoverable failures have occurred (e.g., mis-sequenced block placement or fabric bunched into an entangled knot), leaving no viable recovery path without a complete manual scene reset.
A key challenge towards reliable robotic control is devising computational models that can both learn policies and guarantee robustness when deployed in the field. Inspired by the free energy principle in computational neuroscience, to address these challenges, we propose a model for policy computation that jointly learns environment dynamics and rewards, while ensuring robustness to epistemic uncertainties. Expounding a distributionally robust free energy principle, we propose a modification to the maximum diffusion learning framework. After explicitly characterizing robustness of our policies to epistemic uncertainties in both environment and reward, we validate their effectiveness on continuous-control benchmarks, via both simulations and real-world experiments involving manipulation with a Franka Research~3 arm. Across simulation and zero-shot deployment, our approach narrows the sim-to-real gap, and enables repeatable tabletop manipulation without task-specific fine-tuning.
Hozefa Jesawada, Giovanni Russo, Abdalla Swikir +1
Mechanical Engineering Program, New York University Abu Dhabi, Abu Dhabi, UAE. · Department of Information and Electrical Engineering & Applied Mathematics, University of Salerno, Fisciano, Italy. · Mohammed bin Zayed University for Artificial Intelligence, Abu Dhabi, UAE.
Despite rapid progress in Vision-Language-Action (VLA) models for robotic control, instruction drift remains a persistent failure mode in long-horizon tasks. This paper reconceptualizes this phenomenon, positing that instruction drift is fundamentally a systematic sampling error: local greedy sampling is prone to collapsing into "Negative Pivotal Windows"--irreversible local optima with high local probability that sever global success pathways. To address this, we propose Context-Aware Power Sampling (CAPS), a training-free inference-time computation framework. CAPS leverages power distributions to sharpen global trajectory probabilities, enabling lookahead search over the model's conditional generative trajectory distribution. Furthermore, we introduce a metacognitive control mechanism based on Signal-to-Noise Ratio (SNR). This mechanism triggers adaptive MCMC search solely when drift risk is detected, enabling a dynamic transition from "intuitive fast thinking" to "rational slow search." Experiments on RoboTwin, Simpler-WindowX, and Libero-long benchmarks show that CAPS achieves substantial improvements over strong baselines, including OpenVLA and TACO, without parameter updates. These results support the effectiveness of adaptive inference-time computation for improving long-horizon robustness in embodied control.
Kewei Chen, Yayu Long, Mingsheng Shang
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences · Chongqing School, University of Chinese Academy of Sciences
Currently, Vision-Language-Action (VLA) models have become the most adopted paradigm for robotic manipulation for its great potential for task generalization. While most generative flow-matching action decoders for VLA control are often deployed with fixed sampling horizons, limiting state-dependent compute and temporal reuse across control cycles. We present π0-EqM, which replaces the flow-matching expert in π0 with an Equilibrium Matching (EqM) decoder while leaving the upstream VLA stack unchanged. Under a matched 300-step budget, π0-EqM improves RoboTwin average success from 40.4% to 50.2% across 19 tasks and remains competitive on LIBERO, with its clearest gain on LIBERO-10 (87.0%). Two threshold scans reveal a task-dependent non-monotonic relation between residual and success, which we term the stationarity--executability gap. The results suggest that inference depth in iterative VLA control is part of policy design and introduce an energy-based VLA perspective that may inform future work on composable action generation across tasks and embodiments.
Huanming Liu, Congsheng Xu, Jianmin Ji +1
University of Science and Technology of China, China · Shanghai Jiao Tong University, China