This paper presents Proactive Human-in-the-Loop (PHIL), a framework for proactive policy correction and preference-based post-training. PHIL enables operators to preview policy-predicted future motion and intervene before anticipated undesirable actions. During takeover, PHIL continues querying the policy and displaying its predicted trajectories in the physical workspace through an extended reality (XR) interface. PHIL pairs rejected policy predictions with corrective action chunks subsequently executed from the same physical state over the same horizon, and preference-based post-training encourages the policy to favor these corrections. This paper evaluates PHIL on four real-world manipulation tasks using a diffusion policy and a pretrained vision-language-action policy. With five correction episodes per condition, PHIL increases mean success rates across the evaluated tasks and demonstration budgets from 44.0% to 70.3% for the diffusion policy and from 57.8% to 76.9% for X-VLA, compared with HG-DAgger. Separately, policies trained with proactive corrections show fewer unnecessary recovery behaviors in successful episodes. Project page: https://intuitive-robots.github.io/PHIL-page/.
Figures & tables
Fig. 2: Reactive versus proactive correction under equal correction budgets (two interventions per panel; only the first annotated). Solid and dashed lines denote executed robot motion and previewed policy continuations, respectively. Top: in these examples, reactive intervention follows observed errors and requires recovery. Bottom: XR previews enable intervention before anticipated errors. During correction, rejected predictions A− from scheduled queries are paired with subsequent executed corrective trajectories A+ from the same query state. Takeover and release leave the query schedule unchanged; control returns to the policy when the preview is acceptable.
Fig. 3: Preference pairs from fixed B -step queries during human intervention. Each retained rejected H -step prediction At− pairs with the next H executed actions At+ , yielding At+≻At− . Here, takeover coincides with a query and positive windows fall entirely within the human intervention period; generally, windows may include policy actions after release. Windows overlap when B<H . Takeover and release do not reset queries. Only complete pairs are retained; the latest prediction displayed at release is accepted and excluded from negatives.
Fig. 4: Example of proactive human intervention with PHIL during the Cook the Carrot task. Colored spheres visualize the policy-predicted future end-effector trajectories in XR; green and red frames indicate policy control and human intervention, respectively. (1–2) The policy approaches the pot and moves it onto the stove. (3–4) The operator intervenes when the previewed motion fails to approach the carrot or moves toward the pot before grasping the carrot, providing corrections before these undesirable motions are executed. Policy predictions remain visible during human intervention, allowing the operator to assess when to return control. (5) Once the predicted motion is acceptable, policy control resumes to place the grasped carrot into the pot.
Fig. 5: Diffusion Policy performance across base-demonstration budgets. Panels (a–d) show full-task success rates for Pick Up Banana, Towel Folding, Cook the Carrot, and Lemon in the Drawer, respectively; panels (e, f) show normalized task completion scores for Cook the Carrot and Lemon in the Drawer. Each measured point summarizes 25 evaluation rollouts. The points at (0,0) represent an assumed zero-demonstration baseline and were not evaluated experimentally. HG-DAgger and Residual Policy use reactive corrections, whereas PHIL uses proactive corrections; all three train with five correction episodes and the original base demonstrations.
Fig. 6: Correction-collection and post-training ablations with Diffusion Policy. Panels (a–d) show success rates for Pick Up Banana, Towel Folding, Cook the Carrot, and Lemon in the Drawer, respectively. Base policies use three demonstrations for Pick Up Banana and ten for each remaining task. Each bar summarizes 25 evaluation rollouts after post-training with five correction episodes and the original base demonstrations. R, H, and P denote Reactive, Hybrid, and Proactive collection; Hybrid combines two proactive and three reactive episodes. Methods within each collection mode share the same recorded correction data. Dashed horizontal lines indicate Base Policy success rates.
Fig. 7: Unnecessary recovery in Diffusion Policies post-trained with HG-DAgger or PHIL. Panels (a–d): Pick Up Banana, Towel Folding, Cook the Carrot, and Lemon in the Drawer. Each point averages recovery counts over ten successful episodes per method, collection mode, and base-demonstration budget, excluding recovery required after actual errors. The x-axis shows correction frames as a percentage of base-demonstration plus correction frames, decreasing rightward. Orange circles/blue diamonds denote HG-DAgger/PHIL; filled/open markers denote reactive/proactive collection. Dashed/dotted lines show least-squares linear fits across three budgets per method for reactive/proactive collection, respectively.
Task
Method
Base demonstrations
10
20
30
SR
TCS
SR
TCS
SR
TCS
Towel Folding
Base Policy
32
–
36
–
60
–
HG-DAgger
32
–
40
–
64
–
PHIL
64
–
72
–
72
–
Cook the Carrot
Base Policy
40
73.3
56
77.3
56
85.3
TABLE I: X-VLA results with different numbers of base demonstrations. HG-DAgger and PHIL use corrections collected over five interactive rollout episodes.
Task
Stages
Diffusion Policy demos
X-VLA demos
Metrics
Rollouts
Pick Up Banana
1
3/6/9
—
SR
25
Towel Folding
1
10/20/30
10/20/30
SR
25
Cook the Carrot
3
10/20/30
10/20/30
SR, TCS
25
Lemon in the Drawer
3
10/20/30
10/20/30
SR, TCS
25
TABLE II: Task, demonstration-count, and evaluation summary. Demo counts refer to base-policy demonstrations.
Method
Success Rate (SR)
Task Completion Score (TCS)
Banana
Towel
Carrot
Lemon
Carrot
Lemon
3
6
9
10
20
30
10
20
30
10
20
30
10
20
30
10
20
30
Base Policy
36%
48%
84%
24%
40%
52%
4%
24%
44%
0%
0%
4%
38.7%
65.3%
80.0%
0.0%
0.0%
6.7%
HG-DAgger
56%
76%
76%
48%
52%
64%
24%
28%
40%
20%
20%
24%
57.3%
64.0%
77.3%
40.0%
44.0%
52.0%
Residual Policy
56%
76%
80%
48%
64%
64%
24%
28%
72%
12%
40%
48%
56.0%
62.7%
88.0%
41.3%
62.7%
65.3%
PHIL
56%
76%
96%
68%
72%
72%
52%
80%
80%
40%
68%
84%
84.0%
93.3%
93.3%
60.0%
78.7%
94.7%
TABLE III: Main diffusion-policy comparison with different numbers of base demonstrations. Pick Up Banana uses 3/6/9 base demonstrations; all other tasks use 10/20/30. Long-horizon tasks report both full-task success rate (SR) and normalized task completion score (TCS).
Objective
Collection
Banana
Towel
Carrot
Lemon
Base Policy
—
36%
24%
4%
0%
HG-DAgger
Reactive
56%
48%
24%
20%
HG-DAgger
Proactive
56%
40%
28%
36%
PHIL
Reactive
48%
36%
24%
16%
PHIL
Proactive
56%
68%
52%
40%
HG-DAgger
Hybrid
60%
52%
36%
60%
TABLE IV: Controlled diffusion-policy ablation in the lowest-data regime. Each comparison changes the correction-collection mode or the post-training objective while holding the remaining protocol fixed.
HG-DAgger / collection
Pick Up Banana
Towel Folding
Cook the Carrot
Lemon in the Drawer
3
6
9
10
20
30
10
20
30
10
20
30
Evaluation recovery count
Reactive
21
17
9
24
19
15
23
20
14
24
18
10
Proactive
0
0
1
0
0
1
2
4
4
0
4
5
Training correction-frame ratio (%)
Reactive
23.40
13.66
11.19
40.44
23.39
16.43
32.82
18.00
14.20
23.23
12.72
5.98
TABLE V: Unnecessary recovery behaviors in a separate evaluation of diffusion policies post-trained with HG-DAgger, and the corresponding training-data composition. The comparison covers four tasks and three base-demonstration budgets, with ten successful episodes per condition.
Human-in-the-loop reinforcement learning (HIL-RL) improves sample efficiency in real-robot manipulation through online human intervention. However, successful trajectories may include suboptimal actions that deviate from the desired task-execution path and force human intervention. Existing HIL-RL methods typically apply the consistent credit assignment principle to all transitions, uniformly propagating discounted terminal rewards through suboptimal segments, ignoring the actual contribution of each transition to task success. This overestimates Q-values for critic learning and indirectly misguides actor updates toward suboptimal behavior patterns. To this end, we propose PACT, a Preference-calibrated Actor-Critic Training framework that leverages the implicit preference signals induced by intervention to perform credit reassignment on identified suboptimal segments while directly guiding policy training for unbiased critic-actor learning. Specifically, we first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction. Then, from the human action and resampled policy action at the intervention state, we build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment, enabling directional credit calibration. Moreover, we directly align the policy with human corrective actions in the bounded mean space, providing an additional signal beyond critic-guided updates. Across five real-robot manipulation tasks, PACT improves the average success rate by 24.5% and achieves 1.3 times faster convergence, thereby improving both RL sample efficiency and performance. Code is available at https://anonymous.4open.science/r/HILRL-A1X-BC05.
Zeyi Liu, Guangyao Liu, Yinuo Qu +6
Central South University · Nanyang Technological University · Zhejiang University
Human-in-the-loop (HIL) post-training improves vision-language-action (VLA) policies through expert corrections, but costly human supervision and repeated real-robot evaluation limit algorithm exploration and development. We introduce SimPT (Simulation-based Post-Training), a simulation framework that automates intervention-based VLA post-training by integrating policy rollouts, corrective data collection, policy training, and standalone evaluation. To automatically determine intervention timing and recovery strategies, we develop an Agent-Compiled Progress Graph that tracks task progress using privileged simulator state and retains stable snapshots during policy execution. After a rollout fails, a CuRobo-based expert attempts recovery from these snapshots, starting at more advanced task stages and falling back to earlier ones as needed to generate corrective trajectories. We implement and compare four post-training methods, RECAP, HG-DAgger, DPO+SFT, and FlowPRO, across seven RoboTwin manipulation tasks. Relative to a fixed-interval snapshot baseline, the Progress Graph reduces recovery attempts by 75.0% and increases recovery success from 93.4% to 98.9%. Using SimPT, we develop RECAP-CF-LPF, a refinement of RECAP that combines cross-fitting and low-pass filtering of value predictions. This refinement increases the overall success rate from 76.50% to 79.36% in held-out simulation trials. In real-robot HIL post-training, it increases the success rate from 72% to 96% across 25 scenarios of an Ethernet cable-insertion task. These results demonstrate the utility of SimPT for exploring and developing VLA post-training algorithms.
Vision-Language-Action (VLA) foundation models have recently achieved strong progress in embodied intelligence. To reduce policy-call frequency while preserving temporal coherence, most generative policies adopt an action chunk mechanism, executing multiple future actions in an open-loop manner under a fixed action horizon. However, this "predict-then-blindly-execute" paradigm sacrifices closed-loop reactivity: in contact-rich physical interactions, even small local perturbations can rapidly amplify within the open-loop blind spot, leading to compounding errors and ultimately task failure. To address this limitation, we propose VLA-Corrector, a lightweight corrective inference framework for action-chunked VLA policies. Without modifying the backbone policy weights, VLA-Corrector introduces a lightweight Latent-space Vision Monitor (LVM) that continuously compares predicted and actual visual feature evolution, enabling online detection of visual dynamics deviations. Once persistent deviation is detected, the system triggers a truncation event, discards the remaining stale actions, and invokes corrective replanning via Online Gradient Guidance (OGG). The detect-and-correct mechanism of VLA-Corrector naturally induces an event-triggered adaptive action horizon: it preserves long-horizon execution when the current chunk remains reliable, and invokes short-horizon corrective replanning when execution begins to drift. In doing so, VLA-Corrector mitigates the trade-off imposed by static horizons between execution robustness and policy-call frequency. It can be integrated into different VLA models without further retraining the VLA backbone, interrupting compounding errors while preserving much of the efficiency benefit of action chunking and substantially improving robustness in long-horizon, contact-rich robotic manipulation tasks.