BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models

Weihui Zhao; Xiaohan Yan; Zunian Wan; Xuan Du; Zhaozhan Chi; Jianbo Mao; Ruipu Wu; Rushuai Yang

cs.ROSep 23, 2026

BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models

Authors: Weihui Zhao, Xiaohan Yan, Zunian Wan, Xuan Du, Zhaozhan Chi, Jianbo Mao, Ruipu Wu, Rushuai Yang,

Organizations: South China University of Technology · AgiBot

Abstract

Vision-language-action (VLA) models handle long-horizon manipulation, yet success hinges on a few precision-critical phases where millimeter-scale errors undo all prior progress. Online reinforcement learning (RL) can optimize exactly these actions, but free exploration is far too costly on real robots, which makes human corrections indispensable. However, existing online RL methods for VLAs either cannot incorporate such corrections or fold them into undifferentiated supervision. Yet human corrections are not uniformly noisy but reliable along some action dimensions and variable along others. Building on this, we introduce BEE, an intervention-adaptive framework for real-world RL on a frozen VLA that lets the policy go BEyond Expert imitation. We formulate human corrections not as actions to reproduce but as evidence about a constraint: a Correction Model predicts how a human would correct a given VLA proposal and how consistent the correction is along each action dimension. This predicted consistency sets the per-dimension tightness of a constraint on policy optimization. Where corrections are consistent the policy stays close to the human, and where they vary, the constraint relaxes. We evaluate BEE on three real-world manipulation tasks and one LIBERO-Pro simulation task at a matched online-data budget. BEE attains the highest success rate on every task, 91.2% on average against 57.5% for RLT and 42.1% for DSRL, and the lowest human intervention rate on all real-world tasks.

Figures & tables

Method	Phone Charging		Snack Hanging		Cloth Aligning		Bowl Placing		Overall
Method	SR $\uparrow$	Int. $\downarrow$	SR $\uparrow$	Int. $\downarrow$	SR $\uparrow$	Int. $\downarrow$	SR $\uparrow$	Int. $\downarrow$	SR $\uparrow$	Int. $\downarrow$
Base Policy	90.0 $\pm$ 5.0	–	0.0 $\pm$ 0.0	–	36.7 $\pm$ 7.6	–	28.3 $\pm$ 7.6	–	38.8	–
RLT	58.3 $\pm$ 2.9 †	22.5	21.7 $\pm$ 5.8	62.1	73.3 $\pm$ 7.6	85.3	76.7 $\pm$ 17.6	26.2	57.5	49.0
DSRL	93.3 $\pm$ 2.9	–	13.3 $\pm$ 2.9	–	31.7 $\pm$ 2.9	–	30.0 $\pm$ 5.0	–	42.1	–
Bee	100.0 $\pm$ 0.0	12.2	85.0 $\pm$ 0.0	17.0	90.0 $\pm$ 10.0	65.3	90.0 $\pm$ 13.2	27.4	91.2	30.5

TABLE I : Comparison of Bee with the Base Policy, RLT, and DSRL . Success rate (SR, %,

\uparrow

) and human intervention rate (Int., %,

\downarrow

) on the real-world tasks phone charging, snack hanging, and cloth aligning, and on the LIBERO-PRO simulation task bowl placing. The Base Policy and DSRL involve no online intervention, so no intervention rate is reported.

Fig. 2 : Evaluation Tasks. Keyframes of the three real-world tasks (phone charging, snack hanging, cloth aligning) and of the LIBERO-PRO simulation task, bowl placing.

Fig. 3 : Sample Efficiency on Phone Charging and Snack Hanging . Success rate against the number of online robot episodes, with color distinguishing Bee from RLT and line style distinguishing the two tasks; the horizontal dashed lines mark the Base Policy. The right end of the axis is the matched online-data budget at which Table I is reported. Bee stays above RLT at every budget on both tasks.

Fig. 4 : Correction Model Calibration on Phone Charging , evaluated on human-intervention demonstrations over the seven joint dimensions

q_{1}

–

q_{7}

of the intervened arm. (a) The VLA proposal

\tilde{\mathbf{a}}_{t}

and the corrected action

\hat{\mathbf{a}}^{H}_{t}

produced by the Correction Model, plotted against the human correction. Points closer to the

y{=}x

line match it better. (b) Per-dimension root-mean-square error to the human correction. The corrected action reduces this error below the VLA proposal in every dimension. (c) Predicted per-dimension standard deviation of the Correction Model and empirical standard deviation of human corrections. The prediction follows the same per-dimension pattern as the empirical spread.

Fig. 5 : Dimension-wise Constraint Allocation on Phone Charging . Bars show each dimension’s constraint share (its normalized contribution to

\rho_{\phi}

), and the overlaid curve is the predicted uncertainty

\sigma

of the Correction Model. (a) The state-wise form weights all dimensions equally, so a high-uncertainty dimension such as

q_{5}

claims the largest share. (b) The Mahalanobis form concentrates the share on low-uncertainty, consistently corrected dimensions (e.g.,

q_{2}

) and downweights high-uncertainty ones (e.g.,

q_{5}

Fig. 6 : Ablation of Human Correction Modeling on (a) phone charging and (b) snack hanging, each reporting success rate (left) and human intervention rate (right) for w/o VLA Proposal in Correction Model (the Correction Model conditioned on

s_{t}

alone), w/o Correction Model (direct actor supervision from raw human corrections), w/o Mahalanobis Constraint (a single state-wise constraint in place of the dimension-wise one), w/o Encoder (mean-pooled frozen VLA features in place of the latent-token encoder), and Bee .

Fig. 7 : Training Dynamics of the State-Dependent Multiplier on phone charging. (a) State-wise multiplier statistics. (b) Constraint slack and violation rate. (c) Per-dimension contributions to

\rho_{\phi}

	Phone charging	Snack hanging	Cloth aligning	Bowl placing
Environment	real	real	real	LIBERO-Pro
Control rate	$30$ Hz	$30$ Hz	$30$ Hz	$20$ Hz
Action dims $d$ / $D$	$16$ / $160$	$8$ / $160$	$16$ / $160$	$7$ / $35$
VLA chunk length (approach)	$50$	$50$	$50$	$10$
Learner chunk length $K$	$10$	$20$	$10$	$5$
Trained by online RL	critical phase	full task	critical phase	full task

TABLE II: Per-Task Specification. The action space lists the per-timestep dimension

d

and the chunked dimension

D=Kd

seen by the actor; the subset of dimensions online RL is allowed to edit, with the remaining dimensions held at the VLA proposal, is described in the text. On phone charging and cloth aligning online RL refines the precision-critical phase only, so an online episode covers that phase and success is reported on it. The online episode budget is the one given to the methods that are seeded with pre-collected human corrections. Section -D gives the budget of the methods that are not.

Hyperparameter	Value
Interface and action space
Control frequency	30 Hz
VLA chunk length, approach / online	50 / 10
Learner chunk length $K$	10
Per-timestep action dim. $d$	16
Chunked action dim. $D=Kd$	160

TABLE III : Hyperparameters of Bee on Phone Charging. Values are those of the run analyzed in the main paper; the quantities that vary by task are listed per task in Table II . All distances, thresholds, and noise scales are expressed in normalized action units.

Method	SR (%) $\uparrow$	Int. (%) $\downarrow$
Phone charging
Base Policy	$90.0\pm 5.0$	–
DAgger	$63.3\pm 5.8$	$7.8$
Bee	$100.0\pm 0.0$	$12.2$
Bowl placing
SiLRI	$0.0$	$90.0+$

TABLE IV : Comparison with DAgger, SiLRI, and HIL-SERL under the matched robot-data budget. Each method is evaluated on the task where the comparison is informative. Intervention rates follow the accounting in Section -D and are not directly comparable across methods.

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 472 papers

RL Token: Bootstrapping Online RL with Vision-Language-Action Models
Apr 24, 2026Charles Xu, Jost Tobias Springenberg, Michael Equi +4RL Fine-Tuning Vision-Language-Action Models
Physical Intelligence
Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models
Sep 25, 2026Yuan Fang, Zechu Li, Haolei Tong +2Robotic RL Agentic RL
TU Darmstadt · University of Augsburg · Tongji University +1
ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning
Jun 15, 2026Wei Xiao, Weiliang Tang, Yuying Ge +4RL for Robotics Robotic Manipulation
1XPENG Robotics · 2Fudan University · 3The Chinese University of Hong Kong +1

Apr 24, 2026J/K move · Enter open · S save

RL Token: Bootstrapping Online RL with Vision-Language-Action Models

Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin+3

Physical Intelligence

RL Fine-Tuning Vision-Language-Action Models RL for Robotics Robot Skill Learning Robotic Manipulation Data-Efficient Robot Learning+2

Open paper AskPDF arXiv

Vision-language-action (VLA) models can learn to perform diverse manipulation skills "out of the box," but achieving the precision and speed that real-world tasks demand requires further fine-tuning -- for example, via reinforcement learning (RL). We introduce a lightweight method that enables sample-efficient online RL fine-tuning of pretrained VLAs using just a few hours of real-world practice. We (1) adapt the VLA to expose an "RL token," a compact readout representation that preserves task-relevant pretrained knowledge while serving as an efficient interface for online RL, and (2) train a small actor-critic head on this RL token to refine the actions, while anchoring the learned policy to the VLA. Online RL with the RL token (RLT) makes it possible to fine-tune even large VLAs with RL quickly and efficiently. Across four real-robot tasks (screw installation, zip tie fastening, charger insertion, and Ethernet insertion), RLT improves the speed on the hardest part of the task by up to 3x and raises success rates significantly within minutes to a few hours of practice. It can even surpass the speed of human teleoperation on some of the tasks.

Abstract

RL for VLMs

Explore similar work

RL Token: Bootstrapping Online RL with Vision-Language-Action Models

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning