JevVLA: Jev-Inspired Energy-Based Adaptation for Vision-Language-Action Policies

Wei Wang; Wenqiao Zhang; Zheqi Lv; Zheng Dong; Yingjie Niu; Jiaqi Zhu; Jun Xiao; Chao Li

cs.ROOct 26, 2026

JevVLA: Jev-Inspired Energy-Based Adaptation for Vision-Language-Action Policies

Authors: Wei Wang, Wenqiao Zhang, Zheqi Lv, Zheng Dong, Yingjie Niu, Jiaqi Zhu, Jun Xiao, Chao Li,

Organizations: College of Computer Science and Technology, Zhejiang University · Cornell University · Universal Ubiquitous AI Co., Ltd. · Hangzhou DEEP Robotics Technology Co., Ltd. ·

Abstract

Vision-language-action (VLA) policies can select among discrete candidates for pick-and-place manipulation or generate continuous action chunks for motion control. Improving both forms of decision making through a common mechanism while retaining the capabilities of a pretrained policy is an important adaptation problem. Inspired by Jev's structured, probabilistic decision interface, we propose JevVLA, a lightweight energy-based adaptation framework for VLA policies. JevVLA instantiates this inspiration by modeling the relative quality of action proposals: with the base policy frozen, it learns a differentiable scalar energy conditioned on observations, task context, and actions, using execution feedback, expert corrections, or demonstration-derived preferences. JevVLA uses this single learned action preference function in two ways: its values compare discrete candidates, while its action gradients correct continuous actions or guide their generation. Candidate selection and continuous control thus share one trainable decision core. Each benchmark uses independently trained weights and fixed input/output encodings, while retaining the same core network with approximately 80,000 trainable parameters. On the held-out test sets, JevVLA improves task success over the respective matched base-policy controls by 1.28, 2.31, 1.60, 1.52, and 1.28 percentage points on MetaWorld MT50, VIMA, LanguageTable, LIBERO Plus, and CLIPort, respectively. These results support the feasibility of a lightweight, Jev-inspired energy model for adapting both discrete action selection and continuous action guidance. The JevVLA implementation will be open-sourced at https://github.com/ZJU4EmbodiedAI/JevVLA.

Figures & tables

Figure 2: JevVLA architecture. Fixed encodings connect observations and native action proposals from a frozen policy to a conditional energy core. The core combines a quadratic potential with a nonlinear scalar residual. Its values rank candidates, optionally with the policy prior, while its action gradients provide bounded corrections or guide iterative generation. Execution feedback, expert corrections, or demonstration preferences supervise energy values and action gradients as available; training updates only the adapter. Benchmarks share the 79,817-parameter core architecture and learn separate weights. Robot images illustrate the computation.

Benchmark	Frozen policy	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
MetaWorld MT50	$\pi_{0.5}$ –MT50	2,500	75.84	77.12	+1.28
VIMA L1	VIMA-200M	1,300	84.77	87.08	+2.31
LanguageTable	Language BC	250	78.00	79.60	+1.60
LIBERO Plus	$\pi_{0.5}$ –LIBERO	10,030	78.63	80.15	+1.52
CLIPort	CLIPort	1,800	34.22	35.50	+1.28

Table 1: Task success across five manipulation benchmarks. Each JevVLA adapter uses the corresponding frozen policy and separately trained weights. CLIPort reports strict terminal success from closed-loop execution. Evaluation protocols are specified in the text.

Figure 3: Task success across five manipulation benchmarks. Circles show Direct and diamonds show JevVLA. Connectors pair results within each benchmark. The right columns give the gain in percentage points and the number of examples. Each benchmark uses independently trained adapter weights.

Figure 4: Continuous action refinement on LanguageTable. Both runs start from the same recorded state (seed 900003). JevVLA succeeds at step 41; Direct reaches the 200-step limit without success. The first three rollout columns show steps 0, 20, and 30; the final column shows each actual terminal. Panel (a) compares native and refined actions on the same JevVLA observation at step 23. Panel (c) shows all 41 recorded correction magnitudes and the two action components at that step. The local vector is shown relative to the native action, in millimetres.

Figure 5: Task-level changes. (a) Direct and JevVLA success for individual tasks; the dashed diagonal denotes unchanged success. (b) Numbers of improved, unchanged, and decreased tasks. LIBERO Plus groups its perturbation variants by the 40 original tasks.

Perturbation	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
Background Textures	1,076	88.94	86.99	-1.95
Camera Viewpoints	1,599	72.73	76.30	+3.56
Language Instructions	1,537	79.70	81.33	+1.63
Light Conditions	1,142	85.55	87.92	+2.36
Objects Layout	1,525	82.03	84.13	+2.10
Robot Initial States	1,550	65.03	65.55	+0.52

Table 2: LIBERO Plus success by perturbation. The seven categories partition the 10,030 variants. Task and suite summaries in the appendix regroup the same outcomes.

Selection rule	Count / expectation	Success (%)	$\Delta$ vs. Direct (pp)
Direct ( $c_{0}$ )	616/1800	34.22	0.00
Uniform random (expectation)	616/1800	34.22	0.00
Minimum action norm	592/1800	32.89	-1.33
Nearest alternative to Direct	600/1800	33.33	-0.89
JevVLA (Full)	639/1800	35.50	+1.28

Table 3: Selection rules on the same CLIPort candidate set. Uniform random averages the recorded candidate outcomes to obtain an analytical expectation. Differences are relative to Direct.

Figure 6: CLIPort ablation configurations. Dots show success rates; differences are relative to Full (639/1800, 35.50%, solid line). The dashed line marks Direct (616/1800, 34.22%). The panels group energy structure, preference supervision, and scoring inputs. Full uses 1,032 fit and 258 disjoint validation examples; the 1,800 results are from a separate test set.

Energy structure
Configuration	Success count	Success (%)	$\Delta$ vs. Full (pp)
JevVLA (Full)	639/1800	35.50	0.00
Without quadratic term	601/1800	33.39	-2.11
Without nonlinear residual	615/1800	34.17	-1.33
Without low-rank coupling	624/1800	34.67	-0.83
Preference supervision

Table 4: CLIPort configurations on 1,800 test examples. All gaps are relative to Full (639/1800), using 1,032 fit examples and 258 validation examples for checkpoint and threshold selection. The low-rank ablation also changes the training objective to pairwise preference learning.

Configuration	Changes	Rescues	Harms	Net	Success (%)
Randomly initialized energy	1,347	86	96	-10	33.67
Shuffled preference labels	691	52	63	-11	33.61
Shuffled input features	1,192	62	79	-17	33.28
JevVLA (Full)	1,316	82	59	+23	35.50

Table 5: CLIPort outcome changes relative to Direct (

N=1{,}800

). Changes counts candidate replacements. Rescues turn failures into successes; harms turn successes into failures. Net equals rescues minus harms, and total successes equal

616+\mathrm{Net}

Figure 7: Rescues and harms on CLIPort. (a) Leftward bars count Direct successes lost after replacement; rightward bars count Direct failures rescued. Diamonds indicate the net change. (b) Full retains the Direct action in 484 cases, changes the action with the same terminal outcome in 1,175, rescues 82, and harms 59. These categories sum to 1,800; rescues minus harms yield 23 additional successes.

Figure 8: Candidate selection on CLIPort. The upper case (seed 3900000155) changes the first proposal from $c_{0}$ to $c_{1}$ ; frozen-policy continuation then places both orange blocks in pink bowls. The lower case (seed 3900000148) selects $c_{3}$ , which moves a yellow bowl into the white bowl while leaving the orange block outside. Candidate overlays show recorded pick and place positions on the same initial heightmap. Displayed scores are $\Delta U=U(c_{i})-U(c_{0})$ , with $E=-U$ . Camera frames replay the four recorded branches with the frozen base policy. The aggregate CLIPort evaluation covers 1,800 test examples.

Method	Successes	Success (%)
Direct ( $\pi_{0.5}$ )	1896/2500	75.84
Shared linear value probe Zhang et al. (2026b)	1839/2500	73.56
RSS RAS decoding rule Zhan et al. (2026)	1168/2500	46.72
JevVLA (Full)	1928/2500	77.12

Table 6: Method comparisons on MetaWorld MT50 (50 tasks, 50 test episodes each). All methods use the frozen

\pi_{0.5}

policy.

Method	Successes	Success (%)
Direct (CLIPort)	616/1800	34.22
UA-CLIPort Wu et al. (2024b)	607/1800	33.72
Logistic regression	623/1800	34.61
Shared MLP	617/1800	34.28
TabM Gorishniy et al. (2025)	627/1800	34.83
FT-Transformer Gorishniy et al. (2021)	624/1800	34.67

Table 7: Method comparisons on the 18 CLIPort variants (100 test episodes each). UA-CLIPort uses an online 11

\times

11 grasp rule; no matched online Direct run is available for this configuration.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Adapter optimization	Target and inference settings
MetaWorld MT50	Learning rate $2\times 10^{-4}$ , batch 128, 8 initial epochs; $3\times 10^{-5}$ , batch 128, 3 continuation epochs (1).	Expert shift $\alpha=0.25$ ; $\sigma(t)=0.2+0.8t$ ; correction gain $\eta=0.08$ ; component clip $0.12$ .
VIMA L1	Learning rate $3\times 10^{-4}$ , batch 64, 80 epochs (80); seed 5137.	Five training candidates; target temperature $\tau_{y}=0.235$ ; native-prior scale $\tau_{0}=20$ ; energy gain $\beta=4$ .
LanguageTable	Learning rate $3\times 10^{-4}$ ; each step uses 16 initial anchors and up to 128 future states; 80-epoch selection, 6-epoch refit; seed 260930.	Initial CE and DSM weights $1:1$ , future-state DSM weight $0.2$ ; $\sigma(t)=0.2+0.8t$ ; $\eta=0.0004$ .
LIBERO Plus	Learning rate $3\times 10^{-4}$ , batch 128, up to 80 epochs (46); seed 5137.	Two candidates; $\tau_{y}=0.0312$ ; $\beta=0.25$ ; global Direct margin $m=0.005$ .
CLIPort	Learning rate $3\times 10^{-4}$ , batch 128, up to 80 epochs for CE Full; pairwise ablations use up to 50 epochs.	Five executed candidates; $\beta=1$ ; the checkpoint and global Direct margin $m$ are selected on the 258-example validation set.

Appendix

Table 8: Energy-adapter hyperparameters. A parenthesized epoch is the selected checkpoint; other epoch counts are training limits.

Task	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
follow order	100	85.00	85.00	0.00
manipulate old neighbor	100	92.00	94.00	+2.00
novel adj	100	96.00	98.00	+2.00
novel noun	100	98.00	97.00	-1.00
pick in order then restore	100	63.00	64.00	+1.00
rearrange	100	99.00	100.00	+1.00

Appendix

Table 9: VIMA L1: task counts and success rates.

Task	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
align rope	100	9.00	6.00	-3.00
assembling kits seq seen colors	100	22.00	21.00	-1.00
assembling kits seq unseen colors	100	2.00	0.00	-2.00
packing boxes pairs seen colors	100	52.00	55.00	+3.00
packing boxes pairs unseen colors	100	14.00	16.00	+2.00
packing seen google objects group	100	21.00	25.00	+4.00

Appendix

Table 10: CLIPort: task counts and success rates.

Task	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
assembly	50	98.00	98.00	0.00
basketball	50	28.00	70.00	+42.00
bin picking	50	54.00	44.00	-10.00
box close	50	58.00	60.00	+2.00
button press topdown	50	98.00	98.00	0.00
button press topdown wall	50	100.00	100.00	0.00

Appendix

Table 11: MetaWorld MT50: task counts and success rates.

Task	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
blocktoabsolutelocation	50	70.00	70.00	0.00
blocktoblock	50	90.00	92.00	+2.00
blocktoblockrelativelocation	50	66.00	68.00	+2.00
blocktorelativelocation	50	68.00	72.00	+4.00
separate	50	96.00	96.00	0.00
Total	250	78.00	79.60	+1.60

Appendix

Table 12: LanguageTable: task counts and success rates.

Task	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
libero 10: KITCHEN SCENE3 turn on the stove and put the moka pot on it	281	69.04	81.85	+12.81
libero 10: KITCHEN SCENE4 put the black bowl in the bottom drawer of the cabinet and close it	304	76.97	78.29	+1.32
libero 10: KITCHEN SCENE6 put the yellow and white mug in the microwave and close it	278	66.55	67.99	+1.44
libero 10: KITCHEN SCENE8 put both moka pots on the stove	233	61.80	56.22	-5.58
libero 10: LIVING ROOM SCENE1 put both the alphabet soup and the cream cheese box in the basket	194	79.90	79.38	-0.52
libero 10: LIVING ROOM SCENE2 put both the alphabet soup and the tomato sauce in the basket	212	82.08	83.02	+0.94

Appendix

Table 13: LIBERO Plus: original tasks: task counts and success rates.

Task	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
Background Textures	1,076	88.94	86.99	-1.95
Camera Viewpoints	1,599	72.73	76.30	+3.56
Language Instructions	1,537	79.70	81.33	+1.63
Light Conditions	1,142	85.55	87.92	+2.36
Objects Layout	1,525	82.03	84.13	+2.10
Robot Initial States	1,550	65.03	65.55	+0.52

Appendix

Table 14: LIBERO Plus: perturbations: task counts and success rates.

Task	$N$	Direct (%)	JevVLA (%)	$\Delta$ (pp)
libero 10	2,519	69.91	72.69	+2.78
libero goal	2,591	72.67	76.26	+3.59
libero object	2,518	89.91	89.28	-0.64
libero spatial	2,402	82.39	82.60	+0.21
Total	10,030	78.63	80.15	+1.52

Appendix

Table 15: LIBERO Plus: suites: task counts and success rates.

Task	Direct	JevVLA	Probe	RAS
assembly	49	49	36	5
basketball	14	35	40	12
bin picking	27	22	20	10
box close	29	30	24	29
button press topdown	49	49	48	25
button press topdown wall	50	50	50	34

Appendix

Table 16: MetaWorld MT50: comparison methods by task.

Task	Direct	JevVLA	UA	Logistic	MLP
align rope	9	6	8	11	8
assembling kits seq seen colors	22	21	21	20	18
assembling kits seq unseen colors	2	0	5	2	0
packing boxes pairs seen colors	52	55	57	56	54
packing boxes pairs unseen colors	14	16	30	14	14
packing seen google objects group	21	25	13	22	25

Appendix

Table 17: CLIPort: online and learned comparison methods by task.

Task	Direct	JevVLA	TabM	FT-Trans.	TabNet
align rope	9	6	5	5	4
assembling kits seq seen colors	22	21	23	15	17
assembling kits seq unseen colors	2	0	2	2	0
packing boxes pairs seen colors	52	55	56	59	54
packing boxes pairs unseen colors	14	16	14	12	15
packing seen google objects group	21	25	23	28	21

Appendix

Table 18: CLIPort: tabular model architectures by task.

VLA Model Adaptation

See all 41 papers

Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation
May 15, 2026Jin Shi, Brady Zhang, Yishun LuVLM Distillation Efficient VLA Model Inference
Department of Mechanical Engineering University College London London, United Kingdom · Department of Engineering Science University of Oxford Oxford, United Kingdom
Juno: Taming Predictive Latents for Vision-Language-Action Models
Oct 7, 2026Yuchen Zhu, Chenyi Xu, Yulin Zhang +2Vision-Language-Action Models World Models for Robotics
University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Hangzhou Dianzi University +1
IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies
Sep 10, 2026Kian Hosseinkhani, Qinhe Peng, George Shramko +6Efficient VLA Model Inference Vision-Language-Action Models
Simon Fraser University · University of Pennsylvania · Alberta Machine Intelligence Institute +1

May 15, 2026J/K move · Enter open · S save

Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation

Jin Shi, Brady Zhang, Yishun Lu

Department of Mechanical Engineering University College London London, United Kingdom · Department of Engineering Science University of Oxford Oxford, United Kingdom

VLM Distillation Efficient VLA Model Inference Visuomotor Policy Learning Efficient VLA Models Robotic Manipulation Vision-Language-Action Models+1

Open paper AskPDF arXiv

Billion-parameter Vision-Language-Action (VLA) policies have recently shown impressive performance in robotic manipulation, yet their size and inference cost remain major obstacles for real-time closed-loop control. We introduce VLA-AD, a distillation framework that uses a Vision-Language Model as an offline semantic supervisor to transfer large VLA teachers into lightweight student policies. Instead of relying only on low-level action imitation, VLA-AD augments teacher-provided 7-DoF action targets with high-level semantic guidance, including task phase anchors and multi-frame operating-direction descriptions. These auxiliary signals are used only during training: at test time, the student policy runs independently, with neither the VLA teacher nor the VLM required. We evaluate VLA-AD on three LIBERO benchmark suites. Using OpenVLA-7B as the teacher, our method produces a 158M-parameter student, yielding a

44\times

reduction in model size while matching the teacher with only a

0.27\%

average relative gap. The resulting policy runs at 12.5 Hz on an RTX 4090, achieving a

3.28\times

inference speedup over OpenVLA-7B. We further show that the same semantic distillation pipeline generalizes to a different

π_{0.5}

-4B teacher, where the student outperforms the teacher on two suites and remains within

0.53\%

on libero\_goal. Additional analysis indicates that phase-level supervision and multi-frame directional cues make the student less sensitive to noisy teacher actions, such as erroneous high-frequency gripper changes. Overall, VLA-AD demonstrates that offline semantic guidance from VLMs can substantially improve the efficiency, robustness, and deployability of VLA policy distillation.

Abstract

VLA Model Adaptation

Explore similar work

Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation

Juno: Taming Predictive Latents for Vision-Language-Action Models

IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies