LexiconVLA: Learning Reusable Atomic Action Codebooks for Unseen Tasks

Zeming Wei; Jianheng Ye; Xinshuai Song; Sirui Chen; Yang Liu; Liang Lin

cs.ROSep 29, 2026

LexiconVLA: Learning Reusable Atomic Action Codebooks for Unseen Tasks

Authors: Zeming Wei, Jianheng Ye, Xinshuai Song, Sirui Chen, Yang Liu, Liang Lin

Organizations: Sun Yat-sen University · X-Era AI Lab · Pengcheng Laboratory

Abstract

Vision-language-action (VLA) models struggle to reuse recurring interactions in unseen tasks. Our diagnostic study reveals that reliable task completion does not imply consistent execution of constituent atomic actions across task contexts. We present LexiconVLA, a retrievable atomic-action lexicon for cross-task reuse. Global and detail codebooks capture shared interaction structure and fine-grained execution variation, respectively, preserving both reusable patterns and execution details. Visual-Atomic Action Alignment couples trajectory reconstruction from visual state changes with visual outcome prediction from action codes, grounding the lexicon in motion and its effects. We learn these codebooks with trajectory reconstruction and visual alignment on our AtomAction Dataset of 57,803 segments from 69 tasks. A planner and scene-aware adapter translate new goals into code-conditioned subtasks for a shared policy, without skill-specific experts or deployment-time parameter updates. Across five policy backbones on 26 RLBench tasks, LexiconVLA largely maintains performance on 18 seen tasks while improving success on 8 tasks held out from policy training. With BridgeVLA, unseen-task success rises from 16.67% to 34.17% (+17.50 percentage points), and overall success reaches 71.08%, the highest among methods with reported results. Real-robot experiments demonstrate stepwise execution and failure recovery.

Figures & tables

Figure 1: LexiconVLA and Motivation. Left: LexiconVLA decomposes an instruction into atomic-action subtasks, executes each with a VLA policy conditioned on codes from a shared codebook, and advances, retries, or replans according to execution feedback; real-robot rollouts show step-wise execution and recovery from a wrong button press (Bottom left). Right: A diagnostic probe on seven tasks, where each atomic subtask is prompted alone from the demonstration state. Top right: Lowest and highest cross-task success of the contact-free action - approach (stars), with full-task success on the two corresponding tasks (bars). Bottom right: Full-task success (A) versus mean atomic-subtask success (B). For existing policies, high full-task success does not translate into reliable or task-consistent atomic execution.

Figure 2: The AtomAction Dataset. Left: Atomic-action categories (inner ring) and contributing tasks (outer ring). Top right: Segment counts per category; coloured block widths show individual task contributions, matching the outer ring. Bottom right: Example segment with start/end frames and language descriptions.

Figure 3: Overview of LexiconVLA. A VLM-based planner decomposes the task instruction into atomic subtasks. Given the active subtask instruction and current observations, a scene-aware adapter retrieves entries from the frozen atomic-action lexicon—comprising global and detail codes—to condition a shared VLA policy. Execution feedback drives subtask progression and recovery through four skills: NEXT, CONTINUE, RETRY, and REPLAN. Subtask transitions and replanning refresh the retrieved codes based on updated observations.

Figure 4: Overview of the Vector-Quantized Atomic Action Model (VQAA). (a) Atomic Action-NSVQ encodes an atomic action segment into discrete codes from two learnable codebooks using Noise Substitution in Vector Quantization (NSVQ) ( Vali and Bäckström, 2022 ) . A conditional flow matching decoder provides trajectory reconstruction supervision for learning the codebooks. (b) Visual-Atomic Action Alignment (V3A) incorporates visual context through two auxiliary objectives: reconstructing actions from initial-to-final visual changes and predicting final-frame features conditioned on the initial observation and the global action code.

Method	Unseen (8 tasks)	Seen (18 tasks)	Overall (26 tasks)
Act3D ( Gervet et al., 2023 )	$15.67_{\pm 2.52}$	$65.19_{\pm 1.22}$	$49.95_{\pm 1.43}$
3D Diffuser Actor ( Ke et al., 2024 )	$28.67_{\pm 0.29}$	$78.30_{\pm 1.92}$	$63.03_{\pm 1.34}$
SAM2Act ( Fang et al., 2025 )	$15.50_{\pm 1.50}$	$84.30_{\pm 1.80}$	$63.13_{\pm 1.02}$
PerAct ( Shridhar et al., 2022 )	$19.00_{\pm 1.50}$	$37.11_{\pm 2.12}$	$31.54_{\pm 1.34}$
LexiconVLA (PerAct)	$19.33_{\pm 0.58}$	$39.48_{\pm 2.11}$	$33.28_{\pm 1.62}$
RVT ( Goyal et al., 2023 )	$11.83_{\pm 2.31}$	$59.33_{\pm 2.04}$	$44.72_{\pm 2.00}$

Table 1: Multi-task performance on RLBench ( James et al., 2020 ) . Results are presented as mean±std success rate(%) over three random seeds, with 25 evaluation episodes per task and seed. Shaded rows apply LexiconVLA to the base policy above; green numbers give gains in percentage points. Best result per column in bold . ∗ denotes results we obtained by repeatedly fine-tuning and evaluating from the officially released pre-trained checkpoint.

Method	Unseen (8)	Seen (18)	Overall (26)
LexiconVLA (BridgeVLA)	34.17	87.48	71.08
w/o Planner	28.50	88.22	69.85
w/o Codebooks	26.00	87.78	68.77
w/o Global Codebook	26.50	85.33	67.23
w/o Detail Codebook	29.50	87.11	69.38
w/o V3A	26.50	84.89	66.92

Table 2: Success rates (%) in the ablation study.

	Open drawer	Put item in drawer	Push buttons	Pour water into the bowl	Place fruits on the plate	Push buttons	Place fruits on the plate	Pour water into the bowl	Put item in drawer	Open drawer
Methods	Standard					Generalization
	Sim-Short		Sim-Long	Real only- Short	Real only- Long	Distractor		Height	Category	Instruction
BridgeVLA ( Li et al., 2025 )	50.0	0.0	60.0	20.0	0.0	40.0	0.0	0.0	0.0	40.0
$\pi_{0.5}$ ( Intelligence et al., 2025 )	60.0	0.0	80.0	0.0	0.0	40.0	0.0	0.0	0.0	20.0
LexiconVLA (Ours)	100.0	40.0	80.0	40.0	50.0	60.0	20.0	80.0	40.0	80.0

Table 3: Real-world success rates (%) on sim-to-real and generalization tasks.

Figure 5: Qualitative rollouts in simulation and the real world. Ours : LexiconVLA. Rows 1–2: policy-unseen RLBench tasks; rows 3–4: real-robot tasks. After a missed grasp (yellow box) or human disturbance (red box), our planner triggers RETRY, re-executing the failed subtask with codes re-retrieved from the current observation, whereas the baselines fail to recover.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Task	PerAct	BridgeVLA++	Act3D	RVT-2	3DDA	LexiconVLA
Reach & drag	100.0 / 18.0	80.0 / 26.0	90.0 / 22.0	90.0 / 36.0	100.0 / 22.0	100.0 / 58.0
Place wine	50.0 / 29.7	90.0 / 33.3	80.0 / 26.3	80.0 / 45.0	80.0 / 25.6	70.0 / 69.2
Sweep to dustpan	70.0 / 0.0	100.0 / 40.0	100.0 / 17.5	100.0 / 17.5	80.0 / 25.0	100.0 / 35.0
Slide block	100.0 / 16.0	100.0 / 31.4	80.0 / 26.0	70.0 / 14.0	90.0 / 22.0	100.0 / 66.0
Close jar	40.0 / 14.0	100.0 / 25.0	90.0 / 24.0	20.0 / 20.0	60.0 / 0.0	100.0 / 67.5
Push buttons	20.0 / 40.0	100.0 / 56.7	100.0 / 50.0	100.0 / 40.0	100.0 / 66.7	100.0 / 83.3

Appendix

Table 4: Per-task results corresponding to Figure 1 . Each cell reports Full task SR / Atomic SR, in %. Atomic SR pools valid segments within the task; full-task SR uses RLBench success detection.

Figure 6: Codebook size ablation. Success rates (%) of LexiconVLA (BridgeVLA) with the global and detail codebooks scaled jointly (global/detail entries in parentheses). The default size ( $\times 1$ , shaded) reports mean $\pm$ std over three runs; other sizes are evaluated once.

Figure 7: Effect of the planner model. Success rates (%) of LexiconVLA (BridgeVLA) with different planner models, ordered by MMMU-Pro score (dashed line, right axis). All other components are fixed; each planner is evaluated once.

Figure 8: t-SNE of the learned (a) global and (b) detail codebooks. Dots denote encoder features and crosses denote codewords; cross area scales with usage frequency.

	Input: Task instruction $\ell$ , frozen codebooks, adapter, and VLA policy $\pi_{\theta}$ .
1	Observe $(O,\mathbf{s})$ ; $P\leftarrow\mathrm{PLAN}(O,\ell)$ ; initialize active index $i\leftarrow 1$ .
2	Initialize execution records and counters; initialize the retrieval key and code history.
3	while the episode is active do
4	if the active subtask key $(i,\ell_{i})$ has changed and lexicon use is enabled then
5	Predict codebook indices from $(O,\ell_{i})$ using the adapter.
6	Retrieve and cache $(\mathbf{z}_{g}^{(i)},\mathbf{Z}_{d}^{(i)})$ from the frozen lexicon; update the retrieval key and code history.

Appendix

Algorithm 1 LexiconVLA: online planning and execution

Trigger	Condition
Gripper change	Binary gripper state changes, except for actions where closure serves as preparation or tool use.
Prolonged execution	The active subtask exceeds four keyframe steps.
Persistent inactivity	Inactivity persists for two steps, using joint‑speed and end‑effector displacement thresholds of $0.01$ rad/s and $0.005$ m; detected joint motion overrides the displacement test.
Heartbeat	Three keyframe steps have elapsed since the last monitoring call.

Appendix

Table 5: Trigger conditions for invoking the MONITOR function

Skill	State update
NEXT	Mark the active subtask complete, advance by one step, and reset its execution and retry counters
CONTINUE	Retain the active subtask and cached codes
RETRY	Repeat the indicated failed step; when rolling back, revoke completion records from that step onward
REPLAN	Clear completion records and regenerate the plan from the current observation

Appendix

Table 6: State‑machine updates for each scheduling skill

Input field	Dim.	Preprocessing
End-effector position $\mathbf{p}_{t}$	3	Quantile normalization
End-effector rotation $\mathbf{r}_{t}$	6	Quaternion-to-6D conversion; no statistical normalization
Joint positions $\mathbf{q}_{t}$	7	Z-score normalization
Joint velocities $\dot{\mathbf{q}}_{t}$	7	Quantile normalization
Joint torques $\mathbf{f}_{t}^{\mathrm{joint}}$	7	Quantile normalization
Gripper joint positions $\mathbf{q}_{t}^{\mathrm{grip}}$	2	Z-score normalization

Appendix

Table 7: Preprocessing of the 39-dimensional encoder input.

Figure 9: AtomAction Dataset construction pipeline. Complete RLBench ( James et al., 2020 ) expert trajectories are segmented by a rule-based procedure, checked against expert-provided phase-count priors, and annotated with atomic-action labels and phase-level language descriptions.

Figure 10: Motion-structure comparison between AtomicVLA ( Zhang et al., 2026b ) and AtomAction. Left: four action families. Right: clustering metrics. Gray triangles denote AtomicVLA ( Zhang et al., 2026b ) coarse labels; colored circles denote AtomAction fine-grained labels.

Seen tasks (18)
Close jar	Light bulb in	Open drawer
Place cups	Place shape in shape sorter	Push buttons
Put groceries in cupboard	Reach and drag	Slide block to color target
Stack blocks	Place wine at rack location	Sweep to dustpan of size
Insert onto square peg	Meat off grill	Put item in drawer
Put money in safe	Stack cups	Turn tap

Appendix

Table 8: Simulation task groups and training coverage.

Setting	Value
Optimizer	AdamW
Learning rate	$8\times 10^{-5}$
Batch size	8
Training epochs	3
Optimizer steps	17,904
Training seed	20260914

Appendix

Table 9: Fine-tuning configuration for the BridgeVLA comparison arms. These settings are specific to BridgeVLA.

Figure 11: Real-robot platform and experimental settings. Top: the robot setup and all objects used in the experiments. Middle, from left to right: open drawer, put item in drawer, push buttons, place items on a plate, and pour water into a bowl. Bottom: the corresponding generalization settings involving instruction, category, distractor, and height.

Task	Group	Variants	Success criterion
Open drawer	Sim-Short	Top / middle drawer	The specified drawer is pulled open.
Put item in drawer	Sim-Short	Top / middle drawer	The specified drawer is opened and the block rests inside it.
Push buttons	Sim-Long	Red; red $\to$ blue; red $\to$ blue $\to$ green	All specified buttons are pressed in the given order.
Pour water into the bowl	Real-only-Short	Single setup	The cup is grasped, carried to the bowl, and tilted to pour into it without being knocked over.
Place fruits on the plate	Real-only-Long	Pepper + eggplant; pepper + carrot; eggplant + carrot	Both items rest on the plate at the end of the trial.

Appendix

Table 10: Real-world tasks, variants, and success criteria. Groups follow Table 3 .

Method	Standard			Generalization
Method	Sim (3)	Real-only (2)	All (5)	All (5)
BridgeVLA	36.7	10.0	26.0	16.0
$\pi_{0.5}$	46.7	0.0	28.0	12.0
LexiconVLA (Ours)	73.3	45.0	62.0	56.0

Appendix

Table 11: Aggregated real-world success rates (%). Group averages of Table 3 . Each task appears once in each setting, so both All columns cover the same five tasks.

Method	Avg.	Basketball	Close	Knife on	Lamp	Open	Phone	Pick	Press
Method	Avg.	in Hoop	Drawer	Board	On	Jar	on Base	up Cup	Switch
Act3D	$15.7_{\pm 2.5}$	$1.3_{\pm 2.3}$	$68.0_{\pm 4.0}$	$1.3_{\pm 2.3}$	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$18.7_{\pm 4.6}$	$36.0_{\pm 10.6}$
3D Diffuser Actor	$28.7_{\pm 0.3}$	$0.0_{\pm 0.0}$	$\mathbf{94.7}_{\pm 4.6}$	$5.3_{\pm 2.3}$	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$2.7_{\pm 2.3}$	$42.7_{\pm 2.3}$	$\mathbf{84.0}_{\pm 6.9}$
SAM2Act	$15.5_{\pm 1.5}$	$0.0_{\pm 0.0}$	$42.7_{\pm 2.3}$	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$1.3_{\pm 2.3}$	$65.3_{\pm 10.1}$	$14.7_{\pm 8.3}$
PerAct	$19.0_{\pm 1.5}$	$0.0_{\pm 0.0}$	$45.3_{\pm 8.3}$	$6.7_{\pm 6.1}$	$6.7_{\pm 6.1}$	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$20.0_{\pm 4.0}$	$73.3_{\pm 8.3}$
LexiconVLA (PerAct)	$19.3_{\pm 0.6}$	$0.0_{\pm 0.0}$	$45.3_{\pm 9.2}$	$4.0_{\pm 4.0}$	$6.7_{\pm 4.6}$	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$30.7_{\pm 4.6}$	$68.0_{\pm 6.9}$

Appendix

Table 15: Per-task success rate (%) on the 8 unseen tasks. No method is trained on these tasks. Mean

{}_{\pm\text{std}}

over three random seeds. Best result per column in bold. Shaded rows are LexiconVLA variants.

Method	Avg.	Close	Drag	Insert	Meat off	Open	Place	Place	Push	Put in
Method	Avg.	Jar	Stick	Peg	Grill	Drawer	Cups	Wine	Buttons	Cupboard
Act3D	$65.2_{\pm 1.2}$	$\mathbf{100.0}_{\pm 0.0}$	$66.7_{\pm 8.3}$	$13.3_{\pm 2.3}$	$\mathbf{100.0}_{\pm 0.0}$	$90.7_{\pm 9.2}$	$8.0_{\pm 4.0}$	$65.3_{\pm 4.6}$	$\mathbf{100.0}_{\pm 0.0}$	$65.3_{\pm 6.1}$
3D Diffuser Actor	$78.3_{\pm 1.9}$	$96.0_{\pm 0.0}$	$98.7_{\pm 2.3}$	$60.0_{\pm 8.0}$	$\mathbf{100.0}_{\pm 0.0}$	$77.3_{\pm 12.2}$	$33.3_{\pm 6.1}$	$90.7_{\pm 6.1}$	$\mathbf{100.0}_{\pm 0.0}$	$81.3_{\pm 6.1}$
SAM2Act	$84.3_{\pm 1.8}$	$\mathbf{100.0}_{\pm 0.0}$	$\mathbf{100.0}_{\pm 0.0}$	$80.0_{\pm 0.0}$	$\mathbf{100.0}_{\pm 0.0}$	$90.7_{\pm 4.6}$	$33.3_{\pm 4.6}$	$96.0_{\pm 0.0}$	$\mathbf{100.0}_{\pm 0.0}$	$65.3_{\pm 8.3}$
PerAct	$37.1_{\pm 2.1}$	$34.7_{\pm 4.6}$	$90.7_{\pm 2.3}$	$0.0_{\pm 0.0}$	$88.0_{\pm 4.0}$	$81.3_{\pm 8.3}$	$0.0_{\pm 0.0}$	$50.7_{\pm 15.1}$	$26.7_{\pm 2.3}$	$6.7_{\pm 2.3}$
LexiconVLA (PerAct)	$39.5_{\pm 2.1}$	$9.3_{\pm 2.3}$	$81.3_{\pm 4.6}$	$4.0_{\pm 4.0}$	$78.7_{\pm 2.3}$	$76.0_{\pm 10.6}$	$0.0_{\pm 0.0}$	$50.7_{\pm 2.3}$	$37.3_{\pm 2.3}$	$17.3_{\pm 6.1}$

Appendix

Table 16: Per-task success rate (%) on the 18 seen tasks. Mean

{}_{\pm\text{std}}

over three random seeds. Best result per column in bold. Shaded rows are LexiconVLA variants. The Avg. column covers all 18 tasks.

Explore similar work

GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization
May 12, 2026Xiaosong Jia, Bowen Yang, Zuhao Ge +17Recent Vision-Language Models Guidance
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI · Shanghai Jiao Tong University +1
PAPO-VLA: Planning-Aware Policy Optimization for Vision-Language-Action Models
May 19, 2026Peizheng Guo, Jingyao Wang, Changwen Zheng +1Generalizable Vision-Language-Action Policies Vision-Language-Action Framework
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences
ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Models
Sep 7, 2026Songhua Yang, Ziyu Liu, Xuetao Li +3In-Context Multimodal Fusion
Wuhan University · Wuhan University, Wuhan, China · Institute of Technological Sciences, Wuhan University +1

May 12, 2026, cs.ROJ/K move · Enter open · S save

GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization

Xiaosong Jia, Bowen Yang, Zuhao Ge, Xian Nie+16

Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI · Shanghai Jiao Tong University+1

Recent Vision-Language Models Guidance Task-Relevant Information Plug-And-Play

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models aim for general robot learning by aligning action as a modality within powerful Vision-Language Models (VLMs). Existing VLAs rely on end-to-end supervision to implicitly enable the action decoding process to learn task-relevant features. However, without explicit guidance, these models often overfit to spurious correlations, such as visual shortcuts or environmental noise, limiting their generalization. In this paper, we introduce GuidedVLA, a framework designed to manually guide the action generation to focus on task-relevant factors. Our core insight is to treat the action decoder not as a monolithic learner, but as an assembly of functional components. Individual attention heads are supervised by manually defined auxiliary signals to capture distinct factors. As an initial study, we instantiate this paradigm with three specialized heads: object grounding, spatial geometry, and temporal skill logic. Across simulation and real-robot experiments, GuidedVLA improves success rates in both in-domain and out-of-domain settings compared to strong VLA baselines. Finally, we show that the quality of these specialized factors correlates positively with task performance and that our mechanism yields decoupled, high-quality features. Our results suggest that explicitly guiding action-decoder learning is a promising direction for building more robust and general VLA models.

Abstract

Explore similar work

GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization

PAPO-VLA: Planning-Aware Policy Optimization for Vision-Language-Action Models

ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Models