Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution

Yunze Liu; Jing Wang; Shenrui Wu; Wenkun He; Guanqi Zhan; Xianzheng Ma; Junyu Chen; Wenhang Ge

cs.ROOct 14, 2026

Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution

Authors: Yunze Liu, Jing Wang, Shenrui Wu, Wenkun He, Guanqi Zhan, Xianzheng Ma, Junyu Chen, Wenhang Ge,

Organizations: NVIDIA · UCB · MIT · Oxford ·

Abstract

Retrieving useful context for a robot requires matching its task, execution stage, and the continuation that should follow. We introduce Robota-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. Predictive retrieval training aligns these endpoints, enabling historical continuations to serve as context at inference time. We construct a multi-source corpus of 1,695,684 cleaned training anchors and introduce Robota-Embed-Bench, with 4,000 queries and a shared 8,000-entry gallery. Robota-Embed achieves 47.33% clip-level and 81.77% episode-level R@1, exceeding action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. Controlled retrieval studies show that removing native action degrades retrieval more than removing vision. We study two complementary applications. Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset, while improving task progress on real robots. Embed2Evolve uses execution feedback to grow task-local memory and refine context use around a frozen GPT-6 Astra policy. Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout, counting both successes and failures. These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution.

Figures & tables

Figure 1: One joint VLA embedding, two context applications. Robota-Embed jointly encodes vision, language, and native action from human, robot, and simulated experience to retrieve recorded continuations. Left: retrieval accuracy and modality contributions on Robota-Embed-Bench. Right: retrieved demonstrations condition robot policies in Embed2ICL; task-local memory and context use evolve around a fixed GPT-6 Astra policy in Embed2Evolve. Token savings include failed final tests and exclude development; decision/time savings use matched successful pairs. These efficiency comparisons use Direct as the baseline.

Figure 2: Robota-Embed architecture and objectives. Each query is paired with the next four seconds of recorded video and actions. Three objective groups train a joint VLA space.

Figure 3: Multi-source training-data curation. We split episodes before sampling anchors, localize and verify interaction crops, and pair accepted anchors with video and actions from the next four seconds. Query filtering uses only the instruction and current observations.

CLIP-style text–video embeddings
	Clip-level				Episode-level
Model	R@1	R@5	R@10	MRR	R@1	R@5	R@10	MRR
CLIP ViT-L/14	1.2	5.8	9.8	4.06	3.2	7.7	11.6	5.98
Perception Encoder B16	2.0	8.1	12.8	5.52	5.0	10.1	15.2	8.01
InternVideo2	1.6	7.3	11.4	4.82	4.1	9.1	13.2	6.94
Unified VL embeddings

Table 1: Retrieval on Robota-Embed-Bench. Four-source average recall and MRR (%) on the shared 8,000-clip gallery. Bold marks column-wise best results.

Figure 4: Selected retrieval diagnostics. (a) The paired future clip ranks first. (b) A nearby clip from the same episode ranks first, while the paired clip ranks seventh. Both top-1 returns are episode-level matches; only (a) is also a clip-level match.

Figure 5: Embed2ICL: one retriever, two policy interfaces. Reranked DROID videos warm up DreamZero’s visual cache; top-1 EgoDex matches supply Context Tokens for Fast-WAM’s native cross-attention. Retrieval vectors select experience; videos or tokens condition the policy.

Reference condition	Cube $\rightarrow$ Bowl		Can $\rightarrow$ Mug		Banana $\rightarrow$ Bin		Average
No warm-up	61		13		67		47.0
Random reference	50	( $-11$ )	7	( $-6$ )	51	( $-16$ )	36.0	( $-11.0$ )
Omni-Embed-Nemotron-3B
Top-1	57	( $-4$ )	5	( $-8$ )	68	( $+1$ )	43.3	( $-3.7$ )
+ reranking	62	( $+1$ )	19	( $+6$ )	72	( $+5$ )	51.0	( $+4.0$ )
Robota-Embed Top-1	63	( $+2$ )	16	( $+3$ )	72	( $+5$ )	50.3	( $+3.3$ )

Table 2: Training-free DreamZero-DROID in DROID-Sim. Success rates (%); parentheses show percentage-point changes from no warm-up. Reranked conditions share the same agent.

Suite / ID	Abbreviated task	Official release	Embed2ICL
Spatial / S4	Black bowl $\rightarrow$ plate	46	49	( $+6$ )
Goal / G3	Bowl $\rightarrow$ top drawer	42	50	( $+16$ )
Goal / G9	Wine bottle $\rightarrow$ rack	45	47	( $+4$ )
LIBERO-10 / L4	Two mugs $\rightarrow$ plates	43	44	( $+2$ )
LIBERO-10 / L6	Mug–pudding arrangement	40	42	( $+4$ )
Five-task average (%)		86.4	92.8	( $+6.4$ )

Table 3: Training-based ICL on the LIBERO difficult subset. Successes out of 50 per task; last row: average success (%). Parentheses give percentage-point gains over the official release.

Task	No ICL	Embed2ICL
Training-free: DreamZero
Banana $\rightarrow$ Bowl	21.67	40.00	( $+18.33$ )
Training-based: Fast-WAM
Pick Fruit	75.00	79.17	( $+4.17$ )
Split Cup	66.67	87.50	( $+20.83$ )

Table 5: Embed2ICL on real robots. Mean task progress on task-specific 0–100 scales over 12 rollouts per task and condition. Parentheses show gains computed before rounding.

Figure 6: Retrieved references and real-robot execution. Each task pairs a retrieved reference (top) with real-robot execution (bottom). References come from DROID for DreamZero/YAM and EgoDex for Fast-WAM. Timestamps are relative to each clip’s start.

Figure 7: Embed2Evolve. Development grows memory and optimizes its use; testing fixes both. GPT-6 Astra proposes actions for a fixed controller. The model comparison replaces Robota-Embed in both phases. Images show the PullCubeTool task.

Task	GPT-6 Astra Direct	GPT-6 Astra + random history	Evolve + Omni-Embed- Nemotron-3B	Embed2Evolve
Poke Cube	45	15	45	80
Stack Cube	10	20	20	60
Lift Peg Upright	30	40	55	65
Pull Cube with Tool	10	10	20	50
Two-Robot Stack Cube	25	15	25	85
Average (%)	24.0	20.0	33.0	68.0

Table 6: Embed2Evolve in ManiSkill simulation. Success (%) over 20 trials per task and condition; mean policy tokens per test rollout, including failures.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Domain ID	Action dimensions	Rate (Hz)
EgoDex	3	57	30
DROID	8	10	15
LIBERO	5	10	10
RoboMIND–Franka-dual	12	20	10

Appendix

Table 7: Native action interfaces. Dimensions and rates describe native actions before normalization and padding. Domain IDs route actions to their Cosmos3-Edge projectors.

Objective	Constraint or positive relation	Weight	Margin
Predictive retrieval alignment
Clip-level retrieval	Exact future-clip identity; bidirectional InfoNCE	1.00	–
Multimodal grounding and discrimination
Text auxiliary	Text-only query; matched task	0.15	–
Image auxiliary	Image-only query; matched interaction object	0.20	–
Action auxiliary	Disabled; future identity uses multimodal context	0.00	–

Appendix

Table 8: Robota-Embed training objective. Loss coefficients and ranking margins for each term, organized into the three objective groups in Eq. 2 .

Source domain	Training rows
EgoDex [ 31 ]	1,345,160
DROID [ 32 ]	26,563
LIBERO [ 33 ]	136,671
RoboMIND–Franka-dual [ 34 ]	187,290
Total	1,695,684

Appendix

Table 9: Training-data composition. Filtered anchor-level training examples by source.

Domain	R@1	R@5	R@10	MRR	Median rank	Mean rank
Clip-level retrieval
EgoDex	36.70	96.60	99.30	62.22	2	2.68
DROID	74.10	99.00	100.00	85.27	1	1.41
LIBERO	38.50	87.00	96.60	60.33	2	2.92
RoboMIND-FD	40.00	85.00	91.90	60.39	2	3.87
Average	47.33	91.90	96.95	67.05	–	2.72

Appendix

Table 10: Robota-Embed retrieval by source. Recall and MRR are percentages; lower mean ranks are better. Overall median ranks are 2 for clip retrieval and 1 for episode retrieval.

VLM2Vec-V2
	Clip-level				Episode-level
Domain	R@1	R@5	R@10	MRR	R@1	R@5	R@10	MRR
EgoDex	32.40	92.00	99.20	56.90	78.13	97.81	99.88	86.04
DROID	58.70	97.80	99.90	75.15	99.19	100.00	100.00	99.51
LIBERO	4.70	16.20	28.10	12.68	10.75	30.44	46.25	22.01
RoboMIND-FD	0.40	1.40	3.00	1.96	1.19	3.94	6.94	3.95

Appendix

Table 11: Action-free VL baselines by source. Recall and MRR are percentages on the shared gallery. Average weights sources equally; episode-level metrics first average within each episode.

	Clip-level				Episode-level
Setting	R@1	R@5	R@10	MRR	R@1	R@5	R@10	MRR
Original	47.33	91.90	96.95	67.05	81.77	94.91	98.06	86.96
Candidate text masked	12.78	33.65	43.10	23.06	23.17	36.83	46.69	30.32
Same-instruction gallery	47.62	92.45	97.25	67.46	82.53	95.23	98.20	87.55

Appendix

Table 12: Aggregate benchmark-validity controls. Candidate-text masking and same-instruction retrieval use a fixed model. Recall and MRR (%) are averaged equally across four sources.

	Candidates		R@1		R@5
Domain	Mean	Median	Model	Random	Model	Random
EgoDex	4.06	4	37.60	29.76	98.40	95.09
DROID	2.71	2	74.20	54.64	99.20	97.46
LIBERO	54.51	50	38.60	1.90	87.00	9.50
RoboMIND-FD	473.59	342	40.10	0.24	85.20	1.19
All domains	133.72	25	47.62	21.63	92.45	50.81

Appendix

Table 13: Same-instruction gallery difficulty. Candidate counts and clip-level recall (%) within query-specific galleries. Random recall uses the same restricted candidates. The final row averages source means and recalls equally; its candidate median is the cross-source aggregate. RoboMIND-FD denotes RoboMIND–Franka-dual.

Candidate text masked
	Clip-level				Episode-level
Domain	R@1	R@5	R@10	MRR	R@1	R@5	R@10	MRR
EgoDex	4.00	14.30	21.40	9.73	9.00	16.81	24.75	13.62
DROID	4.40	13.30	21.20	9.69	6.50	16.81	25.25	12.57
LIBERO	13.40	36.40	49.30	25.03	23.19	39.56	53.87	32.09
RoboMIND-FD	29.30	70.60	80.50	47.77	54.00	74.12	82.88	63.01

Appendix

Table 14: Benchmark-validity controls by source. Recall and MRR are percentages. Candidate-text masking preserves video and action; same-instruction retrieval restricts each query’s gallery. Original-protocol results are in Table 10 . Episode-level metrics use the within-episode and per-source reducer from Section 3.3 .

Setting	Clip R@1	Episode R@1
Cosmos3-Edge 4B (frozen readout)	0.65	1.17
Robota-Embed	47.33	81.77

Appendix

Table 15: Retrieval adaptation. Four-source average retrieval (%) on the shared Robota-Embed-Bench gallery. The frozen readout and Robota-Embed use the same VLA query/target fields.

Trainable scope	R@1	MRR
Head only	13.8	24.00
Backbone + head	78.7	88.00

Appendix

Table 16: Trainable scope. Clip-level retrieval on the controlled DROID split (%). Both rows use a learned retrieval head; the second also updates the backbone.

Objective (Eq. 2 )	R@1	MRR
$\mathcal{L}_{\mathrm{align}}$	75.6	86.20
$\mathcal{L}_{\mathrm{align}}+\mathcal{L}_{\mathrm{reg}}$	77.0	87.10
$\mathcal{L}_{\mathrm{align}}+\mathcal{L}_{\mathrm{ground}}+\mathcal{L}_{\mathrm{reg}}$	78.7	88.00

Appendix

Table 17: Objective groups. Clip-level retrieval on the controlled DROID split (%). Each row adds an objective group from Equation 2 .

Visual input	R@1	MRR
w/o vision (retrained)	76.8	86.69
Full frame	77.9	87.40
SAM3 object zoom	78.7	88.00

Appendix

Table 18: Visual input. Clip-level retrieval on the controlled DROID split (%). All rows are retrained for their stated visual interface.

Setting	Input	Clip R@1	$\Delta$ clip R@1
Full VLA	VLA	78.7	—
No action	VL	59.8	$-18.9$
No vision	LA	76.8	$-1.9$
No language	VA	20.2	$-58.5$

Appendix

Table 19: Training-time modalities. Matched DROID retraining after removing one modality (%). V/L/A denote vision, language, and action.

Query input	Clip R@1	Episode R@1	$\Delta$ clip R@1
Full VLA	47.33	81.77	—
VL (no action)	20.08	37.36	$-27.25$
LA (no vision)	23.00	39.50	$-24.33$
VA (no language)	7.13	14.73	$-40.20$
L (text only)	16.23	31.09	$-31.10$
V (image only)	1.68	3.66	$-45.65$

Appendix

Table 20: Inference-time modality availability. Four-source average retrieval (%) with the final checkpoint and gallery fixed. Only the query inputs are changed. V/L/A denote vision/language/action;

\Delta

gives the percentage-point change in clip-level R@1 from Full VLA.

Query setting	Input	R@1	R@5	R@10	MRR	$\Delta$ R@1
Full query	VLA	78.7	99.4	100.0	88.00	$\mathbf{0.0}$
Action masked	VL	50.7	90.3	94.9	67.60	$-28.0$
Image masked	LA	60.9	91.8	96.1	74.17	$-17.8$
Text masked	VA	1.9	7.2	12.0	5.41	$-76.8$

Appendix

Table 21: DROID query-only masking. Clip-level recall and MRR (%) with the model and gallery embeddings fixed.

\Delta

R@1 gives the percentage-point change from the full VLA query.

Gallery	Top-10 search (ms)	Encode + retrieve latency (ms)	Single-stream QPS	Est. GPU memory (GB)
1M	2.66	160.36	6.24	11.29
2M	5.14	162.84	6.14	15.39
5M	12.60	170.30	5.87	27.69
10M	25.04	182.74	5.47	48.19
15M	37.47	195.17	5.12	68.69

Appendix

Table 22: Retrieval scaling on one NVIDIA A100. Measured latencies cover Top-10 search alone and query encoding plus search. Throughput is derived from the latter; GPU memory is estimated.

Figure 8: Selected qualitative evidence across embodiments. Each query searches only the illustrated target source. The top-ranked clips share placement or removal roles with their queries.

Figure 9: Training-free Embed2ICL. Robota-Embed retrieves candidate VLA clips, and the Reranking Agent selects a task-relevant reference containing completion evidence. Its video initializes DreamZero-DROID’s visual key–value cache. The policy remains frozen while live observations update the cache during execution.

Figure 10: DROID references for training-free Embed2ICL. Each row shows one task’s fixed reference from a single camera, before, at, and after the detected final gripper release. DreamZero-DROID receives all three synchronized four-second video streams.

Figure 11: Training-based Embed2ICL. Offline encoding caches each candidate’s retrieval vector, Context Tokens, and context mask. Online cosine top-1 search selects a reference and loads its tokens and mask. The prefilling adapter projects the tokens to Fast-WAM’s context width and appends them to the native text and proprioceptive conditioning for cross-attention.

Figure 12: LIBERO G3: reference and final states. Successes out of 50 per condition.

Suite / ID	Official release	Side branch	Embed2ICL
Spatial / S4	46	46 ( $0$ )	49 ( $+6$ )
Goal / G3	42	44 ( $+4$ )	50 ( $+16$ )
Goal / G9	45	44 ( $-2$ )	47 ( $+4$ )
LIBERO-10 / L4	43	43 ( $0$ )	44 ( $+2$ )
LIBERO-10 / L6	40	43 ( $+6$ )	42 ( $+4$ )
Five-task average (%)	86.4	88.0 ( $+1.6$ )	92.8 ( $+6.4$ )

Appendix

Table 23: LIBERO integration ablation. Successes from 50 trials per task and condition; the final row gives the five-task average (%). Parentheses show percentage-point changes from the official release; bold marks the better ICL setting.

Task	No ICL	Side branch	Embed2ICL
Pick Fruit	75.00	77.08 ( $+2.08$ )	79.17 ( $+4.17$ )
Split Cup	66.67	75.00 ( $+8.33$ )	87.50 ( $+20.83$ )

Appendix

Table 24: Real-robot integration ablation. Mean normalized task progress (%) from 12 rollouts per task and condition. Parentheses show percentage-point changes from Fast-WAM without ICL.

Integration	Random tokens	Retrieved tokens
Side branch	85.6	88.0
Embed2ICL	88.0	92.8

Appendix

Table 25: Retrieved versus random Context Tokens. Each integration is evaluated on the LIBERO difficult subset with 50 trials per task and condition. Results are five-task average success rates (%).

Component	Configuration
Policy	GPT-6 Astra, reasoning effort low
Simulator	ManiSkill 3.0.1; SAPIEN 3.0.3
Current RGB	One $256\times 256$ image per camera per decision
Cameras	Four for single-arm tasks; five for TwoRobotStackCube
Control rate	20 Hz; two ticks per primitive (0.1 s simulation time)
Decision budget	At most eight primitives per GPT decision

Appendix

Table 26: Embed2Evolve runtime and observation interface. Simulation time is distinct from client execution time, which includes API waiting.

Preset	$k$	Cadence $\delta$	History $H$	$\alpha$	$\beta$
A	2	2	12	0.10	0.03
B	1	4	6	0.10	0.10
C	2	1	8	0.15	0.05

Appendix

Table 27: Initial context configurations. All presets enable historical images and are evaluated on the same ten development seeds.

Figure 13: From Direct control to a selected harness. PullCubeTool’s R4 $\rightarrow$ R5 $\rightarrow$ R8 parent–child sequence grows memory while reducing retained history and retrieval frequency. The bottom filmstrips show the same fresh-test initialization completed successfully by both systems. Task execution time includes API waits.

Task	Round	Episodes	Anchors	$k$	Cadence	$H$	$\alpha/\beta$
Poke Cube	8	80	1,610	2	2	6	.15/.05
Stack Cube	2	20	454	2	2	12	.10/.03
Lift Peg Upright	6	60	1,116	2	1	6	.20/.05
Pull Cube with Tool	8	80	1,667	2	3	4	.15/.05
Two-Robot Stack Cube	5	50	1,126	2	3	6	.10/.03

Appendix

Table 28: Selected Embed2Evolve harnesses. Bank sizes count development episodes and indexed anchors. Cadence counts GPT decisions;

H

counts retained action/feedback pairs.

Figure 14: Task-wise development trajectories. Success uses ten trials per round; decisions average that round’s successful trials. Hollow diamonds mark selected harnesses. Lines connect evaluated candidates, not necessarily parent–child configurations.

	Omni-Embed- Nemotron-3B		Robota-Embed
Task	Completed	Selected	Completed	Selected
Poke Cube	8	6	8	8
Stack Cube	6	3	4	2
Lift Peg Upright	8	4	8	6
Pull Cube with Tool	3	1	8	8
Two-Robot Stack Cube	7	5	8	5

Appendix

Table 29: Independent development under matched search rules. Completed rounds count all numbered rounds; selected rounds identify the configuration and memory frozen for testing.

Metric	Test population	Five-task aggregation
Task success	All 20 trials per task in all four conditions	Mean of task success rates.
Policy decisions and execution time	Matched initializations successful under Direct and Embed2Evolve	Mean of task-wise relative reductions; speedup averages Direct/Embed2Evolve time ratios.
Policy tokens	All 20 trials per task for Direct, evolution with Omni-Embed-Nemotron-3B, and Embed2Evolve, including failures	Relative reduction between equal-task per-rollout means.
Reference API cost	All 20 trials per task for Direct and Embed2Evolve, including failures	Relative reduction between equal-task per-rollout means at fixed input/output rates.

Appendix

Table 30: Embed2Evolve test-metric accounting. All five tasks receive equal weight. Paired comparisons and reference API costs use Direct and Embed2Evolve.

Task	Paired $n$	Policy decisions	Execution time (s)	Control ticks	Tick change (%)
Poke Cube	7	34.86 $\rightarrow$ 22.71	419.30 $\rightarrow$ 382.16	133.0 $\rightarrow$ 181.4	$+36.4$
Stack Cube	2	45.00 $\rightarrow$ 10.00	499.03 $\rightarrow$ 186.35	90.0 $\rightarrow$ 65.0	$-27.8$
Lift Peg Upright	5	57.20 $\rightarrow$ 12.40	556.06 $\rightarrow$ 234.46	152.0 $\rightarrow$ 149.6	$-1.6$
Pull Cube with Tool	1	80.00 $\rightarrow$ 26.00	962.27 $\rightarrow$ 489.55	347.0 $\rightarrow$ 337.0	$-2.9$
Two-Robot Stack Cube	4	87.25 $\rightarrow$ 33.00	1054.67 $\rightarrow$ 751.59	206.0 $\rightarrow$ 312.8	$+51.8$

Appendix

Table 31: Efficiency and physical control on shared successes. Direct

\rightarrow

Embed2Evolve means over the same

n

matched successful initializations for all three metrics. Execution time includes API waiting. Tick change is

100(\mathrm{Evolve}/\mathrm{Direct}-1)

; positive values indicate more physical control ticks despite fewer GPT decisions.

Poke Cube
Method	Input tokens	Output tokens	Total tokens	Ref. cost (USD)
Direct	693,733.05	16,052.50	709,785.55	7.7400
Embed2Evolve	375,759.60	12,903.40	388,663.00	4.4028
Stack Cube
Direct	935,086.30	18,401.05	953,487.35	10.2709
Embed2Evolve	656,130.60	19,000.10	675,130.70	7.5113

Appendix

Table 32: Policy token usage and reference cost per rollout. Means include all 20 test rollouts per task and condition, regardless of outcome. Direct denotes GPT-6 Astra Direct; the final group averages task means equally. USD estimates use the fixed input/output rates above.

Method	Modalities	Query $\rightarrow$ candidate	Action role	Cross-source	Policy coupling	Lifetime
Deep Multimodal	3D/L/A	3D+L $\rightarrow$ trajectory	Candidate key	No	Independent	Offline corpus
CLASP	L/S/A	L $\rightarrow$ state+action	Candidate key	No	Independent	Offline corpus
VLA-R	V/L/A	scene V/L $\rightarrow$ action token	Candidate key	No	Direct control	Action vocab.
ROSER	P	prototype $\rightarrow$ window	None	No	Independent	Offline corpus
RAEA	V/L	V/L $\rightarrow$ V/L record	Returned only	Yes	Coupled	Persistent bank
REGENT	S	state $\rightarrow$ demo state	Returned+blend	No	Coupled	Task buffer

Appendix

Table 33: Representative physical-experience retrieval interfaces. V/L/A/S/P denote vision, language, native action, state, and proprioception; 3D denotes point clouds. Training-only action use is listed under action role; cross-source galleries span datasets or interfaces.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,190 papers

Fast-VLA: You Only Think Once for Robot Manipulation
Oct 15, 2026Kehong Gong, Guanli Hou, Xiaoxu Zheng +6Vision-Language-Action Models Closed-Loop Control of VLA Models
Huawei Technologies Co., Ltd · Huawei Central Research Institute
In-Context Robot Learning with VLM Agents
Sep 16, 2026Dongzhou Cheng, Taoran Yi, Ye Fang +12Robot Policy Learning ICL for Robotics
Morphi Robot · Shanghai Innovation Institute · Huazhong University of Science and Technology +7
RoboICL: Embodied In-Context Learning with GPT-6 Astra
Sep 28, 2026Fangcheng Liu, Yeqing Shen, Anda Cheng +7Robotic Control ICL for Robotics
Samsung Research, Beijing, China · Samsung Robotics eXperience · Shanghai Jiao Tong University

Oct 15, 2026J/K move · Enter open · S save

Fast-VLA: You Only Think Once for Robot Manipulation

Kehong Gong, Guanli Hou, Xiaoxu Zheng, Qi Wang+5

Huawei Technologies Co., Ltd · Huawei Central Research Institute

Vision-Language-Action Models Closed-Loop Control of VLA Models Efficient VLA Model Inference Robotic Manipulation

Open paper AskPDF arXiv

Vision-language-action (VLA) policies commonly place a multi-billion-parameter model inside the feedback loop, repeatedly recomputing task context whenever they predict a new action chunk. Yet task understanding changes far more slowly than the robot state. We introduce Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize the instruction and initial views as fixed, multi-layer key-value task memory. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model. Fast-VLA achieves 92.46% and 92.40% success on RoboTwin 2.0 Clean and Randomized, and 95.95% and 79.11% on LIBERO and LIBERO-Plus evaluation. Against three continually conditioned baselines on the same hardware, it reduces RoboTwin amortized policy inference time by 10.9-17.1x and requires 1.22 GiB of post-prefill online resident memory. Task-memory swaps further show that the actor relies on episode-matched task memory. These results show that task context can be established once while closed-loop control remains responsive throughout execution. Project page: https://garfield-kh.github.io/FastVLA

Abstract

Vision-Language-Action Models

Explore similar work

Fast-VLA: You Only Think Once for Robot Manipulation

In-Context Robot Learning with VLM Agents

RoboICL: Embodied In-Context Learning with GPT-6 Astra