VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning

Diyuan Shi; Yiqi Tang; Hongyin Zhang; You Wu; Donglin Wang

cs.AIOct 14, 2026

VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning

Authors: Diyuan Shi, Yiqi Tang, Hongyin Zhang, You Wu, Donglin Wang

Organizations: Zhejiang University · Westlake University

Abstract

Vision-language-action (VLA) models connect visual perception and language understanding with robot control, providing a foundation for instruction-conditioned manipulation. However, conventional pretrained VLA policies lack a mechanism for gradual adaptation through accumulated interaction experience during deployment. We therefore propose VLRLA, a general framework that equips existing VLA models with in-context reinforcement learning capabilities for deployment-time adaptation. Our approach augments a VLA model with a reinforcement learning module that learns from historical observations, actions, rewards, and termination signals. During deployment, the policy gradually refines its action generation and progressively improves task success while keeping model parameters fixed. Our approach is also designed for general applicability and compatibility with diverse learning history construction strategies and model training pipelines. Finally, through extensive experiments, we systematically investigate the learning properties of VLRLA in simulation and demonstrate its ability to adapt to unseen scenarios during real-world deployment.

Figures & tables

Figure 1: Framework overview. VLRLA is designed as a general in-context RL framework supporting diverse learning history construction strategies and model integration strategies.

Seen tasks
	$\pi_{0.5}$	GRPO	RoboTTT	VLRLA-bridge	$\Delta$
In-context adaptation	No	No	Yes	Yes	/
Sample efficiency	/	Low	Moderate	Moderate	/
handover_mic	38.3%	32.8%	52.3%	78.9%	40.6%
pick_diverse_bottles	58.6%	77.3%	71.1%	75.8%	17.2%
place_a2b_left	67.2%	81.2%	70.3%	71.9%	4.7%

Table 1: Algorithm characteristics and success rates (%) on seen and unseen RoboTwin tasks.

\Delta

denotes the improvement of VLRLA-bridge over

\pi_{0.5}

in percentage points. “/” indicates not applicable . The highest success rate in each row is shown in bold, including all ties.

Figure 2: Adaptation across VLM modules. Shaded area is 1 standard deviation across 5 seeds.

Figure 3: Effect of context and memory. Shaded area is 1 standard deviation across 5 seeds.

Figure 4: History transfer to improve initial task performance. Shaded regions indicate one standard deviation across five seeds.

Figure 5: Left 3 figures depict the fruits placement. Right 3 figures depict the pepper placement.

Panel of Figure 6 — Figure 6: Success rates on fruit-placement tasks.

	$\pi_{0.5}$	$\mathrm{VLRLA}_{1}$	$\mathrm{VLRLA}_{2}$	$\mathrm{VLRLA}_{3}$	$\mathrm{VLRLA}_{4}$	$\mathrm{VLRLA}_{4}-\pi_{0.5}$
Apple	46.7%	40.0%	53.3%	53.3%	60.0%	+13.3%
Starfruit	40.0%	26.7%	46.7%	60.0%	66.7%	+26.7%
Grape	53.3%	60.0%	46.7%	66.7%	60.0%	+6.7%

Table 2: Fruit-placement success-rate breakdown by fruit type, with 15 runs per fruit.

Figure 7: Left part: illustrative diagram for cross-episodic-memory task. The digit in circle denotes the reward label passed to policy after the episode. Right part: success rate averaged in 44 runs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Transformer layers / attention heads	6 / 8
Hidden / feedforward dimension	512 / 1024
Number of summary / action queries	36 / 24
Residual-adapter rank	64
Dropout	0.1
Residual output initialization	Zero

Appendix

Table S1: Common architecture and training settings for VLRLA-bridge

Hyperparameter	Value
Policy / embodiment	$\pi_{0.5}$ / ALOHA–AgileX
Visual inputs	Head RGB and two wrist RGB views
Frozen component	Vision–language backbone
Action dimension / chunk length	$14$ / $50$
Flow integration steps	$5$
Training sampler / noise coefficient	Flow-SDE / $0.3$

Appendix

Table S2: PPO fine-tuning configuration for

\pi_{0.5}

on Robotwin tasks.

Stage	Episodes	Successful	Success (%)
0	99	35	35.35
1	95	53	55.79
2	99	80	80.81
3	98	91	92.86
Total	391	259	66.24

Appendix

Table S3: Composition of the HITL dataset. Success denotes the binary human annotation associated with each recorded episode.

Hyperparameter	Value
Backbone	Frozen $\pi_{0.5}$
Trainable module	Temporal residual-KV bridge
Episodes per sequence	4 (one per stage)
Transitions per episode	12
Transitions per sequence	48
Action chunk length	20

Appendix

Table S4: Hyperparameters for fruit-placement tasks.

Dataset and representation
Parameter	Value
Successful source episodes	185: 93 green, 92 yellow
Adapter-training episodes	169: 85 green, 84 yellow
Held-out source episodes	16: 8 per color
Recording frequency	15 Hz
Episode length	180 action steps (12 s)

Appendix

Table S5: Dataset specification and adapter-training hyperparameters.

Test-Time Adaptation of VLA Models

Test-time adaptation of vision-language-action models helps robot policies adjust to new views, tasks, objects, and execution errors during deployment.

See all 57 papers

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2Reasoning for VLA Models ICL for Robotics
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)
Agentic-VLA: Efficient Online Adaptation for Vision-Language-Action Models
May 21, 2026Ruofan Jin, Zaixi ZhangAgentic RL VLM Adaptation
Scinetics.
Self-Adaptive VLA for Robust Robot Deployment
Sep 24, 2026Hongxin Zhang, Chunru Lin, Tsun-Hsuan Wang +2Robot Policy Learning Robot Policy Adaptation
University of Massachusetts Amherst · Genesis AI

Aug 6, 2026J/K move · Enter open · S save

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

Jiarui Yang, Wen Huang, Jiale Zhang, Maowei Hu+1

Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)

Reasoning for VLA Models ICL for Robotics Continual Learning for VLA Models Robotic Manipulation Vision-Language-Action Models VLMs for Robotics+4

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have become the dominant recipe for generalist manipulation, yet they are almost universally trained by behavior cloning: a policy imitates expert action chunks conditioned on a static image and a fixed instruction. A natural remedy is to inject explicit reasoning through textual chain-of-thought (CoT). We show, both empirically and analytically, that free-form textual CoT degrades low-level control: the reasoning it produces is ungrounded, its latency breaks closed-loop timing, and, crucially, the reasoning and action tokens are optimized against conflicting objectives so that the policy learns to narrate rather than to act. We argue that what a VLA needs is not the ability to generate language, but the ability to consume grounded language. To this end we introduce \ourmethod{}, a framework that endows a VLA with language competence through (i) in-context post-training, in which perceptual evidence is injected as structured context and the model is supervised only on actions, and (ii) an agentic tool-use interface, in which the policy queries open-vocabulary detectors, monocular depth, and a vision--language model to actively acquire task-relevant information. Rather than emitting a single templated caption, our data engine produces diverse, paraphrased, and evidence-conditioned spatial descriptions, so that the policy learns to interpret language it has never seen verbatim. Across the RoboCasa-GR1, SimplerEnv, and LIBERO simulation benchmarks, together with 8 real-world robot manipulation tasks, our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches under matched configurations.

Abstract

Test-Time Adaptation of VLA Models

Explore similar work

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

Agentic-VLA: Efficient Online Adaptation for Vision-Language-Action Models

Self-Adaptive VLA for Robust Robot Deployment