Continually Evolving Skill Knowledge in Vision Language Action Model

Yuxuan Wu; Guangming Wang; Zhiheng Yang; Tianchen Deng; Maoqing Yao; Brian Sheil; Hesheng Wang

cs.RONov 22, 2025

Continually Evolving Skill Knowledge in Vision Language Action Model

Authors: Yuxuan Wu, Guangming Wang, Zhiheng Yang, Tianchen Deng, Maoqing Yao, Brian Sheil, Hesheng Wang

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · University of Cambridge · Beihang University ·

Abstract

Vision-language-action (VLA) models show promising knowledge accumulation ability through pretraining, yet continual learning remains challenging. Existing continual imitation learning (CIL) methods often require additional parameters or modules, limiting scalability for large VLA models. We propose Stellar VLA, a knowledge-driven CIL framework without parameter growth, featuring two variants: T-Stellar for task-centric modeling and TS-Stellar for hierarchical modeling of reusable skills across tasks. Stellar VLA jointly optimizes task representations and a self-evolving knowledge space, with knowledge-posterior-guided expert routing using knowledge relations and Top-K semantic embeddings for task specialization. Experiments on LIBERO demonstrate strong performance against VLA and CIL baselines with only 1% replay. Dual-arm real-world experiments validate knowledge transfer across embodiments and scenes. TS-Stellar excels in hierarchical manipulation, while visualizations demonstrate knowledge retention and task discovery. Project Website: https://stellarvla.github.io/

Figures & tables

Figure 1: Illustration of Stellar VLA. T-Stellar and TS-Stellar outperform UniVLA [ 1 ] and $\pi_{0.5}$ [ 2 ] in final average success rates across simulation and real-world dual-arm experiments.

Figure 2: Overall architecture of Stellar VLA. CLIP [ 16 ] and FiLM [ 17 ] -conditioned ResNet encode language and visual inputs respectively. Task-centric representation $z$ and knowledge space are jointly learned through knowledge update and latents aggregation, as detailed in Section 3.3 . The learned knowledge prior guides the MoE action head for motion prediction, as detailed in Section 3.4 .

Figure 3: Knowledge-prior-routed MoE action head. Two knowledge embeddings, relation and top-K semantic, are computed for expert routing, alongside language, noise, observation and noise action tokens fed into the denoising transformer.

tasks

metrics

Scratch VLA Models

Pretrained VLA Models (on cross-embodiment data)

MoDE [ 14 ]

UniVLA [ 1 ]

\pi_{0}

[ 6 ]

T-Stellar

TS-Stellar

MoDE [ 14 ]

UniVLA [ 1 ]

\pi_{0}

[ 6 ]

\pi_{0.5}

[ 2 ]

T-Stellar

TS-Stellar

LIBERO-goal

FWT (↑)

71.0

\pm

2.1

3.1

\pm

0.7

82.9

\pm

3.9

81.4

\pm

0.2

80.2

\pm

2.3

69.7

\pm

1.8

87.5

\pm

0.1

74.9

\pm

2.7

84.5

\pm

2.5

76.6

\pm

3.4

73.6

\pm

4.1

NBT (↓)

36.2

\pm

2.8

3.1

\pm

0.7

49.0

\pm

1.0

20.7

\pm

2.8

22.2

\pm

3.3

23.2

\pm

2.9

65.7

\pm

6.1

25.6

\pm

1.0

38.4

\pm

3.7

19.1

\pm

3.6

28.0

\pm

3.7

AUC (↑)

39.7

\pm

1.0

0.4

\pm

0.2

45.5

\pm

3.1

61.7

\pm

2.2

60.7

\pm

1.2

46.8

\pm

2.9

35.2

\pm

4.7

55.6

\pm

3.3

55.4

\pm

4.9

57.5

\pm

1.1

48.1

\pm

2.2

Final SR (↑)

33.1

\pm

2.4

0.0

\pm

0.0

35.7

\pm

0.5

67.9

\pm

0.5

64.2

\pm

1.5

41.6

\pm

4.8

20.5

\pm

4.7

41.1

\pm

1.1

43.8

\pm

3.8

57.0

\pm

2.0

44.1

\pm

5.2

LIBERO-long

FWT (↑)

71.6

\pm

3.6

6.1

\pm

0.1

57.3

\pm

3.1

76.3

\pm

4.0

75.5

\pm

0.6

71.1

\pm

1.5

70.3

\pm

5.6

60.5

\pm

2.6

55.5

\pm

3.3

80.3

\pm

1.6

81.9

\pm

4.1

Table 1: Overall performance on the LIBERO benchmark. Baselines are grouped into pretrained and scratch categories. Results are reported in percentages. The highest performance is highlighted in bold and the second highest is underline .

tasks	metrics	ER [ 8 ]	MoDE [ 14 ]	UniVLA [ 1 ]	$\pi_{0}$ [ 6 ]	$\pi_{0.5}$ [ 2 ]	T-Stellar	TS-Stellar
Dual arm real world task	FWT (↑)	97.1	15.7	70.0	98.6	95.7	98.6	98.6
	NBT (↓)	21.9	3.2	37.1	34.6	25.8	12.4	7.4
	AUC (↑)	79.9	13.0	43.6	72.6	75.6	89.6	93.4
	Final SR (↑)	70.0	10.0	21.4	57.1	72.9	84.3	90.0

Table 2: Performance on real-world experiments. The highest performance is highlighted in bold and the second highest is underline .

Figure 4: Success rate curves of real-world experiments and behavior visualization on “Handover Toy” after training on “Pull Stick from bag”. Stellar VLAs maintain high overall success rates, while UniVLA [ 1 ] and the $\pi$ series [ 6 , 2 ] exhibit clear forgetting during continual learning. MoDE [ 14 ] , lacking adaptation to dual-arm robots, fails to perform most tasks.

Table 7

Table 8

Figure 5: T-SNE visualization of Stellar VLA Task latents after 1, 4, 6, 8, and 10 tasks on LIBERO-long are shown. Task names are abbreviated for clarity. T-Stellar models discrete task distributions, and TS-Stellar learn relevant skill across tasks, while other VLAs may mix the task modeling.

TV distance $\downarrow$	Task 3-Sub1	Task 3-Sub2
Task 9-Sub2	0.357	0.132
Task 9-Sub3	0.498	0.319

Table 7: Cross-task subskill similarity.

Continual Learning for VLA Models

Continual learning for vision-language-action (VLA) models studies how robots acquire new skills without losing previously learned behaviors.

See all 23 papers

StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models
Aug 12, 2026Siyu Xu, Yunke Wang, Zijian Wang +6Reasoning for VLA Models Continual Learning for VLA Models
StellarEdge AI Technical Team
PHASER: Phase-Aware and Semantic Experience Replay for Vision-Language-Action Models
Jun 2, 2026Ziyang Chen, Shaoguang Wang, Weiyu Guo +5Continual Learning for VLA Models Robotic Manipulation
Thrust of AI, HKUST(Guangzhou), Guangzhou, China · AI2 Robotics, Shenzhen, China
Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning
Mar 12, 2026Jiaheng Hu, Jay Shim, Chen Tang +4Continual Learning for VLA Models VLA Model Adaptation
UT Austin · UCLA · NTU +1

Aug 12, 2026J/K move · Enter open · S save

StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models

Siyu Xu, Yunke Wang, Zijian Wang, Dihao Zhu+5

StellarEdge AI Technical Team

Reasoning for VLA Models Continual Learning for VLA Models Robotic Manipulation Instruction Generalization in VLA Models Robustness of VLA Models Vision-Language-Action Models+3

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models can follow instructions and manipulate objects, but their performance often collapses out of distribution (OOD), when the scene, viewpoint, or object differs from training. Adapting to each new situation typically requires collecting more data and fine-tuning. We present StellaVLA, a framework that instead adapts at test time by conditioning on a single retrieved demonstration. The key idea is to move beyond imitating what an expert did and instead convey why: an automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost. Provided as in-context guidance, this structured demonstration lets the policy reason about the task rather than mimic a pixel trajectory, which also makes it transferable across embodiments (real-robot, human-hand, or XR demonstrations). A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone, preserving real-time, high-frequency control with no added latency. On the VLA-Arena leaderboard(Aug 1, 2026), StellaVLA ranks first with an overall score of 0.63, versus 0.44 and 0.22 for the strong prior models (

π_{0.5}

and LingBot-VLA), and it further leads on LIBERO with 98.8% average success rate and LIBERO-Plus with 85.1% success rate. Our real-robot benchmark demonstrates that StellaVLA can use both human/robot demos and human-to-robot (XR) demos as in-context structured demonstration to help VLA model adapt to OOD tasks.

Abstract

Continual Learning for VLA Models

Explore similar work

StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models

PHASER: Phase-Aware and Semantic Experience Replay for Vision-Language-Action Models

Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning