The Low-Rank Structure of VLA Reinforcement Learning

Minjae Oh; Yoonah Park; Jongwon Lim; Yohan Jo

cs.LGSep 28, 2026

The Low-Rank Structure of VLA Reinforcement Learning

Authors: Minjae Oh, Yoonah Park, Jongwon Lim, Yohan Jo

Organizations: Graduate School of Data Science, Seoul National University

Abstract

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including $π_{0.5}$ and GR00T~N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to $99.6\%$ ). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.

Figures & tables

Figure 1: Overview of the Timestep Module in

\pi_{0.5}

. The timestep embedding is transformed by a TimeMLP and AdaRMS to produce timestep-dependent scale, shift, and gate vectors that modulate each attention or FFN residual sublayer.

Figure 2: Timestep Modules receive dense but low-rank RL updates. (Left) Update density for BC and RL checkpoints trained on LIBERO-Spatial from the same $\pi_{0.5}$ policy. (Right) Effective update rank across $\pi_{0.5}$ BC and RL checkpoints.

Figure 3: AdaRMS updates specialize to denoising timesteps, largely changing the shift vector. (Left) Top three input singular directions of LIBERO-Spatial AdaRMS updates for BC and RL. Unlike standard BC, RL and discrete-timestep BC checkpoints peak near the denoising timesteps used during training. (Right) Absolute cosine similarity between the base AdaRMS output and the RL-induced change (RL $-$ base) in MetaWorld for the scale, shift, and gate vectors, respectively.

		Base Policy		RL Policy
Benchmark	Condition	F1	AUC	F1	AUC
LIBERO-Spatial	Ours	97.1	99.6	96.3	96.6
LIBERO-Spatial	Random	$58.0\pm 20.5$	$46.5\pm 4.0$	$75.2\pm 13.6$	$51.5\pm 3.9$
LIBERO-Object	Ours	98.2	98.6	97.3	97.4
LIBERO-Object	Random	$76.3\pm 10.2$	$53.4\pm 6.3$	$86.2\pm 15.8$	$47.2\pm 13.3$
LIBERO-Goal	Ours	77.5	98.9	74.9	68.0

Table 2: Linear probing of shift vectors in

\pi_{0.5}

. F1 and ROC-AUC (%) for probes applied to the base and RL policies. Random denotes probes trained with shuffled outcome labels.

Figure 4: Cross-task structure in $\pi_{0.5}$ on LIBERO-Spatial subtasks. (Top) Absolute cosine similarity between shift vectors. (Bottom) Correlation of cross-task transfer effects.

Checkpoint	RL	Random	Steered
LIBERO-Spatial	86.7	$87.2\pm 0.3$	90.0 $(+3.3)$
LIBERO-Object	94.7	$94.7\pm 0.0$	96.7 $(+2.0)$
LIBERO-Goal	90.7	$90.7\pm 0.0$	94.7 $(+4.0)$
ManiSkill	90.0	$88.4\pm 3.4$	92.0 $(+2.0)$
MetaWorld	66.7	$64.1\pm 1.8$	68.7 $(+2.0)$

Table 3:

\pi_{0.5}

adaptive steering results. Success rates (%) on the 30% test split for RL, random vector steering, and shift vector steering.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

	$\pi_{0.5}$	GR00T N1.5	GR00T N1.6	SmolVLA
Expert hidden size	1,024	1,536	1,536	720
Adaptive normalization	AdaRMSNorm	AdaLayerNorm	AdaLayerNorm	–
Modulation outputs	scale, shift, gate	scale, shift	scale, shift	–
Timestep parameters (M)	118.60	158.52	234.07	0.52
Action expert parameters (M)	430.10	1,068.81	1,418.62	99.88
Timestep share of action expert (%)	27.58	14.83	16.50	0.52

Appendix

Table 4: Architecture and size of Timestep Modules. For GR00T, the count includes the shared Time MLP, the adaptive normalization projections, and the timestep-input blocks of the embodiment-specific action encoders. For SmolVLA, it includes only the timestep-input block of the action–timestep MLP, not the full joint MLP.

$\bm{\pi_{0.5}}$
Update	Benchmark	Reference	Post-trained checkpoint
BC	LIBERO	OpenPI $\pi_{0.5}$ base	LIBERO few-shot SFT
BC	MetaWorld	OpenPI $\pi_{0.5}$ base	MetaWorld SFT
BC	ManiSkill	OpenPI $\pi_{0.5}$ base	ManiSkill-25Main SFT
BC	CALVIN	OpenPI $\pi_{0.5}$ base	CALVIN ABC-D SFT
BC	LIBERO	LIBERO few-shot SFT	LIBERO full-shot SFT

Appendix

Table 5: Public checkpoints and comparison references.

Optimization
	$\bm{\pi}_{0.5}$			GR00T N1.5			GR00T N1.6
Hyperparameter	Spatial	Object	Goal	Spatial	Object	Goal	Spatial
Max steps	150	120	150	25	5	5	5
Global batch size	2048	2048	2048	1024	768	1024	720
Update epochs	1	1	4	4	4	4	4
Actor learning rate	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$

Appendix

Table 6: In-house RL hyperparameters on LIBERO.

Hyperparameter	LIBERO
Optimizer steps	1800
Global batch size	2048
Learning rate	$5\times 10^{-6}$
Weight decay	0.01
Gradient clipping norm	1.0
Action horizon $H$	10

Appendix

Table 7: In-house BC hyperparameters for

\pi_{0.5}

Figure 5: Sensitivity to measurement thresholds. (Top) Update density across absolute-change thresholds. (Bottom) Effective update rank across explained-energy thresholds.

Figure 6: GR00T RL updates on LIBERO. (Left) Update density. (Right) Effective update rank. Parentheses indicate percentages of maximum rank.

Figure 7: SmolVLA updates on LIBERO. (Left) Spatial BC and RL update density. (Right) Effective update rank, including Object RL. Parentheses indicate percentages of maximum rank.

Model	Reference	RL checkpoint
DanceGRPO	FLUX.1-dev	xzyhku/flux_hpsv2.1_dancegrpo , checkpoint-300-0
Pref-GRPO	FLUX.1-dev	CodeGoat24/FLUX.1-dev-PrefGRPO
SeFi-Image	SeFi-Image-5B-Base-diffusers	SeFi-Image-5B-RL-diffusers
VideoRLVR	DarthZhu/VideoRLVR-Wan2.2-Base	DarthZhu/VideoRLVR-Wan2.2

Appendix

Table 8: Reference and RL checkpoint pairs for image and video parameter analyses.

Figure 8: RL updates in image and video models. (Left) Update-energy share. (Right) Mean per-matrix Rank95 and normalized rank.

Condition	Spatial	Object	Goal	ManiSkill	MetaWorld	CALVIN
	$\pi_{0.5}$
Base	85.0	95.4	82.8	42.2	42.8	61.8
Timestep Modules only	94.8	99.0	91.4	87.5	69.6	87.0
w/o Top Singular Directions	79.6	51.8	78.4	53.8	22.6	48.7
w/ Top Singular Directions Only	90.4	96.6	89.4	88.4	66.6	86.7

Appendix

Table 9: Top singular direction parameter replacement for

\pi_{0.5}

. Best results are bolded .

	GR00T N1.5			GR00T N1.6
Condition	Spatial	Object	Goal	Spatial
Base	49.4	62.4	51.4	75.6
Timestep Modules only	52.6	81.6	49.8	84.8
w/o Top Singular Directions	51.0	65.2	46.2	81.2
w/ Top Singular Directions Only	51.8	81.2	51.8	84.0

Appendix

Table 10: Top singular direction parameter replacement for GR00T. Best results are bolded .

LoRA target	0	10	20	30	40	50	60	70	80	90	100
Time MLP + AdaRMS	85.0	85.6	90.0	91.4	91.8	92.0	90.8	92.0	94.2	93.0	93.0
Attn + FFN	85.0	81.0	83.4	87.0	89.4	89.0	91.6	91.4	92.0	92.0	91.4

Appendix

Table 11: Targeted LoRA on LIBERO-Spatial with

\pi_{0.5}

. Success rates (%) over training steps with LoRA, rank=32. The higher success rate at each step is bolded .

Figure 10: Layerwise BC and RL AdaRMS input directions on MetaWorld (Layers 0–8).

Figure 11: Layerwise BC and RL AdaRMS input directions on MetaWorld (Layers 9–17).

Figure 12: Layerwise BC and RL AdaRMS input directions on CALVIN (Layers 0–8).

Figure 13: Layerwise BC and RL AdaRMS input directions on CALVIN (Layers 9–17).

Figure 14: Layerwise AdaRMS input directions for LIBERO-Spatial RL with three denoising steps.

Figure 15: Layerwise AdaRMS input directions for LIBERO-Spatial with RL five denoising steps.

Figure 16: Layerwise AdaRMS input directions for BC trained with five discrete timesteps (Layers 0–8).

Figure 17: Layerwise AdaRMS input directions for BC trained with five discrete timesteps (Layers 9–17).

Figure 18: Position-wise cosine similarity between each base output and its RL-induced change in GR00T checkpoints.

Figure 19: Position-wise cosine similarity between each base output and its RL-induced change in $\pi_{0.5}$ checkpoints.

		Base Policy		RL Policy
Benchmark	Method	F1	AUC	F1	AUC
LIBERO Spatial	Logistic	97.1	99.6	96.3	96.6
	Single-Position	93.6	99.0	85.7	78.0
	Random	$58.0\pm 20.5$	$46.5\pm 4.0$	$75.2\pm 13.6$	$51.5\pm 3.9$
LIBERO Object	Logistic	98.2	98.6	97.3	97.4
	Single-Position	97.8	98.2	97.3	94.7

Appendix

Table 12: Extended linear probing results with shift vectors in

\pi_{0.5}

. F1 and ROC-AUC (%) for probes applied to the base and RL policies. We use per-benchmark 70:30 train–test splits. Logistic denotes our

\ell_{2}

-regularized logistic regression probes. Single-Position uses a single sublayer–timestep shift vector, with the position selected by the best F1 on the training split. Random denotes probes trained on randomly shuffled outcome labels.

Model	Benchmark	Original (+)	Reverse ( $-$ )
Base	LIBERO Spatial	58.4	41.6
	LIBERO Object	62.7	37.3
	LIBERO Goal	43.2	56.8
	ManiSkill	44.6	55.4
	MetaWorld	52.4	47.6
	Average	52.6	47.4

Appendix

Table 13: Probe direction at single positions in

\pi_{0.5}

. Fraction of single positions where the original or reverse probe direction is predictive (%) across benchmarks and policies. Original indicates that larger projections are associated with success, whereas Reverse indicates the opposite.

Figure 20: Single-position probing results in $\pi_{0.5}$ base policies. Each sublayer–timestep direction is used individually as a probe. Red indicates that larger projections are associated with success (Original), while blue indicates the opposite (Reverse). Values denote F1 scores on the test split.

Figure 21: Single-position probing results in $\pi_{0.5}$ RL policies. Each sublayer–timestep direction is used individually as a probe. Red indicates that larger projections are associated with success (Original), while blue indicates the opposite (Reverse). Values denote F1 scores on the test split.

Figure 22: Episode-level probe trajectories across different $\pi_{0.5}$ checkpoints. Episode progress is normalized to $0$ – $100\%$ for each action horizon. Probe outputs from successful (orange) and failed (blue) episodes are interpolated over normalized progress, and we plot the mean prediction with $95\%$ confidence intervals.

Benchmark	Base	RL	Steering coefficient $\alpha$				Best Recovery
			0.5	1.0	1.5	2.0	(%)
LIBERO Goal	82.8	93.6	88.2 $(+5.4)$	84.4 $(+1.6)$	79.8 $(-3.0)$	73.2 $(-9.6)$	50.0
LIBERO Object	95.4	99.2	95.2 $(-0.2)$	97.6 $(+2.2)$	98.0 $(+2.6)$	98.8 $(+3.4)$	89.5
LIBERO Spatial	85.0	96.6	86.4 $(+1.4)$	92.0 $(+7.0)$	90.0 $(+5.0)$	90.2 $(+5.2)$	60.3
ManiSkill	42.2	89.1	48.4 $(+6.2)$	55.3 $(+13.1)$	57.5 $(+15.3)$	63.8 $(+21.6)$	46.1
MetaWorld	42.8	69.2	56.4 $(+13.6)$	59.4 $(+16.6)$	58.6 $(+15.8)$	57.2 $(+14.4)$	62.9

Appendix

Table 14: Base-policy steering across different strengths in

\pi_{0.5}

. Success rates (%) across steering coefficients

\alpha

, where

\alpha

scales the RL-induced shift vector before addition. Values in parentheses denote changes relative to the base policy. Best Recovery is the fraction of the RL improvement recovered by the best steering result.

Explore similar work

RL $^2$ -VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models
Jul 29, 2026Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer +4Inference-Time Steering Video Latents
1National University of Singapore · 3Singapore Technologies Engineering · University of Toronto
Learning from Hindsight for VLA Reinforcement Learning
Jul 10, 2026Iris Xu, Sunshine Jiang, John Marangola +2Hindsight
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab
What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models
May 13, 2026Yuanfang Peng, Jingjing Fu, Chuheng Zhang +6Reinforcement Fine-Tuning Model Fine-Tuning
1Hong Kong University of Science and Technology · 2Microsoft Research Asia · 3Zhejiang University

Jul 29, 2026, cs.ROJ/K move · Enter open · S save

RL $^2$ -VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer, William Wei Jie Teo+3

1National University of Singapore · 3Singapore Technologies Engineering · University of Toronto

Inference-Time Steering Video Latents

Open paper AskPDF arXiv

Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce

RL^2

, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight,

RL^2

activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks,

RL^2

improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing

RL^2

as a practical and modular steering framework for VLA deployment.

Abstract

Explore similar work

RL2^22-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Learning from Hindsight for VLA Reinforcement Learning

What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models

RL $^2$ -VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models