Controlling Action-Head Co-Adaptation to VLM Representation Anisotropy

Ruijie Zhang; Jiayin Zheng; Yequan Zhao; Liyan Tan; Ziyue Liu; Zhengyang Wang; Kunpeng Song; Nathan Zhao

cs.AINov 2, 2026

Controlling Action-Head Co-Adaptation to VLM Representation Anisotropy

Authors: Ruijie Zhang, Jiayin Zheng, Yequan Zhao, Liyan Tan, Ziyue Liu, Zhengyang Wang, Kunpeng Song, Nathan Zhao,

Organizations: University of California, Santa Barbara · Foundation Model Team, XPeng

Abstract

Modern learning systems increasingly combine pretrained models with downstream heads. Despite the growing importance of such modular designs, how the downstream model co-adapts to the geometry of pretrained representations remains relatively underexplored. We study this problem through vision-language-action models (VLAs), where a pretrained vision-language model (VLM) conditions an action model for control. We find that the action model progressively concentrates its sensitivity on high-variance dimensions of the VLM outputs. Through controlled experiments across different backbone-head models, we further show that counteracting this sensitivity concentration can be beneficial under representation--task mismatch settings. Motivated by these observations, we introduce Adaptive Interface Equalizer (AdaIE), a parameter-free mechanism that reweights pretrained representations according to the evolving downstream sensitivity during training. AdaIE improves VLA success rates on simulation benchmarks, with gains of up to $+5.28$ percentage points (pp) on RoboCasa-GR1 and $+1.68$ pp on RoboTwin 2.0, and achieves an average gain of $+19$ pp across five real-world manipulation tasks. It further generalizes to diffusion policies and world models, improving downstream performance in both settings. Our results highlight interface co-adaptation as an important optimization problem when coupling pretrained representations with downstream models.

Figures & tables

Figure 1: Action-head sensitivity progressively aligns with the anisotropic VLM representation during training. Top : per-channel variance of the VLM output dimensions at different training steps. Bottom : corresponding per-channel action-head sensitivity. While sensitivity is relatively uniform early in training, it progressively concentrates on high-variance VLM dimensions as training proceeds, revealing increasing alignment between VLM anisotropy and action-head sensitivity.

Aligned
Task	Dataset	Interface	Metric	Equalize	Baseline	Exploit
Depth	NYUv2	DINOv2-S $\rightarrow$ DPT	$\delta_{1}$ (%) $\uparrow$	93.4 $\pm$ 0.3	93.5 $\pm$ 0.1	93.2 $\pm$ 0.3
Classification	CIFAR-100	DeiT-S $\rightarrow$ Cls.	Acc. (%) $\uparrow$	90.0 $\pm$ 0.2	89.6 $\pm$ 0.4	89.8 $\pm$ 0.4
Mismatched
Diffusion Policy	Push-T	ResNet $\rightarrow$ DP	Score $\uparrow$	0.936 $\pm$ 0.003	0.922 $\pm$ 0.003	0.909 $\pm$ 0.009
VLA	RoboCasa	VLM $\rightarrow$ DiT	Succ. (%) $\uparrow$	22.9 $\pm$ 0.5	19.4 $\pm$ 0.6	16.2 $\pm$ 1.1

Table 1: Effect of sensitivity reweighting under aligned and mismatched interfaces.

Figure 2: Overview of Adaptive Interface Equalizer (AdaIE) on VLAs. AdaIE operates at the interface between the VLM and the action DiT. It estimates the action head’s per-channel sensitivity online using stochastic probes, maintains a running sensitivity estimate, and inversely rescales the VLM conditioning channels before they are consumed by the DiT.

Method	Backbone	Steps	Avg. (%)
$\pi_{0.5}$	–	–	37.00
GR00T-N1.6	–	–	47.60
StarVLA-GR00T	Qwen3-VL-4B	100K	47.80
StarVLA- $\pi$	Qwen3-VL-4B	100K	43.90
StarVLA-OFT	Qwen3-VL-4B	100K	48.80
GR00T	Qwen3-VL-2B	100K	42.36

Table 2: Average success rate on RoboCasa-GR1 (24 tasks). Results above the separator are reported by StarVLA ( Community, 2026 ) , while results below are our runs, evaluated with 60 rollouts per task. Detailed per-task results of our runs are provided in Appendix A.5 .

Method	Backbone	Steps	Success Rate (%)
GR00T	Cosmos-Reason2-2B	150K	66.92
GR00T + AdaIE	Cosmos-Reason2-2B	150K	68.60 (+1.68)

Table 3: Success rates on RoboTwin 2.0 (50 tasks). Both methods are trained for 150K optimization steps using identical training configurations. See per task comparison in the Appendix A.5 .

Figure 3: Real-world manipulation evaluation. For each task, we conduct 10 trials with the left hand and 10 trials with the right hand. See Table 4 for the detailed results.

In-Distribution (ID)
		Baseline		AdaIE
Task	Hand	Task Succ.	Interaction Succ.	Task Succ.	Interaction Succ.
Banana	Left	5/10	10/10	6/10	9/10
	Right	4/10	10/10	7/10	10/10
Press Bell	Left	6/10	9/10	5/10	10/10
	Right	2/10	9/10	4/10	8/10

Table 4: Real-robot evaluation results on in-distribution (ID) and out-of-distribution (OOD) tasks.

Push-T			Reacher
Method	Success (%)	Std.	Method	Success (%)	Std.
DINO-WM	71.4	7.0	DINO-WM	78.6	3.9
DINO-WM + AdaIE	73.2 (+1.8)	6.5	DINO-WM + AdaIE	82.8 (+4.2)	6.3

Table 6: Performance of DINO-WM and DINO-WM + AdaIE.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	GR1	RoboTwin 2.0
VLM backbone	Qwen3-VL-2B-Instruct	Cosmos-Reason2-2B
Action head	28-layer DiT, 1024-d	28-layer DiT, 1024-d
Training steps	100K	150K
Effective batch size	256	192
Optimizer	AdamW	AdamW
VLM learning rate	$1\times 10^{-5}$	$1\times 10^{-5}$

Appendix

Table 7: Training configurations for the GR1 and RoboTwin 2.0 experiments. Within each benchmark, the baseline and AdaIE models use identical architectures, data, and optimization settings, differing only by the insertion of AdaIE at the VLM–DiT interface (bottom block).

Figure 5: Representative examples from the real-robot training dataset.

Configuration	Baseline	AdaIE
VLM backbone	Qwen3-VL-2B-Instruct	Qwen3-VL-2B-Instruct
Action head	28-layer DiT, 1024-d	28-layer DiT, 1024-d
Training steps	32K	32K
Effective batch size	512	512
Optimizer	AdamW	AdamW
VLM learning rate	$1\times 10^{-5}$	$1\times 10^{-5}$

Appendix

Table 8: Training configurations for the real-robot baseline and AdaIE models. Both variants use identical model architectures, data, and optimization settings, differing only by the insertion of AdaIE at the VLM–DiT interface.

Configuration	LeWM	DINO-WM
Predictor	6-block AdaLN-Zero DiT, 192-d	6-layer ViT, 16 heads, MLP 2048 ( $\sim$ 19M)
Training loss	next-emb. MSE + $0.09\,$ SIGReg	latent next-state MSE (DINOv2 space)
Proprioception	unused	excluded (pixels + actions only)
Epochs	10	10
Batch size	128	32
Optimizer	AdamW	AdamW

Appendix

Table 9: Training configurations for the LeWM and DINO-WM experiments. We use pre-trained backbone for two world model, with Tiny-ViT for LeWM and DINOv2 for DINO-WM.

Configuration	DINOv2-S + DPT / NYUv2
Task / dataset	NYU Depth V2, BTS/Eigen split (24,231 train pairs, 654 test images)
Input	$480\times 640$ RGB, ImageNet mean/std
Resize / crop (train / eval)	short side to 518 (multiple of 14) + random $518\times 518$ crop / resize only
Depth target	16-bit PNG / 1000 (m), valid range $[0.001,10]$ m
Backbone	DINOv2 ViT-S/14, 12 blocks, width 384
Backbone init	DINOv2 self-supervised weights ( dinov2_vits14_pretrain )

Appendix

Table 10: Training configuration for the DINOv2-S + DPT experiments on NYUv2, following the official Depth Anything V2 metric-depth recipe (SiLog loss, AdamW, polynomial decay,

518\times 518

crops) with the train.py default of 40 epochs and a global batch of 16. The baseline and AdaIE models are identical except for the insertion of AdaIE at the encoder–decoder interface (bottom block). The standard deviation reflects variation across 3 random seed training in Table 1 .

Configuration	DeiT-S / CIFAR-100
Task / dataset	CIFAR-100 (49k train, 1k minival held out from train, 10k test)
Input	$32\times 32$ RGB, bicubic upsampling to $224\times 224$ , ImageNet mean/std
Backbone	DeiT-S ( deit_small_patch16_224 , timm), 12 blocks, width 384, patch 16
Backbone init	ImageNet-1k supervised DeiT weights
Classifier head	Linear $384\to 100$ , random init (trunc. normal, std 0.02; zero bias)
Head input	CLS token after final LayerNorm (pre-logits)

Appendix

Table 11: Training configuration for the DeiT-S experiments on CIFAR-100. The schedule follows the ViT fine-tuning budget (10k steps, batch 512) with DeiT AdamW defaults. The baseline and AdaIE models are identical except for the insertion of AdaIE at the backbone–classifier interface. The standard deviation reflects variation across the 5 random seed training in Table 1 .

Configuration	DiffusionPolicy-C
Task / dataset	PushT image ( pusht_cchi_v7_replay , 90 training episodes, 2% val)
Observation	$96\times 96$ RGB + 2-D agent position, $T_{o}=2$
Vision encoder	Pre-trained ResNet-18, GroupNorm, spatial-softmax pooling
Random crop (train / eval)	$84\times 84$ / fixed
Denoiser	1-D convolutional UNet (512, 1024, 2048)
Conditioning	FiLM (scale + shift), diffusion-step embed 128

Appendix

Table 12: Training configuration for the DiffusionPolicy-C experiments on Push-T, following the official Diffusion Policy recipe. The baseline and AdaIE use identical configurations except for the insertion of AdaIE at the observation–denoiser interface. Following the original evaluation protocol, we report the best checkpoint performance for each variant as the mean and standard deviation of test scores across three random seeds.

Task	GR00T-2B	GR00T-2B + AdaIE	GR00T-4B	GR00T-4B + AdaIE
PnPBottleToCabinetClose	50.00	56.67	65.00	48.33
PnPCanToDrawerClose	65.00	78.33	80.00	75.00
PnPCupToDrawerClose	23.33	23.33	23.33	15.00
PnPMilkToMicrowaveClose	53.33	51.67	60.00	60.00
PnPPotatoToMicrowaveClose	26.67	33.33	41.67	43.33
PnPWineToCabinetClose	48.33	48.33	51.67	51.67

Appendix

Table 13: Per-task success rates on RoboCasa-GR1 for our GR00T baselines and AdaIE. 60 rollouts per task.

Task	Clean		Randomized		Avg
	GR00T-2B	+ AdaIE	GR00T-2B	+ AdaIE	GR00T-2B	+ AdaIE
adjust_bottle	94.00	99.00	97.00	98.00	95.50	98.50
beat_block_hammer	62.00	76.00	74.00	74.00	68.00	75.00
blocks_ranking_rgb	65.00	71.00	68.00	64.00	66.50	67.50
blocks_ranking_size	60.00	47.00	54.00	46.00	57.00	46.50
click_alarmclock	100.00	99.00	100.00	100.00	100.00	99.50

Appendix

Table 14: Per-task success rates on RoboTwin 2.0 for the GR00T-2B baseline and AdaIE, evaluated with 100 rollouts per task per setup. Clean uses the standard environment, while Randomized introduces variations in backgrounds, lighting, distractors, tabletop height, and language instructions. Avg denotes the mean over both setups. AdaIE improves average performance primarily under Clean. The limited gain under Randomized may reflect additional perceptual and generalization demands that are not directly addressed by interface sensitivity reweighting.

Task	$\alpha$	Ep. 1	Ep. 2	Ep. 3	Ep. 4	Ep. 5	Ep. 6	Ep. 7	Ep. 8	Ep. 9	Ep. 10
Two Room	None	94	90	88	90	88	88	90	88	90	92
	0.05	94	94	94	90	94	94	92	94	92	90
	0.10	92	90	90	90	88	90	92	92	92	92
	0.15	94	94	92	88	92	90	90	92	94	92
	0.25	92	92	90	88	92	94	94	94	94	94
	0.35	92	90	88	88	92	94	94	92	96	94

Appendix

Table 15: Ablation of the AdaIE equalization strength

\alpha

on LeWM (seed 42). We report planning success rates (%) at each training epoch. None denotes the baseline without AdaIE. Each checkpoint is evaluated over 50 episodes using the same evaluation seed.

Figure 6: The co-adaption of DiffusionPolicy-C. The Policy is trained on Push-T.

Figure 7: The co-adaption of DiffusionPolicy-T.The Policy is trained on Push-T.

Figure 8: The co-adaption of DINO-WM.The model is trained on Two Room.

VLA Model Adaptation

See all 48 papers

Rethinking VLM Representation for VLA Initialization
May 25, 2026Weifeng Lin, Siyuan Huang, Hao Li +5VLM Adaptation Low-Rank Adaptation
1CUHK · 4ACE Robotics · 3Peking University +1
PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models
May 11, 2026Xinyu Guo, Bin Xie, Wei Chai +4VLA Model Adaptation Robot Policy Adaptation
Institute of Automation, Chinese Academy of Sciences · Dexmal · Nanjing University of Aeronautics and Astronautics +1
Behavioral Steering of Vision-Language-Action Models in the Low-Parameter Regime
Oct 31, 2026James Tcheng, Feiyu Gavin ZhuVLA Model Adaptation RL for VLMs
Carnegie Mellon University

May 25, 2026J/K move · Enter open · S save

Rethinking VLM Representation for VLA Initialization

Weifeng Lin, Siyuan Huang, Hao Li, Tingwei Chen+4

1CUHK · 4ACE Robotics · 3Peking University+1

VLM Adaptation Low-Rank Adaptation Vision-Language-Action Models Vision-Language Pretraining

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models widely adopt pretrained Vision-Language Models (VLMs) as policy backbones, yet it remains unclear what kind of pretrained VLM representation is useful as a VLA initialization. In this paper, we study VLA initialization as a controlled representation-design problem along three axes: capability-level embodied VQA supervision, parameter-update strategy, and robot-data pretraining. Our experiments show that the original pretrained VLM representation is a key source of action performance. However, embodied VQA adaptation does not yield uniform gains: its benefit depends on downstream bottlenecks, and gains from different capability domains are not simply additive. For update strategy, LoRA provides a more reliable initialization than Full Finetune, indicating that overly reshaping the pretrained representation can weaken VLA initialization. Robot-data pretraining further improves VLA initialization, with the strongest variant obtained by staged LoRA-based training. Together, these findings suggest that effective VLM-to-VLA adaptation should inject action-relevant embodied and robot-trajectory signals while preserving the pretrained VLM representation that remains useful for action learning.

Abstract

VLA Model Adaptation

Explore similar work

Rethinking VLM Representation for VLA Initialization

PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models

Behavioral Steering of Vision-Language-Action Models in the Low-Parameter Regime