Role-Conditioned Sub-Token Routing for Efficient Vision-Language-Action Policies

Wei Jiang; Wei Wang

cs.LGAug 19, 2026

Role-Conditioned Sub-Token Routing for Efficient Vision-Language-Action Policies

Authors: Wei Jiang, Wei Wang

Organizations: Futurewei Technologies San Jose, CA 95131, USA

Abstract

Vision-Language-Action (VLA) models process long multimodal token sequences, making inference expensive in both memory and computation. Existing efficiency methods mainly reduce visual tokens, but aggressive token pruning becomes fragile because removing a token discards its entire representation. Sub-token compression provides a complementary alternative by retaining more tokens while reducing their value width. However, directly applying sub-token compression to VLA policies is less effective because information important for perception, language understanding, and control is distributed differently across the multimodal representation. We introduce Role-Conditioned Sub-Token Routing (RoleSub), which learns how to compress the value representations of retained tokens. After visual token reduction, RoleSub partitions each retained value representation into groups in an orthogonal space and uses a lightweight router to determine which groups should be preserved. The routing decision is conditioned on the token representation, a learned latent role representation, and language context. The same mechanism can also be applied to language values, allowing visual and language representations to be compressed without removing additional tokens. We evaluate RoleSub on OpenVLA-OFT-7B across the four LIBERO suites. At matched visual-KV budgets, RoleSub outperforms a trained token-only control in 33 of 36 settings, with the largest gains under aggressive compression. Combining visual and language compression reduces total KV to 9.2--11.3% of the original while retaining strong control performance on most tasks. These results show that reducing the representation within retained tokens provides an effective complement to token pruning for aggressive VLA compression.

Figures & tables

Visual keep	$K_{\mathrm{vis}}$	Visual KV	Method	Goal	LIBERO-10	Object	Spatial
8/512	1	0.83%	RoleSub	93.5	74.2	95.5	95.6
			Token-only	85.7	73.6	96.5	88.3
	2	0.88%	RoleSub	91.8	76.9	95.9	96.7
			Token-only	88.7	67.7	95.2	96.8
	4	0.98%	RoleSub	93.4	71.8	95.8	95.3
			Token-only	84.7	67.7	96.9	94.6

Table 1: Visual compression at matched KV budgets. RoleSub retains 8, 16, or 32 of the original 512 visual tokens and varies the mean retained value-group budget

K_{\mathrm{vis}}

. The trained token-only pruning retains complete values but uses fewer visual tokens to match the same visual-KV budget.

Visual keep	Visual KV	Method	Goal	LIBERO-10	Object	Spatial
8/512	0.83%	RoleSub	93.5	74.2	95.5	95.6
		Plain	89.4	59.8	94.6	96.2
16/512	1.66%	RoleSub	95.3	83.9	96.5	97.0
		Plain	93.2	83.4	97.2	95.2
32/512	3.32%	RoleSub	97.2	88.4	96.4	97.8
		Plain	94.8	85.8	96.6	97.8

Table 2: Comparison with plain sub-token routing at

K_{\mathrm{vis}}=1

$K_{\mathrm{lang}}$	Language V	Language KV	Goal	LIBERO-10	Object	Spatial
8	50.0%	75.0%	97.0	94.8	98.2	98.2
4	25.0%	62.5%	98.0	94.6	98.2	98.4
2	12.5%	56.25%	98.4	93.8	98.4	98.4
1	6.25%	53.13%	98.0	94.8	98.4	98.0
Dense	100%	100%	98.8	94.8	97.8	98.2

Table 3: Language-only value routing with

S=16

value groups.

Suite	Visual keep	Combined KV	Combined	Visual-only KV	Visual-only	$\Delta$
Goal	8/512	9.2%	92.7	15.4%	93.5	$-0.8$
Goal	16/512	9.9%	93.6	16.1%	95.3	$-1.7$
Goal	32/512	11.3%	94.3	17.5%	97.2	$-2.9$
LIBERO-10	8/512	9.2%	64.2	15.4%	74.2	$-10.0$
LIBERO-10	16/512	9.9%	72.7	16.1%	83.9	$-11.2$
LIBERO-10	32/512	11.3%	77.6	17.5%	88.4	$-10.8$

Table 4: Combined visual and language compression with

K_{\mathrm{vis}}=K_{\mathrm{lang}}=1

. “Visual-only” uses the same visual-token retention and

K_{\mathrm{vis}}=1

, but keeps the language values uncompressed.

Language budget	8 visual tokens	16 visual tokens	32 visual tokens
$K_{\mathrm{lang}}=1$	64.2	72.7	77.6
$K_{\mathrm{lang}}=8$	68.4	79.7	84.8
$K_{\mathrm{lang}}=16$	83.6	86.8	87.6

Table 5: LIBERO-10 success rate with different language value budgets.

K_{\mathrm{vis}}=1

in all settings.

Efficient VLA Models

Efficient vision-language-action (VLA) models reduce the compute, memory, and latency needed to turn visual and language inputs into actions.

See all 98 papers

FastVLA: Task-Conditioned and Temporally Adaptive Visual Token Compression for Efficient Vision-Language-Action Models
Oct 15, 2026Jiuyi Xu, Chengyuan Qian, Zhengzhong Tu +2Efficient VLA Model Inference Visual Token Compression
Colorado School of Mines · Texas A&M University · Microsoft AI
Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models
Mar 11, 2026Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7Efficient VLM Inference Visual Token Pruning
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China
Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity
Mar 10, 2026Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3Visual Token Pruning Visual Token Compression
Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory

Oct 15, 2026J/K move · Enter open · S save

FastVLA: Task-Conditioned and Temporally Adaptive Visual Token Compression for Efficient Vision-Language-Action Models

Jiuyi Xu, Chengyuan Qian, Zhengzhong Tu, Yang Sui+1

Colorado School of Mines · Texas A&M University · Microsoft AI

Efficient VLA Model Inference Visual Token Compression Vision-Language-Action Models VLMs for Robotics

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models present a promising paradigm for general-purpose robot manipulation, but closed-loop deployment is often limited by inference efficiency: at each timestep, high-resolution, often multi-view images are encoded into hundreds of visual tokens that dominate transformer computation and latency. Token compression via attention is a natural solution but is challenging for VLA control because (1) decision-critical tokens are determined by the task, not generic visual saliency, and (2) the token budget varies substantially across steps, so static metrics and budgets waste computation or discard control-critical tokens. We introduce FastVLA, a training-free, task-aware, and temporally adaptive visual token compression framework. At each step, FastVLA scores token importance via task-to-vision attention and adaptively sets the compression rate from the importance-distribution uniformity, measured by an inverse-Simpson metric with online sliding-window calibration. We evaluate FastVLA on OpenVLA, OpenVLA-OFT, and CogACT across LIBERO and SIMPLER, and on a physical Franka Research 3 robot. In simulation, it reduces LLM-backbone FLOPs by 35-59% and action latency by 9-32% (a 1.09

\times

-1.46

\times

speedup) while maintaining task success, achieving a comparable or better efficiency-performance trade-off than existing plug-and-play token compression/pruning/reuse baselines. On the real robot, it cuts end-to-end latency by 20% (1.25

\times

speedup) at comparable success. Code is available at https://github.com/jiuyixu25/FastVLA.

Abstract

Efficient VLA Models

Explore similar work

FastVLA: Task-Conditioned and Temporally Adaptive Visual Token Compression for Efficient Vision-Language-Action Models

Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity