cs.LGAug 19, 2026

Role-Conditioned Sub-Token Routing for Efficient Vision-Language-Action Policies

Authors: Wei Jiang, Wei Wang

Organizations: Futurewei Technologies San Jose, CA 95131, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-Language-Action (VLA) models process long multimodal token sequences, making inference expensive in both memory and computation. Existing efficiency methods mainly reduce visual tokens, but aggressive token pruning becomes fragile because removing a token discards its entire representation. Sub-token compression provides a complementary alternative by retaining more tokens while reducing their value width. However, directly applying sub-token compression to VLA policies is less effective because information important for perception, language understanding, and control is distributed differently across the multimodal representation. We introduce Role-Conditioned Sub-Token Routing (RoleSub), which learns how to compress the value representations of retained tokens. After visual token reduction, RoleSub partitions each retained value representation into groups in an orthogonal space and uses a lightweight router to determine which groups should be preserved. The routing decision is conditioned on the token representation, a learned latent role representation, and language context. The same mechanism can also be applied to language values, allowing visual and language representations to be compressed without removing additional tokens. We evaluate RoleSub on OpenVLA-OFT-7B across the four LIBERO suites. At matched visual-KV budgets, RoleSub outperforms a trained token-only control in 33 of 36 settings, with the largest gains under aggressive compression. Combining visual and language compression reduces total KV to 9.2--11.3% of the original while retaining strong control performance on most tasks. These results show that reducing the representation within retained tokens provides an effective complement to token pruning for aggressive VLA compression.

Figures & tables

Visual keepKvisK_{\mathrm{vis}}Visual KVMethodGoalLIBERO-10ObjectSpatial
8/51210.83%RoleSub93.574.295.595.6
   Token-only85.773.696.588.3
 20.88%RoleSub91.876.995.996.7
   Token-only88.767.795.296.8
 40.98%RoleSub93.471.895.895.3
   Token-only84.767.796.994.6
Table 1: Visual compression at matched KV budgets. RoleSub retains 8, 16, or 32 of the original 512 visual tokens and varies the mean retained value-group budget KvisK_{\mathrm{vis}} . The trained token-only pruning retains complete values but uses fewer visual tokens to match the same visual-KV budget.
Visual keepVisual KVMethodGoalLIBERO-10ObjectSpatial
8/5120.83%RoleSub93.574.295.595.6
  Plain89.459.894.696.2
16/5121.66%RoleSub95.383.996.597.0
  Plain93.283.497.295.2
32/5123.32%RoleSub97.288.496.497.8
  Plain94.885.896.697.8
Table 2: Comparison with plain sub-token routing at Kvis=1K_{\mathrm{vis}}=1 .
KlangK_{\mathrm{lang}}Language VLanguage KVGoalLIBERO-10ObjectSpatial
850.0%75.0%97.094.898.298.2
425.0%62.5%98.094.698.298.4
212.5%56.25%98.493.898.498.4
16.25%53.13%98.094.898.498.0
Dense100%100%98.894.897.898.2
Table 3: Language-only value routing with S=16S=16 value groups.
SuiteVisual keepCombined KVCombinedVisual-only KVVisual-onlyΔ\Delta
Goal8/5129.2%92.715.4%93.5−0.8-0.8
Goal16/5129.9%93.616.1%95.3−1.7-1.7
Goal32/51211.3%94.317.5%97.2−2.9-2.9
LIBERO-108/5129.2%64.215.4%74.2−10.0-10.0
LIBERO-1016/5129.9%72.716.1%83.9−11.2-11.2
LIBERO-1032/51211.3%77.617.5%88.4−10.8-10.8
Table 4: Combined visual and language compression with Kvis=Klang=1K_{\mathrm{vis}}=K_{\mathrm{lang}}=1 . “Visual-only” uses the same visual-token retention and Kvis=1K_{\mathrm{vis}}=1 , but keeps the language values uncompressed.
Language budget8 visual tokens16 visual tokens32 visual tokens
Klang=1K_{\mathrm{lang}}=164.272.777.6
Klang=8K_{\mathrm{lang}}=868.479.784.8
Klang=16K_{\mathrm{lang}}=1683.686.887.6
Table 5: LIBERO-10 success rate with different language value budgets. Kvis=1K_{\mathrm{vis}}=1 in all settings.
More in this topic

Efficient VLA Models

Efficient vision-language-action (VLA) models reduce the compute, memory, and latency needed to turn visual and language inputs into actions.

See all 98 papers

Explore similar work

CardsList
Oct 15, 2026

FastVLA: Task-Conditioned and Temporally Adaptive Visual Token Compression for Efficient Vision-Language-Action Models

Vision-Language-Action (VLA) models present a promising paradigm for general-purpose robot manipulation, but closed-loop deployment is often limited by inference efficiency: at each timestep, high-resolution, often multi-view images are encoded into hundreds of visual tokens that dominate transformer computation and latency. Token compression via attention is a natural solution but is challenging for VLA control because (1) decision-critical tokens are determined by the task, not generic visual saliency, and (2) the token budget varies substantially across steps, so static metrics and budgets waste computation or discard control-critical tokens. We introduce FastVLA, a training-free, task-aware, and temporally adaptive visual token compression framework. At each step, FastVLA scores token importance via task-to-vision attention and adaptively sets the compression rate from the importance-distribution uniformity, measured by an inverse-Simpson metric with online sliding-window calibration. We evaluate FastVLA on OpenVLA, OpenVLA-OFT, and CogACT across LIBERO and SIMPLER, and on a physical Franka Research 3 robot. In simulation, it reduces LLM-backbone FLOPs by 35-59% and action latency by 9-32% (a 1.09×\times-1.46×\times speedup) while maintaining task success, achieving a comparable or better efficiency-performance trade-off than existing plug-and-play token compression/pruning/reuse baselines. On the real robot, it cuts end-to-end latency by 20% (1.25×\times speedup) at comparable success. Code is available at https://github.com/jiuyixu25/FastVLA.

Jiuyi Xu, Chengyuan Qian, Zhengzhong Tu +2

Colorado School of Mines · Texas A&M University · Microsoft AI

Efficient VLA Model InferenceVisual Token CompressionVision-Language-Action ModelsVLMs for Robotics
AskPDF
Mar 11, 2026

Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models

Large Vision-Language Models (LVLMs) incur substantial inference costs due to the processing of a vast number of visual tokens. Existing methods typically struggle to model progressive visual token reduction as a multi-step decision process with sequential dependencies and often rely on hand-engineered scoring rules that lack adaptive optimization for complex reasoning trajectories. To overcome these limitations, we propose TPRL, a reinforcement learning framework that learns adaptive pruning trajectories through language-guided sequential optimization tied directly to end-task performance. We formulate visual token pruning as a sequential decision process with explicit state transitions and employ a self-supervised autoencoder to compress visual tokens into a compact state representation for efficient policy learning. The pruning policy is initialized through learning from demonstrations and subsequently fine-tuned using Proximal Policy Optimization (PPO) to jointly optimize task accuracy and computational efficiency. Our experimental results demonstrate that TPRL removes up to 66.7% of visual tokens and achieves up to a 54.2% reduction in FLOPs during inference while maintaining a near-lossless average accuracy drop of only 0.7%. Code is released at mypink{https://github.com/MagicVicCoder/TPRL}.

Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China

Efficient VLM InferenceVisual Token PruningVisual Token CompressionRL for VLMs
AskPDF
Mar 10, 2026

Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity

Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens. While prior work shows that a large fraction of visual tokens are redundant, existing compression methods struggle to balance importance preservation and information diversity. To address this, we propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline: (1) Principal Semantic Components Analysis (PSCA) for clustering tokens into semantically coherent groups, ensuring comprehensive concept coverage, and (2) Intra-group Non-Maximum Suppression (NMS) for pruning redundant tokens while preserving key representative tokens within each group. Additionally, PruneSID incorporates an information-aware dynamic compression ratio mechanism that optimizes token compression rates based on image complexity, enabling more effective average information preservation across diverse scenes. Extensive experiments demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT, outperforming prior methods by 2.5% with 7.8 ×\times faster prefilling speed compared to the original model. Our framework generalizes across diverse VLMs and both image and video modalities, showcasing strong cross-modal versatility. Code is available at https://github.com/ZhengyaoFang/PruneSID.

Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3

Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory

Visual Token PruningVisual Token CompressionEfficient VLM InferenceVideo Token Compression
AskPDF