cs.CVNov 16, 2026

POKE: Probe Offline, Keep Efficiently for Vision-Language-Action Models

Authors: Jaehyuk Lee, Yeajin Lee, Suhyun Bae, Donghun Lee

Organizations: Department of Mathematics, Korea University, Seoul, Republic of Korea · Program in Actuarial Science and Financial Engineering, Korea University, Seoul, Republic of Korea

Abstract

Visual token pruning reduces the inference cost of Vision-Language-Action (VLA) models, but attention-based scores do not directly measure a token's effect on the predicted action. We define action sensitivity as the magnitude of the first-order action change when a token is replaced by its camera's mean token and estimate it for all tokens with a single backward pass. Computing it online is costly: scoring on the full (unpruned) graph needs an extra forward pass at every control step. Scoring on the pruned forward pass is cheaper but causes scoring-graph lock-in: removed tokens lose their direct paths through later layers and tend to receive lower scores, making those locations less likely to return. On replayed observations, even with a random conditioning mask, pruned-graph scoring re-selects 59% of the mask's tokens, versus 6% expected by chance; attention histories show a similar bias. POKE (Probe Offline, Keep Efficiently) computes full-graph action sensitivities offline and distills them into a student with under 0.3M parameters. At inference, the student scores the current observation without backpropagation or previous masks, and score-guided diversity selection retains the tokens. Keeping 32 of 512 visual tokens on LIBERO, POKE achieves 95.2% success on OpenVLA-OFT and 90.8% on π0.5π_{0.5} at 1.85×1.85\times and 1.26×1.26\times speedups, outperforming attention-, diversity-, and distillation-based pruners, with larger margins on LIBERO-Plus.

Figures & tables

Explore similar work

CardsList
  1. ActPrune: Action-Grounded Visual Token Pruning for Efficient Vision-Language-Action Models

    Nov 6, 2026Sungwon Moon, Minji Kim, Hyemin Boo +2Efficient VLA Model InferenceVisual Token Pruning

  2. SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning

    Sep 6, 2025Hanzhen Wang, Jiaming Xu, Yushun Xiang +4Efficient VLA Model InferenceVision-Language-Action Models