POKE: Probe Offline, Keep Efficiently for Vision-Language-Action Models
Authors: Jaehyuk Lee, Yeajin Lee, Suhyun Bae, Donghun Lee
Organizations: Department of Mathematics, Korea University, Seoul, Republic of Korea · Program in Actuarial Science and Financial Engineering, Korea University, Seoul, Republic of Korea
Visual token pruning reduces the inference cost of Vision-Language-Action (VLA) models, but attention-based scores do not directly measure a token's effect on the predicted action. We define action sensitivity as the magnitude of the first-order action change when a token is replaced by its camera's mean token and estimate it for all tokens with a single backward pass. Computing it online is costly: scoring on the full (unpruned) graph needs an extra forward pass at every control step. Scoring on the pruned forward pass is cheaper but causes scoring-graph lock-in: removed tokens lose their direct paths through later layers and tend to receive lower scores, making those locations less likely to return. On replayed observations, even with a random conditioning mask, pruned-graph scoring re-selects 59% of the mask's tokens, versus 6% expected by chance; attention histories show a similar bias. POKE (Probe Offline, Keep Efficiently) computes full-graph action sensitivities offline and distills them into a student with under 0.3M parameters. At inference, the student scores the current observation without backpropagation or previous masks, and score-guided diversity selection retains the tokens. Keeping 32 of 512 visual tokens on LIBERO, POKE achieves 95.2% success on OpenVLA-OFT and 90.8% on π0.5 at 1.85× and 1.26× speedups, outperforming attention-, diversity-, and distillation-based pruners, with larger margins on LIBERO-Plus.
Figures & tables
Figure 1: Motivation on OpenVLA-OFT. (a) LIBERO-Goal, 64/512 tokens; rows: third-person/wrist; clear patches: retained. Attention misses the target plate kept by action sensitivity. (b) LIBERO-Long, B=32 , 9 closed-loop episodes (mean, IQR). Left: cumulative number of distinct tokens ever retained; pruned-graph scoring keeps returning to the same tokens. Right: share of score assigned to tokens outside the previous mask; 19% on the pruned graph vs. 77% on the full graph (dotted: uniform scores).
Figure 2: Method overview. (a) Offline training: a frozen full-graph VLA produces single-probe action-sensitivity labels, which are distilled into a lightweight student. (b) Inference: student predicts token scores, and MMDP selects diverse tokens from scoring candidates.
B=128
B=64
B=32
Method
Spa.
Obj.
Goal
Long
Avg.( ↑ )
× ( ↑ )
Spa.
Obj.
Goal
Long
Avg.( ↑ )
× ( ↑ )
Spa.
Obj.
Goal
Long
Avg.( ↑ )
× ( ↑ )
OpenVLA-OFT
Vanilla
98.6
97.2
97.6
93.4
96.70
1.00
98.6
97.2
97.6
93.4
96.70
1.00
98.6
97.2
97.6
93.4
96.70
1.00
Random
95.6
94.6
95.0
89.8
93.75
1.43
72.4
80.8
52.4
33.8
59.85
1.78
43.8
15.6
8.8
10.0
19.55
1.89
FastV
97.4
97.2
95.6
91.8
95.50
1.40
95.8
94.6
93.6
84.2
92.05
1.73
91.6
80.8
92.6
43.2
77.05
1.88
DivPrune
98.0
96.4
93.6
90.2
94.55
1.37
95.8
93.2
88.8
72.4
87.55
1.67
76.4
90.4
80.8
40.8
72.10
1.84
VLA-Cache
97.0
98.0
97.4
90.8
95.80
1.28
96.0
92.4
94.2
80.2
90.70
1.44
90.4
88.8
90.4
70.4
85.00
1.47
Table 1: LIBERO success rates (%) and speedups over Vanilla ( × ) with B retained visual tokens. † : our adaptation. Gray: online diagnostic probes (§ 4.5 ), excluded from bold along with Vanilla; on π0.5 their backward passes run in eager mode, so their speedups are against eager Vanilla (Table 26 ). POKE uses its B=32 checkpoints at all budgets; timing details in Appendix 13 .
Method
Spatial
Object
Goal
Long
Avg.( ↑ )
OpenVLA
Vanilla
84.6
84.0
74.0
51.4
73.50
FastV
61.2
74.2
58.8
36.4
57.65
VLA-Cache
67.4
71.0
60.2
34.2
58.20
VLA-Pruner
75.6
76.2
61.4
42.0
63.80
POKE (top- k )
72.8
81.2
70.0
44.2
67.05
POKE (MMDP)
78.8
82.2
72.4
53.6
71.75
Table 2: Additional policies and benchmark at B=32 of 256 visual tokens (success, %). Top: autoregressive OpenVLA on LIBERO. Bottom: open-pi-zero π0 (Bridge checkpoint) on SimplerEnv WidowX visual matching, 240 episodes per task; its student is trained on real BridgeData V2 frames. Details and full results appear in Appendices 10 and 11 .
Figure 3: Qualitative rollouts on OpenVLA-OFT Long task 6 ( B=32 ) from the same initial state, showing frames and the applied third-person mask. VLA-Pruner times out at step 520 (red); POKE (top- k ) succeeds at step 220 (green). Figure 4 adds more results.
Method
Camera
Robot
Lang.
Light
Backgr.
Noise
Layout
Total
OpenVLA-OFT
Vanilla (%)
66.4
41.0
80.7
91.9
89.6
74.4
83.7
74.4
FastV
0.56
0.51
0.74
0.63
0.73
0.58
0.66
0.64
DivPrune
0.46
0.35
0.54
0.47
0.56
0.38
0.54
0.48
VLA-Pruner
0.66
0.69
0.89
0.86
0.83
0.78
0.81
0.79
MAP †
0.72
0.87
0.85
0.89
0.95
0.69
0.85
0.82
POKE (top- k )
0.82
0.89
0.97
0.98
0.96
0.93
0.92
0.92
Table 3: LIBERO-Plus retention by perturbation category at B=32 on a stratified 25% subset (2,508 perturbed tasks, one episode each), pooled over the four suites. Vanilla: success (%); other rows: success relative to Vanilla. Policies and students are unchanged (trained on unperturbed demonstrations). Per-suite success appears in Appendix 12 .
OpenVLA-OFT
π0.5
Suite
Teacher
Top- k
MMDP
Top- k
MMDP
Object
Attention
93.4
96.0
82.8 ± 1.62
94.4 ± 2.02
Action sens.
96.8
97.6
86.0 ± 1.44
96.4 ± 2.84
Long
Attention
80.40 ± 2.38
83.20 ± 1.28
62.8 ± 0.48
71.8 ± 2.46
Action sens.
83.93 ± 0.81
87.53 ± 1.86
68.8 ± 0.98
76.4 ± 1.12
Table 4: Teacher supervision at B=32 (success, %). Only the teacher labels differ (Appendix 8 ). ± : mean ± SD over three training seeds, separate from the Table 1 checkpoint; other entries use the Table 1 checkpoint.
Success rate
Overlap
Method
Current-frame signal
pruned
full
pruned
full
VP: action attention only
–
3.4
69.4
0.995
0.664
VP: combined
Semantic attention
78.6
82.4
0.515
0.364
SpecPrune † : history only
–
4.8
84.4
1.000
0.431
SpecPrune † : combined
Local attention + frame difference
72.6
80.6
0.545
0.343
Action-sensitivity probe (top- k )
–
59.6
81.8
0.85
0.41
Table 5: History-graph interventions (OFT Long, B=32 , single runs). Pruned/full: graph used to compute history or probe scores; current-frame signals are computed before the cut; overlap: Jaccard. VP denotes VLA-Pruner; SpecPrune † uses its fixed-budget static stage (Appendix 9.5 ).
Long
Goal
Teacher
GPU-h
Top- k
MMDP
GPU-h
Top- k
MMDP
P=1
0.7
68.4
76.4
0.3
77.0
93.6
P=3
2.0
70.2
80.0
0.9
86.8
94.0
P=5
3.3
72.4
80.4
1.1
85.2
93.4
P=8
5.4
75.4
83.0
2.3
92.8
94.2
Online probe (full, top- k )
–
79.4
–
–
93.2
–
Table 6: Teacher probe count on π0.5 Long and Goal ( B=32 ; success %, 500 episodes). One checkpoint per P ; P=1 is from Table 1 . Label costs are GPU-hours; Long P=3,5 are scaled from measured P=8 . Inference cost is unchanged.
Spatial
Object
Goal
Long
Task
Top- k
MMDP
Top- k
MMDP
Top- k
MMDP
Top- k
MMDP
0
88
100
92
94
92
92
66
72
1
86
98
74
100
88
98
66
88
2
100
100
86
100
94
98
74
84
3
92
100
88
94
50
84
84
90
4
60
88
88
90
96
98
60
58
Table 11: Task-level POKE success rates (%) on π0.5 at B=32 . Each task is evaluated over 50 episodes. Top- k and MMDP denote the two POKE selection rules. Entries report single-checkpoint results, rather than averages over training seeds; task IDs are suite-specific.
Figure 4: All methods on OpenVLA-OFT Long task 6 ( B=32 ), from the same initial state as Figure 3 . Layout and borders follow Figure 3 .
Method
Carrot
Eggplant
Spoon
Stack
Pooled
FastV
19/24 (0.54)
9/40 ( 9.3×10−6 )
21/37 (0.048)
40/31 (0.34)
89/132 (0.0046)
POKE (top- k )
22/24 (0.88)
37/38 (1.00)
30/45 (0.11)
39/21 (0.027)
128/128 (1.00)
POKE (MMDP)
19/21 (0.87)
52/27 (0.0066)
35/24 (0.19)
46/19 (0.0011)
152/91 ( 1.1×10−4 )
Table 21: SimplerEnv paired comparison with VLA-Pruner ( π0 , B=32 ). Entries: episodes where only the method succeeds / only VLA-Pruner succeeds (two-sided exact McNemar p ), 240 episodes per task.
Method
Spatial
Object
Goal
Long
Avg.( ↑ )
Ret.( ↑ )
OpenVLA-OFT
Vanilla
82.7
83.8
64.0
67.8
74.6
1.00
FastV
77.0
33.2
44.4
36.0
47.7
0.64
DivPrune
53.3
27.8
42.6
19.4
35.8
0.48
VLA-Pruner
71.7
63.3
50.6
50.5
59.0
0.79
MAP †
71.8
63.2
59.3
50.8
61.3
0.82
POKE (top- k )
81.3
75.7
59.6
58.6
68.8
0.92
Table 22: LIBERO-Plus success (%) by suite at B=32 . Avg.: unweighted mean over suites; Ret.: Avg. relative to Vanilla. The Total column of Table 3 pools tasks across suites and can differ slightly.
Method
Dev./random ↓
Below random ↑
Tasks < VP
FastV
0.371
0.914
2/10
VLA-Pruner
0.332
0.938
–
POKE (top- k )
0.252
0.981
10/10
POKE (MMDP)
0.244
0.973
10/10
Table 36: Mask fidelity at shared observations (OFT Long, B=32 , 625 frames from Vanilla rollouts). Deviation is normalized by a random mask (lower is better); below random: fraction of frames with a smaller deviation than the random mask. VP’s temporal signal is updated along the shared trajectory. This measures mask quality on common reference states, not on each method’s own deployment states.
Human-Centered Artificial Intelligence Research Institute, Ewha Womans University · Division of Artificial Intelligence & Software, Ewha Womans University