Vision-Language-Action (VLA) models excel in robotic manipulation but suffer from significant inference latency due to processing dense visual tokens. Existing token reduction methods predominantly rely on attention magnitude as a static selection. In this work, we challenge this assumption, revealing that high-attention tokens are task-dependent and can even degrade policy performance. To address this, we introduce TIES (Tau-guided Inter-layer Efficient Selection), a dynamic framework guided by inter-layer token ranking consistency. By adaptively balancing attention magnitude with ranking consistency, TIES ensures robust token selection without requiring additional training. On the CogACT + SIMPLER benchmark, TIES improves average success rates by 6% while reducing token usage by 78%, and demonstrate strong generalization across diverse decoders and benchmarks.
Figures & tables
Figure 1: Informative vs. misleading high-attention tokens. Patches with high attention magnitude are unshaded, with the top 5 highlighted in red. (a) Successful case: high-attention tokens effectively localize task-relevant information. (b) Failure case: high-attention tokens focus on spurious features, leading to policy errors. Our approach introduces a consistency-based indicator to distinguish these scenarios and adaptively select tokens.
Figure 2: TIES framework. TIES dynamically computes the Kendall τ and adaptively decides the token selection strategy.
Figure 3: Performance of Top - k and Bottom - k strategies . We observe performance inversion in the Drawer task while conventional attention importance holds in the MoveNear task
Figure 4: Entropy distribution.
Figure 5: Kendall τ distribution.
SIMPLER
Method
PickCan
MoveNear
Drawer
DrawerApple
Average
Visual Matching
CogACT
91.7
83.8
72.7
42.6
72.7
FastV
92.6
81.4
69.8
52.4
74.1
VLA-Cache
92.0
83.3
70.5
51.6
74.4
EfficientVLA
94.7
82.4
69.8
55.4
75.5
Soft-TIES
93.3
79.6
76.9
57.4
76.8
Hard-TIES
91.3
79.2
77.1
64.8
78.1
Table 1: Performance of TIES compared with other baselines on the SIMPLER environment. All pruning methods are evaluated with 56 retained tokens.
Figure 6: Impact of pruning ratio.
Figure 7: Comparisons of alternative strategies. All strategies yield consistent performance gains.
SIMPLER VM
SIMPLER VA
LIBERO
Full tokens
34.30
39.30
96.90
FastV
27.99
31.47
92.70
TIES
28.31
39.44
94.30
Table 2: Average success rate comparison across benchmarks. All pruning methods are evaluated with 56 retained tokens.
Institute of Software, Chinese Academy of Sciences, Beijing, China. · University of Chinese Academy of Sciences, Beijing, China. · LimX Dynamic, Shenzhen, China. +3