Training-Free Interaction-Aligned Visual Token Pruning for Efficient Embodied Manipulation

Jintao Cheng; Weibin Li; Haozhe Wang; Gang Wang; Yipu Zhang; Xiaoyu Tang; Jin Wu; Xieyuanli Chen

cs.CVMar 24, 2026

Training-Free Interaction-Aligned Visual Token Pruning for Efficient Embodied Manipulation

Authors: Jintao Cheng, Weibin Li, Haozhe Wang, Gang Wang, Yipu Zhang, Xiaoyu Tang, Jin Wu, Xieyuanli Chen,

Organizations: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China · University of Macau, Macau, China · School of Data Science and Engineering, Xingzhi College, South China Normal University, Guangzhou, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China ·

Abstract

Efficient visual representation is a central image-processing challenge in embodied manipulation, where policies repeatedly process dense visual-token sequences during closed-loop control. Existing methods rank or prune tokens using semantic relevance, VLM attention, cross-frame redundancy, or motion in the action space. These signals may discard task-relevant regions when instruction-related appearance and observed image motion are not yet spatially aligned. We introduce Interaction-Aligned Pruning (IAprune), a training-free method that treats per-frame budget setting and within-budget token selection as two linked decisions. Semantic--motion spatial agreement guides the choice between Conservative and Aggressive coverage, and the resulting region size is mapped to a calibrated dynamic budget. Continuous semantic and motion responses rank the existing tokens, while geometric residual correction redirects fixed selection slots toward under-represented boundaries without increasing the sequence length. Across four embodied manipulation policies, three simulation benchmarks, and a real-robot platform, IAprune provides a favorable accuracy--efficiency trade-off, matching the unpruned policy on LIBERO with a $1.54\times$ speedup and reaching $1.48\times$ acceleration on a real robot. Phase-wise analysis shows that retention gains are largest under tight budgets early in an episode, while fixed-budget analysis confirms that geometry replaces low-priority tokens with boundary and contact-region evidence rather than retaining more tokens. Our project website is: IAprune.com.

Figures & tables

Fig. 1: Comparison of Perception-First vs. Interaction-First token pruning paradigms. Perception-First baselines (Top) prematurely lose the manipulation target due to early semantic misalignment—a vulnerability that simple temporal stacking fails to resolve without explicit interaction modeling. In contrast, IAprune (Bottom) combines IoU-aware conservative-to-aggressive scheduling with budget-preserving structural reallocation, preserving the physical target while reducing redundant visual computation.

Fig. 2: Overview of IAprune. Semantic and motion evidence serve two complementary roles: their spatial agreement selects a Conservative or Aggressive coverage mode, while their continuous responses define the base relevance of each visual token. The selected mode defines a provisional region, whose size is converted into a dynamic per-frame budget $K_{t}$ under a target average retention ratio $r_{\mathrm{tar}}$ . Geometric evidence then performs budget-preserving structural reallocation through residual correction without changing $K_{t}$ . The resulting compact visual sequence is fed into the unchanged embodied manipulation policy for action generation.

Fig. 3: Overview of the evaluation benchmarks and tasks. The simulation evaluation includes LIBERO, VLABench, and CALVIN ABC-D. The real-world evaluation covers simple, long-horizon, and dual-arm manipulation.

Fig. 4: Physical platform for real-world evaluation. The head-mounted primary camera and wrist cameras provide complementary observations to the embodied manipulation policy, while the external RealSense camera is used only to record task execution. The shared workspace supports both single-arm manipulation and coordinated dual-arm operation.

Robot Platform	Self-developed dual-arm system; 2 $\times$ 7-DoF arms + 1-DoF parallel-jaw gripper; fixed base
Global View	ZED 2i binocular camera mounted on the head; RGB stream from the left lens only
Wrist View	Wide-angle RGB wrist camera per arm; single-arm uses head + active wrist; dual-arm uses head + both wrists
Data Spec.	All observations standardized to 640 $\times$ 480; Intel RealSense D455 used only for external recording
OS / Hardware	Ubuntu 22.04 workstation with a single NVIDIA A100 GPU
Backbone	$\pi_{0.5}$
Inference	Action chunk size 50; first 20 actions of each predicted chunk are executed

TABLE I: Real-Robot Deployment Details. Hardware and inference configuration used for physical experiments.

Method	DreamVLA										OpenPi $(\pi_{0},\pi_{0.5})$
	LIBERO Success (%)				CALVIN Seq. Length $\uparrow$						LIBERO Success (%)				VLABench Success (%)
	Spa.	Obj.	Goal	Long	1	2	3	4	5	Avg.	Spa.	Obj.	Goal	Long	Con.	Paint.	Poker.	Avg.
Vanilla Baseline: 100% Token Retention
Vanilla	84.5	91.5	89.5	89.5	97.8	94.5	89.5	83.8	77.5	4.43	94.2	98.2	94.6	82.8	56.0	30.0	52.0	46.0
Retention: 70%
FastV	86.5	86.5	84.0	87.5	97.6	94.5	89.0	82.8	76.8	4.40	93.5	96.1	92.2	80.5	44.0	28.0	32.0	34.6

TABLE II: Comprehensive Performance Comparison. Success rates are reported on LIBERO and VLABench, and average sequence length is reported on CALVIN at nominal retention labels of 30%, 50%, and 70%. For IAprune, these labels denote target average retention ratios. Baseline labels follow their original budget definitions and may not correspond to identical realized token counts.

Embodied Manipulation Policies
Method	CKPT	Success Rate (%)					Speedup
Method	CKPT	Spa.	Obj.	Goal	Long	Avg.	Speedup
OpenVLA [ 2 ]	OpenVLA (7B)	84.7	88.4	79.2	53.7	76.5	-
WorldVLA [ 4 ]	Cham. (7B)	87.6	96.2	83.4	60.0	81.8	-
NORA [ 42 ]	Qwen-VL (3B)	85.6	87.8	77.0	45.0	73.9	-
SmolVLA [ 43 ]	SmolVLM (2.2B)	93.0	94.0	91.0	77.0	88.8	-

TABLE III: LIBERO results for embodied manipulation policies and visual-token pruning methods. Representative policies are included for context. All pruning methods use the OFT 7B backbone, and speedup is reported relative to the unpruned OpenVLA-OFT policy.

Ret.	Method	Early segment			Late segment
Ret.	Method	TTR $\uparrow$	BTR $\uparrow$	CZR $\uparrow$	TTR $\uparrow$	BTR $\uparrow$	CZR $\uparrow$
30%	SparseVLM	0.016	0.049	0.100	0.077	0.083	0.125
	VLA-Cache	0.267	0.271	0.312	0.304	0.395	0.356
	DivPrune	0.374	0.311	0.313	0.444	0.415	0.400
	IAprune	0.733	0.672	0.710	0.543	0.507	0.565
50%	SparseVLM	0.207	0.334	0.571	0.273	0.391	0.545

TABLE IV: Segmentation-grounded token retention on VLABench. TTR, BTR, and CZR measure the retained fractions of target-object, target-boundary, and contact-zone tokens. The early segment is the first 30% of each episode, and the late segment is the remaining 70%.

Fig. 5: Visualization of interaction-aligned pruning on LIBERO. Bottom: visual-token retention changes from Conservative to Aggressive coverage. Middle: the alignment score (IoU) selects the coverage mode. Top: the overlap (purple) between the semantic (red) and observed-motion (blue) masks determines the alignment score.

Fig. 6: Effect of budget-preserving structural reallocation. Under the same per-frame Top- $K_{t}$ budget, geometry-guided residual correction retains more object-boundary tokens near potential contact regions.

Token type	Available	Semantic only	+Sobel	Avg. change
Object body	22	21.3 (97%)	22.0 (100%)	$+0.7$
Semantic drift	18	14.5 (81%)	0 6.8 (38%)	$-7.7$
Object edge	26	0 7.9 (30%)	24.1 (93%)	$\mathbf{+16.2}$
Noise edge	38	0 2.6 (7%)	0 6.4 (17%)	$+3.8$
Other/background	152	30.7 (20%)	17.7 (12%)	$-13.0$
Top- $K$ total	256 (100%)	77.0 (keep30%)	77.0 (keep30%)	–

TABLE V: Budget-preserving structural reallocation. Geometry-guided residual correction reallocates a fixed budget toward object edges near potential contact regions. Both variants use

K_{t}=77

OpenVLA-OFT
Budget	Sem.	+Mot.	+Edge	+M+E	Full IAP	w/o Edge
30%	91.0	90.7 (-0.3)	91.8 (+0.8)	91.4 (+0.4)	97.1 (+6.1)	96.1 (+5.1)
50%	94.0	94.0 (+0.0)	94.8 (+0.8)	94.2 (+0.2)	97.5 (+3.5)	96.8 (+2.8)
70%	95.3	95.5 (+0.2)	95.9 (+0.6)	96.0 (+0.7)	97.8 (+2.5)	97.2 (+1.9)
DreamVLA
30%	80.8	80.2 (-0.6)	82.0 (+1.2)	81.5 (+0.7)	88.8 (+8.0)	87.2 (+6.4)

TABLE VI: Mechanism ablation across backbones. Average success rate under matched average retention. Sem. is the semantic-only variant; +Mot. and +Edge add one cue at a time; +M+E combines both cues without coverage-mode switching; and w/o Edge. removes structural reallocation from full IAprune. Parentheses report percentage-point changes from Sem.

Fig. 7: Qualitative visual comparison under an extreme target average retention of 12.5%. Across four manipulation cases, IAprune preserves more continuous target- and manipulator-adjacent regions, while FastV and VLA-Cache tend to miss the target or keep fragmented redundant patches.

Average-Budget Tracking
Target	Observed	Dev. (pp)	Ret. Range	Lat. Range (ms)
30%	30.9%	$\mathbf{+0.9}$	29.5–31.9%	78.2–82.6
50%	49.7%	$\mathbf{-0.3}$	48.9–50.0%	89.2–91.7
70%	67.7%	$\mathbf{-2.3}$	67.6–67.9%	98.5–99.9

TABLE VII: Budget tracking and threshold sensitivity on OpenVLA-OFT/LIBERO. The upper table reports observed average retention together with retention and latency ranges across the evaluated IoU thresholds. The lower table reports latency in milliseconds and success rate in percent as

\theta_{iou}

varies.

Memory in GB; CUDA time in ms; lower is better.
Method	70% retention		50% retention		30% retention
Method	Mem.	Time	Mem.	Time	Mem.	Time
DreamVLA on LIBERO
Vanilla (100%)	Unpruned: 2.661 GB memory / 101.47 ms CUDA time
FastV	2.626	84.55	2.615	76.87	2.608	72.96
SparseVLM	2.621	83.85	2.612	75.16	2.602	73.76

TABLE VIII: Memory and runtime on LIBERO. Peak GPU memory and per-step CUDA runtime at nominal retention labels of 70%, 50%, and 30%. For IAprune, these labels denote the target average retention ratio

r_{\mathrm{tar}}

; baseline labels follow their original budget definitions.

Fig. 8: Latency breakdown of IAprune. Visual-token processing in the multimodal backbone is the main component that decreases with token retention. In the measured setting, vision encoding, text-token processing, and action decoding remain nearly constant.

Fig. 9: Real-World Task Demonstrations. From top to bottom: simple (single-step), long-horizon (multi-step), and dual-arm (coordinated) manipulation tasks, showing head and wrist camera perspectives.

Method	Success Rate (%)				Latency (ms) $\downarrow$
Method	Simple	Long	Dual-Arm	Avg.	Single	Dual
Vanilla (100%)	80.0	64.0	44.0	62.7	88.1	124.3
FastV	60.0	44.0	20.0	41.3	61.6	88.2
VLA-ADP	72.0	52.0	32.0	52.0	65.3	94.2
IAprune	84.0	68.0	44.0	65.3	59.7 (1.48 $\times$ )	84.6 (1.47 $\times$ )

TABLE IX: Real-robot results on three tasks. We report success rate and inference latency for the Simple, Long, and Dual-arm tasks. Vanilla denotes the unpruned

\pi_{0.5}

policy, and values in parentheses are speedups relative to Vanilla.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 276 papers

VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models
Oct 6, 2026Owen Du, Yang Yue, Jie Zhang +3Efficient VLA Models VLMs for Robotics
ETH Zurich · Tsinghua University
Beyond Token Importance: Preserving Spatial Scaffolds for Efficient Vision-Language-Action Inference
Sep 29, 2026Jiayu Chen, Shuyong Gao, Jingkai Jia +6Efficient VLA Model Inference VLMs for Robotics
Fudan University · The Hong Kong Polytechnic University
When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA
Oct 4, 2026Tianjun Shi, Haotian Xiong, Ziyu Gong +2Efficient VLA Model Inference Vision-Language-Action Models
University of Macau

Oct 6, 2026J/K move · Enter open · S save

VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models

Owen Du, Yang Yue, Jie Zhang, Jiaqi Pi+2

ETH Zurich · Tsinghua University

Efficient VLA Models VLMs for Robotics Visual Token Pruning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models achieve strong robotic manipulation performance but incur high computational costs from processing long token sequences at every control step, limiting real-time deployment. Visual token pruning offers a direct solution, as visual patches dominate the input sequence and contain considerable redundancy. Existing approaches, however, either rely on indirect training-free heuristics, such as attention scores and motion thresholds, or require costly fine-tuning of the base VLA model. We introduce VLA-ACL (Action Consistency Learning), which learns a lightweight visual token pruning policy through action-level supervision while keeping the base VLA model entirely frozen. The training objective encourages actions produced from pruned visual contexts to remain consistent with the full-context teacher, with ground-truth actions as auxiliary supervision. This directly ties token selection to its effect on the downstream control output. Experiments on LIBERO and real-world manipulation tasks show that VLA-ACL prunes up to 87.5% of visual tokens while retaining competitive performance, reduces computation by up to 75%, and achieves a 1.5x inference speedup. These results establish a stronger performance-efficiency trade-off than existing frozen-VLA pruning methods and demonstrate the value of action-level supervision for visual token selection. Code is available at https://github.com/du-owen/VLA-ACL.

Abstract

Visual Token Pruning

Explore similar work

VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models

Beyond Token Importance: Preserving Spatial Scaffolds for Efficient Vision-Language-Action Inference

When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA