Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration

Qiwei Chen; Kaijun Zhou; Nuohui Shi; Zhiyang Li; Yuxuan Feng; Jinyu Gu

cs.ROSep 29, 2026

Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration

Authors: Qiwei Chen, Kaijun Zhou, Nuohui Shi, Zhiyang Li, Yuxuan Feng, Jinyu Gu

Organizations: Shanghai Jiao Tong University

Abstract

Simulated manipulation benchmarks are the standard tool for evaluating vision-language-action (VLA) policies and the acceleration methods that reduce their inference latency for on-robot deployment. On these benchmarks, we observe that some training-free acceleration methods, which approximate the baseline policy's computation, achieve higher measured success rates than the baseline itself. Success rates alone cannot establish whether such gains come from better task execution or from evaluation flaws. We therefore investigate two kinds of benchmark flaws behind these gains: bugs, where the implementation does not match the intended task or evaluation protocol, and design limitations, where success criteria and simulation settings do not fully capture how acceleration affects task execution. Starting from tasks with anomalous gains, we localize root causes by plotting object trajectories against checker acceptance regions, and classify the resulting bugs into task consistency, initialization, and reproducibility. Extending this audit to seven benchmarks, including RoboTwin, LIBERO-Plus, and VLABench, we identify 22 bugs of these types and 4 design limitations. For the latter, we revise permissive success checkers, correct unrealistic object masses, and add a motion-aware score that favors smoother actions. Experiments show that bug fixes can reverse method rankings, moving the baseline from last to first on one task. Addressing design limitations can likewise remove anomalous gains: on another task, the baseline moves from 21 percentage points behind an accelerated method to 5 points ahead. Gains attributed to acceleration can therefore be artifacts of the benchmark rather than better task execution. We release our bug fixes and revised benchmark settings to support trustworthy evaluation of VLA acceleration.

Figures & tables

Benchmark	Instruction	Demos	Evaluation
LIBERO ( Liu et al., 2023 )	Not specified	Human teleoperation	Binary
LIBERO-PRO ( Zhou et al., 2025 )	Based on LIBERO	Reused from LIBERO	Binary
LIBERO-Plus ( Fei et al., 2026 )	LLM-rewritten	Reused from LIBERO	Binary
RoboTwin ( Chen et al., 2026b )	LLM-generated	Scripted experts, motion planning	Binary
RoboCasa ( Nasiriany et al., 2024 )	Not specified	Human teleoperation, MimicGen	Binary
VLABench ( Zhang et al., 2025 )	LLM-rewritten	Scripted experts, motion planning	Binary, stage scores

Table 1: Benchmark components. Instruction, Demos, and Evaluation denote instruction construction, training demonstration sources, and the success signal, respectively.

Figure 1: Task Place Shoe . Simplified prompt: “Pick up the footwear from the table and place it on the mat.” (a) Baseline and DP-Cache-Fast rollouts alongside a rule-based expert demonstration. (b) The final poses show that both shoes lie on the mat.

Figure 2: Task Put Bottles Dustbin . Simplified prompt: “Pick the bottle with two distinct parts, drop it into the garbage bin.” (a) The recording rollout videos. (b) The trajectory visualizations show how both bottles enter the checker’s acceptance region, serving as an alternative to rollout videos.

Figure 3: Distribution of the 22 benchmark bugs. (a) Counts across all categories; (b)–(d) subcategory counts within task consistency, initialization, and reproducibility, respectively.

Figure 4: Benchmark design limitations arising from fixed acceptance thresholds and unrealistic object masses. (a,b) Spatial distributions of Scan Object samples relative to the acceptance region and success rates under different acceptance thresholds, respectively. (c) Discrepancies between simulated and real-world object masses.

Figure 5: Policy-action fluctuations and executed motion in simulation (top) and on the real robot (bottom). Acceleration and jerk magnitude are abbreviated as acc. and jm., respectively.

Benchmark	Task / subset	Baseline	DP-Cache-Fast	DP-Cache-Slow	ProbeFlow
RoboTwin	place shoe	$\mathstrut 65$ $\,/\,$ $\mathstrut 74$	$\mathstrut 68$ $\,/\,$ $69$	$66$ $\,/\,$ $\mathstrut 67$	$\mathstrut 68$ $\,/\,$ $69$
	rotate qrcode	$56$ $\,/\,$ $\mathstrut 66$	$\mathstrut 60$ $\,/\,$ $62$	$\mathstrut 48$ $\,/\,$ $\mathstrut 57$	$59$ $\,/\,$ $60$
	$\cdots$
	blocks ranking size	$21$ $\,/\,$ $\mathstrut 27$	$\mathstrut 22$ $\,/\,$ $25$	$\mathstrut 16$ $\,/\,$ $\mathstrut 21$	$19$ $\,/\,$ $23$
	handover block	$18$ $\,/\,$ $\mathstrut 29$	$\mathstrut 15$ $\,/\,$ $26$	$\mathstrut 20$ $\,/\,$ $27$	$16$ $\,/\,$ $\mathstrut 25$
VLABench	select mahjong	$\mathstrut 18$ $\,/\,$ $\mathstrut 35$	$\mathstrut 7$ $\,/\,$ $\mathstrut 17$	$14$ $\,/\,$ $20$	$8$ $\,/\,$ $\mathstrut 17$

Table 2: Original / revised SR (%); blue/pink marks each row’s worst/best values. The listed LIBERO-PRO and LIBERO-Plus tasks are enhanced variants of a subset of LIBERO tasks.

Permissive checkers
Benchmark	Task	Baseline	DP-Cache-Fast	DP-Cache-Slow	ProbeFlow
RoboTwin	put bottles dustbin	$\mathstrut 24$ $\,/\,$ $\mathstrut 24$	$26$ $\,/\,$ $\mathstrut 22$	$\mathstrut 28$ $\,/\,$ $\mathstrut 22$	$\mathstrut 28$ $\,/\,$ $\mathstrut 22$
VLABench	select drink	$\mathstrut 35$ $\,/\,$ $\mathstrut 31$	$\mathstrut 17$ $\,/\,$ $12$	$20$ $\,/\,$ $17$	$19$ $\,/\,$ $\mathstrut 9$
Mass adjustment
RoboTwin	beat block hammer	$\mathstrut 79$ $\,/\,$ $\mathstrut 27$	$\mathstrut 87$ $\,/\,$ $\mathstrut 19$	$82$ $\,/\,$ $\mathstrut 19$	$86$ $\,/\,$ $22$
	place fan	$\mathstrut 40$ $\,/\,$ $\mathstrut 38$	$\mathstrut 61$ $\,/\,$ $\mathstrut 33$	$51$ $\,/\,$ $\mathstrut 38$	$59$ $\,/\,$ $34$

Table 3: Original SR (%) / revised SR or scores; blue/pink marks each row’s worst/best values.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Pick Dual Bottles on the real robot (left pair) and in simulation (right pair).

Repository	Commit SHA
RoboTwin	2eeec322d95799f537cbfe5f291a8220d965ccb8
LIBERO	8f1084e3132a39270c3a13ebe37270a43ece2a01
LIBERO-PRO	eafdb809426b13153aa1e4c42d6601844217dfec
RoboCasa365	be22d659b02db8f6d7f3a3c3edc742934fdcbaae
LIBERO-Plus	4976dc30028e805ff8094b55501d532c48fec182
VLABench	cf588fe60c0c7282174fe979f5913170cfe69017

Appendix

Table 4: Repository revisions used for benchmark evaluation.

Configuration	Latency (ms)	Speedup
Baseline	313.73	$1.00\times$
DP-Cache-Fast	122.61	$2.56\times$
DP-Cache-Slow	194.12	$1.62\times$
ProbeFlow	210.87	$1.49\times$

Appendix

Table 5: Mean inference latency and speedup.

No.	Benchmark	Bug	Repair
Task consistency — Semantic omission
1	RoboTwin	place shoe : the checker and demonstrations restrict toe orientation, but the instruction only specifies mat placement.	Remove the unrequested orientation constraint; retain the mat-placement check.
2	RoboTwin	rotate qrcode : the checker adds table-height and gripper-release requirements absent from the instruction.	Remove the height and release constraints; retain the orientation check.
3	RoboTwin	adjust bottle : the checker restricts lateral position by initial side, although the prompt only requests grasping and lifting.	Accept either lateral region when no arm is specified; see Figure 8 for arm-selection analysis.
4	RoboTwin	blocks ranking size : the instruction omits parts of the ordering rule used by the checker and demonstrations.	Explicitly state the required size order and placement in the instruction.
5	RoboTwin	handover block : grasp-only prompts are scored against a longer handover, placement, and release sequence.	Adapt the checker to the delivered prompt and check only its required operations.

Appendix

Table 6: Benchmark bugs and corresponding repairs.

Figure 7: (a) The gripper-presence check accepts Baseline and rejects DP-Cache-Slow. (b) An edge grasp holds the 10 g fan but fails at 600 g. Gray dashed poses and grippers are schematic references; pad mass and inertia also change.

Figure 8: Step-by-step flow-matching denoising on adjust bottle . Top/bottom rows show left/right TCP paths for the first 32-action chunk; color indicates action order within the chunk. The rightmost column shows the accelerated comparison. From Step 2, motion remains concentrated in the left arm and becomes smoother over subsequent denoising steps.

Task consistency
Benchmark	Task	Baseline	DP-Cache-Fast	DP-Cache-Slow	ProbeFlow
RoboTwin	adjust bottle	$\mathstrut 92$ $\,/\,$ $\mathstrut 96$	$94$ $\,/\,$ $\mathstrut 94$	$\mathstrut 95$ $\,/\,$ $95$	$94$ $\,/\,$ $\mathstrut 94$
	place bread skillet	$42$ $\,/\,$ $\mathstrut 35$	$\mathstrut 47$ $\,/\,$ $32$	$\mathstrut 39$ $\,/\,$ $\mathstrut 29$	$46$ $\,/\,$ $32$
	place bread basket	$\mathstrut 55$ $\,/\,$ $\mathstrut 54$	$\mathstrut 59$ $\,/\,$ $53$	$\mathstrut 55$ $\,/\,$ $\mathstrut 52$	$58$ $\,/\,$ $\mathstrut 52$
Initialization
LIBERO-PRO	Goal-Language	$97.2$ $\,/\,$ $96.4$	$\mathstrut 98.4$ $\,/\,$ $96.6$	$\mathstrut 96.6$ $\,/\,$ $\mathstrut 95.8$	$\mathstrut 98.4$ $\,/\,$ $\mathstrut 96.8$

Appendix

Table 7: Additional bug-fix comparisons: original / repaired SR (%); blue/pink marks each row’s worst/best values separately before and after repair.

Mass adjustment
Benchmark	Task	Baseline	DP-Cache-Fast	DP-Cache-Slow	ProbeFlow
RoboTwin	place a2b right	$71$ $\,/\,$ $\mathstrut 68$	$74$ $\,/\,$ $\mathstrut 59$	$\mathstrut 78$ $\,/\,$ $62$	$\mathstrut 68$ $\,/\,$ $62$
	stack blocks three	$26$ $\,/\,$ $\mathstrut 34$	$26$ $\,/\,$ $31$	$\mathstrut 18$ $\,/\,$ $\mathstrut 26$	$\mathstrut 28$ $\,/\,$ $31$
LIBERO	Plate Pushing	$\mathstrut 100$ $\,/\,$ $86$	$\mathstrut 100$ $\,/\,$ $\mathstrut 90$	$\mathstrut 100$ $\,/\,$ $\mathstrut 80$	$\mathstrut 100$ $\,/\,$ $88$
Motion-aware scoring
RoboTwin	pick diverse bottles	$37$ $\,/\,$ $\mathstrut 37.0$	$\mathstrut 36$ $\,/\,$ $18.2$	$38$ $\,/\,$ $30.4$	$\mathstrut 41$ $\,/\,$ $\mathstrut 17.8$

Appendix

Table 8: Additional results: before / after SR (%) for mass adjustment; SR (%) / score (0–100) for motion-aware scoring. Blue/pink marks each row’s worst/best values separately for each condition, including ties.

RoboTwin
Task	Object	Original (g)	Adjusted (g)
beat block hammer	Hammer	1	600
place a2b right	Bell	50	150
stack blocks three	Block	10	50
place fan	Fan	10	600
LIBERO

Appendix

Table 9: Object masses used in the RoboTwin and LIBERO mass-adjustment comparisons (g).

Figure 9: Visual examples of benchmark issues. Bug details and repairs are provided in Table 6 .

Explore similar work

Jul 14, 2026cs.RO

Reducing Temporal Redundancy for Efficient Vision-Language-Action Inference

Vision-Language-Action (VLA) models exhibit strong generalization for robotic manipulation, yet their high inference latency limits real time deployment. We identify two primary sources of temporal redundancy in existing VLA pipelines: repeated visual encoding of highly similar consecutive frames and multi step iterative sampling in diffusion based policies. To address this, we propose a system level acceleration strategy that reduces computation in both perception and action generation. On the perception side, we incrementally update only tokens corresponding to dynamic scene regions instead of re-encoding entire frames. On the policy side, we compress diffusion sampling into a compact 2-step schedule through efficiency oriented training while preserving action precision. Experiments on Libero, RobotWin, and Real Robot Platforms demonstrate over 2 times speedup while maintaining high performance, achieving up to 98% success rate on general manipulation benchmarks. Our codes will be released on Github.

Yuzhou Wu, Yuxin Zheng, Muchun Niu +6

1Tianji KernalMind co ltd · 2Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · 3Shanghai Jiao Tong University, Shanghai, China +2

Diffusion-Based Vision-Language-Actions Vision-Language-Action Framework Action Generation Inference Latency Robotic Manipulation Redundancy

Ask PDF

Mar 14, 2026cs.AI

vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models

Vision-Language-Action (VLA) models are increasingly evaluated across multiple simulation benchmarks, yet adding each benchmark to an evaluation pipeline requires resolving incompatible dependencies, matching underspecified evaluation protocols, and reverse-engineering undocumented preprocessing. This burden scales with the number of models and benchmarks, making comprehensive evaluation impractical for most teams. We present vla-eval, an open-source evaluation harness that eliminates this per-benchmark cost by decoupling model inference from benchmark execution through a WebSocket+msgpack protocol with Docker-based environment isolation. Models integrate once by implementing a single predict() method; benchmarks integrate once via a four-method interface; the full cross-evaluation matrix works automatically. The framework supports 14 simulation benchmarks and six model servers. Parallel evaluation via episode sharding and batch inference achieves up to 47x wall-clock speedup, completing 2,000 LIBERO episodes in ~18 minutes. To validate the framework, we reproduce published scores across six VLA codebases and three benchmarks, documenting previously undocumented pitfalls. We additionally release a VLA leaderboard aggregating 657 published results across 17 benchmarks. Framework, evaluation configs, and all reproduction results are publicly available at https://github.com/allenai/vla-evaluation-harness and https://allenai.github.io/vla-evaluation-harness/leaderboard.

Suhwan Choi, Yunsung Lee, Yubeen Park +4

Allen Institute for AI (AI2) · Seoul National University

Preprocessing

Ask PDF

Dec 27, 2025cs.RO

VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

While Vision-Language-Action models (VLAs) are rapidly advancing towards generalist robot policies, it remains difficult to quantitatively understand their limits and failure modes. To address this, we introduce a comprehensive benchmark called VLA-Arena. We propose a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For Task Structure, VLA-Arena's 170 tasks are grouped into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon. Each task is designed with three difficulty levels (L0-L2), with fine-tuning performed exclusively on L0 to assess general capability. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task to enable a decoupled analysis of robustness. Our extensive evaluation of state-of-the-art VLAs reveals several critical limitations, including a strong tendency toward memorization over generalization, asymmetric robustness, a lack of consideration for safety constraints, and an inability to compose learned skills for long-horizon tasks. To foster research addressing these challenges and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, data, models, and leaderboard are available at https://vla-arena.github.io.

Borong Zhang, Jiahao Li, Jiachen Shen +7

Institute for Artificial Intelligence, Peking University. · Zhongguancun Academy. · PKU-PsiBot Joint Lab. +2

Diffusion-Based Vision-Language-Actions Robot Policies Open-Source

Ask PDF