VLA Model Evaluation

VLA: Vision-Language-Action

Follow this topic Ask about this topic

Latest papers

8 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models
Oct 31, 2026Rafiqul IslamVLA Model Evaluation
Independent Researcher
Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks
Aug 31, 2026Chunyun Ma, Lun Luo, Xingjian Luo +7Long-Horizon Robotic Manipulation Mobile Manipulation
College of Intelligence Science and Technology and National Key Laboratory of Equipment State Sensing and Smart Support, National University of Defense Technology, Changsha, China. · XPeng Inc., Guangzhou, China. · The Chinese University of Hong Kong, Hong Kong, China. +1
MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models
Jun 23, 2026Pablo Valle, Shaukat Ali, Aitor Arrieta +1Automated Software Testing Vision-Language-Action Models
Mondragon University, Guipuzcoa, Spain · Simula Research Laboratory, Oslo, Norway · University of Ottawa, Ottawa, ON K1N 6N5, Canada +1
VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving
Jun 10, 2026Thach Nguyen, Danhua Guo, Tom Lampo +2Autonomous Driving Benchmarks CoT Faithfulness
Uber AV Labs
A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation
Jun 9, 2026Shuo Wang, Hanyuan Xu, Yingdong Hu +2Robotics Simulation Sim-to-Real Transfer
Tsinghua University · Shanghai Qi Zhi Institute
VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesis
May 16, 2026Zeqin Liao, Peifan Ren, Zixu Gao +6VLM Adaptation Synthetic Data Generation for Robotics
School of computing and data science, Nanyang Technological University · School of Software Engineering, Sun Yat-sen University, China and GuangDong Engineering Technology Research Center of Blockchain, China · Northwest A&F University
Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation
May 7, 2026Yixin Zhu, Zixiong Wang, Jian Yang +4Robotics Simulation Sim-to-Real Transfer
Nanjing University · Nankai University · ShanghaiTech University
vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
Mar 14, 2026Suhwan Choi, Yunsung Lee, Yubeen Park +4VLM Evaluation Vision-Language-Action Models
Allen Institute for AI (AI2) · Seoul National University

Oct 31, 2026J/K move · Enter open · S save

Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models

Rafiqul Islam

Independent Researcher

VLA Model Evaluation

Open paper AskPDF arXiv

Simulation-based VLA evaluation often treats matching task, seed, and episode identifiers as sufficient evidence that two policies faced identical test conditions. Under specific vectorized reset and autoreset conditions, different termination timing can change subsequent initial-state assignment, so the same episode index need not mean the same test state. We present an evaluation-integrity framework for paired comparison - invariants I1-I5, provenance logging, optional scene fingerprints, and an automated checker - and a controlled LeRobot 0.6.1 LIBERO harness study with emulated controllers. Four evidence levels are kept separate: (A) mixed-horizon Sync wave-1 IDs [20, 5, 22, 7] vs [4, 5, 6, 7]; (B) homogeneous separate-job Sync control (16 cells, 0 ID/fingerprint mismatches); (C) Async+Freeze batch-size map differences (12 I3 mismatches; bs=8 incomplete; mechanism unisolated); (D) a synthetic McNemar illustration only. Controllers were emulated and no real-VLA shared-vector experiment was completed. We recommend checking declared/resolved init-state identity and provenance before paired statistics. We do not claim that published VLA leaderboards are generally invalid.