Robot Manipulation Benchmarks

Follow this topic Ask about this topic

Momentum

47 papers in the last four weeks, up 262% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 200

May 20, 2026cs.RO

roto 2.0: The Robot Tactile Olympiad

Tactile-based reinforcement learning (RL) is currently hindered by fragmented research and a focus on over-saturated orientation tasks. We introduce v2 of the Robot Tactile Olympiad (\texttt{roto 2.0}), a GPU-parallelised benchmark designed to standardise tactile-based RL across four distinct robotic morphologies (16-DOF to 24-DOF). Unlike prior benchmarks, roto focuses on end-to-end "blind" manipulation, utilising only proprioception and tactile sensing without state information or distillation. We demonstrate a significant performance leap, with our blind agents achieving 13 Baoding ball rotations in 10 seconds, an order of magnitude faster than current state-of-the-art speeds. By open-sourcing our environments and robustly tuned baselines, we reduce the barrier to entry and enable researchers to prioritise fundamental algorithmic challenges over tedious RL tuning. Website: https://elle-miller.github.io/roto/

Elle Miller, Jayaram Reddy, Ayush Deshmukh +4

University of Edinburgh, UK · National University Singapore, Singapore

RL for Robotics Dexterous Manipulation Robotic Manipulation Dynamic Robotic Manipulation Robot Manipulation Benchmarks Tactile Sensing for Robotic Manipulation

May 20, 2026cs.RO

Humanoid Whole-Body Manipulation via Active Spatial Brain and Generalizable Action Cerebellum

In this paper, we explore spatial-aware humanoid whole-body manipulation task. Compared with tabletop settings, this task poses two key challenges: 1) Spatial understanding is challenging in complex 3D environments with diverse spatial relations. 2) Action generation is difficult to generalize, as limited and costly real-robot data restricts data-driven models generalization. To address these challenges, we propose a generalizable humanoid loco-manipulation framework that leverages the spatial perception and action generation capabilities of multi-agent large models. Specifically, our framework includes two components: Active Spatial Brain for active spatial perception and decision-making, and Generalizable Action Cerebellum for executable robot action generation. The first component actively perceives the spatial scene and makes decisions on task planning and subtask decomposition. The second component generate executable robot actions based on the decisions made by the first module without needs of task-specific real robot data. To benchmark our framework, we design a set of spatial manipulation tasks from two perspectives: evaluating spatial perception and understanding, and assessing real-robot task performance. The results demonstrate strong performance on both aspects across diverse tasks and environments.

Zhizhao Liang, Yi-Lin Wei, Xuhang Chen +6

School of Computer Science and Engineering, Sun Yat-sen University, China · Peng Cheng Laboratory, China · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China

Humanoid Loco-Manipulation Large Language Model-Based Robot Planning 3D Spatial Reasoning Generalization in Robotic Manipulation Whole-Body Robotic Manipulation Robot Motion Generation+2

May 20, 2026cs.RO

VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models

Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, but their real-world evaluation remains limited by a lack of accessible, reproducible, and consistent benchmarks. Simulation benchmarks fail to capture real-world complexity, while existing real-world benchmarks often require expensive hardware, centralized evaluation, or are limited in task diversity. We introduce VLA-REPLICA, a low-cost, easily reproducible real-world benchmark for evaluating VLA models. Built from off-the-shelf components, our system can be quickly assembled and replicated across laboratories, providing a consistent environment for policy evaluation anywhere in the world. VLA-REPLICA includes a diverse suite of manipulation tasks and a small-scale demonstration dataset for target-domain adaptation, with real-world evaluation protocols for both in-distribution and out-of-distribution settings. Experiments with imitation learning and state-of-the-art VLA models reveal model strengths and limitations, while consistent results across independently constructed setups demonstrate the reproducibility of our benchmark.

Alex S. Huang, Jiahui Zhang, Shiqing Tang +1

Intelligent Robotics and Vision Lab, University of Texas at Dallas · Allen High School

VLM Evaluation Vision-Language-Action Models Robot Manipulation Benchmarks

May 19, 2026cs.CV

Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation

Language instructions offer a flexible way to specify task semantics but can become ambiguous in cluttered scenes containing multiple visually similar objects and candidate targets. We study Spatially Prompted Visual Trajectory Prediction (SP-VTP), an open-loop setting in which points or bounding boxes in the first egocentric frame specify an object--target pair and condition the prediction of future 3D end-effector (EE) trajectory chunks. To support this setting, we introduce EgoSPT, a controlled dataset comprising 2,841 egocentric fork pick-and-place episodes with first-frame object and target annotations and reconstructed 3D EE trajectories. SP-VTP evaluates whether a static task specification remains informative as the viewpoint and scene configuration evolve throughout execution. We also introduce SPOT, a spatially prompted object--target trajectory predictor that integrates first-frame visual and coordinate prompts with current visual observations and motion history to generate future trajectories. Rather than relying on an explicit object tracker, SPOT maintains a fixed representation of the first-frame task specification. Across three training seeds, spatial prompting consistently reduces translation and endpoint errors relative to a no-prompt baseline, while controlled prompt interventions reveal distinct roles for object and target cues at different stages of execution. Together, EgoSPT and SPOT provide a dataset and baseline for spatially conditioned egocentric trajectory prediction in open-loop pick-and-place settings.

Yifan Li, Xinyu Zhou, Yunhao Ge +2

Michigan State University · NVIDIA Research

Robotic Manipulation Robot Manipulation Benchmarks Trajectory Inference Trajectory Prediction

May 19, 2026cs.RO

Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation

Fine-grained manipulation marks a regime where global scene context no longer suffices, and success hinges on the tight coupling of local attribute grounding, high-fidelity spatial perception, and constraint-respecting motor execution. However, current embodied AI benchmarks collapse these capacities into binary success rates, systematically inflating reported capabilities by up to 70% and masking the architectural bottlenecks that impede real-world deployment. We introduce MetaFine, a diagnostic meta-evaluation framework that disentangles manipulation competency along three axes: understanding, perception, and controlled behavior. Built on a compositional task graph, MetaFine absorbs heterogeneous external benchmarks and reconstructs them into diagnostic scenarios of varying complexity under a unified protocol. Evaluating state-of-the-art vision-language-action (VLA) models through this lens exposes severe dimension-specific failures invisible to conventional metrics. Through targeted causal intervention, we identify the visual encoder's ability to preserve local spatial structure as a key bottleneck for fine-grained precision: improving it directly unlocks previously inaccessible manipulation capabilities without modifying downstream policies. MetaFine further supports hybrid real-sim validation, using limited paired real-world rollouts to calibrate scalable simulation-based estimates for more stable physical benchmarking. By shifting evaluation from ranking to diagnosis, MetaFine turns benchmarking into an actionable compass for repairing the layered capacities underlying genuine physical dexterity. The MetaFine framework, benchmarks, and supporting resources will be publicly released at our project page: https://metafine.github.io/.

He-Yang Xu, Pengyuan Zhang, Zongyuan Ge +5

Southeast University, China · Monash University, Australia · Xiaomi EV, China +2

VLM Evaluation Robotic Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks Fine-Grained Visual Perception

May 18, 2026cs.RO

DexHoldem: An Agentic Robotics Benchmark for Dexterous Manipulation in Texas Hold'em

Evaluating embodied systems with real dexterous hardware requires more than isolated motor-skill tests: an agent must perceive a changing scene (e.g. a tabletop), choose a context-appropriate action, execute it with a dexterous hand, and leave the scene usable for later decisions. We introduce DexHoldem, a comprehensive real-world benchmark evaluating Texas Hold'em related dexterous manipulations with a ShadowHand. DexHoldem provides 1,470 teleoperated demonstrations across 14 Texas Hold'em manipulation primitives, a standardized physical policy benchmark, and an agentic perception benchmark that tests whether agents can recover the structured game state needed for embodied decision making. On primitive execution,

π_{0.5}

obtains the highest task completion rate (

61.2\%

), while

π_{0.5}

and

π_0

tie on scene-preserving success rate (

47.5\%

). On agentic perception, Opus 5.5 narrowly leads on both strict problem-level accuracy (

49.1\%

) and average field-wise accuracy (

80.6\%

); the gap between the two exposes the distance between isolated visual sub-capabilities and complete routing-relevant state recovery. Finally, we instantiate the full embodied-agent loop with one agent--policy pairing over 33 closed-loop hand-level rollouts, in which only

12.1\%

of hands complete; retries restore the failed primitive in 12 of 34 dispatches and resolve prolonged execution stalls in three of the four completed hands, which would otherwise have required manual termination. Only one hand completes with neither a retry nor a human-help request. DexHoldem therefore evaluates dexterous tabletop execution, agentic perception, and embodied decision routing in a shared physical setting. Project website: https://dexholdem.github.io/Dexholdem/

Feng Chen, Tianzhe Chu, Li Sun +6

Dexterous Manipulation AI Agent Evaluation Robotic Manipulation Robot Manipulation Benchmarks Embodied AI

May 18, 2026cs.RO

ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics

Most existing vision-language manipulation research targets rigid robotic arms, whose fixed morphology limits adaptability in cluttered or confined spaces. Soft robotic arms offer an appealing alternative due to their deformability, but confront challenges such as unreliable proprioception and distributed low-level actuation. To investigate these challenges, we introduce \ManiSoft, a benchmark for vision-language manipulation with soft arms. ManiSoft features a tailored simulator that couples realistic soft-body dynamics with contact-rich interactions via an elastic force constraint. On this basis, ManiSoft defines four tasks, each highlighting distinct aspects of deformable control, from basic end-effector coordination to obstacle avoidance. To support policy training and evaluation, \ManiSoft{} includes an automated pipeline that generates

6{,}300

diverse scenes and corresponding expert trajectories. To produce high-quality trajectories at scale, we first employ a high-level planner to decompose each task into a sequence of waypoints, followed by a low-level reinforcement learning policy that generates torque commands to track waypoints. Benchmarking three representative policy models shows relatively promising results in clean scenes but substantial performance drop under randomization. Visualization analysis indicates that failures stem primarily from inaccurate visual estimation of proprioceptive state and limited exploitation of deformability for adaptive obstacle avoiding. We anticipate ManiSoft to serve as a valuable testbed, bridging the gap between rigid and soft arms in the context of vision-language manipulation. Out codes and datasets are released at https://buaa-colalab.github.io/ManiSoft.

Ziyu Wei, Luting Wang, Chen Gao +2

Beihang University · Hangzhou Innovation Institute, Beihang University · National University of Singapore

Language-Conditioned Robot Manipulation Robotic RL Robot Manipulation Benchmarks Continuum Robot Control VLMs for Robotics Soft Robotics

May 15, 2026cs.RO

DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo

Achieving human-level manipulation requires dexterous robotic hands capable of complex object interactions. Advancing such capabilities further demands standardized benchmarks for systematic evaluation. However, prior work does not focus on task-oriented dexterous manipulation. Existing benchmarks also often lack high quality human demonstrations and an easy-to-use evaluation pipeline. In this paper, we present DexJoCo, a benchmark and toolkit for task-oriented dexterous manipulation, comprising 11 functionally grounded tasks that evaluate tool-use, bimanual coordination, long-horizon execution, and reasoning. We develop a low-cost data collection system and collect 1.1K trajectories across these tasks, with support for domain randomization to assess robustness. We benchmark modern models under diverse settings, including visual and dynamics randomization, multi-task training, and action-head adaptation. Through extensive empirical analysis, we identify several important insights and common limitations of current policies in dexterous manipulation, highlighting key challenges for future research in dexterous hand robot learning. Code is available at: https://dexjoco.github.io

Hanwen Wang, Weizhi Zhao, Xiangyu Wang +11

NLPR & MAIS, CASIA · SJTU · MBZUAI +3

Dexterous Manipulation Long-Horizon Robotic Manipulation Bimanual Robotic Manipulation Robot Manipulation Benchmarks

May 15, 2026cs.RO

KaRMA: A Kinematic Metric for Fine Manipulation Ability in Robotic Hands

Traditional robotic hand metrics focus on static properties such as workspace, manipulability, and grasp stability. However, these metrics do not directly measure dexterity under the standard definition in robotic manipulation: the ability to continuously change an object's pose within the hand while maintaining contact from an initial grasp. We introduce Kinematic Rolling Manipulation Ability (KaRMA), a kinematic-only metric for fine manipulation that quantifies reachable in-hand translation and reorientation of a spherical test object within a two-finger precision pinch through feasible rolling motions. KaRMA enforces joint limits, collision constraints, rolling contact, and antipodal force feasibility, then investigates reachable in-hand object poses via breadth-first search over translation and rotation primitives. KaRMA reports three scores: translational coverage (KaRMA-T), rotational coverage (KaRMA-R), and sensitivity to the initial grasp (KaRMA-S). We evaluate KaRMA on 16 widely used robotic hands and compare against static baselines, showing that KaRMA separates hands that rank identically under static proxies, reveals translation-rotation tradeoffs invisible to existing baselines, and is qualitatively consistent with selected published task benchmarks where Jacobian-based metrics can be misleading.

Martin Peticco, Pulkit Agrawal

Improbable AI Lab, Massachusetts Institute of Technology

Dexterous Manipulation In-Hand Manipulation Robot Manipulation Benchmarks

May 14, 2026cs.RO

IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation

Robot imitation data are often multimodal: similar visual-language observations may be followed by different action chunks because human demonstrators act with different short-horizon intents, task phases, or recent context. Existing frame-conditioned VLA policies infer each chunk from the current observation and instruction alone, so under partial observability they may resample different intents across adjacent replanning steps, leading to inter-chunk conflict and unstable execution. We introduce IntentVLA, a history-conditioned VLA framework that encodes recent visual observations into a compact short-horizon intent representation and uses it to condition chunk generation. We further introduce AliasBench, a 12-task ambiguity-aware benchmark on RoboTwin2 with matched training data and evaluation environments that isolate short-horizon observation aliasing. Across AliasBench, SimplerEnv, LIBERO, and RoboCasa, IntentVLA improves rollout stability and outperforms strong VLA baselines

Shijie Lian, Bin Yu, Xiaopeng Lin +8

1HUST · 2ZGCA · 4HIT +7

Robot Policy Learning Language-Conditioned Robot Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks

May 13, 2026cs.RO

Towards Robotic Dexterous Hand Intelligence: A Survey

Robotic dexterous hands are central to contact-rich manipulation, with rapid progress driven by advances in hardware, sensing, control, simulation, and data generation. However, existing studies are often developed under different assumptions regarding hand embodiments, sensory configurations, task settings, training data, and evaluation protocols, making systematic comparison difficult and obscuring the developmental trajectory of the field. This survey provides a holistic review of dexterous hand research from four complementary aspects. First, we present a hardware-level analysis covering actuation, transmission, perception, and representative hand designs, highlighting the key trade-offs in force capability, compliance, bandwidth, integration, and system complexity. Furthermore, we review control and learning methods for dexterous manipulation from a methodological perspective, grouping representative works by major paradigms and tracing their evolution in chronological order. In addition, we consolidate datasets, modality design, and evaluation practices, which enables methodological progress to be interpreted together with the ways in which it is trained, benchmarked, and assessed. Finally, we discuss the major limitations of current dexterous hand research and summarize the corresponding future directions. By connecting hardware analysis, methodological development, data resources, and evaluation, this survey aims to provide a structured understanding of dexterous hand research and to clarify the most important open challenges for future study.

Weiguang Zhao, Tian Liang, Xihao Guo +3

University of Liverpool · Xi’an Jiaotong-Liverpool University · Duke Kunshan University +1

Robotic Control Robotic Hand Design Dexterous Manipulation Robot Manipulation Benchmarks

May 13, 2026cs.CV

AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects

Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified scenarios, overlooking shape complexities and assembly trajectories in industrial assemblies. We introduce AssemblyBench, a synthetic dataset of 2,789 industrial objects with multimodal instruction manuals, corresponding 3D part models, and part assembly trajectories. We also propose a transformer-based model, AssemblyDyno, which uses the instructional manual and the 3D shape of each part to jointly predict assembly order and part assembly trajectories. AssemblyDyno outperforms prior works in both assembly pose estimation and trajectory feasibility, where the latter is evaluated by our physics-based simulations.

Danrui Li, Jiahao Zhang, Bernhard Egger +4

Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1

Object Pose Estimation Robot Manipulation Benchmarks Robotic Assembly Trajectory Prediction

May 12, 2026cs.RO

SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

Robotic manipulation is typically evaluated by task success, but reaching the correct final state does not guarantee safe execution. A robot may succeed despite contamination, premature release, or incorrect sequencing. These failures are poorly captured by task-completion metrics or isolated state checks because safety often depends on how behavior unfolds over time. We introduce SafeManip, a benchmark for evaluating temporal safety directly from low-level manipulation rollouts across eight physical and semantic categories. SafeManip grounds symbolic manipulation predicates into reusable temporal properties over finite executions, formalized with LTLf. We further introduce ManipVerse, a property ontology enabling semantic safety rules generalize across objects, tasks, and environments. Finally, SafeManip introduces per-trigger safety metrics that normalize violations by the safety-relevant opportunities that activate each obligation, accounting for differences in how often those opportunities arise across policies and tasks. We instantiate SafeManip on RoboCasa365 and LIBERO to audit seven state-of-the-art robot foundation-model policies. Our results reveal a gap between task capability and temporal safety: higher task success does not necessarily imply safer execution, successful rollouts can still contain violations, and failure patterns vary systematically across safety categories, task horizons, and manipulation suites. SafeManip provides a reusable layer for evaluating how safely manipulation tasks are executed, not only whether they are completed.

Chengyue Huang, Khang Vo Huynh, Mellon M. Zhang +3

Department of Machine Learning, Georgia Institute of Technology · Department of Computer Science, University of Virginia

Robot Safety Robotic Manipulation Robot Manipulation Benchmarks Linear Temporal Logic

May 11, 2026cs.RO

RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark

Memory is a critical component of robotic intelligence, as robots must rely on past observations and actions to accomplish long-horizon tasks in partially observable environments. However, existing robotic memory benchmarks still lack multimodal annotations for memory formation, provide limited task coverage and structural complexity, and remain restricted to simulation without real-world evaluation. We address this gap with RoboMemArena, a large-scale benchmark of 26 tasks, with average trajectory lengths exceeding 1,000 steps per task and 68.9% of subtasks being memory-dependent. The generation pipeline leverages a vision-language model (VLM) to design and compose subtasks, generates full trajectories through atomic functions, and provides memory-related annotations, including subtask instructions and native keyframe annotations, while paired real-world memory tasks support physical evaluation. We further design PrediMem, a dual-system VLA in which a high-level VLM planner manages a memory bank with recent and keyframe buffers and uses a predictive coding head to improve sensitivity to task dynamics. Extensive experiments on RoboMemArena show that PrediMem outperforms all baselines and provides insights into memory management, model architecture, and scaling laws for complex memory systems.

Huashuo Lei, Wenxuan Song, Huarui Zhang +10

1The Hong Kong University of Science and Technology (Guangzhou) · 5Zhejiang University of Technology · 2Zhejiang University +3

Long-Horizon Robotic Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks Memory-Augmented Agents

May 8, 2026cs.RO

TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

Active vision -- where a policy controls its own gaze during manipulation -- has emerged as a key capability for imitation learning, with multiple independent systems demonstrating its benefits in the past year. Yet there is no shared benchmark to compare approaches or quantify what active vision contributes, on which task types, and under what conditions. We introduce TAVIS, evaluation infrastructure for active-vision imitation learning, with two complementary task suites -- TAVIS-Head (5 tasks, global search via pan/tilt necks) and TAVIS-Hands (3 tasks, local occlusion via wrist cameras) -- on two humanoid torso embodiments (GR1T2, Reachy2), built on IsaacLab. TAVIS provides three evaluation primitives: a paired headcam-vs-fixedcam protocol on identical demonstrations; GALT (Gaze-Action Lead Time), a novel metric grounded in cognitive science and HRI that quantifies anticipatory gaze in learned policies; and procedural ID/OOD splits. Baseline experiments with Diffusion Policy and

π_0

reveal that (i) active vision generally helps, consistently across training runs, but benefits are task-conditional rather than uniform; (ii) multi-task policies degrade sharply under controlled distribution shifts on both suites; and (iii) imitation alone yields anticipatory gaze that lands on the task-relevant object, with lead times comparable to those of the human demonstrations, although head motion is less smooth than in the demonstrations, an effect of action chunking that success rate does not reveal. Code and evaluation scripts are released at https://github.com/spiglerg/tavis; demonstrations (LeRobot v3.0; ~2200 episodes) and trained baselines at https://huggingface.co/tavis-benchmark.

Giacomo Spigler

Department of Intelligent Systems Tilburg University Netherlands

Active Vision for Robot Manipulation Active Sensing Robotic Manipulation Robot Imitation Learning Robot Manipulation Benchmarks Active View Selection+1

May 7, 2026cs.RO

Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation

Reliable simulation evaluation of robot manipulation policies serves as a high-fidelity proxy for real-world performance. Although existing benchmarks cover a wide range of task categories, they lack visual realism, creating a large domain gap between simulation and reality. This undermines the reliability of simulation-based evaluation in predicting real-world performance. To mitigate the sim-to-real visual gap, we conduct a systematic analysis to isolate the effects of lighting and material. Our results show that these factors play a critical role in geometric reasoning and spatial grounding, yet are largely overlooked in existing benchmarks. Motivated by the analysis, we propose VISER, a visually realistic benchmark for evaluating robot manipulation in simulation. VISER features a high-fidelity dataset of over 1,000 3D assets with physically-based rendering (PBR) materials, along with 3D scenes created from these assets through curated layouts or generation. To this end, we propose an automated pipeline leveraging Multi-modal Large Language Models (MLLMs) for material-aware part segmentation and material retrieval, enabling scalable generation of physically plausible assets. Building on the high-fidelity 3D asset dataset, we construct diverse evaluation tasks, such as grasping, placing, and long-horizon tasks, enabling scalable and reproducible assessment of Vision-Language-Action (VLA) models. Our benchmark shows a strong correlation between simulation and real-world performance, achieving an average Pearson correlation coefficient of 0.92 across different policies.

Yixin Zhu, Zixiong Wang, Jian Yang +4

Nanjing University · Nankai University · ShanghaiTech University

Robotics Simulation Sim-to-Real Transfer Synthetic Data Generation for Robotics Robot Manipulation Benchmarks

Apr 28, 2026cs.RO

KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning

Robotic systems that interact with the physical world must reason about kinematic and dynamic constraints imposed by their own embodiment, their environment, and the task at hand. We introduce KinDER, a benchmark for Kinematic and Dynamic Embodied Reasoning that targets physical reasoning challenges arising in robot learning and planning. KinDER comprises 25 procedurally generated environments, a Gymnasium-compatible Python library with parameterized skills and demonstrations, and a standardized evaluation suite with 13 implemented baselines spanning task and motion planning, imitation learning, reinforcement learning, and foundation-model-based approaches. The environments are designed to isolate five core physical reasoning challenges: basic spatial relations, nonprehensile multi-object manipulation, tool use, combinatorial geometric constraints, and dynamic constraints, disentangled from perception, language understanding, and application-specific complexity. Empirical evaluation shows that existing methods struggle to solve many of the environments, indicating substantial gaps in current approaches to physical reasoning. We additionally include real-to-sim-to-real experiments on a mobile manipulator to assess the correspondence between simulation and real-world physical interaction. KinDER is fully open-sourced and intended to enable systematic comparison across diverse paradigms for advancing physical reasoning in robotics. Website and code: https://prpl-group.com/kinder-site/

Yixuan Huang, Bowen Li, Vaibhav Saxena +9

Princeton University · Carnegie Mellon University · Georgia Tech +3

Robotics Simulation Robotics Robot Task Planning Robot Manipulation Benchmarks Embodied Reasoning Robot Learning

Apr 28, 2026cs.RO

HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

Dexterous robot hands offer rich opportunities for multifunctional manipulation, where a robot must execute multiple skills in sequence while maintaining control over previously grasped objects. Most prior work in dexterous manipulation focuses on single-object, single-skill tasks. In contrast, our insight is that many sequential tasks require resource-aware grasps that conserve fingers for future actions. In this paper, we study sequential grasp-conditioned dexterous manipulation, where a robot first grasps an object and then performs a second, distinct manipulation subtask while preserving the initial grasp. We introduce HANDFUL, a learning framework that models finger usage as a limited resource and encourages exploration of resource-aware grasps through finger-level contact rewards. These grasps are subsequently selected for downstream tasks via curriculum-based policy learning. We further propose HANDFUL-Bench, a simulation benchmark that introduces sequential dexterous manipulation tasks across multiple secondsubtask objectives, including pushing, pulling, and pressing, under a shared grasp-conditioned setup. Extensive simulation results demonstrate that prioritizing resource-aware grasps improves second-subtask success and robustness compared to a baseline that greedily optimizes the initial grasp before attempting the second subtask. We additionally validate our approach on a real dexterous LEAP hand. Together, this work establishes resource-aware grasp planning as a key principle for multifunctional dexterous manipulation. Supplementary material is available on our website: https://handful-dex.github.io.

Ethan Foong, Yunshuang Li, Hao Jiang +2

Department of Computer Science at Northwestern University · Viterbi School of Engineering at the University of Southern California

Robotic Grasping Dexterous Manipulation Robot Manipulation Benchmarks

Apr 27, 2026cs.LG

SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning

Specification-guided reinforcement learning (RL) provides a principled framework for encoding complex, temporally extended tasks using formal specifications such as linear temporal logic (LTL). While recent methods have shown promising results, their ability to generalize across unseen specifications and diverse environments remains insufficiently understood. In this work, we introduce SpecRLBench, a benchmark designed to evaluate the generalization capabilities of LTL-based specification-guided RL methods. The benchmark spans multiple difficulty levels across navigation and manipulation domains, incorporating both static and dynamic environments, diverse robot dynamics, and varied observation modalities. Through extensive empirical evaluation, we characterize the strengths and limitations of existing approaches and reveal the challenges that emerge as specification and environment complexity increase. SpecRLBench provides a structured platform for systematic comparison and supports the development of more generalizable specification-guided RL methods. Code is available at https://github.com/BU-DEPEND-Lab/SpecRLBench.

Zijian Guo, İlker Işık, H. M. Sabbir Ahmad +1

Boston University

RL Benchmarks Reinforcement Learning Robot Manipulation Benchmarks Robot Policy Generalization Linear Temporal Logic

Apr 24, 2026cs.RO

LeHome: A Simulation Environment for Deformable Object Manipulation in Household Scenarios

Household environments present one of the most common, impactful yet challenging application domains for robotics. Within household scenarios, manipulating deformable objects is particularly difficult, both in simulation and real-world execution, due to varied categories and shapes, complex dynamics, and diverse material properties, as well as the lack of reliable deformable-object support in existing simulations. We introduce LeHome, a comprehensive simulation environment designed for deformable object manipulation in household scenarios. LeHome covers a wide spectrum of deformable objects, such as garments and food items, offering high-fidelity dynamics and realistic interactions that existing simulators struggle to simulate accurately. Moreover, LeHome supports multiple robotic embodiments and emphasizes low-cost robots as a core focus, enabling end-to-end evaluation of household tasks on resource-constrained hardware. By bridging the gap between realistic deformable object simulation and practical robotic platforms, LeHome provides a scalable testbed for advancing household robotics. Webpage: https://lehome-web.github.io/ .

Zeyi Li, Yushi Yang, Shawn Xie +15

Institute of Automation, Chinese Academy of Sciences. · Peking University. · Lightwheel. +1

Robotics Simulation Deformable Object Manipulation Robotics Robot Manipulation Benchmarks Deformable Object Simulation

Apr 23, 2026cs.RO

How VLAs (Really) Work In Open-World Environments

Vision-language-action models (VLAs) have been extensively used in robotics applications, achieving great success in various manipulation problems. More recently, VLAs have been used in long-horizon tasks and evaluated on benchmarks, such as BEHAVIOR1K (B1K), for solving complex household chores. The common metric for measuring progress in such benchmarks is success rate or partial score based on satisfaction of progress-agnostic criteria, meaning only the final states of the objects are considered, regardless of the events that lead to such states. In this paper, we argue that using such evaluation protocols say little about safety aspects of operation and can potentially exaggerate reported performance, undermining core challenges for future real-world deployment. To this end, we conduct a thorough analysis of state-of-the-art models on the B1K Challenge and evaluate policies in terms of robustness via reproducibility and consistency of performance, safety aspects of policies operations, task awareness, and key elements leading to the incompletion of tasks. We then propose evaluation protocols to capture safety violations to better measure the true performance of the policies in more complex and interactive scenarios. At the end, we discuss the limitations of the existing VLAs and motivate future research.

Amir Rasouli, Yangzheng Wu, Zhiyuan Li +4

Noahs Ark Laboratory, Huawei Technologies Canada. · while at Huawei Canada.

VLM Robustness Long-Horizon Robotic Manipulation AI Agent Benchmarks AI Safety Evaluation Vision-Language-Action Models Robot Manipulation Benchmarks+2

Apr 21, 2026cs.RO

RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

Recent advances in large-scale video world models have enabled increasingly realistic future prediction, raising the prospect of using generated videos as scalable supervision for robot learning. However, for embodied manipulation, perceptual realism alone is not sufficient: generated interactions must also be physically consistent and executable by robotic agents. Existing benchmarks provide valuable assessments of visual quality and physical plausibility, but they do not systematically evaluate whether predicted behaviors can be translated into executable actions that complete manipulation tasks. We introduce RoboWM-Bench, a manipulation-centric benchmark for embodiment-grounded evaluation of video world models. RoboWM-Bench converts generated human-hand and robotic manipulation videos into embodied action sequences and validates them through execution in physically grounded simulation environments. Built on real-to-sim scene reconstruction and diverse manipulation tasks, RoboWM-Bench enables standardized, reproducible, and scalable evaluation of physical executability. Using RoboWM-Bench, we evaluate state-of-the-art video world models and observe that visual plausibility and embodied executability are not always aligned. Our analysis highlights several recurring factors that affect execution performance, including spatial reasoning, contact prediction, and non-physical geometric distortions, particularly in complex and long-horizon interactions. These findings provide a more fine-grained view of current model capabilities and underscore the value of embodiment-aware evaluation for guiding physically grounded world modeling in robotic manipulation.

Feng Jiang, Yang Chen, Kyle Xu +8

Peking University · Tsinghua University · Lightwheel

Robotic Manipulation Robot Manipulation Benchmarks World Model Evaluation Video World Models

Apr 21, 2026cs.RO

Gated Memory Policy

Robotic manipulation tasks exhibit varying memory requirements, ranging from Markovian tasks that require no memory to non-Markovian tasks that depend on historical information spanning single or multiple interaction trials. Surprisingly, simply extending observation histories of a visuomotor policy often leads to a significant performance drop due to distribution shift and overfitting. To address these issues, we propose Gated Memory Policy (GMP), a visuomotor policy that learns both when to recall memory and what to recall. To learn when to recall memory, GMP employs a learned memory gate mechanism that selectively activates history context only when necessary, improving robustness and reactivity. To learn what to recall efficiently, GMP introduces a lightweight cross-attention module that constructs effective latent memory representations. To further enhance robustness, GMP injects diffusion noise into historical actions, mitigating sensitivity to noisy or inaccurate histories during both training and inference. On our proposed non-Markovian benchmark MemMimic, GMP achieves a 30.1% average success rate improvement over long-history baselines, while maintaining competitive performance on Markovian tasks in RoboMimic. All code, data and in-the-wild deployment instructions are available on our project website https://gated-memory-policy.github.io/.

Yihuai Gao, Jinyun Liu, Shuang Li +1

Stanford University

Sequential Decision Making Visuomotor Policy Learning Robot Imitation Learning Robot Manipulation Benchmarks Memory-Augmented Agents

Apr 20, 2026cs.RO

COFFAIL: A Dataset of Successful and Anomalous Robot Skill Executions in the Context of Coffee Preparation

In the context of robot learning for manipulation, curated datasets are an important resource for advancing the state of the art; however, available datasets typically only include successful executions or are focused on one particular type of skill. In this short paper, we briefly describe a dataset of various skills performed in the context of coffee preparation. The dataset, which we call COFFAIL, includes both successful and anomalous skill execution episodes collected with a physical robot in a kitchen environment, a couple of which are performed with bimanual manipulation. In addition to describing the data collection setup and the collected data, the paper illustrates the use of the data in COFFAIL to learn a robot policy using imitation learning.

Alex Mitrevski, Ayush Salunke

Division of Systems and Control, Chalmers University of Technology, Gothenburg, Sweden · Autonomous Systems Group, Bonn-Rhein-Sieg University of Applied Sciences, Sankt Augustin, Germany

Robot Skill Learning Robotic Data Collection Robotic Manipulation Robot Manipulation Benchmarks Robot Learning

Apr 20, 2026cs.RO

Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models

Recent Vision-Language-Action (VLA) models report impressive success rates on standard robotic benchmarks, fueling optimism about general-purpose physical intelligence. However, recent evidence suggests a systematic misalignment between standard benchmark success and true embodied reasoning, raising the question of whether these high scores reflect genuine cognitive capability. To address this gap, we introduce BeTTER, a diagnostic Benchmark for Testing True Embodied Reasoning in robotic policies. BeTTER applies targeted causal interventions (e.g., spatial layout shifts, temporal extrapolation) while enforcing kinematic isolation to explicitly decouple high-level reasoning failures from low-level execution limits. Through systematic evaluation, we reveal that state-of-the-art VLAs catastrophically fail in dynamic scenarios, exhibiting severe lexical-kinematic shortcuts, behavioral inertia, and semantic feature collapse. Crucially, our mechanistic analysis traces these symptoms to fundamental architectural bottlenecks - such as capacity compression and myopic downsampling - which systematically degrade the model's foundational semantic representation. We demonstrate that highly static evaluation protocols effectively mask this degradation by allowing optimization to overfit to sensorimotor priors. Supported by real-world robotic validation, our findings confirm that this representational breakdown is not a simulation artifact, highlighting the critical need for future VLA paradigms to resolve the structural tension between high-frequency control and high-level reasoning.

Haiweng Xu, Sipeng Zheng, Hao Luo +3

1Peking University · 3BeingBeyond · 2Tsinghua University

VLM Evaluation VLM Reasoning Vision-Language-Action Models Robot Manipulation Benchmarks Embodied AI Shortcut Learning+2

Apr 18, 2026cs.RO

Chain Of Interaction Benchmark (COIN): When Reasoning meets Embodied Interaction

Generalist embodied agents must perform interactive, causally-dependent reasoning, continually interacting with the environment, acquiring information, and updating plans to solve long-horizon tasks before they could be adopted in real-life scenarios. For instance, retrieving an apple from a cabinet may require opening multiple doors and drawers before the apple becomes visible and reachable, demanding sequential interaction under partial observability. However, existing benchmarks fail to systematically evaluate this essential capability. We introduce COIN, a benchmark designed to assess interactive reasoning in realistic robotic manipulation through three key contributions. First, we construct COIN-50: 50 interactive tasks in daily scenarios, and create COIN-Primitive required by causally-dependent tasks, and COIN-Composition with mid-term complexity for skill learning and generalization evaluation. Second, we develop a low-cost mobile AR teleoperation system and collect the COIN-Primitive Dataset with 50 demonstrations per primitive task (1,000 in total). Third, we develop systematic evaluation metrics about execution stability and generalization robustness to evaluate CodeAsPolicy, VLA, and language-conditioned H-VLA approaches. Our comprehensive evaluation reveals critical limitations in current methods: models struggle with interactive reasoning tasks due to significant gaps between visual understanding and motor execution. We provide fine-grained analysis of these limitations.

Xianhao Wang, Xiaojian Ma, Haozhe Hu +7

University of Science and Technology of China · Beijing Institute for General Artificial Intelligence (BIGAI) · Xidian University +2

Long-Horizon Robotic Manipulation Robotics Robot Manipulation Benchmarks VLMs for Robotics Embodied Reasoning

Apr 18, 2026cs.RO

LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

Robotic manipulation policies often degrade over extended horizons, yet existing benchmarks provide limited insight into why such failures occur. Most prior benchmarks are either simulation-based or report aggregate success, making it difficult to disentangle the distinct sources of temporal difficulty in real-world execution. We introduce LongBench, a real-world benchmark for evaluating long-horizon manipulation. LongBench consists of over 1,000 real-world episodes, covering two complementary regimes: Context-Independent (fully observable) and Context-Dependent (ambiguity-driven). By organizing tasks into capability- and ambiguity-specific subsets, LongBench enables mechanism-aware evaluation of execution robustness, temporal consistency, and context-dependent reasoning. Evaluating six state-of-the-art policies reveals that long-horizon performance is not governed by a single factor. We observe that performance in fully observable settings is more strongly associated with execution robustness, while contextual difficulty varies across tasks and is not consistently improved by memory-based methods. We hope that LongBench serves as a useful benchmark for studying long-horizon manipulation and for developing policies with stronger robustness across both execution and contextual challenges.

Xueyao Chen, Jingkai Jia, Tong Yang +3

Fudan University

Long-Horizon Robotic Manipulation Robotic Policy Evaluation Robotic Manipulation Robot Manipulation Benchmarks

Apr 16, 2026cs.RO

HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments

We present HRDexDB, a real-world 4D dexterous grasping dataset capturing 3D hand-object interaction trajectories over time across five embodiments. The dataset comprises 3.2K trials over 100 diverse objects. Using a synchronized multi-camera system and an integrated reconstruction pipeline, HRDexDB provides multi-view and egocentric RGB observations, 3D hand geometry, robot states, and object 6D pose trajectories, together with success/failure annotations. Human and robotic hands interact with shared objects, enabling the study of embodiment-dependent grasp strategies and contact patterns. We demonstrate the dataset's utility through human-to-robot contact map transfer, visual robot-object contact estimation, and retrieval-assisted grasping. Together, these results establish HRDexDB as a resource for studying and learning dexterous interactions across human and robotic embodiments.

Jongbin Lim, Taeyun Ha, Seongho Cha +6

Seoul National University · RLWRLD

Robotic Grasping Dexterous Manipulation Robotic Data Collection Cross-Embodiment Robot Learning Robot Manipulation Benchmarks Human-Object Interaction

Apr 16, 2026cs.AI

ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints

Intelligent embodied agents should not simply follow instructions, as real-world environments often involve unexpected conditions and exceptions. However, existing methods usually focus on directly executing instructions, without considering whether the target objects can actually be manipulated, meaning they fail to assess available affordances. To address this limitation, we introduce DynAfford, a benchmark that evaluates embodied agents in dynamic environments where object affordances may change over time and are not specified in the instruction. DynAfford requires agents to perceive object states, infer implicit preconditions, and adapt their actions accordingly. To enable this capability, we introduce ADAPT, a plug-and-play module that augments existing planners with explicit affordance reasoning. Experiments demonstrate that incorporating ADAPT significantly improves robustness and task success across both seen and unseen environments. We also show that a domain-adapted, LoRA-finetuned vision-language model used as the affordance inference backend outperforms a commercial LLM (GPT-4o), highlighting the importance of task-aligned affordance grounding.

Pei-An Chen, Yong-Ching Liang, Jia-Fong Yeh +4

National Taiwan University · National Yang Ming Chiao Tung University · National Tsing Hua University

Robot Task Planning Robot Manipulation Benchmarks VLMs for Robotics

Mar 30, 2026cs.RO

ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation

Vision-Language-Action (VLA) models and world-action models have emerged as central paradigms for general-purpose robotic intelligence, yet their empirical progress remains constrained by the absence of evaluation protocols that are both physically realistic and diagnostically controlled. Simulator-centric benchmarks provide scale and reproducibility, but cannot fully capture the reality gap induced by perception noise, contact dynamics, latency, calibration error, and hardware constraints. Conversely, real-robot evaluations are often fragmented across platforms, scenes, objects, and scoring rules, making fair comparison and failure attribution difficult. We introduce ManipArena, a standardized real-robot evaluation framework for studying manipulation generalization under matched physical conditions. ManipArena comprises 20 tasks, 10,812 expert trajectories, 13.5M frames, and approximately 188 robot hours across tabletop and mobile manipulation. The framework combines schema-defined task variation, stratified in-domain, visualshift, and semantic-OOD trials, subtask-level partial-credit scoring, three-level language annotations, low-level motor signals, and paired real-to-sim environments reconstructed from physical scenes. Using ManipArena, we evaluate seven tabletop configurations spanning VLA and world-action-model policies. The results show that real-robot conclusions depend not only on architecture, but also on model provenance, fine-tuning regime, data sampling, and annotation granularity. ManipArena thus provides a reproducible and interpretable foundation for diagnosing capability boundaries and failure modes in embodied generalization.

Yu Sun, Meng Cao, Yang Ping +24

Sun Yat-sen University · X Square Robot · MBZUAI +2

Generalization in Robotic Manipulation Robotic Manipulation Robot Manipulation Benchmarks VLMs for Robotics