As learned manipulation policies become increasingly capable of solving practical manipulation tasks, assessing how well a task is performed becomes a fundamental challenge. While binary labels confirm task success, they cannot distinguish successful executions with different levels of geometric precision. To address this, we introduce SPECTRA, an object-centric framework for quantifying execution quality in manipulation of rigid objects. From a small set of oracle demonstrations generated in simulation, SPECTRA represents object interactions through progress-aligned relative poses. SPECTRA identifies which relative-pose directions are constrained and their admissible variations from demonstrations. These constraints establish a reference for evaluating robotic manipulation tasks. To score an unseen execution, deviations beyond the admissible variation are summarized by a Task Quality Index Q in [0,1]. Across seven simulated tasks, SPECTRA recovers constraints with 94% accuracy using 30 demonstrations per task. On rollouts from three learned policies, its scores achieve pooled Spearman correlations with human judgments of p=0.81 over all rollouts and p=0.61 among successful rollouts, exceeding all baselines in both subsets. Using the task reference learned in simulation, we use SPECTRA to select the highest-scoring demonstrations for training a diffusion policy on hardware. Exploratory results show higher success rate than training on randomly selected demonstrations.
Figures & tables
Fig. 2: Overview of SPECTRA. 1. Constraints estimation. Oracle demonstrations are segmented and aligned to shared progress samples. At each progress sample l , the task is represented by the relative poses between the reference object r , the manipulated object m , the gripper g , and the support object s . Deviations from the nominal pose Tˉj,l are mapped to its tangent space, and eigen decomposition of their covariance Σj,l identifies the constrained directions Ij , whose standard deviation falls below σpin . 2. Policy evaluation. An unseen execution incurs cost only where its normalized deviation along constrained directions exceeds the tolerance z0 . The average cost and a terminal gate give the Task Quality Index Q∈[0,1] .
Q (Ours) at commanded deviation
Task
Axis
Clean
5∘
10∘
25∘
45∘
ρ
PlaceCan
tilt
0.99
0.97
0.94
0.82
0.65
−0.98
CloseDrawer
tilt
1.00
0.88
0.81
0.58
0.42
−0.97
PlugCharger
roll
0.99
0.98
0.94
0.84
0.76
−0.94
PegInsertion
roll
0.99
0.95
0.90
0.71
0.56
−0.94
StackCube
tilt
0.99
0.99
0.98
0.90
0.75
−0.88
TABLE I: Mean Q under controlled deviations. ρ : Spearman’s correlation between deviation and Q per task.
Policy
Q (Ours) ↑
Qwen3-VL ↑
SQS ↑
P (m) ↓
J (m/s 3 ) ↓
DP
0.75
0.52
0.79
0.82
4.14
ACT
0.64
0.55
0.78
0.63
2.13
π0.5
0.71
0.45
0.74
0.77
4.89
TABLE II: Mean of each metric per policy rollouts; darker blue indicates the better value within each metric.
(a) All rollouts ( N=105 )
Task
Q (Ours)
Qwen3-VL
SQS
CloseDrawer
0.37
0.25
0.16
PlaceCan
0.71
0.44
0.20
PlaceSphere
0.90
0.37
0.79
PokeCube
0.81
0.22
0.31
StackCube
0.64
0.56
0.42
TABLE III: Spearman correlation with mean human ratings. Each task has 15 rollouts, ns of them successful; Overall pools all tasks. SQS: PRM-as-a-Judge success quality score; P : TCP path length; J : mean Cartesian jerk, correlated as −P and −J . Bold: best per row. –: undefined, as Qwen3-VL assigns all rollouts the same score.
Fig. 3: Example of successful rollouts with different SPECTRA Quality score Q. Numbered keyframes mark events in the frame cost c(t) ; Qpath(t) shows the cumulative score. The bin dragged during release and the peg pushed against the box during alignment lower their final scores.
Fig. 4: SPECTRA quality index Q of the 180 teleoperated demonstrations per task, sorted by rank.
Setting
Rec. (%)
Qˉclean
ΔQ10
ΔQ45
Demonstration budget K
5
82.4
0.91
0.088
0.347
10
92.6
0.98
0.075
0.316
30 †
94.3
0.99
0.055
0.294
70
96.0
0.99
0.052
0.287
Within-phase resolution M
TABLE IV: Ablations, one parameter at a time with the others at their defaults ( † ). Rec.: constraint recovery. Qˉclean : mean Q on unseen clean executions. ΔQd : mean decrease in Q under a d∘ deviation. σpin , σ0 in mm.
Although Vision--Action (VA) and Vision--Language--Action (VLA) policies have advanced robotic manipulation, their evaluation remains dominated by binary success rates, which obscure process-level differences among executions that complete the same task. We introduce Eval-Actions, a diagnostic evaluation methodology and real-robot benchmark for fine-grained execution-quality assessment of learned manipulation policies. Eval-Actions combines criteria-based Expert Grading (EG), Rank-Guided (RG) labels that align measurable motion indicators with expert rankings, and Chain-of-Thought-style (CoT) annotations that explain observable quality differences. The benchmark contains 13K+ teleoperated and policy-generated real-robot episodes covering 150+ tasks and approximately 52 hours of recordings with RGB-D videos, robot-state trajectories, task descriptions, and success/failure labels. Its densely annotated subset provides EG/RG/CoT supervision for training and evaluation. We further provide AutoEval, a reference multimodal evaluator that predicts quality scores, task outcomes, and diagnostic explanations from RGB temporal evidence and compact kinematic summaries. On the annotated Eval-Actions test split, AutoEval-S achieves Spearman rank correlations (SRCCs) of 0.81 and 0.84 under EG and RG, with success detection accuracies of 90.6% and 91.0%; AutoEval-P reaches 0.70 SRCC under CoT. Analyses of expert consistency, physical-metric baselines, modality ablations, structured generalization, and offline policy ranking show that Eval-Actions provides standardized, interpretable diagnostic signals complementary to success-rate evaluation.
Manipulation policies are evaluated predominantly by binary task success, which cannot tell whether the commanded gesture is the one the object's kinematics admit. We introduce Constraint-Referenced Strategy (CRS), which projects the commanded end-effector twist onto the object's joint axis, separating progress along the joint from off-constraint waste. CRS requires a joint model and a joint-following scripted controller derived from it, but no recorded demonstration or learned judge. On 74 simulated articulated objects, success registers three transfer failures as lower rates but attributes none, while the two CRS channels give each a distinct signature. A door-trained behavior-cloning policy opens drawers in 59 of 110 episodes, and in these passes it commands about 40× the off-axis rotation of a drawer-trained policy's passes (1.27 vs.\ 0.032~rad/s). Relative to a scripted controller scored with the same axis, CRS verdicts are unchanged under joint-axis errors up to 30∘; absolute thresholds tolerate about 8∘.
Baptiste Bonin, Caro Strickland, Audrey Durand
Universit´e Laval · Mila – Quebec AI Institute · CIFAR AI Chair
Training robot manipulation policies relies on costly robot demonstrations, making large-scale data collection impractical. Meanwhile, to improve policy generalization, existing approaches seek greater diversity in visual observations by varying object placements, viewpoints, and robot configurations during data collection. However, under a limited demonstration budget, this strategy forces the policy to model diverse visual observations, providing insufficient supervision to learn reliable observation-action correspondences under similar local conditions. Our study reveals that policies trained under this strategy achieve lower task success rates than those trained within a compact, visually and kinematically stable region. We refer to these stable regions as comfortable manipulation regions. To exploit this finding, we propose ComManip, a learning paradigm that specializes manipulation policies to comfortable manipulation regions. During inference, ComManip repositions the mobile base until the detected target center enters the familiar image-space range estimated from comfortable-region demonstrations. It then executes the same manipulation policy, enabling effective manipulation across diverse target locations. We conduct extensive experiments across multiple manipulation tasks, demonstration budgets, and policy families including ACT, π0.5, RDT, OpenVLA-OFT, and SmolVLA. The results demonstrate that ComManip improves task success by roughly 20 percentage points or more across different policy architectures in large workspaces under limited demonstration budgets, suggesting that specializing manipulation policies to comfortable regions provides a more data-efficient learning paradigm for manipulation.
Yidan Lai, Xinyi Chen, Shuquan Man +1
South China University of Technology, Guangzhou, China · Nanyang Technological University, Singapore