As learned manipulation policies become increasingly capable of solving practical manipulation tasks, assessing how well a task is performed becomes a fundamental challenge. While binary labels confirm task success, they cannot distinguish successful executions with different levels of geometric precision. To address this, we introduce SPECTRA, an object-centric framework for quantifying execution quality in manipulation of rigid objects. From a small set of oracle demonstrations generated in simulation, SPECTRA represents object interactions through progress-aligned relative poses. SPECTRA identifies which relative-pose directions are constrained and their admissible variations from demonstrations. These constraints establish a reference for evaluating robotic manipulation tasks. To score an unseen execution, deviations beyond the admissible variation are summarized by a Task Quality Index Q in [0,1]. Across seven simulated tasks, SPECTRA recovers constraints with 94% accuracy using 30 demonstrations per task. On rollouts from three learned policies, its scores achieve pooled Spearman correlations with human judgments of p=0.81 over all rollouts and p=0.61 among successful rollouts, exceeding all baselines in both subsets. Using the task reference learned in simulation, we use SPECTRA to select the highest-scoring demonstrations for training a diffusion policy on hardware. Exploratory results show higher success rate than training on randomly selected demonstrations.
Figures & tables
Fig. 2: Overview of SPECTRA. 1. Constraints estimation. Oracle demonstrations are segmented and aligned to shared progress samples. At each progress sample l , the task is represented by the relative poses between the reference object r , the manipulated object m , the gripper g , and the support object s . Deviations from the nominal pose Tˉj,l are mapped to its tangent space, and eigen decomposition of their covariance Σj,l identifies the constrained directions Ij , whose standard deviation falls below σpin . 2. Policy evaluation. An unseen execution incurs cost only where its normalized deviation along constrained directions exceeds the tolerance z0 . The average cost and a terminal gate give the Task Quality Index Q∈[0,1] .
Q (Ours) at commanded deviation
Task
Axis
Clean
5∘
10∘
25∘
45∘
ρ
PlaceCan
tilt
0.99
0.97
0.94
0.82
0.65
−0.98
CloseDrawer
tilt
1.00
0.88
0.81
0.58
0.42
−0.97
PlugCharger
roll
0.99
0.98
0.94
0.84
0.76
−0.94
PegInsertion
roll
0.99
0.95
0.90
0.71
0.56
−0.94
StackCube
tilt
0.99
0.99
0.98
0.90
0.75
−0.88
TABLE I: Mean Q under controlled deviations. ρ : Spearman’s correlation between deviation and Q per task.
Policy
Q (Ours) ↑
Qwen3-VL ↑
SQS ↑
P (m) ↓
J (m/s 3 ) ↓
DP
0.75
0.52
0.79
0.82
4.14
ACT
0.64
0.55
0.78
0.63
2.13
π0.5
0.71
0.45
0.74
0.77
4.89
TABLE II: Mean of each metric per policy rollouts; darker blue indicates the better value within each metric.
(a) All rollouts ( N=105 )
Task
Q (Ours)
Qwen3-VL
SQS
CloseDrawer
0.37
0.25
0.16
PlaceCan
0.71
0.44
0.20
PlaceSphere
0.90
0.37
0.79
PokeCube
0.81
0.22
0.31
StackCube
0.64
0.56
0.42
TABLE III: Spearman correlation with mean human ratings. Each task has 15 rollouts, ns of them successful; Overall pools all tasks. SQS: PRM-as-a-Judge success quality score; P : TCP path length; J : mean Cartesian jerk, correlated as −P and −J . Bold: best per row. –: undefined, as Qwen3-VL assigns all rollouts the same score.
Fig. 3: Example of successful rollouts with different SPECTRA Quality score Q. Numbered keyframes mark events in the frame cost c(t) ; Qpath(t) shows the cumulative score. The bin dragged during release and the peg pushed against the box during alignment lower their final scores.
Fig. 4: SPECTRA quality index Q of the 180 teleoperated demonstrations per task, sorted by rank.
Setting
Rec. (%)
Qˉclean
ΔQ10
ΔQ45
Demonstration budget K
5
82.4
0.91
0.088
0.347
10
92.6
0.98
0.075
0.316
30 †
94.3
0.99
0.055
0.294
70
96.0
0.99
0.052
0.287
Within-phase resolution M
TABLE IV: Ablations, one parameter at a time with the others at their defaults ( † ). Rec.: constraint recovery. Qˉclean : mean Q on unseen clean executions. ΔQd : mean decrease in Q under a d∘ deviation. σpin , σ0 in mm.