cs.ROOct 31, 2026

Shades of Success: Evaluating Manipulation Quality from Object-Centric Constraints

Authors: Lorena Maria Genua, Sarvesh Prajapati, Ananya Trivedi, Taskin Padir

Organizations: Institute for Experiential Robotics, Northeastern University, Boston, Massachusetts, USA.

Abstract

As learned manipulation policies become increasingly capable of solving practical manipulation tasks, assessing how well a task is performed becomes a fundamental challenge. While binary labels confirm task success, they cannot distinguish successful executions with different levels of geometric precision. To address this, we introduce SPECTRA, an object-centric framework for quantifying execution quality in manipulation of rigid objects. From a small set of oracle demonstrations generated in simulation, SPECTRA represents object interactions through progress-aligned relative poses. SPECTRA identifies which relative-pose directions are constrained and their admissible variations from demonstrations. These constraints establish a reference for evaluating robotic manipulation tasks. To score an unseen execution, deviations beyond the admissible variation are summarized by a Task Quality Index Q in [0,1]. Across seven simulated tasks, SPECTRA recovers constraints with 94% accuracy using 30 demonstrations per task. On rollouts from three learned policies, its scores achieve pooled Spearman correlations with human judgments of p=0.81 over all rollouts and p=0.61 among successful rollouts, exceeding all baselines in both subsets. Using the task reference learned in simulation, we use SPECTRA to select the highest-scoring demonstrations for training a diffusion policy on hardware. Exploratory results show higher success rate than training on randomly selected demonstrations.

Figures & tables

Explore similar work

CardsList
  1. Kinematic Constraint-Referenced Evaluation for Articulated Object Manipulation

    Oct 27, 2026Baptiste Bonin, Caro Strickland, Audrey DurandArticulated Object ManipulationRobotic Policy Evaluation

  2. ComManip: Overfitting Manipulation Policies to Comfortable Regions

    Sep 29, 2026Yidan Lai, Xinyi Chen, Shuquan Man +1Visuomotor Policy LearningRobotic Manipulation