Learning Manipulation-Sufficient Representations via Outcome Bottlenecks

Md Selim Sarowar; Sungho Kim

cs.ROSep 2, 2026

Learning Manipulation-Sufficient Representations via Outcome Bottlenecks

Authors: Md Selim Sarowar, Sungho Kim

Organizations: Department of Electronics Engineering, Yeungnam University, Gyeongsan 38541, Republic of Korea.

Abstract

Networked manipulation endpoints couple perception to actuation across compute- and bandwidth-limited links, yet commonly exchange dense geometric states optimized for fidelity rather than action outcomes. A stochastic representation is learned with a policy-free, action-conditioned outcome bottleneck: marginal outcome log-loss supplies distortion and a KL term regularizes rate. The construction is motivated by the minimal statistic that preserves the outcome distribution of every admissible action, while the implemented finite model is evaluated as a rate-regularized mixture predictor. The same encoder and outcome head support grasp selection, singleton conformal filtering, active viewpoint selection, and latent test-time adaptation. A finite-probe theorem identifies the local level-set tangent space with the null space of an outcome Jacobian. The synthetic oracle verifies this result; on scanned objects, an analytic surrogate agrees with measured simulator invariances within $0.56^\circ$ . Across 11,979 simulated grasps on 13 objects, a reconstructed-geometry wrench score attains 0.542 AUC against lift success and falls below chance on curved objects, while our representation attains 0.876. At 25% commitment, executed-grasp success is 0.503 versus 0.984. The 512-byte interface is $288\times$ smaller than one RGB-D frame and runs at 16,ms per CPU decision. Independent synthetic points track the tested conformal levels; scanned-object all-pair coverage is reported as a clustered empirical diagnostic. On unseen objects, within-scene AUC falls to 0.569, and a full-feedback update raises empirical mean pairwise coverage from 0.728 to 0.883.

Figures & tables

Symbol	Meaning
$x\in\mathcal{X}$	continuous state ( 1 ), $d_{\mathcal{X}}=14$ per fixed mesh
$o\in\mathcal{O}$	RGB-D observation (or view set)
$a\in\mathcal{A}$	grasp action ( 2 ), $a\in\mathbb{R}^{7}$
$b\in\mathcal{B}$	sensing action (viewpoint index)
$y\in\mathcal{Y}$	outcome $(\mathrm{succ},\mathrm{margin},\mathrm{slip})$
$M(\cdot\|x,a)$	task-outcome kernel used by the theory

TABLE I : Principal notation. Symbol collisions with common single letters are avoided: the latent prior is

p_{0}(z)

, box half-extents are

e

, and the sensing budget is

B_{\mathrm{sense}}

Quantity	Value
Encoder parameters	11.41 M
Head parameters	0.251 M
Encoder payload $(\mu,\log\sigma^{2})$ , fp32 / fp16	512 B / 256 B
Single RGB-D frame, fp32	147.5 kB
Eight-view set, fp32	1.18 MB
Encoder forward, 1 / 2 threads	25.6 / 14.9 ms

TABLE II : Measured interface budget (Ours; released

\alpha{=}0.1

checkpoint, fp32, two x86-64 CPU cores, median of 60 runs). No GPU or accelerator is used.

Fig. 1 : The outcome-bottleneck perception interface. A sensor node encodes the RGB-D observation (or a fused view set) into a stochastic distribution $q_{\theta}(z|o)$ ; its 128 scalar parameters are the only traffic across the perception-to-decision boundary. The decision node propagates $K$ encoder samples through the outcome head $p_{\psi}(y|z,a)$ to obtain $s(o,a)$ and $v(o,a)$ per candidate grasp, from which four capabilities derive without further learning: risk-averse selection ( 24 ), singleton-filtered abstention ( 27 ), ambiguity-triggered viewpoint acquisition ( 30 ), and probe assimilation ( 34 ).

Predictor	Pooled	Within-scene	Best acc.
Chance / majority	0.500	0.500	0.597
Ferrari–Canny (recon.)	0.542	0.548	0.612
Ours	0.876	0.639	0.784
Privileged state	–	0.685	–

TABLE III : Predictive validity against the realized lift outcome. 11,979 executable grasps on 13 scanned objects; base success rate 0.597. The privileged-state row is an MLP on the 14-vector ( 1 ), an input unavailable to any perception system. Because that vector omits object identity and mesh, it does not by itself determine the simulator outcome across the corpus, so it is a baseline, not a true upper bound; residual rollout stochasticity bounds all predictors below 1.

Fig. 2 : Pooled AUC against the realized lift, stratified by whether the bounding-box reconstruction is exact (box-hulled objects) or wrong (curved objects). The analytic score degrades to below chance where its geometry is hallucinated; the outcome-trained representation improves there.

Act rate	Ferrari–Canny	Ours
0.25 (selective)	0.503	0.984
1.00 (always act)	0.665	0.692
Random grasp	0.623

TABLE IV : Deployment value: the system ranks a scene’s candidates, executes its favorite, and commits only on its most confident scenes. Entries are success rates of executed grasps over 1,997 test scenes.

Fig. 3 : Risk-coverage behavior. Success of executed grasps as the act rate sweeps from selective to always-on. The proxy’s curve falls as it grows more selective, crossing below the random-grasp floor; ours rises to 0.984 at a 25% act rate.

	$\alpha=0.05$	$\alpha=0.10$	$\alpha=0.20$
Target coverage $1-\alpha$	0.950	0.900	0.800
Pairwise coverage (Ours)	0.946	0.897	0.800
Singleton precision	0.792	0.744	0.673
Abstention rate	0.823	0.743	0.598
Retained action fraction	0.169	0.235	0.373

TABLE V : Empirical singleton filtering on the scanned-object corpus (

\beta_{\mathrm{succ}}=30

; 16,000 action records over

\approx

2,000 scenes). Because calibration pooled eight correlated actions per scene, these values are descriptive and Theorem 6 does not apply. Singleton precision is a distinct selection-conditional estimand.

Fig. 4 : Empirical against target coverage on the synthetic oracle (64,000 test points per level, where the outcome kernel is the analytic map with a closed-form indistinguishability class). The split-conformal sets track the diagonal at every level; the right panel relates the result to the KL rate surrogate.

	Fixed $\hat{q}$	ACI (anchored)
Marginal coverage	0.899	0.900
Worst 500-decision window	0.702	0.744
Worst object segment	0.720	0.834
Per-object dispersion (std)	0.111	0.035

TABLE VI : Empirical coverage under an object-sorted stream (Ours; 16,000 decisions sorted by object, target 0.90), a full-feedback stress test: every decision supplies a label, unlike the selective feedback of a deployed policy that abstains. The fixed column uses the calibration-fold quantile; the ACI column uses ( 28 ) with

\gamma=0.01

. Marginal coverage is order-invariant; local coverage is not.

Object	$\mathrm{rank}\,J_{N}$	$\dim\ker J_{N}$	Max angle [ ∘ ]
alphabet soup	12	2	0.05
BBQ sauce	14	0	–
butter	14	0	–
chocolate pudding	14	0	–
cookies	14	0	–
cream cheese	14	0	–

TABLE VII : Finite-probe identifiability per object.

\dim\ker J_{N}

is the dimension of the analytic probe-map level-set tangent space at the settled state; the principal angle compares the analytic-tier

\ker J_{N}(x)

against the simulator’s independently measured outcome-invariant subspace on the true mesh. This is a surrogate-agreement measurement, not a computation of the theorem’s

J_{N}

for the simulator kernel (see text). For 6 of 13 objects

J_{N}

has full rank and the local level set is zero-dimensional (Remark 3 ). The comparison uses the fixed probe set, perturbation scale, and rank tolerance of the companion diagnostic configuration. Those numerical settings are not included in the present manuscript artifact, so reproducing the angles requires that configuration.

Fig. 5 : Verification of Theorem 5 on the synthetic oracle, whose indistinguishability class is known in closed form: principal angles between the predicted

\ker J_{N}(x)

and the measured invariant subspace concentrate near zero (mean

0.013^{\circ}

, max

0.056^{\circ}

over 50 states), and the recovered rank and null dimensions match the ground truth exactly.

$\beta$	$R$ [nats]	$D_{\mathrm{succ}}$	$\sum_{j}D_{j}$	Coverage	Abstention
1	0.004	0.656	0.725	0.896	0.643
3	0.088	0.622	0.767	0.892	0.674
10	0.280	0.598	0.870	0.904	0.736
30	0.534	0.582	0.993	0.897	0.743
100	3.286	0.412	0.956	0.905	0.367

TABLE VIII : Rate-distortion frontier over the sufficiency budget (Ours; 16,000 test points per row).

\beta

multiplies the success relevance term, so larger

\beta

emphasizes success prediction; the unweighted total

\sum_{j}D_{j}

is dominated by the deliberately sacrificed margin and slip likelihoods and need not fall. Descriptive all-pair coverage remains near 0.90 along the frontier, without a finite-sample claim for the clustered calibration.

Fig. 6 : The measured rate-distortion frontier ( 12 ). Each point is one trained model; the abscissa is the KL rate surrogate

R

(Section III ), not a coded wire length, and the ordinate is outcome distortion.

Variant	Within	Pooled	Resp.
Ours (full)	0.639	0.875	0.221
Uniform outcome budget	0.511	0.710	0.002
Constant $z$	0.506	0.615	0.031
$K{=}1$ sample	0.602	0.844	0.174
$d{=}16$	0.634	0.873	0.219
$d{=}32$	0.587	0.846	0.185

TABLE IX : Ablations on the scanned-object corpus (

\beta_{\mathrm{succ}}=300

except where the budget itself is ablated). The pooled AUC is reported to expose its failure as a referee: it scores the uniform-budget and no-perception variants near 0.7 although both have stopped ranking grasps.

Fig. 7 : Per-object base success rates (0.043 to 0.999) with pooled and within-scene AUC. The spread is why a pooled AUC can be won by object recognition alone, and why single-class or near-single-class scenes make within-scene AUC undefined or high-variance for the extreme-base-rate objects (Section VIII-I ).

Moderate base rate ( $0.3<\text{base}<0.7$ ): within-scene AUC is meaningful
Held-out object	base	Within	Top-1	Cov. fix	Cov. ACI
butter	0.48	0.569	0.474	0.703	0.895
cream cheese	0.41	0.546	0.429	0.543	0.896
tomato sauce	0.51	0.511	0.473	0.785	0.896
alphabet soup	0.52	0.479	0.500	0.774	0.897
choc. pudding	0.58	0.695	0.778	0.845	0.898

TABLE X : Leave-one-object-out transfer (Ours). Each row is a model trained with the named object absent and evaluated only on that object’s test scenes; the legacy all-pair filter is calibrated on the other twelve objects (nominal target

1-\alpha=0.90

). Within-scene AUC is averaged only over scenes containing both labels. BBQ sauce has too few valid scenes for a stable estimate and is reported as unavailable; the aggregate within-scene mean therefore excludes that entry. Top-1 success and pairwise coverage remain well defined.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 798 papers

More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning
Jul 10, 2026Yi Li, Alexandre Chapin, Liming Chen +2Robot Skill Learning Visuomotor Policy Learning
TU Darmstadt · 1TU Darmstadt · LIRIS +3
ComManip: Overfitting Manipulation Policies to Comfortable Regions
Sep 29, 2026Yidan Lai, Xinyi Chen, Shuquan Man +1Visuomotor Policy Learning Robotic Manipulation
South China University of Technology, Guangzhou, China · Nanyang Technological University, Singapore
Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field
Jun 29, 2026Licheng Zhong, Gim Hee LeeRL for Robotics Robotic Grasping
Department of Computer Science, National University of Singapore, Singapore

Jul 10, 2026J/K move · Enter open · S save

More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning

Yi Li, Alexandre Chapin, Liming Chen, Jan Peters+1

TU Darmstadt · 1TU Darmstadt · LIRIS+3

Robot Skill Learning Visuomotor Policy Learning Object-Centric Representation Learning Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Robotic manipulation policies rely on pre-trained vision models that give either a global scene embedding or a dense patch grid. Both mix task-relevant and task-irrelevant features. Object-centric slot representations are a structured alternative: they group features into a few per-object slots. We test what this structure buys on ManiSkill3 PickCube-v1, with a frozen encoder and a held-out-seed evaluation. Holding the policy, goal token, rendering, and calibration fixed and changing only the encoder, a frozen object-centric SPOT representation (DINO ViT-B/16 + Slot Attention) reaches 55.0

\pm

2.9% success, 22.4% above a dense DINO global-feature baseline (32.6

\pm

1.5%), with the same trainable policy and no encoder fine-tuning. More tokens alone do not help: a dense patch grid with 16x the tokens performs no better than the global feature. Adding an explicit 2D spatial goal and native-resolution rendering raises the full system to 68.7

\pm

4.2%, just below a privileged 3D-oracle upper bound (71.7

\pm

4.1%). An automated kinematic failure taxonomy then separates spatial-precision (Near-Miss) failures from object-tracking (No-Grasp) failures: spatial grounding reduces Near-Miss while leaving No- Grasp unchanged. The same taxonomy transfers to the harder StackCube-v1 and points to occlusion as the main bottleneck.

Abstract

Representation Learning

Explore similar work

More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning

ComManip: Overfitting Manipulation Policies to Comfortable Regions

Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field