cs.ROSep 27, 2026

Learning with Object-centric Representations of Tactile Interactive Perception for Robot Manipulation

Authors: Xinyi Yang, Zilin Si, Zhuowei Xu, Zeynep Temel, Oliver Kroemer

Organizations: Robotics Institute Carnegie Mellon University United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Implicit object properties that are difficult to directly infer from vision, such as material, container contents, or softness, can be revealed through tactile sensing and exploratory interactions. However, because tactile signals are transient and sparse, extracting informative tactile events and effectively incorporating them into robotic manipulation remains a challenge. In this work, we present an object-centric context-aware manipulation framework that learns task-agnostic object representations through tactile exploration. A token learner autonomously selects representative tactile segments from long-horizon exploration, while contrastive alignment with descriptive text embeddings enables a latent space that captures multiple physical object properties. These learned representations are then used as semantic context to guide object-centric manipulation policies and adapt strategies based on object properties. Experiments show that the learned representations achieve 93% and 84% property estimation accuracy on seen and unseen objects. Evaluated on three tasks involving visually ambiguous objects, i.e. multi-object rearrangement, pouring, and box opening, the proposed framework improves both target selection and property-dependent manipulation adaptation, raising task success, aggregated over all evaluation trials, from 41% to 92% on seen objects and from 19% to 67% on unseen objects over baseline policies without object-context conditioning. Videos and additional results are available at https://xinyiyxyx.github.io/tactile-object-centric/.

Figures & tables

Figure 1: An object-centric manipulation learning framework. Task-agnostic object representations are learned through tactile exploration and then used as semantic context for downstream manipulation. In a multi-object rearrangement task, the policy adapts grasping strategies based on object rigidity and placement strategies based on object contents.
Figure 2: Object representation learning through tactile exploration. Each object is explored with a sequence of pre-defined actions while vibrotactile signals, 6-axis end-effector force measurements, and robot end-effector linear velocities are recorded. A token learner selects the most informative KK tokens, which are aligned with text embeddings of physical-property descriptions using contrastive learning to produce semantic object representations.
Figure 3: Object-centric and context-aware manipulation policy learning. Built on the Action Chunking Transformer, the policy conditions on learned object context and object position to focus on target-specific visual, tactile, and proprioceptive observations for adaptive manipulation.
Figure 4: Evaluation of learned tactile object representations. (a,b) Object property estimation on seen and unseen objects. Multi-modal sensing (vibrotactile, force, and velocity) achieves the best performance and generalizes to unseen objects. (c,d) Token learner ablation against trajectory-compression baselines. Bars in (a–d) show mean accuracy across eight training seeds (0–7), and error bars denote ±1\pm 1 sample standard deviation. Numerical results are provided in Table 5 . (e) Objects for model training. (f,g) Seen and unseen objects for evaluation. (h) Examples of selected representative tokens. This shows that the model prioritizes informative interactions such as squeezing, swinging, shaking, and cavity tapping.
Figure 5: Visualization of learned object embeddings and aligned text embeddings on each property axis using UMAP dimension reduction. Embedding clusters show semantic alignment between tactile representations and text anchors.
Figure 6: (a) Multi-object rearrangement task. The policy selects appropriate grasping strategies and places objects into task-specific target regions based on their rigidity and contents. (b) Pouring task. The policy chooses a top grasp for the bottle if it is empty and directly recycles it, or uses a side grasp to pour out the liquid before recycling it. (c) Box opening task with a multi-finger hand. The policy chooses to open heavy boxes containing screws and takes no action for empty boxes.
Figure 7: Policy evaluation results. (a) Performance of the multi-object rearrangement task with seen and unseen objects. (b) Ablation with a subset of visually identical objects on context type and reactive tactile sensing. (c, d) Performance of the pouring and box opening tasks.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

ParameterSwingShakeTap on cavityTap on tableSqueezeSlide
Amplitude[20,28]∘[20,28]^{\circ}[20,30]∘[20,30]^{\circ}––––
Lifting distance––[3,3.3][3,3.3] cm[1.5,2.5][1.5,2.5] cm––
Grasp force––––0.1 N–
Sliding distance–––––[5,8][5,8] cm
Period[0.8,1.0][0.8,1.0] s[0.35,0.5][0.35,0.5] s[0.2,0.8][0.2,0.8] s[0.4,0.6][0.4,0.6] s–[1.6,2.2][1.6,2.2] s
Repetitions3[3,5][3,5][5,8][5,8][4,7][4,7]33
Appendix
Table 1: Data randomization configurations for object exploration data collection
 ContentFill levelMaterialRigidity 
Splitemptygran.liquidsolidemptyhalffullglassmetalplasticdeform.rigidTotal
Train304238263059472446663799136
Seen / Unseen1296012966615121527
Appendix
Table 2: Per-axis label distribution of the training pool and the evaluation sets. The Seen and Unseen sets share identical distributions and are shown in a single row.
Table 3: Attribute configurations of the object exploration data used for representation learning and evaluation.
AxisClassTraining promptsInference paraphrases
contentsolidsolid contenta solid object inside; one solid piece inside
granulargranular; particlegrains inside ; powder ; pellets
liquidliquid; fluidwater inside ; sloshing liquid
content / fill levelemptyemptyhollow ; nothing inside ; contains nothing ; no contents ; vacant ; unfilled
fill levelfullfull; fully filledfilled to the top; completely filled
halfhalfat half capacity; midway level
Appendix
Table 4: Training prompts and the 29-paraphrase inference vocabulary. The empty class of the content and fill-level axes shares one training prompt; its paraphrases are split between the two axes during evaluation. Paraphrases in italics share no word stem with any training prompt and form the strictest evaluation subset (17 in total).
Method (split)ContentFill LevelMaterialRigidityAvg.
Modality ablation
Vibrotactile only (S)0.77±0.040.77\pm 0.040.82±0.070.82\pm 0.070.88±0.060.88\pm 0.060.95±0.030.95\pm 0.030.86±0.040.86\pm 0.04
Vibrotactile only (U)0.75±0.110.75\pm 0.110.71±0.110.71\pm 0.110.71±0.120.71\pm 0.120.88±0.050.88\pm 0.050.76±0.050.76\pm 0.05
Vibrotactile + force (S)0.81±0.070.81\pm 0.070.86±0.050.86\pm 0.050.84±0.150.84\pm 0.150.98±0.020.98\pm 0.020.87±0.060.87\pm 0.06
Vibrotactile + force (U)0.80±0.080.80\pm 0.080.78±0.060.78\pm 0.060.71±0.230.71\pm 0.230.92±0.040.92\pm 0.040.80±0.080.80\pm 0.08
Vibrotactile + velocity (S)0.86±0.060.86\pm 0.060.86±0.030.86\pm 0.030.87±0.090.87\pm 0.090.96±0.030.96\pm 0.030.89±0.030.89\pm 0.03
Appendix
Table 5: Object property estimation ablations on seen (S) and unseen (U) objects at different locations. Values are mean ±\pm sample standard deviation over 8 seeds (0–7). Avg. is the mean accuracy across the four property axes; its standard deviation is computed across the per-seed averages.
Figure 8: (a) Per-axis property estimation accuracy of ours and the CE baseline on seen (S) and unseen (U) objects, mean ±\pm std over 8 seeds. (b) Axis-average accuracy vs. exploration time with a causal top- KK token buffer decoded every 15 s and at trajectory completion, using only observation windows completed by the decoding time. Accuracy is averaged over four object attributes and 27 test trajectories per split. Curves show the mean across eight training seeds; shaded bands indicate ±1\pm 1 standard deviation across seeds. The horizontal axis expresses the decoding times as a fraction of full duration, computed as the mean elapsed exploration time divided by the mean full trajectory duration within each split.
MethodContentFill LevelMaterialRigidityAvg.
Ours (Seen)0.87±0.030.87\pm 0.030.94±0.040.94\pm 0.040.94±0.040.94\pm 0.040.96±0.030.96\pm 0.030.93±0.020.93\pm 0.02
Ours (Unseen)0.85±0.080.85\pm 0.080.81±0.040.81\pm 0.040.80±0.140.80\pm 0.140.91±0.050.91\pm 0.050.84±0.060.84\pm 0.06
CE (Seen)0.89±0.050.89\pm 0.050.92±0.060.92\pm 0.060.90±0.030.90\pm 0.030.95±0.040.95\pm 0.040.92±0.020.92\pm 0.02
CE (Unseen)0.84±0.060.84\pm 0.060.85±0.050.85\pm 0.050.82±0.100.82\pm 0.100.89±0.040.89\pm 0.040.85±0.040.85\pm 0.04
Appendix
Table 6: Numerical values of Fig. 8 (a).
Figure 9: Object representation learning with a multi-finger robotic hand (DeltaHand [ 29 ] ). (a–e) Five exploration procedures: squeezing the object by closing all fingers, sliding one finger along the object surface while the remaining fingers provide support, lifting the object, in-hand horizontal shifting, and in-hand tilting. (f, g) Object property estimation is evaluated on both (h) seen and (i) unseen objects. (j) Three examples of selected tokens from the token learner are visualized, showing that most tokens are selected from squeezing, horizontal shifting, and tilting motions.
Figure 10: Object-centric and context-aware manipulation policy learning. Built on the Action Chunking Transformer, the policy conditions on learned object context and object position to focus on target-specific visual, tactile, and proprioceptive observations for adaptive manipulation.
Table 7: Manipulation policy evaluation across different task settings.
TaskRequired hidden property
Multi-object rearrangementrigidity + content
Three-container subsetrigidity + content
Pouringfill level
Box openingweight / fill level
Appendix
Table 8: Manipulation tasks and the properties required for strategy selection.
More in this topic

Tactile Sensing for Robotic Manipulation

Tactile sensing for robotic manipulation uses contact signals to help robots perceive forces, prevent slips, and handle objects reliably.

See all 178 papers

Explore similar work

CardsList

  1. TacO: Benchmarking Tactile Sensors for Object Manipulation

    May 21, 2026Anya Zorin, Zilin Si, Myungsun Park +11Tactile SensingContact-Rich Robotic Manipulation

    UC San Diego United States · CMU · UC San Diego +2

  2. Tactile-Based Robotic Manipulation: From Mechanics-Driven to Data-Driven Perspectives

    Oct 21, 2026Jingyi Xiang, Yixuan Wang, Wenzhen YuanTactile Sensing for Robotic ManipulationRobotic Tactile Sensing

    Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, Illinois, USA, 61801

  3. TactX: Learning Shared Tactile Representations Across Diverse Sensors

    Jun 30, 2026Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2Cross-Modal Representation LearningContact-Rich Robotic Manipulation

    UC San Diego · Seoul National University · Amazon FAR