cs.ROSep 27, 2026

Learning with Object-centric Representations of Tactile Interactive Perception for Robot Manipulation

Authors: Xinyi Yang, Zilin Si, Zhuowei Xu, Zeynep Temel, Oliver Kroemer

Organizations: Robotics Institute Carnegie Mellon University United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Implicit object properties that are difficult to directly infer from vision, such as material, container contents, or softness, can be revealed through tactile sensing and exploratory interactions. However, because tactile signals are transient and sparse, extracting informative tactile events and effectively incorporating them into robotic manipulation remains a challenge. In this work, we present an object-centric context-aware manipulation framework that learns task-agnostic object representations through tactile exploration. A token learner autonomously selects representative tactile segments from long-horizon exploration, while contrastive alignment with descriptive text embeddings enables a latent space that captures multiple physical object properties. These learned representations are then used as semantic context to guide object-centric manipulation policies and adapt strategies based on object properties. Experiments show that the learned representations achieve 93% and 84% property estimation accuracy on seen and unseen objects. Evaluated on three tasks involving visually ambiguous objects, i.e. multi-object rearrangement, pouring, and box opening, the proposed framework improves both target selection and property-dependent manipulation adaptation, raising task success, aggregated over all evaluation trials, from 41% to 92% on seen objects and from 19% to 67% on unseen objects over baseline policies without object-context conditioning. Videos and additional results are available at https://xinyiyxyx.github.io/tactile-object-centric/.

Figures & tables

Figure 1: An object-centric manipulation learning framework. Task-agnostic object representations are learned through tactile exploration and then used as semantic context for downstream manipulation. In a multi-object rearrangement task, the policy adapts grasping strategies based on object rigidity and placement strategies based on object contents.
Figure 2: Object representation learning through tactile exploration. Each object is explored with a sequence of pre-defined actions while vibrotactile signals, 6-axis end-effector force measurements, and robot end-effector linear velocities are recorded. A token learner selects the most informative KK tokens, which are aligned with text embeddings of physical-property descriptions using contrastive learning to produce semantic object representations.
Figure 3: Object-centric and context-aware manipulation policy learning. Built on the Action Chunking Transformer, the policy conditions on learned object context and object position to focus on target-specific visual, tactile, and proprioceptive observations for adaptive manipulation.
Figure 4: Evaluation of learned tactile object representations. (a,b) Object property estimation on seen and unseen objects. Multi-modal sensing (vibrotactile, force, and velocity) achieves the best performance and generalizes to unseen objects. (c,d) Token learner ablation against trajectory-compression baselines. Bars in (a–d) show mean accuracy across eight training seeds (0–7), and error bars denote ±1\pm 1 sample standard deviation. Numerical results are provided in Table 5 . (e) Objects for model training. (f,g) Seen and unseen objects for evaluation. (h) Examples of selected representative tokens. This shows that the model prioritizes informative interactions such as squeezing, swinging, shaking, and cavity tapping.
Figure 5: Visualization of learned object embeddings and aligned text embeddings on each property axis using UMAP dimension reduction. Embedding clusters show semantic alignment between tactile representations and text anchors.
Figure 6: (a) Multi-object rearrangement task. The policy selects appropriate grasping strategies and places objects into task-specific target regions based on their rigidity and contents. (b) Pouring task. The policy chooses a top grasp for the bottle if it is empty and directly recycles it, or uses a side grasp to pour out the liquid before recycling it. (c) Box opening task with a multi-finger hand. The policy chooses to open heavy boxes containing screws and takes no action for empty boxes.
Figure 7: Policy evaluation results. (a) Performance of the multi-object rearrangement task with seen and unseen objects. (b) Ablation with a subset of visually identical objects on context type and reactive tactile sensing. (c, d) Performance of the pouring and box opening tasks.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

ParameterSwingShakeTap on cavityTap on tableSqueezeSlide
Amplitude[20,28]∘[20,28]^{\circ}[20,30]∘[20,30]^{\circ}––––
Lifting distance––[3,3.3][3,3.3] cm[1.5,2.5][1.5,2.5] cm––
Grasp force––––0.1 N–
Sliding distance–––––[5,8][5,8] cm
Period[0.8,1.0][0.8,1.0] s[0.35,0.5][0.35,0.5] s[0.2,0.8][0.2,0.8] s[0.4,0.6][0.4,0.6] s–[1.6,2.2][1.6,2.2] s
Repetitions3[3,5][3,5][5,8][5,8][4,7][4,7]33
Appendix
Table 1: Data randomization configurations for object exploration data collection
 ContentFill levelMaterialRigidity 
Splitemptygran.liquidsolidemptyhalffullglassmetalplasticdeform.rigidTotal
Train304238263059472446663799136
Seen / Unseen1296012966615121527
Appendix
Table 2: Per-axis label distribution of the training pool and the evaluation sets. The Seen and Unseen sets share identical distributions and are shown in a single row.
Table 3: Attribute configurations of the object exploration data used for representation learning and evaluation.
AxisClassTraining promptsInference paraphrases
contentsolidsolid contenta solid object inside; one solid piece inside
granulargranular; particlegrains inside ; powder ; pellets
liquidliquid; fluidwater inside ; sloshing liquid
content / fill levelemptyemptyhollow ; nothing inside ; contains nothing ; no contents ; vacant ; unfilled
fill levelfullfull; fully filledfilled to the top; completely filled
halfhalfat half capacity; midway level
Appendix
Table 4: Training prompts and the 29-paraphrase inference vocabulary. The empty class of the content and fill-level axes shares one training prompt; its paraphrases are split between the two axes during evaluation. Paraphrases in italics share no word stem with any training prompt and form the strictest evaluation subset (17 in total).
Method (split)ContentFill LevelMaterialRigidityAvg.
Modality ablation
Vibrotactile only (S)0.77±0.040.77\pm 0.040.82±0.070.82\pm 0.070.88±0.060.88\pm 0.060.95±0.030.95\pm 0.030.86±0.040.86\pm 0.04
Vibrotactile only (U)0.75±0.110.75\pm 0.110.71±0.110.71\pm 0.110.71±0.120.71\pm 0.120.88±0.050.88\pm 0.050.76±0.050.76\pm 0.05
Vibrotactile + force (S)0.81±0.070.81\pm 0.070.86±0.050.86\pm 0.050.84±0.150.84\pm 0.150.98±0.020.98\pm 0.020.87±0.060.87\pm 0.06
Vibrotactile + force (U)0.80±0.080.80\pm 0.080.78±0.060.78\pm 0.060.71±0.230.71\pm 0.230.92±0.040.92\pm 0.040.80±0.080.80\pm 0.08
Vibrotactile + velocity (S)0.86±0.060.86\pm 0.060.86±0.030.86\pm 0.030.87±0.090.87\pm 0.090.96±0.030.96\pm 0.030.89±0.030.89\pm 0.03
Appendix
Table 5: Object property estimation ablations on seen (S) and unseen (U) objects at different locations. Values are mean ±\pm sample standard deviation over 8 seeds (0–7). Avg. is the mean accuracy across the four property axes; its standard deviation is computed across the per-seed averages.
Figure 8: (a) Per-axis property estimation accuracy of ours and the CE baseline on seen (S) and unseen (U) objects, mean ±\pm std over 8 seeds. (b) Axis-average accuracy vs. exploration time with a causal top- KK token buffer decoded every 15 s and at trajectory completion, using only observation windows completed by the decoding time. Accuracy is averaged over four object attributes and 27 test trajectories per split. Curves show the mean across eight training seeds; shaded bands indicate ±1\pm 1 standard deviation across seeds. The horizontal axis expresses the decoding times as a fraction of full duration, computed as the mean elapsed exploration time divided by the mean full trajectory duration within each split.
MethodContentFill LevelMaterialRigidityAvg.
Ours (Seen)0.87±0.030.87\pm 0.030.94±0.040.94\pm 0.040.94±0.040.94\pm 0.040.96±0.030.96\pm 0.030.93±0.020.93\pm 0.02
Ours (Unseen)0.85±0.080.85\pm 0.080.81±0.040.81\pm 0.040.80±0.140.80\pm 0.140.91±0.050.91\pm 0.050.84±0.060.84\pm 0.06
CE (Seen)0.89±0.050.89\pm 0.050.92±0.060.92\pm 0.060.90±0.030.90\pm 0.030.95±0.040.95\pm 0.040.92±0.020.92\pm 0.02
CE (Unseen)0.84±0.060.84\pm 0.060.85±0.050.85\pm 0.050.82±0.100.82\pm 0.100.89±0.040.89\pm 0.040.85±0.040.85\pm 0.04
Appendix
Table 6: Numerical values of Fig. 8 (a).
Figure 9: Object representation learning with a multi-finger robotic hand (DeltaHand [ 29 ] ). (a–e) Five exploration procedures: squeezing the object by closing all fingers, sliding one finger along the object surface while the remaining fingers provide support, lifting the object, in-hand horizontal shifting, and in-hand tilting. (f, g) Object property estimation is evaluated on both (h) seen and (i) unseen objects. (j) Three examples of selected tokens from the token learner are visualized, showing that most tokens are selected from squeezing, horizontal shifting, and tilting motions.
Figure 10: Object-centric and context-aware manipulation policy learning. Built on the Action Chunking Transformer, the policy conditions on learned object context and object position to focus on target-specific visual, tactile, and proprioceptive observations for adaptive manipulation.
Table 7: Manipulation policy evaluation across different task settings.
TaskRequired hidden property
Multi-object rearrangementrigidity + content
Three-container subsetrigidity + content
Pouringfill level
Box openingweight / fill level
Appendix
Table 8: Manipulation tasks and the properties required for strategy selection.
More in this topic

Tactile Sensing for Robotic Manipulation

Tactile sensing for robotic manipulation uses contact signals to help robots perceive forces, prevent slips, and handle objects reliably.

See all 178 papers

Explore similar work

CardsList
May 21, 2026

TacO: Benchmarking Tactile Sensors for Object Manipulation

Vision-based learning from demonstrations has achieved remarkable success in enabling robots to perform manipulation tasks and high-level semantic reasoning, yet it remains insufficient for complex, contact-rich manipulation. While there is broad agreement that tactile sensing improves manipulation, there is no empirical guidance on which tactile sensors are best suited for which manipulation tasks. In this paper, we provide a systematic, task-driven evaluation of tactile sensors for robot manipulation and propose a framework for selecting and evaluating sensors based on manipulation policy performance. Separate manipulation policies are trained for tactile sensors of four distinct modalities: visual, acoustic, magnetic, and resistive, across three tasks: pick-and-place with unknown mass, object reorientation, and plug insertion. For each task, an analysis of how sensor properties such as spatial resolution, shear sensing, and tactile representation, and the inherent material friction affect task performances is done. Rather than tactile sensing being universally beneficial in the same way, our results show that the usefulness of tactile information depends strongly on sensor modality, material properties, and the specific manipulation tasks. All of the tactile sensors, code, data, and hardware setup will be publicly available on the project website.

Anya Zorin, Zilin Si, Myungsun Park +11

UC San Diego United States · CMU · UC San Diego +2

Tactile SensingContact-Rich Robotic ManipulationRobot Manipulation BenchmarksTactile Sensing for Robotic Manipulation
AskPDF
Oct 21, 2026

Tactile-Based Robotic Manipulation: From Mechanics-Driven to Data-Driven Perspectives

Tactile sensing provides robots with direct access to physical interactions at the contact interface, making it a critical modality for robust and dexterous manipulation. Over the past several decades, tactile-based manipulation research has evolved from mechanics-driven approaches, which explicitly interpret tactile signals through contact mechanics and task-specific reasoning, to data-driven approaches that learn tactile representations and manipulation policies directly from data. This paper reviews the development of tactile-based manipulation from these two complementary perspectives. We first introduce the fundamentals of tactile sensing, and then review, from a mechanics-driven perspective, how those signals form manipulation-relevant state representations and subsequently influence the robot action. We then examine the recent advances in data-driven tactile manipulation, covering imitation learning, reinforcement learning, vision-language-action models, and predictive models. Finally, we discuss open challenges and future opportunities toward scalable, generalizable, and robust tactile-enabled robotic systems.

Jingyi Xiang, Yixuan Wang, Wenzhen Yuan

Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, Illinois, USA, 61801

Tactile Sensing for Robotic ManipulationRobotic Tactile Sensing
AskPDF
Jun 30, 2026

TactX: Learning Shared Tactile Representations Across Diverse Sensors

Tactile sensors provide critical information for contact-rich manipulation, yet tactile representations and policies remain tightly coupled to each specific sensor, limiting transferability across robots and hardware platforms. We propose TactX, a framework for learning a transferable tactile representation across sensors spanning three fundamentally different transduction modalities: resistive, magnetic, and vision-based. TactX maps heterogeneous tactile observations into a shared latent space through modality-specific encoders trained on paired contact data. Such paired interactions provide a natural alignment signal across modalities, and the encoders are jointly trained across all sensor pairs, inducing a consistent latent space for all sensor types. Our experiments show that TactX aligns tactile representations across sensors while preserving object-level contact information, as evidenced by sensor-identity prediction and object classification in the learned latent space. We evaluate TactX on four contact-rich manipulation tasks: pick-and-place, plug insertion, board wiping, and object reorientation, and show that policies trained with one sensor transfer zero-shot to physically distinct sensors through the shared latent. This improves the average success rate from 27.5% for vision-only policy to 45.9%, providing a step toward sensor-agnostic tactile manipulation.

Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2

UC San Diego · Seoul National University · Amazon FAR

Cross-Modal Representation LearningContact-Rich Robotic ManipulationVisuo-Tactile Representation LearningGeneralization in Robotic ManipulationCross-Embodiment Robot LearningTactile Sensing for Robotic Manipulation
AskPDF