TaF-Foundation: Wrench-Grounded Cross-Sensor Tactile Representations
Organizations: ShanghaiTech University · Beijing Institute for General Artificial Intelligence · Beihang University · Beijing Jiaotong University ·
Abstract
Tactile sensing is essential for contact-rich manipulation, yet heterogeneous sensing mechanisms and observation formats hinder cross-hardware transfer. Since physical contact is governed by Newtonian mechanics, we ground tactile representations in six-dimensional wrenches, establishing a common physical reference across sensing mechanisms. Specifically, we pair tactile observations from all sensors with wrench measurements, enabling shared physical supervision without requiring direct tactile-to-tactile pairing between sensors. To support learning at scale, we develop a dedicated acquisition platform and collect over 100 million paired tactile observations and wrench measurements from 24 sensors spanning four families of sensing principles. Building on this dataset, our framework combines sensor-specific input adaptation with a shared tactile encoder. By learning to reconstruct wrenches, the encoder maps observations from different sensors into a shared, continuous feature space grounded in contact physics. To leverage these representations for manipulation, we introduce TaF-Policy, which integrates tactile features with visual observations and robot states to generate action chunks. We evaluate the learned representations through wrench estimation, cross-sensor retrieval, few-shot adaptation to unseen sensors, and robotic manipulation. On the core benchmark, our method reduces single-frame force RMSE by 17.1% and improves cross-sensor retrieval R@1 by 13.11 percentage points over the strongest baseline. In manipulation experiments, TaF-Policy achieves average success rates of 51.33% across three simulated tasks and 46.25% across four real-world tasks, compared with 47.33% and 23.75%, respectively, for the strongest baselines evaluated on the same task sets. Project page: https://mrjiangjm.github.io/TaF-Foundation/.
Figures & tables
| Dataset | Modality (Types) | Sensor Types | Contact Diversity | Paired Observations |
|---|---|---|---|---|
| Sparsh Force [ 10 ] | Vision tactile (1) | 2 | 3 indenters | pairs |
| RCT [ 8 ] | Vision tactile (1) | 1 | 122 materials | pairs |
| TaF [ 15 ] | Vision/Pressure tactile (2) | 6 | 408 probe–sensor pairs | pairs |
| ToucHD-Force [ 7 ] | Vision tactile (1) | 5 | 71 indenters | pairs |
| Ours | Heterogeneous tactile (4) | 24 | 1,445 probe–sensor pairs | pairs |
| Single-frame estimation | Temporal estimation | Cross-sensor retrieval | ||||
|---|---|---|---|---|---|---|
| Method | F RMSE (N) | T RMSE (N m) | F RMSE (N) | T RMSE (N m) | R@1 | R@5 |
| DINOv2 [ 27 ] | ||||||
| Sparsh [ 10 ] | ||||||
| T3 [ 35 ] | ||||||
| AnyTouch 2 [ 7 ] | ||||||
| HTT [ 2 ] | ||||||
| Single-frame estimation | Temporal estimation | Cross-sensor retrieval | |||||
|---|---|---|---|---|---|---|---|
| Variant | F RMSE (N) | T RMSE (N m) | F RMSE (N) | T RMSE (N m) | R@1 | R@5 | |
| Full model | 2.415 | 0.0648 | 1.242 | 0.0318 | 33.52 | 78.90 | 1.581 |
| Direct wrench supervision | 2.494 | 0.0684 | 1.385 | 0.0379 | 29.95 | 73.14 | 1.648 |
| w/o Alignment | 2.481 | 0.0685 | 1.310 | 0.0370 | 28.00 | 76.10 | 1.652 |
| w/o Sensor-specific MLPs | 3.291 | 0.0884 | 1.490 | 0.0384 | 25.90 | 67.81 | 1.761 |
| w/o Sensor Identity Token | 2.628 | 0.0678 | 1.291 | 0.0323 | 33.90 | 78.10 | 1.592 |
| Method | Lift Can | Pull Out Key | Put Bottle in Shelf | Avg. |
|---|---|---|---|---|
| UniVTAC [ 4 ] | 40 | 64 | 38 | 47.33 |
| T3 [ 35 ] | 30 | 62 | 32 | 41.33 |
| AnyTouch2 [ 7 ] | 32 | 68 | 28 | 42.67 |
| TaF-Policy (Ours) | 44 | 70 | 40 | 51.33 |
| Method | Lift Bottle | Remove Power Bank | Key Insert | Board Clean | Avg. |
|---|---|---|---|---|---|
| UniVTAC | 15 | 20 | 10 | – | 15.00 |
| T3 | 20 | 30 | 5 | 35 | 22.50 |
| AnyTouch2 | 20 | 25 | 10 | 40 | 23.75 |
| TaF-Policy (Ours) | 35 | 60 | 25 | 65 | 46.25 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Sensor | Family | Input shape | Episodes | Frames |
| 9DTact [ 20 ] | Optical-based | 2,366 | 6,555,482 | |
| DM-Tac WL | Optical-based | 2,098 | 6,190,543 | |
| DM-Tac WS | Optical-based | 849 | 3,126,433 | |
| NLiPsTac [ 29 ] | Optical-based | 1,640 | 4,881,827 | |
| PPTac type 1 [ 21 ] | Optical-based | 922 | 3,755,271 | |
| PPTac type 2 [ 21 ] | Optical-based | 2,088 | 6,236,062 |
| Partition | Objects per held-out sensor |
|---|---|
| Adaptation | 2 |
| Validation | 1 |
| Test | 5 |
| Stage | Learning rate | Batch | Training budget | Frozen modules |
|---|---|---|---|---|
| Representation learning | Tactile: Wrench: | 256 | 100,000 steps | None in online branches; EMA updated separately |
| Frozen probes | 256 | 10,000 steps per head | Tactile encoder | |
| Few-shot adaptation | Inputs: Head: | 256 | Single-frame: 20 equivalent epochs, at most 5,000 steps Temporal: 10 equivalent epochs, at most 2,500 steps | Backbone; also the prediction head for input-only adaptation |
| Robot-policy training | 64 | Task-dependent steps; demonstrations per task: 50 in simulation, 100 in the real world | Tactile encoder |
| Method | 5% | 10% | 25% | 50% | 100% |
|---|---|---|---|---|---|
| Single-frame joint adaptation | |||||
| DINOv2 | 0.5453 | 0.5265 | 0.4943 | 0.4486 | 0.4718 |
| Sparsh | 0.5810 | 0.5483 | 0.5156 | 0.4932 | 0.4810 |
| T3 | 0.5805 | 0.5394 | 0.4841 | 0.4547 | 0.4224 |
| AnyTouch2 | 0.5457 | 0.5786 | 0.5026 | 0.4900 | 0.4773 |
| HTT | 0.5458 | 0.5717 | 0.5432 | 0.5271 | 0.5180 |
| Method | Single-frame | Temporal |
|---|---|---|
| DINOv2 | 0.5103 | 0.4238 |
| Sparsh | 0.5832 | 0.4163 |
| T3 | 0.5311 | 0.4079 |
| AnyTouch2 | 0.5905 | 0.4423 |
| HTT | 0.6414 | 0.4030 |
| Ours | 0.5046 | 0.4203 |
| Method | Hierarchical score | Equal-weight mean |
|---|---|---|
| DINOv2 | 0.4524 | 0.4487 |
| Sparsh | 0.4729 | 0.4661 |
| T3 | 0.4497 | 0.4448 |
| AnyTouch2 | 0.4747 | 0.4643 |
| HTT | 0.4794 | 0.4687 |
| Ours | 0.4468 | 0.4428 |
Robotic Manipulation
Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.
See all 1,842 papers