Organizations: ShanghaiTech University · Beijing Institute for General Artificial Intelligence · Beihang University · Beijing Jiaotong University · AMAP, Alibaba
Tactile sensing is essential for contact-rich manipulation, yet heterogeneous sensing mechanisms and observation formats hinder cross-hardware transfer. Since physical contact is governed by Newtonian mechanics, we ground tactile representations in six-dimensional wrenches, establishing a common physical reference across sensing mechanisms. Specifically, we pair tactile observations from all sensors with wrench measurements, enabling shared physical supervision without requiring direct tactile-to-tactile pairing between sensors. To support learning at scale, we develop a dedicated acquisition platform and collect over 100 million paired tactile observations and wrench measurements from 24 sensors spanning four families of sensing principles. Building on this dataset, our framework combines sensor-specific input adaptation with a shared tactile encoder. By learning to reconstruct wrenches, the encoder maps observations from different sensors into a shared, continuous feature space grounded in contact physics. To leverage these representations for manipulation, we introduce TaF-Policy, which integrates tactile features with visual observations and robot states to generate action chunks. We evaluate the learned representations through wrench estimation, cross-sensor retrieval, few-shot adaptation to unseen sensors, and robotic manipulation. On the core benchmark, our method reduces single-frame force RMSE by 17.1% and improves cross-sensor retrieval R@1 by 13.11 percentage points over the strongest baseline. In manipulation experiments, TaF-Policy achieves average success rates of 51.33% across three simulated tasks and 46.25% across four real-world tasks, compared with 47.33% and 23.75%, respectively, for the strongest baselines evaluated on the same task sets. Project page: https://mrjiangjm.github.io/TaF-Foundation/.
Figures & tables
Figure 2: Dataset coverage across heterogeneous tactile sensors. Left: a hierarchy of tactile learning tasks, adapted from AnyTouch 2 [ 7 ] . Center: the distribution of paired observations across 24 sensors in four families; the inner ring shows family proportions, and the outer ring shows individual sensors. Right: sensor inventory and observation counts by family.
Dataset
Modality (Types)
Sensor Types
Contact Diversity
Paired Observations
Sparsh Force [ 10 ]
Vision tactile (1)
2
3 indenters
1.5×105 pairs
RCT [ 8 ]
Vision tactile (1)
1
122 materials
2.9×104 pairs
TaF [ 15 ]
Vision/Pressure tactile (2)
6
408 probe–sensor pairs
107 pairs
ToucHD-Force [ 7 ]
Vision tactile (1)
5
71 indenters
7.2×105 pairs
Ours
Heterogeneous tactile (4)
24
1,445 probe–sensor pairs
108 pairs
Table 1: Comparison of tactile datasets paired with force or wrench measurements.
Figure 3: Acquisition apparatus for synchronized tactile–wrench recording, shown in exploded and assembled views.
Figure 4: Overview of our methodology for wrench-grounded tactile representation learning and downstream evaluation. (a) Sensor-specific input adapters and family-specific projections map heterogeneous tactile observations into tokens for a shared Transformer. A learnable physical query aggregates the encoded tokens into a 768-dimensional continuous tactile representation. (b) Six-dimensional wrench reconstruction and alignment with EMA wrench embeddings provide shared physical supervision. The learned representations are evaluated on single-frame wrench estimation, temporal wrench estimation with historical wrench inputs, cross-sensor retrieval, and robotic manipulation using TaF-Policy in simulation and the real world.
Single-frame estimation
Temporal estimation
Cross-sensor retrieval
Method
F RMSE (N) ↓
T RMSE (N ⋅ m) ↓
F RMSE (N) ↓
T RMSE (N ⋅ m) ↓
R@1 ↑
R@5 ↑
DINOv2 [ 27 ]
3.032±0.053
0.0821±0.0015
1.642±0.064
0.0413±0.0003
14.00±0.27
40.25±2.56
Sparsh [ 10 ]
2.886±0.155
0.0788±0.0021
1.579±0.016
0.0364±0.0022
14.21±0.59
37.83±1.01
T3 [ 35 ]
2.820±0.131
0.0780±0.0035
1.482±0.019
0.0372±0.0006
17.38±1.05
51.52±1.63
AnyTouch 2 [ 7 ]
2.799±0.203
0.0775±0.0026
1.515±0.040
0.0387±0.0026
21.27±0.80
64.48±5.04
HTT [ 2 ]
2.971±0.024
0.0783±0.0014
1.438±0.094
0.0343±0.0008
14.78±0.46
45.33±0.73
Table 2: Benchmark results on the same 21 seen sensors spanning four tactile sensor families. F and T denote force and torque RMSE, respectively. R@ K (%) measures retrieval accuracy based on standardized 6D wrench distance, with candidates ranked by tactile-representation distance. Results are reported as mean ± standard deviation over three seeds.
Figure 5: Few-shot adaptation on three unseen tactile sensors. Curves show temporal joint-adaptation results at different data budgets; the inset reports the hierarchical overall score across adaptation settings. Details are provided in Appendix C.4.
Single-frame estimation
Temporal estimation
Cross-sensor retrieval
Variant
F RMSE (N) ↓
T RMSE (N ⋅ m) ↓
F RMSE (N) ↓
T RMSE (N ⋅ m) ↓
R@1 ↑
R@5 ↑
D1↓
Full model
2.415
0.0648
1.242
0.0318
33.52
78.90
1.581
Direct wrench supervision
2.494
0.0684
1.385
0.0379
29.95
73.14
1.648
w/o Alignment
2.481
0.0685
1.310
0.0370
28.00
76.10
1.652
w/o Sensor-specific MLPs
3.291
0.0884
1.490
0.0384
25.90
67.81
1.761
w/o Sensor Identity Token
2.628
0.0678
1.291
0.0323
33.90
78.10
1.592
Table 3: Ablation and design comparisons on the core benchmark. F and T denote force and torque RMSE; D1 is the standardized 6D wrench distance of the top-ranked retrieval. Results are from a single random seed.
Figure 6: Real-world manipulation setups and UniVTAC simulation scenes. Four real-world tasks are presented: bottle lifting, power bank removal, key insertion, and board cleaning. The first three real-world tasks use Daimon tactile sensors, while the last task uses a PaXini tactile sensor. Three simulation tasks are presented: can lifting, key extraction, and bottle placement on a shelf.
Method
Lift Can
Pull Out Key
Put Bottle in Shelf
Avg. ↑
UniVTAC [ 4 ]
40
64
38
47.33
T3 [ 35 ]
30
62
32
41.33
AnyTouch2 [ 7 ]
32
68
28
42.67
TaF-Policy (Ours)
44
70
40
51.33
Table 4: Task success rates (%) on three UniVTAC simulation tasks.
Method
Lift Bottle
Remove Power Bank
Key Insert
Board Clean
Avg. ↑
UniVTAC
15
20
10
–
15.00
T3
20
30
5
35
22.50
AnyTouch2
20
25
10
40
23.75
TaF-Policy (Ours)
35
60
25
65
46.25
Table 5: Task success rates (%) on four real-world manipulation tasks, evaluated over 20 trials per task. Board Clean uses a magnetic-based tactile sensor. UniVTAC is not evaluated on Board Clean because its encoder expects visuo-tactile images. Its average is computed over the first three tasks; averages for the other methods are computed over all four tasks.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Sensor
Family
Input shape
Episodes
Frames
9DTact [ 20 ]
Optical-based
240×320×3
2,366
6,555,482
DM-Tac WL
Optical-based
240×320×3
2,098
6,190,543
DM-Tac WS
Optical-based
240×320×3
849
3,126,433
NLiPsTac [ 29 ]
Optical-based
240×320×3
1,640
4,881,827
PPTac type 1 [ 21 ]
Optical-based
240×320×3
922
3,755,271
PPTac type 2 [ 21 ]
Optical-based
240×320×3
2,088
6,236,062
Appendix
Table S1 : Sensor inventory and recorded data volume. Input shapes describe saved observations before model preprocessing: H×W×C for images, channel counts for vectors, and array dimensions for pressure matrices. † marks the three sensors held out from source training.
Partition
Objects per held-out sensor
Adaptation
2
Validation
1
Test
5
Appendix
Table S2 : Indenter object splits for adaptation to the three held-out sensors. Each sensor uses disjoint object sets for adaptation, validation, and testing.
Stage
Learning rate
Batch
Training budget
Frozen modules
Representation learning
Tactile: 3×10−5 Wrench: 10−3
256
100,000 steps
None in online branches; EMA updated separately
Frozen probes
10−3
256
10,000 steps per head
Tactile encoder
Few-shot adaptation
Inputs: 10−4 Head: 10−3
256
Single-frame: 20 equivalent epochs, at most 5,000 steps Temporal: 10 equivalent epochs, at most 2,500 steps
Backbone; also the prediction head for input-only adaptation
Robot-policy training
10−5
64
Task-dependent steps; demonstrations per task: 50 in simulation, 100 in the real world
Tactile encoder
Appendix
Table S3 : Training and adaptation settings. All stages use AdamW. Batch sizes are global. Weight decay is 0.01 for representation learning and 10−4 for frozen probes and few-shot adaptation. Few-shot schedules follow Appendix C.4 ; robot-policy training steps vary by task.
Method
5%
10%
25%
50%
100%
Single-frame joint adaptation
DINOv2
0.5453
0.5265
0.4943
0.4486
0.4718
Sparsh
0.5810
0.5483
0.5156
0.4932
0.4810
T3
0.5805
0.5394
0.4841
0.4547
0.4224
AnyTouch2
0.5457
0.5786
0.5026
0.4900
0.4773
HTT
0.5458
0.5717
0.5432
0.5271
0.5180
Appendix
Table S4 : Normalized RMSE ( ↓ ) for joint adaptation of input modules and prediction heads on three unseen tactile sensors. Percentages indicate the fraction of the adaptation pool. The shared backbone remains frozen. Bold denotes the lowest error in each column within each evaluation setting.
Method
Single-frame
Temporal
DINOv2
0.5103
0.4238
Sparsh
0.5832
0.4163
T3
0.5311
0.4079
AnyTouch2
0.5905
0.4423
HTT
0.6414
0.4030
Ours
0.5046
0.4203
Appendix
Table S5 : Normalized RMSE ( ↓ ) for input-only adaptation using 10% of the adaptation pool. Both the shared backbone and prediction head remain frozen. Bold denotes the lowest error in each column.
Method
Hierarchical score
Equal-weight mean
DINOv2
0.4524
0.4487
Sparsh
0.4729
0.4661
T3
0.4497
0.4448
AnyTouch2
0.4747
0.4643
HTT
0.4794
0.4687
Ours
0.4468
0.4428
Appendix
Table S6 : Aggregate few-shot normalized RMSE ( ↓ ). Hierarchical scores weight the three adaptation settings equally; equal-weight scores average all 12 conditions equally. Values are computed before rounding to four decimal places.