WiTac: Wrist-Informed Tactile Representations for Contact-Rich Manipulation
Organizations: Sharpa Robotics · PKU · SJTU · CUHK ·
Abstract
Similar tactile contacts can call for different corrections depending on the configuration of the hand, object and target. Rather than supplying touch and wrist vision separately, fusing them before policy injection better grounds tactile features in visual context, yielding more accurate contact representations. We introduce Wrist-Informed Tactile Representations (WiTac), a pretrained encoder that learns such representations before action generation. Pretrained on nearly 300 hours of SharpaDex data, WiTac deeply fuses tactile signals with hand motion and wrist views into a comprehensive contact representation. Conditioning downstream VLA policies (GR00T N1.7 and ) on WiTac substantially improves task success over vision-only base policies across four contact-rich manipulation tasks (+28.8% and +15.0% on average), and further outperforms a wrist-free tactile encoder by 18.8% and 7.5%, respectively. Offline analyses further show that incorporating wrist vision yields more accurate contact descriptors, stronger joint task-contact discrimination, greater feature stability under tactile noise, and clearer action-direction readout from similar contacts. Project page: https://wrist-informed-tactile-representations.github.io/.
Figures & tables
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Objective | Target | Offsets / horizons | Loss & weight ( ) |
|---|---|---|---|
| Contact descriptor ( ) | Contact area, intensity, 2D center | Current frame ( ) | Smooth L1 ( ) |
| Deformation change ( ) | ( ) | past frames | Balanced L1 ( ) |
| Fingertip motion ( ) | Wrist-frame | past frames | Smooth L1 ( ) |
| Visual recovery ( ) | Masked full-image DINOv3 features | Current frame ( ) | LayerNorm MSE ( ) |
| Wrench response ( ) | 6D wrench change | future frames | Smooth L1 ( ) |
| Policy / task | Feature drift | Attention JS | Contact-output drift | |||
|---|---|---|---|---|---|---|
| WiTac-NW | WiTac | WiTac-NW | WiTac | WiTac-NW | WiTac | |
| GR00T / Insert battery | 0.5078 | 0.3177 | 0.1626 | 0.0658 | 1.4321 | 1.1506 |
| GR00T / Wipe bowl | 0.5413 | 0.3484 | 0.1437 | 0.0878 | 1.1974 | 0.8520 |
| GR00T / Reinsert USB | 0.4966 | 0.3130 | 0.1711 | 0.0348 | 1.6406 | 1.2741 |
| GR00T / Open cabinet | 0.5415 | 0.3028 | 0.2133 | 0.1560 | 1.4558 | 1.1163 |
| / Insert battery | 0.5078 | 0.3177 | 0.1023 | 0.0353 | 1.0884 | 0.8215 |
| Policy / task | Clean MAE ( ∘ ) | Noisy MAE ( ∘ ) | MAE ( ∘ ) | Reduction in MAE 95% interval ( ∘ ) | |||
|---|---|---|---|---|---|---|---|
| WiTac-NW | WiTac | WiTac-NW | WiTac | WiTac-NW | WiTac | ||
| GR00T / Insert battery | 0.8785 | 0.8931 | 1.4027 | 1.0835 | 0.5242 | 0.1904 | [+0.2877, +0.3805] |
| GR00T / Wipe bowl | 0.7757 | 0.7801 | 0.9709 | 0.8996 | 0.1952 | 0.1195 | [+0.0393, +0.1198] |
| GR00T / Reinsert USB | 0.7787 | 0.7592 | 0.9950 | 0.9037 | 0.2163 | 0.1445 | [+0.0348, +0.1067] |
| GR00T / Open cabinet | 0.6370 | 0.6425 | 1.1156 | 0.8748 | 0.4786 | 0.2322 | [+0.1232, +0.3845] |
| / Insert battery | 0.8263 | 0.8250 | 0.8734 | 0.8783 | 0.0470 | 0.0533 | [-0.0157, +0.0035] |
Contact-Rich Robotic Manipulation
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
See all 453 papers