BiView-Touch: Learning Bimanual Tactile Representations by Cross-Hand Completion

Chenxin Liang; Youchen Lai; Chuqiao Lyu; Tianxing Chen; Shoujie Li; Wenbo Ding

cs.CVSep 20, 2026

BiView-Touch: Learning Bimanual Tactile Representations by Cross-Hand Completion

Authors: Chenxin Liang, Youchen Lai, Chuqiao Lyu, Tianxing Chen, Shoujie Li, Wenbo Ding

Organizations: Tsinghua University · XSpark AI · The University of Hong Kong · Nanyang Technological University

Abstract

Bimanual interaction produces complementary tactile views of the same physical process, yet existing tactile representation learning largely models the two hands independently or combines them only for downstream prediction, leaving their cross-hand relationship unexplored. To exploit this overlooked structure, we introduce BiView-Touch, a tactile-only framework that completes masked target-hand latents from the remaining visible target-hand regions and the synchronized full contralateral hand. A student encoder with a geometry-conditioned directional decoder predicts full-view EMA latent targets, while temporal and layout counterfactuals encourage sensitivity to synchronized and anatomically organized source information. Controlled ablations and source-context interventions show that BiView-Touch learns structured cross-hand dependence on temporally aligned and anatomically organized contralateral tactile context, rather than benefiting from bilateral input alone. On the public HumanTouch dataset, its frozen representations consistently outperform representative self-supervised baselines across low-label settings. With only 5% downstream labels, BiView-Touch achieves relative balanced-accuracy gains of 7.1% on bilateral wrist-motion recognition and 14.1% on force-derived interaction-phase recognition. We further introduce BVT-20, a 20-task bilateral tactile dataset, and demonstrate transfer across recording sessions and pretraining corpora, including transfer to a held-out bimanual task. Our code and dataset details are available on the anonymous project page: https://anonymous.4open.science/w/biview-touch-review-site-050C/.

Figures & tables

Fig. 1: Overview of BiView-Touch. Cross-hand latent completion learns region-aware bimanual tactile embeddings that support frozen-backbone recognition, regression, forecasting, and transfer.

Fig. 2: Overview of BiView-Touch. (a) A synchronized bimanual tactile window is organized into a masked target hand $\mathbf{X}_{t}$ and a complete contralateral source hand $\mathbf{X}_{s}$ . (b) The shared online encoder and geometry-conditioned decoder predict the masked target representation $\hat{\mathbf{Z}}$ , matched to the stop-gradient full-view target $\mathbf{Z}^{T}$ from the EMA encoder. Counterfactual branches reuse the online pathway and are omitted for clarity.

Fig. 3: BiView-Touch encoder and interaction decoder. (a) CrossFormer-based temporal and cross-taxel encoding produces taxel–time tokens. (b) Anatomy-constrained region queries aggregate taxels into region–time tokens. (c) The geometry-conditioned decoder predicts masked target tokens from target context and contralateral source features using self- and cross-attention.

Fig. 4: Structured functional masking. (a) A 60-frame window is divided into 12 temporal patches. (b) Taxels form 11 anatomical regions. (c) One functional group is masked and scored; additional masked regions act as distractors.

Class	Rule
No contact	$E<c,\ L<c$
Stable contact	$E\geq c,\ L\geq c,\ \|L-E\|\leq\delta$
Onset	$E<c,\ L\geq c$
Loading	$E\geq c,\ L\geq c,\ L-E>\delta$
Unloading	$E\geq c,\ L\geq c,\ L-E<-\delta$
Release	$E\geq c,\ L<c$

TABLE I: Force-derived interaction-phase rules.

Fig. 5: The 20 tasks in our self-collected bimanual tactile dataset, spanning insertion and fastening, handover and loading, tool-mediated manipulation, shared stabilization, and bimanual transport. Each task is recorded under both active–support role configurations.

Method	Decoder representation	Readout
Within-hand, per-hand	$\mathbf{R}^{L,\mathrm{wh}},\mathbf{R}^{R,\mathrm{wh}}$	$H_{L},H_{R}$
Within-hand, bilateral	$\mathbf{R}^{L,\mathrm{wh}},\mathbf{R}^{R,\mathrm{wh}}$	$[\mathbf{R}^{L,\mathrm{wh}};\mathbf{R}^{R,\mathrm{wh}}]\!\rightarrow\!H$
BiView-Touch	$\mathbf{R}^{L\leftarrow R},\mathbf{R}^{R\leftarrow L}$	$[\mathbf{R}^{L\leftarrow R};\mathbf{R}^{R\leftarrow L}]\!\rightarrow\!H$

TABLE II: Frozen-feature downstream readout protocols.

Method		Force-derived phase (6-way) $\uparrow$			Contact (binary) $\uparrow$			Wrist motion (4-way) $\uparrow$
Method		5%	10%	25%	5%	10%	25%	5%	10%	25%
Raw tactile + MLP ‡ (193K)		39.47	42.30	46.00	71.34	73.42	75.44	51.99	56.38	61.51
Raw Dual CNN ‡ (1.60M) [ 16 ]		43.82	46.95	49.62	74.25	75.71	77.25	62.67	64.13	66.64
PatchTST-SSL † [ 21 ]		33.29	35.76	39.76	66.41	68.28	70.72	51.67	55.33	60.20
Sparsh-Skin adapted † [ 17 ]		38.47	42.46	46.42	71.16	73.78	75.94	60.76	62.08	65.27
Within-hand EMA †	Per-hand head	36.56	39.43	43.13	67.38	69.77	72.14	60.95	63.12	65.21

TABLE III: Low-label recognition on HumanTouch (test balanced accuracy, %, mean over three runs).

\dagger

: frozen backbone with a 193K-parameter head;

\ddagger

: supervised training from random initialization;

\ast

: dense-fusion architecture pretrained as a separate checkpoint.

Method	Phase bAcc $\uparrow$		Wrist motion bAcc $\uparrow$
	5%	25%	5%	25%
w / o $\mathcal{L}_{\mathrm{sync}}$	37.13	44.19	60.16	66.86
w / $\mathcal{L}_{\mathrm{sync}}$	44.12	49.91	63.30	69.01

TABLE IV: Ablation of the synchronization objective on HumanTouch frozen-feature recognition (balanced accuracy, %).

Budget	Phase $\Delta G\,\uparrow$	Contact $\Delta G\,\uparrow$	Dynamic–static interaction $\uparrow$
5%	+0.033 [-0.063, 0.129]	-0.129 [-0.223, -0.036]	-0.066 [-0.296, 0.158]
25%	+0.290 [0.171, 0.414]	+0.107 [0.009, 0.202]	+0.337 [0.114, 0.565]
100%	+0.203 [0.087, 0.320]	+0.045 [-0.037, 0.128]	+0.166 [-0.079, 0.413]

TABLE V: Effect of functional-group masking on regional temporal alignment.

\Delta G

denotes the difference in regional alignment gain between functional-group and random masking. The last column reports the interaction between dynamic and static phase groups. Values are percentage points with paired session-level bootstrap 95% confidence intervals.

Budget	Method	Adjacent $\uparrow$	F $\leftrightarrow$ F $\uparrow$	P $\leftrightarrow$ P $\uparrow$	F $\leftrightarrow$ P $\uparrow$	Global $\uparrow$
5%	w / o $\mathcal{L}_{\mathrm{layout}}$	0.008	0.004	0.022	0.075	0.232
5%	w / $\mathcal{L}_{\mathrm{layout}}$	0.139	0.025	0.072	0.632	2.268
25%	w / o $\mathcal{L}_{\mathrm{layout}}$	-0.006	0.009	-0.015	-0.031	0.051
25%	w / $\mathcal{L}_{\mathrm{layout}}$	0.229	0.004	0.071	1.015	2.792

TABLE VI: Layout-loss ablation under source-region reassignment on HumanTouch. Values are mean bAcc drops (pp); larger values indicate greater sensitivity to content–anatomy mismatch. Adjacent denotes neighboring-region swaps; F

\leftrightarrow

F and P

\leftrightarrow

P denote within-fingertip and within-palm swaps; F

\leftrightarrow

P denotes fingertip–palm swaps; Global averages eight full-region permutations.

Method	Masked region		Masked hand
Method	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$
PatchTST-SSL [ 21 ]	1.026	2.845	1.115	3.175
Sparsh-Skin adapted [ 17 ]	0.937	2.558	1.080	3.075
BiView-Touch	0.892	2.388	1.075	3.065

TABLE VII: Frozen-feature force regression on HumanTouch (N; lower is better).

Method		Accuracy $\uparrow$	bAcc $\uparrow$	Macro-F1 $\uparrow$
Raw Dual CNN ‡ [ 16 ]		72.66	71.03	69.89
PatchTST-SSL † [ 21 ]		64.47	61.81	62.01
Sparsh-Skin adapted † [ 17 ]		72.20	70.72	69.83
Within-hand EMA †		59.51	61.67	57.99
BiView-Touch †	Dense fusion ∗	72.86	71.50	70.29
BiView-Touch †	Cross-attention	71.71	71.13	69.07

TABLE VIII: HumanTouch operation-stage recognition (%, mean over four runs).

\dagger

: frozen backbone;

\ddagger

: end-to-end training;

\ast

: separately pretrained dense-fusion architecture.

Target	Pretraining dataset	Session-disjoint test			Held-out task
Target	Pretraining dataset	5% $\uparrow$	10% $\uparrow$	25% $\uparrow$	5% $\uparrow$	10% $\uparrow$	25% $\uparrow$
Tactile-derived phase	HumanTouch [ 32 ]	$63.49$	$69.07$	$73.78$	$51.54$	$58.28$	$63.30$
Tactile-derived phase	BVT-20	$\mathbf{66.80}$	$\mathbf{72.24}$	$\mathbf{77.44}$	$\mathbf{54.87}$	$\mathbf{60.76}$	$\mathbf{66.46}$
Contact	HumanTouch [ 32 ]	$92.21$	$93.55$	$94.54$	$90.02$	$90.49$	$92.13$
Contact	BVT-20	$\mathbf{93.25}$	$\mathbf{94.34}$	$\mathbf{95.31}$	$\mathbf{90.43}$	$\mathbf{91.36}$	$\mathbf{93.42}$

TABLE IX: Cross-corpus transfer on BVT-20 (balanced accuracy, %). Both models use frozen BiView-Touch backbones and complete bilateral input. Best results are shown in bold.

Horizon	Rate	Balanced accuracy $\uparrow$			AUPRC $\uparrow$
Horizon	Rate	Paired context	Zero source	No cross- attention	AUPRC $\uparrow$
100 ms	3.16	66.21	61.39	52.87	15.23
250 ms	6.97	78.02	67.40	51.73	30.71
500 ms	10.73	80.14	71.47	52.11	42.22
1000 ms	16.18	80.71	73.57	52.83	54.67

TABLE X: Future contact-transition prediction on held-out BVT-20 task T17 (%, mean over three runs). Rate denotes transition prevalence, and AUPRC is reported for the paired context model. Source ablations are inference-time interventions and may introduce distribution shift.

Source context	Wrist bAcc $\uparrow$
Correct paired	54.93
Different stage, same motion	54.16
Different stage, different motion	19.31
Region permutation	31.35
Zero source	25.00

TABLE XI: HumanTouch wrist-motion recognition under source-context interventions with the target hand fully masked. Values are bAcc (%, mean over three head seeds).

Fig. 6: Source-context sensitivity on BVT-20 for phase and wrist-motion recognition. Cells show balanced-accuracy drops relative to correctly paired context under three target-visibility levels. Region permutation disrupts source anatomical assignment, while zero-source and no-cross-attention are auxiliary destructive controls. Full, Region, and Hand denote full target visibility, masked target-region input, and fully masked target-hand input; CS and ST denote cross-session and same-task substitution.

ID	Manipulation sequence
T01	Peg insertion: large- and small-angle failed attempts, followed by aligned insertion and removal.
T02	Three-pin plug insertion: two angular-offset attempts, followed by aligned insertion and unplugging.
T03	Bottle-cap threading: misaligned attempt, back-off, aligned tightening, and removal.
T04	Nut and bolt: align and thread the nut onto the bolt, then unscrew it.
T05	Food-container lid: misaligned closing attempt, realignment, latching, and reopening.
T06	Fabric-bag zipper: open and close the zipper while supporting the bag.

TABLE XII: BVT-20 task definitions. Initial and final idle states are omitted for brevity.

Fig. 7: Residual region-dependency maps for three BVT-20 tasks after removing additive effects and random-control predictions. Rows/columns denote target/source regions; upper/lower panels predict the left/right hand. Colors share a symmetric-log scale. Labels show mean Spearman $\rho$ and the 2.5th–97.5th percentile range over 100 repeated splits.

Tactile Sensing for Robotic Manipulation

Tactile sensing for robotic manipulation uses contact signals to help robots perceive forces, prevent slips, and handle objects reliably.

See all 180 papers

VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation
Apr 22, 2026Qianxi Hua, Xinyue Li, Zheng Yan +4Vision-Based Tactile Sensing Robot Manipulation Datasets
Humanoid Robot (Shanghai) Co., Ltd., Shanghai, China · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · School of Astronautics, Harbin Institute of Technology, Harbin, China
TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video
May 13, 2026Jianyi Zhou, Ziteng Gao, Feiyang Hong +11Vision-Based Tactile Sensing Visual-to-Tactile Prediction
Harbin Institute of Technology, Shenzhen · Meituan Academy of Robotics · Tsinghua Shenzhen International Graduate School, Tsinghua University
HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision
Jun 17, 2026Yuzhe Huang, Jiaping Wu, Jiaming Jiang +6Representation Learning Visuo-Tactile Representation Learning
Beihang University · Rimbot · BUPT +3

Apr 22, 2026J/K move · Enter open · S save

VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation

Qianxi Hua, Xinyue Li, Zheng Yan, Yang Li+3

Humanoid Robot (Shanghai) Co., Ltd., Shanghai, China · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · School of Astronautics, Harbin Institute of Technology, Harbin, China

Vision-Based Tactile Sensing Robot Manipulation Datasets Tactile Sensing Bimanual Robotic Manipulation Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging. This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale. To address these limitations, we introduce the VTOUCH dataset. It leverages vision based tactile sensing to provide high-fidelity physical interaction signals, adopts a matrix-style task design to enable systematic learning, and employs automated data collection pipelines covering real-world, demand-driven scenarios to ensure scalability. To further validate the effectiveness of the dataset, we conduct extensive quantitative experiments on cross-modal retrieval as well as real-robot evaluation. Finally, we demonstrate real-world performance through generalizable inference across multiple robots, policies, and tasks.

Abstract

Tactile Sensing for Robotic Manipulation

Explore similar work

VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision