SenseAlign: Understanding Vision and Touch through Language at Test Time

Chuyang Ye; Qinting Jiang; Jingyan Jiang

cs.CVOct 20, 2026

SenseAlign: Understanding Vision and Touch through Language at Test Time

Authors: Chuyang Ye, Qinting Jiang, Jingyan Jiang

Organizations: New York University · Tsinghua Shenzhen International Graduate School, Tsinghua University · Shenzhen Technology University

Abstract

Tactile-visual-language models combine vision and touch through a shared language space to recognize objects and materials. In real-world deployment, changes in lighting, contact quality or sensor hardware can shift visual or tactile inputs and degrade recognition performance. When both modalities contribute to a prediction, an unreliable modality can override correct evidence from the other. Since labeled data are generally unavailable during deployment, test-time adaptation offers a way to recover recognition performance from unlabeled observations. In this setting, we find that comparing accumulated observations with the text embeddings of candidate classes reveals modality reliability differences that confidence can miss. Refining features against competing classes also improves recognition beyond geometric alignment alone. These findings motivate SenseAlign, which uses a common semantic reference for test-time modality weighting and feature correction. Alignment Trust weights each modality according to its support for shared predicted classes. Anchor Realignment uses the same class text embeddings to fit a shared feature correction that separates predicted classes from alternatives. Both modules use the same unlabeled history while keeping all encoders frozen. Extensive experiments across diverse datasets demonstrate strong recognition performance under both natural shifts and controlled corruptions.

Figures & tables

Figure 1: Vision and touch under sensing and hardware shifts. Natural changes in lighting or contact can affect the modalities unequally. The upper panel uses brightness and defocus corruptions to illustrate three possible modality relationships. The lower panel shows separate wire contacts recorded by GelSight Mini, DIGIT and DuraGel, illustrating differences across tactile sensor hardware and capture conditions.

Figure 2: Modality reliability varies with backbone and stream. (a) Selected class pairs projected along their text-embedding decision direction. Stars mark text anchors; percentages give each class’s share on its own side of the dashed boundary, not full-vocabulary accuracy. Arrows show shifts from equal-fusion to SenseAlign class centres; $w_{v}$ is the mean vision weight. (b) Stream-level accuracy; green arrows and labels give gains over frozen equal fusion in percentage points.

Figure 3: Semantic alignment distinguishes modalities when confidence misranks them. (a) Standardized scores between pseudo-label group means and text embeddings. Columns follow the row-label order; grey rows lack accepted observations. The red box highlights the visual branch’s strong scores for plastic across groups. (b) Frozen accuracy, mean top-1 probability, final-memory alignment $\alpha$ and mean online vision weight. Green and red outlines indicate whether confidence or alignment favors the more or less accurate modality; crosses show alignment with randomized memory labels. Purple diamonds show SenseAlign’s vision weights.

Figure 4: A shared history supports modality weighting and feature correction. Past unlabeled observations supply both modules with shared group statistics. Anchor Realignment fits

R

using fused means (circles) and class text embeddings (stars), then refines class separation. Alignment Trust compares standardized group–text scores to assign fusion weights. Dashed arrows pass the shared correction and weights to the prediction path; the same correction is applied to both branches before weighted fusion. Encoders remain frozen. Positions, colors and bar lengths are schematic.

						Frozen accuracy
Regime	Benchmark	Model	Streams	Presentations	Classes	Vision	Touch	Fusion
	TacQuad (clean)	UniTouch	3 sensors	1,536	24	54.5	0 0.6	46.4
	TacQuad (15 corruptions)	UniTouch	45	23,040	24	53.7	0 0.6	39.4
	ObjectFolder Real	UniTouch	5 splits	3,395	7	48.7	16.2	40.8
Touch unreliable	RCT	UniTouch	1	9,620	7	62.8	11.4	48.1
Touch dominant	ObjectFolder Real	AnyTouch	5 splits	3,395	7	37.3	74.6	71.7

Table 1: Evaluation protocol and frozen accuracy (%), one original prompt per class. Presentations count paired observations. Touch is unreliable near/below chance (4.2% for 24 classes, 14.3% for seven), dominant when more accurate; both sensors are informative when well above chance. Labels are retrospective and unavailable to adaptation. Natural shifts weight groups and streams equally; TAG-C aggregation is detailed in the appendix.

	UniTouch					AnyTouch
Method	TacQuad clean	TacQuad corr.	OFR	RCT	Mean	TacQuad clean	TacQuad corr.	OFR	RCT	Mean
Frozen ∘	46.4	39.4	40.8	48.1	43.7	42.2	37.4	71.7	29.5	45.2
Frozen Vision	54.5	53.7	48.7	62.8	54.9	42.2	32.0	37.3	47.1	39.6
Frozen Tactile	0.6	0.6	16.2	11.4	7.2	18.1	18.1	74.6	18.1	32.3
TENT	45.5	36.8	35.0	26.5	35.9	41.1	35.1	69.7	19.4	41.3
SAR	46.4	39.3	40.6	39.7	41.5	42.0	37.2	71.6	24.7	43.9

Table 2: Natural-shift accuracy (%) with one prompt per class. Both models share four panels; Mean weights panels equally. OFR uses five released-order splits. Frozen ∘ uses equal fusion.

Figure 5: Modality weights adapt across streams, with gains over equal fusion. (a) Online vision weights: the median across TacQuad streams and individual trajectories elsewhere. Grey TAG-C curves are solid for AnyTouch and dashed for UniTouch; V/T indicate visual/tactile corruption. (b) UniTouch TacQuad accuracy averaged over three sensors per condition. Hollow diamonds denote frozen equal fusion, filled diamonds SenseAlign, and grey dots other baselines. (c) TAG-C gains over matched equal fusion in percentage points under visual or tactile corruption; blue denotes UniTouch and teal AnyTouch.

	AnyTouch				UniTouch				Overall
Method	Visual	Tactile	Mixed	Mean	Visual	Tactile	Mixed	Mean	Mean
Frozen ∘	56.1	59.1	55.2	56.8	52.8	50.3	49.6	50.9	53.8
TENT	53.0	60.2	2.4	38.6	46.3	32.2	33.6	37.4	38.0
SAR	52.4	56.9	44.1	51.1	58.9	53.1	57.7	56.6	53.9
SuMi	56.7	60.8	51.6	56.4	49.8	45.4	16.0	37.1	46.7
RoTTA	62.2	57.1	50.0	56.4	42.4	47.4	31.3	40.4	48.4

Table 3: TAG-C severity-3 accuracy (%), one prompt per class; means weight scenarios equally. ∘ : equal fusion. Bold/underline: best/second.

Configuration	GelSight Mini	DIGIT	DuraGel	Mean
Equal fusion	55.15	42.31	30.23	42.56
Trust only	46.76	45.06	41.01	44.27
Realign only	54.18	38.34	32.69	41.74
SenseAlign	56.50	46.23	41.24	47.99

Table 4: Components on AnyTouch TacQuad. One prompt per class; accuracy (%). Groups are weighted equally within each sensor; Mean weights sensors equally.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Operation
1	Encode both modalities with frozen encoders; normalize their features.
2	If $C>2$ , $N\geq 16$ and history changed, form $\mu_{c}=S_{c}/\Lambda_{c}$ , construct $Q$ , initialize $P$ by minimum-change Procrustes, and refine $P$ using Equation 9 .
3	Apply the correction fitted to earlier observations to each branch, fuse with equal weights, classify with the shared text anchors, and record every prediction. Use the identity map before warm-up or when $C=2$ .
4	Compute the pseudo-label and positive-margin weight from raw frozen fusion.
5	Append the current batch’s raw statistics using Equation 13 . Mark the fit for recomputation if any observation was accepted.

Appendix

Table 5: Anchor Realignment alone for one batch. All fitted state is based on earlier batches; fusion weights remain equal. SenseAlign also computes modality weights from the shared memory, as specified in Section 4.2 .

Setting	Value
Encoder parameters updated	None
Classification reference	One normalized text embedding per class
Initialization targets	Same anchors as the classifier
History weighting	Positive top-two cosine margin
Refinement class weighting	Norm of accumulated class mean
Default residual coefficient	$\beta=0.2$

Appendix

Table 6: Default SenseAlign settings. Main-table exceptions are listed in Table 7 . All comparisons use one original prompt per class.

Model	Scenario	$\beta$	$g$
AnyTouch	TacQuad clean	0.2	8
AnyTouch	TacQuad corrupted	0.4	1
AnyTouch	ObjectFolder Real	0.2	1
AnyTouch	RCT	0.1	64
AnyTouch	TAG-C Visual	0.2	4
AnyTouch	TAG-C Tactile	0.2	4

Appendix

Table 7: SenseAlign settings for every main-table scenario. All rows use 16 accepted observations before adaptation.

Figure 6: Sensitivity to feature correction and modality weighting. Cells report accuracy (%); colours show the drop from each panel’s best tested setting on a shared scale.

Condition	Equal fusion	SenseAlign	SenseAlign $-$ fusion
Clean	46.45	58.59	+12.14
Brightness	40.55	54.24	+13.70
Contrast	38.29	59.53	+21.24
Defocus blur	42.99	53.97	+10.98
Elastic	33.43	49.52	+16.09
Fog	40.16	57.26	+17.10

Appendix

Table 8: UniTouch on each TacQuad condition (accuracy, %), averaging named targets within a sensor and then the three sensors. The last column is SenseAlign minus equal fusion.

Condition	Equal fusion	SenseAlign	SenseAlign $-$ fusion
Clean	48.09	60.49	+12.40
Brightness	42.74	60.77	+18.03
Contrast	37.97	61.60	+23.63
Defocus blur	38.73	52.84	+14.10
Elastic	37.63	47.55	+9.92
Fog	38.06	52.87	+14.82

Appendix

Table 9: GelSight Mini on TacQuad with UniTouch (accuracy, %). The last column is SenseAlign minus equal fusion.

Condition	Equal fusion	SenseAlign	SenseAlign $-$ fusion
Clean	43.46	57.81	+14.35
Brightness	35.45	51.57	+16.12
Contrast	31.56	63.13	+31.57
Defocus blur	38.22	54.22	+16.00
Elastic	28.29	54.46	+26.17
Fog	35.03	63.26	+28.23

Appendix

Table 10: DIGIT on TacQuad with UniTouch (accuracy, %). The last column is SenseAlign minus equal fusion.

Condition	Equal fusion	SenseAlign	SenseAlign $-$ fusion
Clean	47.79	57.47	+9.68
Brightness	43.45	50.40	+6.94
Contrast	45.33	53.85	+8.52
Defocus blur	52.01	54.86	+2.85
Elastic	34.38	46.55	+12.17
Fog	47.39	55.66	+8.27

Appendix

Table 11: DuraGel on TacQuad with UniTouch (accuracy, %). The last column is SenseAlign minus equal fusion.

Split	Equal fusion	SenseAlign	SenseAlign $-$ fusion
0	41.11	59.91	+18.79
1	46.63	41.91	-4.73
2	48.71	51.93	+3.22
3	38.89	57.88	+18.98
4	28.46	30.88	+2.42

Appendix

Table 12: UniTouch on the five ObjectFolder Real test splits (accuracy, %, objects weighted equally). Default

\beta=0.2

g=4

results; the selected main-table configuration is reported separately. The last column is SenseAlign minus equal fusion.

	UniTouch					AnyTouch
Method	TacQuad clean	TacQuad corr.	OFR	RCT	Mean	TacQuad clean	TacQuad corr.	OFR	RCT	Mean
READ	9.9	9.1	17.9	26.4	15.9	12.1	9.7	18.0	3.3	10.8

Appendix

Table 13: READ natural-shift accuracy (%) on the tactile–visual backbones. The four panels, input order and aggregation match Table 2 ; each Mean weights its four panels equally.

	AnyTouch				UniTouch				Overall
Method	Visual	Tactile	Mixed	Mean	Visual	Tactile	Mixed	Mean	Mean
READ	6.4	10.2	0.1	5.6	1.3	23.2	6.5	10.3	7.9

Appendix

Table 14: READ TAG-C accuracy (%) at severity 3 on the tactile–visual backbones. Scenarios and aggregation match Table 3 ; each model Mean weights three scenarios equally, and Overall weights all six.

Method	Clean	Corruptions
READ	-36.50	-30.23
SuMi	-0.08	-0.53
TDA	+2.68	+4.20
OGA	+5.65	+6.71
StatA	+4.86	+5.82
TSA	-2.98	-3.08

Appendix

Table 15: Reported external method minus equal fusion with one prompt per class on TacQuad. Differences are in percentage points. StatA uses FP64 arithmetic.

Method	ObjectFolder Real	RCT
READ	-22.81	-21.65
SuMi	-1.42	-21.85
TDA	+1.82	-10.90
OGA	+7.31	-39.02
StatA	+5.48	+12.15
TSA	-3.21	-36.18

Appendix

Table 16: Reported external method minus equal fusion with one prompt per class on natural shifts. Differences are in percentage points. StatA uses FP64 arithmetic.

Dataset	FP32 invalid batches	Finite rows	Disagreements	FP32 accuracy
TacQuad	52	23744	10438	—
ObjectFolder Real	0	3395	399	42.00
RCT	217	6160	470	—

Appendix

Table 17: StatA numerical precision diagnostic on the completed full streams.

Visual corruptions
Condition	SenseAlign minus fusion
Visual mean	+4.04
Tactile mean	+10.53
Overall mean	+7.29
Brightness	+4.90
Defocus blur	+5.40

Appendix

Table 18: UniTouch TAG-C differences from single-prompt equal fusion at default

\beta=0.2

g=4

(percentage points).

Figure 7: Geometric matching and class discrimination. Raw denotes uncorrected features; Procrustes and CE denote geometric initialization and subsequent cross-entropy refinement. (a) Mean cosine similarity to the correct-class text embedding and the mean across all class embeddings. (b) Mean gap between the correct-class score and the strongest incorrect-class score. Scores are evaluated retrospectively using the final memory.

Backbone	Stage	Correct	Rival	Gap	Accuracy (%)
AnyTouch	Raw	0.1044	0.0946	0.0098	72.30
	Procrustes, $\beta=.2$	0.2843	0.2610	0.0233	81.25
	CE, $\beta=.2$	0.1148	0.0823	0.0325	89.35
	Same CE map, $\beta=1$	-0.0001	-0.0628	0.0627	94.32

Appendix

Table 24: Stage scores and retrospective accuracy on the same ObjectFolder Real replay. Correct and Rival are the means of

r_{iy_{i}}

and

\max_{j\neq y_{i}}r_{ij}

; Gap is their difference. The last row applies the same refined map with

\beta=1

without refitting a map optimized at

\beta=0.2

Backbone	Raw	$O$ only	$H$ only	Full $B$	Common norm.
AnyTouch	72.30	86.93	77.27	89.35	90.06

Appendix

Table 25: Frozen-factor controls, reported as retrospective accuracy (%). Full

B

applies both polar factors and normalizes each branch separately. Common norm. uses the same

B

but normalizes only after summing the two transformed branches. All maps and explicit fusion weights are fixed.

Backbone	Mean $\widehat{w}_{v}$	Corrected	Introduced
AnyTouch	51.46%	0	5

Appendix

Table 26: Coefficients relative to unnormalized branches and effects of separate branch normalization on the same 704 observations. Corrected and Introduced count errors corrected and introduced relative to the common-normalization control in Table 25 .

Multimodal Understanding

Multimodal understanding studies how AI systems interpret and reason across text, images, video, audio, and other signals.

See all 479 papers

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs
Jul 1, 2026Yoonhyung Park, Minji Kim, Sungwon Moon +1Visuo-Tactile Representation Learning Multimodal Large Language Models
Division of Artificial Intelligence & Software, Ewha Womans University
Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift
Apr 27, 2026Lixian Chen, Yanhui Chen, Junyi LinVLM Robustness Test-Time Adaptation
Guangdong University of Technology · Guangzhou, China
TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models
Jun 27, 2026Mark Van der Merwe, Mohamad Louai Shehab, Jayjun Lee +4Vision-Language-Action Models Tactile Sensing for Robotic Manipulation
Robotics Department, University of Michigan · Computer Science and Engineering Department, University of Michigan

Jul 1, 2026J/K move · Enter open · S save

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

Yoonhyung Park, Minji Kim, Sungwon Moon, Jiyoung Lee

Division of Artificial Intelligence & Software, Ewha Womans University

Visuo-Tactile Representation Learning Multimodal Large Language Models Multimodal Foundation Models Continual Learning for VLMs Parameter-Efficient Fine-Tuning Multimodal Understanding

Open paper AskPDF arXiv

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.

Abstract

Multimodal Understanding

Explore similar work

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift

TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models