SomaAlign: Reliability-Aware Multimodal Representation Learning for Soft Robotic Grasping

Haoyuan Wang; Shuai Peng; Yongkai Liao; Yuxiao Sun; Wei Luo; Xinxing Chen; Jian Huang

cs.RONov 2, 2026

SomaAlign: Reliability-Aware Multimodal Representation Learning for Soft Robotic Grasping

Authors: Haoyuan Wang, Shuai Peng, Yongkai Liao, Yuxiao Sun, Wei Luo, Xinxing Chen, Jian Huang

Organizations: the Hubei Key Laboratory of Brain-inspired Intelligent Systems and the Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology (HUST), Wuhan 430074, Hubei, China. · Department of Innovation Center, China Ship Development and Design Center, Wuhan 430064, Hubei, China.

Abstract

Robust multimodal perception for soft robotic grasping is challenged by modality-dependent noise, semantic ambiguity, and limited coordinated multisensory data. We introduce Soma, a dataset containing 100 objects, 1,000 continuous grasping trajectories, and 74,914 multimodal samples grouped by software acquisition cycle, spanning remote RGB, hand RGB, depth, tactile, IMU, and structured text observations. We also collect Soma-Success, an independent grasp-outcome dataset with 1,000 trials (998 retained after quality control), to evaluate frozen-encoder transfer for post-hoc grasp outcome classification and early grasp success prediction. Built on Soma, SomaAlign learns multimodal representations in three stages: masked reconstruction estimates relative modality reliability; reliability-aware contrastive alignment uses pair-specific adaptive temperatures to adjust alignment sharpness; and conflict-aware semantic constraints improve intra-sample cross-modal consistency while preserving global feature geometry. SomaAlign achieves an average Recall@10 of 90.66% across 30 directed cross-modal retrieval tasks. The learned representations also support physical attribute recognition, multimodal question answering, cross-modal generation, and cross-dataset transfer to grasp outcome prediction. On Soma-Success, frozen SomaAlign encoders with lightweight probes achieve 94.59% post-hoc classification accuracy and 84.07% early prediction accuracy. Together, Soma, Soma-Success, and SomaAlign provide a data and representation-learning foundation for robust and transferable multimodal perception in soft robotic manipulation.

Figures & tables

Fig. 1: Overview of Soma, Soma-Success, and the SomaAlign framework. The soft robotic platform acquires six coordinated modalities: Remote RGB, Hand RGB, Depth, Tactile, IMU, and Text. Soma comprises 100 objects, 1,000 trajectories, and 74,914 aligned multimodal samples, while Soma-Success contains 1,000 collected grasp trials, of which 998 valid trials are retained for downstream evaluation. SomaAlign learns a unified semantic representation through Reliability Assessment, Adaptive Feature Alignment, and Conflict-aware Refinement. The learned representation supports cross-modal retrieval, physical attribute recognition, multimodal question answering, cross-modal generation, and frozen-encoder grasp success prediction.

Dataset	Acquisition Coordination	Modalities	Embodiment	Sliding States	Text Annotation	Independent Evaluation Set
SoftGrasp [ 29 ]	Not reported	RGB, joint angles, and joint torques	Articulated dexterous hand	Not explicitly included	None	No
The Feeling of Success [ 13 ]	Not reported	RGB and dual GelSight tactile images	Parallel-jaw gripper	Not explicitly included	None	No
Touch and Go [ 14 ]	Timestamped visual and tactile streams	Egocentric RGB and GelSight tactile video	Human-held tactile sensor	No grasp-state annotation	Material labels only	No
VisGel [ 16 ]	Paired visual–tactile sequences	RGB and GelSight tactile images	Robot-mounted tactile sensor	No explicit grasp-state annotation	None	No
TVL [ 9 ]	Paired vision–touch samples	RGB, tactile images, and language	Human-held tactile sensor	No grasp-state annotation	Natural-language descriptions	No
ObjectFolder series [ 10 , 11 , 12 ]	Object-level aligned data	Vision, touch, and audio	Object-centric measurement and simulation	No explicit grasp-state annotation	Object metadata only	No separate subset

TABLE I: Comparison of representative multimodal tactile and grasping datasets along six dimensions relevant to soft robotic perception.

Fig. 2: Overview of the Soma acquisition system. A unified software polling loop coordinates remote RGB-D sensing, hand-mounted RGB cameras, finger-mounted IMUs, and flexible tactile pressure arrays. Structured text descriptions are generated from experimental metadata using predefined templates. Representative sensor outputs and an example text annotation are shown on the right.

Fig. 3: Object collection in Soma. The 100 everyday objects cover diverse materials, shapes, sizes, hardness levels, colors, and functional purposes.

Annotation Dimension	$\boldsymbol{K}$	$\boldsymbol{H}$	$\boldsymbol{H_{\max}}$	$\boldsymbol{J}$
Material	10	1.549	2.303	0.673
Hardness	5	1.419	1.609	0.882
Size	6	1.330	1.792	0.742
Shape	14	2.184	2.639	0.828
Color	28	2.929	3.332	0.879
State	2	0.566	0.693	0.816

TABLE II: Diversity and distribution evenness of the grasping-state and semantic annotations in Soma.

K

denotes the number of categories,

H

is the Shannon entropy,

H_{\max}=\ln K

, and

J=H/H_{\max}

is the normalized evenness.

Fig. 4: Soma-Success collection protocol. The action-triggered snapshots $T_{a}$ , $T_{b}$ , and $T_{c}$ correspond to the pre-grasp, closed-gripper, and terminal-lifting stages, respectively. The same snapshot-selection rule is applied to successful and failed trials. Representative examples are shown for the two-finger (2F) and three-finger (3F) configurations.

Fig. 5: Overview of the SomaAlign framework. SomaAlign follows three progressive optimization stages: (1) Reconstruction-Based Modality Reliability Estimation estimates modality reliability from normalized masked-reconstruction difficulty and constructs pairwise reliability scores; (2) Reliability-Conditioned Contrastive Alignment uses the pairwise reliability scores to adapt cross-modal contrastive temperatures and learn a shared representation; and (3) Batch-Adaptive Conflict-Aware Geometry-Preserving Refinement calibrates pairwise similarities, estimates fuzzy sample-level conflict, and adaptively modulates intra-sample refinement while imposing conflict-consistency and reference-center anchoring constraints. The reference geometry constructed from Stage-II representations, including the sample-specific centers and global representation scale, is frozen and reused during Stage-III optimization. After training, only the modality-specific encoders are retained for inference.

Fig. 6: Conceptual illustration of pair-specific distribution calibration and fuzzy membership assignment. Different modality pairs may exhibit different raw similarity distributions; therefore, the same similarity value can represent different degrees of relative inconsistency. Pair-specific ECDFs convert raw similarities into comparable conflict indicators $\xi_{i,\pi}$ . The calibrated indicator is then softly assigned to overlapping ordered Gaussian fuzzy sets, allowing transitional cross-modal conflicts to partially activate multiple adjacent conflict levels.

Fig. 7: Motivation of the center-protection loss. Local cross-modal refinement may cause sample centers to deviate from the geometry established in Stage II. The center-protection constraint penalizes excessive center drift while allowing local multimodal representations to become more compact.

Fig. 8: Recall@1, Recall@5, and Recall@10 across 30 directed cross-modal retrieval tasks. Sensory-only retrieval uses paired temporal-window correspondence; text-related retrieval uses identical structured descriptions. Remote and Hand denote remote-view and hand-mounted RGB, respectively. Same-modality retrieval is excluded.

Method	R@1	R@5	R@10
Pairwise InfoNCE [ 36 ]	42.00	60.10	65.77
Hub Alignment [ 21 ]	44.55	64.30	70.55
SomaAlign	63.04	85.55	90.66

TABLE III: Comparison with controlled multimodal alignment baselines on Soma. All Recall values are percentages.

Method	Avg. R@1	Avg. R@5	Avg. R@10
TVL-Adapted [ 9 ]	55.64	75.09	81.02
TLV-Link-Adapted [ 37 ]	40.51	60.65	69.89
UniTouch-Adapted [ 23 ]	48.16	64.47	76.86
SomaAlign	69.71	88.07	92.05

TABLE IV: Comparison with published multimodal methods under the common V–T–L retrieval protocol on Soma. Recall values are percentages macro-averaged over six directed tasks.

Representation	Linear	MLP
Pairwise InfoNCE	92.06	94.40
Uniform Prior	92.89	94.90
SomaAlign	93.84	95.45

TABLE V: Mean probe accuracy across the five classification tasks (%).

Representation	Post-hoc		Early
	Linear	MLP	Linear	MLP
Pairwise InfoNCE	89.78 $\pm$ 1.95	90.28 $\pm$ 1.43	82.87 $\pm$ 2.61	84.07 $\pm$ 3.24
Uniform Prior	92.49 $\pm$ 1.64	92.38 $\pm$ 1.16	82.87 $\pm$ 1.75	85.07 $\pm$ 1.84
Hub Alignment	89.18 $\pm$ 1.06	90.98 $\pm$ 0.70	81.67 $\pm$ 2.19	86.48 $\pm$ 2.52
SomaAlign	93.18 $\pm$ 2.18	94.59 $\pm$ 1.94	83.07 $\pm$ 2.60	84.07 $\pm$ 1.55

TABLE VI: Frozen-encoder grasp-outcome classification accuracy on Soma-Success (%), reported as mean

\pm

standard deviation over five folds.

Fig. 9: Attribute classification accuracy retention under cross-modal generation, measured relative to the corresponding real-data baseline. (a) Single-modality probes. (b) Multi-modality probes with one generated modality replacing its real counterpart. (c) Average retention for the two strategies.

Metric	Value
Test samples	15,085
Exact Match	92.0%
Mean normalized Edit Distance	0.0059
Mean BLEU-4	0.9878

TABLE VII: Overall multimodal question-answering metrics on the Soma test split.

Model	R@1 (%)	R@5 (%)	R@10 (%)	k-NN (%)	Fisher	CHI
SomaAlign	63.04	85.55	90.66	87.2	1967.1	7.318
w/o Stage I	47.94	73.92	85.79	85.6	1863.9	7.241
w/o Adaptive Temp ( $\tau=0.15$ )	47.27	73.63	84.67	86.6	1937.9	7.168
w/o Stage III	22.72	50.85	68.45	86.4	2220.7	7.315
w/o Center loss	34.48	65.26	73.64	86.2	1928.0	6.282
w/o Intra loss	26.98	57.51	65.17	86.2	2261.7	7.661

TABLE VIII: Ablation of the core components in SomaAlign.

Method	R@1	R@5	R@10
No-Conflict	54.44	80.72	86.38
Fixed-Weight	58.61	83.57	89.39
Sample-Gamma	59.00	83.48	89.05
Mean-Gamma	63.04	85.55	90.66

TABLE IX: Ablation of Stage-III conflict modulation strategies. Recall values are percentages.

Reliability-Aware Multimodal Fusion

See all 34 papers

MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes
Oct 26, 2026Rui Cao, Jia-Hui Pan, Yichuan Li +46-DOF Grasp Pose Estimation Robotic Grasping
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, Shenzhen, China
DA-GRD: Decision-Aware Grasp-Relevant Disambiguation for tactile recovery under perception-to-execution mismatches
Sep 24, 2026Haoran Wang, Yuteng Sun, Yuanjie Li +6Robotic Grasping Active Sensing
Michael · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore
What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory
Sep 29, 2026Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene +5Robot Task Progress Estimation Contact-Rich Robotic Manipulation
Computer Science Department, University of Maryland, College Park, MD, USA

Oct 26, 2026J/K move · Enter open · S save

MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes

Rui Cao, Jia-Hui Pan, Yichuan Li, Yunfan Yu+3

Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, Shenzhen, China

6-DOF Grasp Pose Estimation Robotic Grasping Multimodal Sensor Fusion

Open paper AskPDF arXiv

Grasping is a foundational capability in robotic manipulation. Scene-level 6-DoF grasp synthesis relies on point clouds for metric geometry, yet real-world point observations are often noisy, incomplete, and over-smoothed. RGB images provide object-region, boundary, and appearance cues, yet two bottlenecks limit their effective use in scene representation and local grasp reasoning. We present MMGNet, a Multi-Modal Grasp Synthesis Network that addresses these bottlenecks through hierarchical image-to-3D feature injection and rotational graspness modeling. Hierarchical injection integrates multi-scale image features throughout 3D representation formation, while the rotational graspness jointly predicts the grasp approach direction and in-plane rotation before organizing local image-informed features through grasp-shaded grouping. Ablation studies show that these two modules provide complementary gains. MMGNet achieves a mean AP of 60.34 on GraspNet-1Billion with RealSense, surpassing all compared methods. In real-world experiments with adversarial grasp objects, MMGNet improves the grasp success rate over AnyGrasp by 8.46 percentage points. Code, models, and data will be released upon publication.

Abstract

Reliability-Aware Multimodal Fusion

Explore similar work

MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes

DA-GRD: Decision-Aware Grasp-Relevant Disambiguation for tactile recovery under perception-to-execution mismatches

What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory