Organizations: the Hubei Key Laboratory of Brain-inspired Intelligent Systems and the Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology (HUST), Wuhan 430074, Hubei, China. · Department of Innovation Center, China Ship Development and Design Center, Wuhan 430064, Hubei, China.
Robust multimodal perception for soft robotic grasping is challenged by modality-dependent noise, semantic ambiguity, and limited coordinated multisensory data. We introduce Soma, a dataset containing 100 objects, 1,000 continuous grasping trajectories, and 74,914 multimodal samples grouped by software acquisition cycle, spanning remote RGB, hand RGB, depth, tactile, IMU, and structured text observations. We also collect Soma-Success, an independent grasp-outcome dataset with 1,000 trials (998 retained after quality control), to evaluate frozen-encoder transfer for post-hoc grasp outcome classification and early grasp success prediction. Built on Soma, SomaAlign learns multimodal representations in three stages: masked reconstruction estimates relative modality reliability; reliability-aware contrastive alignment uses pair-specific adaptive temperatures to adjust alignment sharpness; and conflict-aware semantic constraints improve intra-sample cross-modal consistency while preserving global feature geometry. SomaAlign achieves an average Recall@10 of 90.66% across 30 directed cross-modal retrieval tasks. The learned representations also support physical attribute recognition, multimodal question answering, cross-modal generation, and cross-dataset transfer to grasp outcome prediction. On Soma-Success, frozen SomaAlign encoders with lightweight probes achieve 94.59% post-hoc classification accuracy and 84.07% early prediction accuracy. Together, Soma, Soma-Success, and SomaAlign provide a data and representation-learning foundation for robust and transferable multimodal perception in soft robotic manipulation.
Figures & tables
Fig. 1: Overview of Soma, Soma-Success, and the SomaAlign framework. The soft robotic platform acquires six coordinated modalities: Remote RGB, Hand RGB, Depth, Tactile, IMU, and Text. Soma comprises 100 objects, 1,000 trajectories, and 74,914 aligned multimodal samples, while Soma-Success contains 1,000 collected grasp trials, of which 998 valid trials are retained for downstream evaluation. SomaAlign learns a unified semantic representation through Reliability Assessment, Adaptive Feature Alignment, and Conflict-aware Refinement. The learned representation supports cross-modal retrieval, physical attribute recognition, multimodal question answering, cross-modal generation, and frozen-encoder grasp success prediction.
Dataset
Acquisition Coordination
Modalities
Embodiment
Sliding States
Text Annotation
Independent Evaluation Set
SoftGrasp [ 29 ]
Not reported
RGB, joint angles, and joint torques
Articulated dexterous hand
Not explicitly included
None
No
The Feeling of Success [ 13 ]
Not reported
RGB and dual GelSight tactile images
Parallel-jaw gripper
Not explicitly included
None
No
Touch and Go [ 14 ]
Timestamped visual and tactile streams
Egocentric RGB and GelSight tactile video
Human-held tactile sensor
No grasp-state annotation
Material labels only
No
VisGel [ 16 ]
Paired visual–tactile sequences
RGB and GelSight tactile images
Robot-mounted tactile sensor
No explicit grasp-state annotation
None
No
TVL [ 9 ]
Paired vision–touch samples
RGB, tactile images, and language
Human-held tactile sensor
No grasp-state annotation
Natural-language descriptions
No
ObjectFolder series [ 10 , 11 , 12 ]
Object-level aligned data
Vision, touch, and audio
Object-centric measurement and simulation
No explicit grasp-state annotation
Object metadata only
No separate subset
TABLE I: Comparison of representative multimodal tactile and grasping datasets along six dimensions relevant to soft robotic perception.
Fig. 2: Overview of the Soma acquisition system. A unified software polling loop coordinates remote RGB-D sensing, hand-mounted RGB cameras, finger-mounted IMUs, and flexible tactile pressure arrays. Structured text descriptions are generated from experimental metadata using predefined templates. Representative sensor outputs and an example text annotation are shown on the right.
Fig. 3: Object collection in Soma. The 100 everyday objects cover diverse materials, shapes, sizes, hardness levels, colors, and functional purposes.
Annotation Dimension
K
H
Hmax
J
Material
10
1.549
2.303
0.673
Hardness
5
1.419
1.609
0.882
Size
6
1.330
1.792
0.742
Shape
14
2.184
2.639
0.828
Color
28
2.929
3.332
0.879
State
2
0.566
0.693
0.816
TABLE II: Diversity and distribution evenness of the grasping-state and semantic annotations in Soma. K denotes the number of categories, H is the Shannon entropy, Hmax=lnK , and J=H/Hmax is the normalized evenness.
Fig. 4: Soma-Success collection protocol. The action-triggered snapshots Ta , Tb , and Tc correspond to the pre-grasp, closed-gripper, and terminal-lifting stages, respectively. The same snapshot-selection rule is applied to successful and failed trials. Representative examples are shown for the two-finger (2F) and three-finger (3F) configurations.
Fig. 5: Overview of the SomaAlign framework. SomaAlign follows three progressive optimization stages: (1) Reconstruction-Based Modality Reliability Estimation estimates modality reliability from normalized masked-reconstruction difficulty and constructs pairwise reliability scores; (2) Reliability-Conditioned Contrastive Alignment uses the pairwise reliability scores to adapt cross-modal contrastive temperatures and learn a shared representation; and (3) Batch-Adaptive Conflict-Aware Geometry-Preserving Refinement calibrates pairwise similarities, estimates fuzzy sample-level conflict, and adaptively modulates intra-sample refinement while imposing conflict-consistency and reference-center anchoring constraints. The reference geometry constructed from Stage-II representations, including the sample-specific centers and global representation scale, is frozen and reused during Stage-III optimization. After training, only the modality-specific encoders are retained for inference.
Fig. 6: Conceptual illustration of pair-specific distribution calibration and fuzzy membership assignment. Different modality pairs may exhibit different raw similarity distributions; therefore, the same similarity value can represent different degrees of relative inconsistency. Pair-specific ECDFs convert raw similarities into comparable conflict indicators ξi,π . The calibrated indicator is then softly assigned to overlapping ordered Gaussian fuzzy sets, allowing transitional cross-modal conflicts to partially activate multiple adjacent conflict levels.
Fig. 7: Motivation of the center-protection loss. Local cross-modal refinement may cause sample centers to deviate from the geometry established in Stage II. The center-protection constraint penalizes excessive center drift while allowing local multimodal representations to become more compact.
Fig. 8: Recall@1, Recall@5, and Recall@10 across 30 directed cross-modal retrieval tasks. Sensory-only retrieval uses paired temporal-window correspondence; text-related retrieval uses identical structured descriptions. Remote and Hand denote remote-view and hand-mounted RGB, respectively. Same-modality retrieval is excluded.
Method
R@1
R@5
R@10
Pairwise InfoNCE [ 36 ]
42.00
60.10
65.77
Hub Alignment [ 21 ]
44.55
64.30
70.55
SomaAlign
63.04
85.55
90.66
TABLE III: Comparison with controlled multimodal alignment baselines on Soma. All Recall values are percentages.
Method
Avg. R@1
Avg. R@5
Avg. R@10
TVL-Adapted [ 9 ]
55.64
75.09
81.02
TLV-Link-Adapted [ 37 ]
40.51
60.65
69.89
UniTouch-Adapted [ 23 ]
48.16
64.47
76.86
SomaAlign
69.71
88.07
92.05
TABLE IV: Comparison with published multimodal methods under the common V–T–L retrieval protocol on Soma. Recall values are percentages macro-averaged over six directed tasks.
Representation
Linear
MLP
Pairwise InfoNCE
92.06
94.40
Uniform Prior
92.89
94.90
SomaAlign
93.84
95.45
TABLE V: Mean probe accuracy across the five classification tasks (%).
Representation
Post-hoc
Early
Linear
MLP
Linear
MLP
Pairwise InfoNCE
89.78 ± 1.95
90.28 ± 1.43
82.87 ± 2.61
84.07 ± 3.24
Uniform Prior
92.49 ± 1.64
92.38 ± 1.16
82.87 ± 1.75
85.07 ± 1.84
Hub Alignment
89.18 ± 1.06
90.98 ± 0.70
81.67 ± 2.19
86.48 ± 2.52
SomaAlign
93.18 ± 2.18
94.59 ± 1.94
83.07 ± 2.60
84.07 ± 1.55
TABLE VI: Frozen-encoder grasp-outcome classification accuracy on Soma-Success (%), reported as mean ± standard deviation over five folds.
Fig. 9: Attribute classification accuracy retention under cross-modal generation, measured relative to the corresponding real-data baseline. (a) Single-modality probes. (b) Multi-modality probes with one generated modality replacing its real counterpart. (c) Average retention for the two strategies.
Metric
Value
Test samples
15,085
Exact Match
92.0%
Mean normalized Edit Distance
0.0059
Mean BLEU-4
0.9878
TABLE VII: Overall multimodal question-answering metrics on the Soma test split.
Model
R@1 (%)
R@5 (%)
R@10 (%)
k-NN (%)
Fisher
CHI
SomaAlign
63.04
85.55
90.66
87.2
1967.1
7.318
w/o Stage I
47.94
73.92
85.79
85.6
1863.9
7.241
w/o Adaptive Temp ( τ=0.15 )
47.27
73.63
84.67
86.6
1937.9
7.168
w/o Stage III
22.72
50.85
68.45
86.4
2220.7
7.315
w/o Center loss
34.48
65.26
73.64
86.2
1928.0
6.282
w/o Intra loss
26.98
57.51
65.17
86.2
2261.7
7.661
TABLE VIII: Ablation of the core components in SomaAlign.
Method
R@1
R@5
R@10
No-Conflict
54.44
80.72
86.38
Fixed-Weight
58.61
83.57
89.39
Sample-Gamma
59.00
83.48
89.05
Mean-Gamma
63.04
85.55
90.66
TABLE IX: Ablation of Stage-III conflict modulation strategies. Recall values are percentages.
Grasping is a foundational capability in robotic manipulation. Scene-level 6-DoF grasp synthesis relies on point clouds for metric geometry, yet real-world point observations are often noisy, incomplete, and over-smoothed. RGB images provide object-region, boundary, and appearance cues, yet two bottlenecks limit their effective use in scene representation and local grasp reasoning. We present MMGNet, a Multi-Modal Grasp Synthesis Network that addresses these bottlenecks through hierarchical image-to-3D feature injection and rotational graspness modeling. Hierarchical injection integrates multi-scale image features throughout 3D representation formation, while the rotational graspness jointly predicts the grasp approach direction and in-plane rotation before organizing local image-informed features through grasp-shaded grouping. Ablation studies show that these two modules provide complementary gains. MMGNet achieves a mean AP of 60.34 on GraspNet-1Billion with RealSense, surpassing all compared methods. In real-world experiments with adversarial grasp objects, MMGNet improves the grasp success rate over AnyGrasp by 8.46 percentage points. Code, models, and data will be released upon publication.
Rui Cao, Jia-Hui Pan, Yichuan Li +4
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, Shenzhen, China
Grasping is a fundamental robotic capability that bridges perception and physical task execution. This paper studies grasp pose recovery under a perception-to-execution mismatch, where a grasp generated from visual perception may become spatially stale if the object moves before execution, using only sparse tactile interactions and no further visual observations. We propose DA-GRD, Decision-Aware Grasp-Relevant Disambiguation, which maintains a weighted planar belief over possible object configurations and selects tactile probes according to their ability to eliminate hypotheses and improve agreement among candidate task grasps. Rather than fully relocalizing the object, DA-GRD stops when the remaining hypotheses support a common executable grasp. In MuJoCo experiments on ten rigid objects with translations up to 5~cm and yaw perturbations up to ±45∘, DA-GRD achieves an 84.7% physical lift success rate, compared with 9.1% for stale AnyGrasp, 21.2% for the original fix-scan baseline, and 63.7% for fix-scan method adapted with an SE(2) belief. DA-GRD also achieves a 57.3% Task conditioned Success rate. Across objects, it uses a success-average of 4.13 tactile probes over the ten per-object means, corresponding to a 72.5% reduction relative to the fixed 15-probe baselines. Real-world experiments on six objects achieve 71.7% physical lift success and 38.3% task-conditioned success with 4.20 probes on average. These results show that tactile sensing can recover task-relevant grasps under vision-off conditions with limited physical interaction, without requiring complete object localization.
Haoran Wang, Yuteng Sun, Yuanjie Li +6
Michael · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore
Robotic manipulation integrates vision, touch, and language, whose importance shifts across stages: vision guides reaching, while touch, through its evolution over time, decides grasping, alignment, and contact. Yet existing multi-modal manipulation policies typically use fixed temporal contexts and fusion strategies, despite shifts in what each modality contributes across different skills. We study how vision and touch should be combined at the level of primitive skills, asking what each skill needs from each sensor, and propose a skill-conditioned representation in which the queried skill conditions fusion over modality-specific short-term observation tokens while attending to a sparse event memory that retains terminal observations from the last K executed skills. Evaluated by skill progress estimation on three contact-rich tasks, it reduces slip-detection delay by 87% against fine-tuned SOTA progress models, twist-completion delay by 67.5% against a vision-only ablation, and progress error on a blind search task by 92% through sparse event memory. Gains concentrate exactly where completion is defined by contact or task history. More broadly, our results suggest that observation formation not only policy architecture is a central challenge in multi-modal representation. Project Website: http://what-to-attend-what-to-keep.github.io/