Organizations: MMLab, The Chinese University of Hong Kong · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen · Southeast University · China Singapore International Joint Research Institute
Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose Unisense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. Unisense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate Unisense-1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate Unisense on 12 benchmarks spanning depth, point cloud, IMU, tactile, thermal, radar, motion, and smell understanding. Across these heterogeneous sensing tasks, Unisense achieves strong performance and outperforms both generalist and specialized models on most benchmarks, demonstrating the potential of pretrained MLLMs as unified models for embodied sensory understanding.
Figures & tables
Figure 1: Different senses, shared intelligence : broad embodied understanding requires diverse physical evidence while drawing on shared knowledge and reasoning. Examples from depth, smell, thermal, and radar show how these signals resolve ambiguities left by basic-modality observations. Rather than duplicating general intelligence across separate sensory models, UniSense combines modality-specific experts with a shared LLM.
Figure 2: The architecture of UniSense. UniSense unifies basic and embodied sensory modalities in a shared framework. Basic modalities retain pretrained pathways, while sensory inputs pass through modality-specific tokenizers, a unified encoder, and modality-specific projectors before entering the LLM. Training proceeds in two stages: (1) modality captions supervise sensory–language alignment; (2) modality-specific experts are then tuned within the LLM.
Figure 3: Overview of the sensory representation alignment stage. Conditional captioning provides language-level supervision by decoding the paired caption from sensory representations, while contrastive alignment aligns sensory and caption representations using the first k LLM layers.
Figure 4: Modality-specific expert computation within a Transformer layer. After shared self-attention, a modality router directs sensory tokens to modality-specific expert pools, where token-wise routers produce residual transformations that are added to the shared base FFN output.
Table 1: Evaluation on General Multimodal benchmarks. (a) Image-text results on TextVQA, MME, and MMBench. (b) Audio-image-text results on the image-audio variants of TextVQA, DocVQA, and ChartQA. (c) Video-text results on Video-MME and EgoSchema. Video-MME results are reported without subtitles. “–“ denotes unavailable results. Due to space limitations, the citations of all benchmarks and baseline models reported in the main-text tables are provided in Appendix § B.1 .
SpatialBench
RoboSpatial-Home
Model
Pos.
Exist.
Count.
Reach.
Size
4-Task Agg.
Agg.
Config.
Closed-Source Models
GPT-4o
61.8
90.0
85.2
51.7
40.0
68.1
65.3
77.2
Claude-3.5
55.9
65.0
72.2
–
26.7
55.1
–
–
General MLLMs
LLaVA-1.5-7B
44.1
45.0
82.8
–
30.0
49.5
–
–
Table 2: Evaluation on Depth-Text benchmarks, including SpatialBench and RoboSpatial-Home. SpatialBench has Position, Existence, Counting, Reaching, and Size subtask. 4-Task Agg. is the sample-weighted aggregate over all subtasks except Reaching. UniSense is evaluated zero-shot on RoboSpatial-Home.
Table 7Table 8
Table 7: Evaluation on Radar-Text, Motion-Text, and Smell-Text Benchmarks. (a) Radar results on HoloLLM’s language-grounded MM-Fi benchmark under the Cross-Environment (CrossEnv) split; Human Action QA and Action Caption use accuracy and METEOR, respectively. (b) Motion results on QuAnTS; binary and multiple-choice QA use F1, while open-ended QA uses ROUGE-L and METEOR. TS, GT, and AE denote motion time series, ground-truth actions, and action-encoder representations, respectively. (c) Smell classification on SmellNet using Acc@1 and Macro-F1 (%).
Method
Benchmarks
Setting
Proj.
Alignment
Experts
RoboSpatial-Home
ScanQA
IMU-4-Task
Infrared-Benchmark
MMFi-VQA
Config.
CIDEr
Avg. F1
Acc.
Acc.
Baseline
✓
74.0
96.7
86.0
88.2
60.1
+ Alignment
✓
✓
77.2
97.7
88.0
89.0
63.5
Full
✓
✓
✓
80.5
99.6
89.0
89.9
65.2
Table 8: Component ablation of UniSense. Starting from the baseline, we progressively introduce sensory representation alignment and modality-specific expert tuning. Results are reported on representative benchmarks spanning depth, point cloud, IMU, thermal, and radar.
Figure 5: Ablation of alignment objectives and expert specialization. (a) Comparison of language-modeling supervision, conditional captioning, contrastive alignment, and their combination during sensory representation alignment. (b) Comparison between a shared expert pool and modality-specific expert pools during expert tuning.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Modality
Benchmark
Citation
Benchmark
Citation
General Multimodal Benchmarks
Image-Text
TextVQA
Singh et al. (2019)
MME
Fu et al. (2025a)
MMBench
Liu et al. (2024c)
Audio-Image-Text
DocVQA
Mathew et al. (2021)
ChartQA
Masry et al. (2022)
Video-Text
Video-MME
Fu et al. (2025b)
EgoSchema
Mangalam et al. (2023)
Embodied Sensory Benchmarks
Appendix
Table B9: Supplementary citations of the evaluation benchmarks. Benchmarks are grouped by modality following the organization of § B.2 .
Model
Citation
Model
Citation
Closed-Source Models
GPT-4o
Hurst et al. (2024)
Claude-3.5
Anthropic (2024)
Gemini-2.5-Pro
Comanici et al. (2025)
General MLLMs
LLaVA-1.5
Liu et al. (2024a)
LLaVA-NeXT
Liu et al. (2024b)
LLaVA-OneVision
Li et al. (2025a)
LLaVA-Video
Zhang et al. (2025a)
Appendix
Table B10: Supplementary citations of the closed-source and general multimodal baselines. Models follow the same groups as the main-text tables.
Group
Model
Citation
Model
Citation
Spatial
SpatialVLM
Chen et al. (2024a)
Bunny
He et al. (2024)
SpatialBot
Cai et al. (2025)
SpaceThinker
Batra et al. (2025)
SpaceOm
Remyx AI (2025)
SSR
Liu et al. (2026)
VPT
Yu et al. (2025)
SpatialVTS
Liang et al. (2026)
3D
ScanQA
Azuma et al. (2022)
Scene-LLM
Fu et al. (2025c)
LL3DA
Chen et al. (2024b)
ChatScene
Huang et al. (2024)
Appendix
Table B11: Supplementary citations of the modality-specialized baselines. Models follow the same groups as the main-text tables.
Figure B6: Overview of the training data composition. The donut charts illustrate the distribution of QA and unique data instances across basic and sensory modalities. The bottom panel provides a dataset-level breakdown for each modality, with the number of training QA and unique instances reported for every data source.
Table C12: Stage-wise training configuration of UniSense. The reported learning rate is the base learning rate; modality projectors use 1.5×10−4 .
Setting
Alignment Stage
Tuning Stage
Baseline
LLM
Shared residual FFN
Representation alignment
LLM+Lcap+Lcon
Shared residual FFN
Full
LLM+Lcap+Lcon
Modality-specific experts
Appendix
Table C13: Configurations used for the main component ablation.
Figure C7: Illustration of the tuning Stage adaptation architectures. The shared residual FFN in (a) is used in the component ablation, while (b) and (c) compare shared and modality-specific expert pools.
Model
Avg.
SU
IT
VC
TL
SRU
OC
TFU
AR
TFR
CR
General MLLMs
InternVL3-2B
65.6
91.7
82.1
61.9
56.3
44.7
42.1
55.7
70.0
68.2
83.1
InternVL3.5-2B
62.3
90.5
86.3
61.9
55.7
44.0
38.5
47.7
67.5
57.6
73.0
InternVL3.5-2B-Thinking
65.5
91.0
89.5
62.5
50.2
46.5
44.1
58.9
68.5
69.9
74.0
Qwen2.5-VL-7B
71.1
93.3
86.5
74.4
60.3
49.5
42.7
58.9
74.0
88.6
83.1
InternVL3-8B
71.2
95.0
87.8
71.6
66.4
51.9
45.0
57.7
79.0
70.8
87.1
Appendix
Table D14: Evaluation on IF-Bench. We report the overall average ( Avg. ) and performance across ten infrared understanding and reasoning categories.
Model
Exist.
Count.
Comp. Int.
Query Attr.
Query Obj.
Comp. Attr.
Overall
Cls. Avg.
3D-Specialized Models
ReferIt3D
85.4
26.6
57.5
38.3
11.3
58.5
41.8
46.2
BEV+MCAN
85.7
27.3
60.0
39.5
14.4
53.7
42.5
46.8
ScanQA
84.8
26.3
55.6
41.6
13.2
54.7
42.6
46.0
TransVQA3D
88.2
30.7
63.1
37.3
21.7
57.1
43.7
49.6
PointQA
85.9
29.0
60.3
39.9
23.2
57.5
43.9
49.3
Appendix
Table D15: Evaluation on CLEVR3D-REAL. We report accuracy (%) across six point-cloud compositional reasoning categories: existence ( Exist. ), counting ( Count. ), integer comparison ( Comp. Int. ), attribute querying ( Query Attr. ), object querying ( Query Obj. ), and attribute comparison ( Comp. Attr. ). Overall denotes accuracy over all test questions, while Cls. Avg. denotes the unweighted average across question categories.
Table D16: Effect of expert placement. We vary the Transformer layers equipped with modality-specific experts while keeping all other training settings unchanged. Early-half and Late-half denote placing experts in the first and second halves of the full-attention layers, respectively. Full-attn. denotes our default setting, where experts are attached only to full-attention layers.
Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.
Yoonhyung Park, Minji Kim, Sungwon Moon +1
Division of Artificial Intelligence & Software, Ewha Womans University
Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints and execution feedback. We introduce MachEmbodied-VLM (ME-VLM), a unified vision-language model with two variants, 4B and 35B-A3B, that brings together embodied cognition and multimodal agent capabilities. Our work emphasizes physical perception and spatiotemporal reasoning, together with planning, interaction, and outcome assessment in both digital and physical environments. We construct training data spanning embodied and multimodal agent tasks, including execution observations and feedback to support outcome assessment and decision refinement. The training pipeline comprises embodied capability injection, separate reinforcement learning of embodied and multimodal-agent experts, and multi-teacher on-policy distillation that consolidates their complementary capabilities into a single model. Experiments show competitive performance on both embodied and agent benchmarks, as well as on autonomous-driving and embodied-navigation tasks. For edge deployment, visual token compression, W4A8 quantization, and hardware-software co-optimization enable on-device inference of the 4B variant on the M100, reducing prefill latency from 400 ms to 188 ms. Project Page: https://machembodied.com/ME-Brain/ME-VLM.html Code Repository: https://github.com/MachEmbodied/ME-VLM
Despite rapid progress, multimodal large language models (MLLMs) still fail on tasks that humans solve effortlessly, such as navigating a grid maze from a screenshot or selecting the correct puzzle piece. Rather than retraining the model, we ask a complementary question: how much capability can be elicited from a frozen MLLM purely by improving the execution scaffold around it? We introduce MUSE, a multimodal unified structured execution harness that wraps any off-the-shelf MLLM with composable modules for task representation, visual processing, perception tool use, structured parsing, deterministic verification, and verifier-guided repair, without any model retraining. We evaluate MUSE across diverse benchmarks spanning visual spatial planning, visual perception, multimodal reasoning, and fine-grained visual discrimination, using multiple state-of-the-art MLLMs. MUSE delivers consistent gains over the bare model in all settings, with the largest jumps on challenging instances. Further analysis reveals that many MLLM failures arise from harness-level shortcomings rather than fundamental model deficits, and can be addressed through verifier-guided repair without touching the model. These findings highlight the agentic multimodal harness as a critical yet underexplored design dimension, offering an orthogonal avenue for improving MLLMs beyond model-centric optimization.