Multimodal Sensor Fusion

Ask about this topic

Overview

The field studies how to combine complementary measurements—such as camera and LiDAR, radar and vision, or wearable motion and physiological signals—when no single sensor provides a complete or reliable picture. Common approaches include probabilistic filters and factor graphs for state estimation, feature- or decision-level fusion, and attention-based networks that learn cross-modal relationships; alignment and uncertainty modeling help account for mismatched timing, geometry, or sensor quality. Methods are evaluated in settings such as autonomous driving on nuScenes and KITTI, robotic manipulation, wearable activity recognition, and satellite image analysis. A central challenge is making fusion robust to missing, degraded, or misaligned sensors: adding modalities can help, but can also introduce noise, negative transfer, extra computation, and poor performance when conditions differ from training.

Latest papers

CardsList

418 papers37 papers in the last four weeks, down 16% on the four weeks before. 0.2% of all new papers.

  1. AW-MoE: All-Weather Mixture of Experts for Robust Multi-Modal 3D Object Detection

    Mar 17, 2026Hongwei Lin, Xun Huang, Chenglu Wen +13D Object DetectionAutonomous Driving

    Fujian Key Laboratory of Urban Intelligent Sensing and Computing and the School of Informatics, Xiamen University, Xiamen, FJ 361005, China · Zhongguancun Academy, Beijing, China

  2. Ground Reaction Inertial Poser: Physics-based Human Motion Capture from Sparse IMUs and Insole Pressure Sensors

    Mar 17, 2026Ryosuke Hori, Jyun-Ting Song, Zhengyi Luo +43D Human Motion Reconstruction3D Human Pose Estimation

    Carnegie Mellon University · Keio University · Keio AI Research Center

  3. ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation

    Mar 16, 2026Aditya Iyer, Jack Roberts, Nora AyanianMulti-Object TrackingUncertainty-Aware Multimodal Fusion

    Department of Computer Science, Brown University, Providence, RI, USA · School of Engineering, Brown University, Providence, RI, USA

  4. Context-Aware Sensor Modeling for Asynchronous Multi-Sensor Tracking in Stone Soup

    Mar 16, 2026Martin Vonheim Larsen, Kim MathiassenRadar-LiDAR FusionMulti-Object Tracking

    Norwegian Defence Research Establishment (FFI) · University of Oslo, Department of Technology Systems

  5. GraspALL: Adaptive Structural Compensation from Illumination Variation for Robotic Garment Grasping in Any Low-Light Conditions

    Mar 16, 2026Haifeng Zhong, Wenshuo Han, Zhouyu Wang +10Robotic GraspingMultimodal Sensor Fusion

    School of Artificial Intelligence, Jilin University · Chinese Academy of Sciences · National Cheng-Kung University +3

  6. Towards Versatile Opti-Acoustic Sensor Fusion and Volumetric Mapping

    Mar 15, 2026Ivana Collado-Gonzalez, John McConnell, Brendan EnglotUnderwater 3D ReconstructionUnderwater Robotics

    I. Collado-Gonzalez, and B. Englot are with Stevens Inst. of Technology, Hoboken, NJ, USA · J. McConnell is with the U.S. Naval Academy, Annapolis, MD, USA

  7. Sonar-MASt3R: Real-Time Opti-Acoustic Fusion in Turbid, Unstructured Environments

    Mar 13, 2026Amy Phung, Richard CamilliUnderwater 3D Reconstruction3D Reconstruction

    Applied Ocean Physics and Engineering, Woods Hole Oceanographic Institution, Deep Submergence Laboratory, Woods Hole, MA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA

  8. Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robots

    Mar 13, 2026Guoqiang Zhao, Zhe Yang, Sheng Wu +53D Occupancy PredictionRobotic Perception

    The authors are with the School of Artificial Intelligence and Robotics, Hunan University, Changsha 410012, China · The author is also with the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China.

  9. GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space

    Mar 13, 2026Wentao Wang, Haoran Xu, Guang TanHeterogeneous Collaborative PerceptionAutonomous Driving Perception

    Sun Yat-sen University, Shenzhen Campus, Shenzhen, China · Peng Cheng Laboratory (PCL), Shenzhen, China

  10. AoI-FusionNet: Age-Aware Tightly Coupled Fusion of UWB-IMU under Sparse Ranging Conditions

    Mar 13, 2026Tehmina Bibi, Anselm Köhler, Jan-Thomas Fischer +1GNSS-Denied LocalizationMultimodal Sensor Fusion

    TU Berlin, Germany · School of Electrical Engineering and Computer Science, TU Berlin, Germany · Austrian Research Centre for Forests, Austria

  11. HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI

    Mar 12, 2026Chengwen Zhang, Chun Yu, Borong Zhuang +9Human-Robot InteractionMultimodal Sensor Fusion

    Department of Computer Science and Technology, BNRist Tsinghua University Beijing, China · Department of Computer Science and Technology, BNRist, College of AI Tsinghua University Beijing, China · Key Laboratory of Pervasive Computing, Ministry of Education +4

  12. A Hybrid Neural-Assisted Unscented Kalman Filter for Unmanned Ground Vehicle Navigation

    Mar 12, 2026Gal Versano, Itzik KleinRobot NavigationUnscented Kalman Filtering

    Autonomous Navigation and Sensor Fusion Lab, Hatter Department of Marine Technologies, Charney School of Marine Sciences, University of Haifa, Israel.

  13. R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

    Mar 12, 2026Zhongyu Xia, Yousen Tang, Yongtao Wang +23D Object DetectionCamera-Radar Fusion

    Wangxuan Institute of Computer Technology, Peking University · EBTech Co. Ltd

  14. DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding

    Mar 11, 2026Mingzhe Tao, Ruiping Liu, Junwei Zheng +6VLMs for Autonomous DrivingMultimodal Large Language Models

    Karlsruhe Institute of Technology · Mercedes-Benz Tech Innovation · Hunan University

  15. SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG

    Mar 10, 2026Fredrik K. Gustafsson, Xiao Gu, Mattia Carletti +3Electrocardiogram Foundation ModelsMultimodal Model Evaluation

    Department of Engineering Science, University of Oxford · NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance · Big Data Institute, Nuffield Department of Population Health, University of Oxford +2

  16. OTPL-VIO: Robust Visual-Inertial Odometry with Optimal Transport Line Association and Adaptive Uncertainty

    Mar 10, 2026Zikun Chen, Wentao Zhao, Yihe Niu +2Visual-Inertial OdometryStereo Vision

    School of Automation and Intelligent Sensing, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai 200240, China. · School of Mathematical Sciences Shanghai Jiao Tong University.

  17. SiMO: Single-Modality-Operable Multimodal Collaborative Perception

    Mar 9, 2026Jiageng Wen, Shengjie Zhao, Bing Li +3Cooperative PerceptionMultimodal Sensor Fusion

    Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · School of Computer Science and Technology, Tongji University · School of Mechatronic Engineering and Automation, Shanghai University

  18. Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors

    Mar 9, 2026Ishrat Jahan, Molla E Majid, M Murugappan +5UAV DetectionMultimodal Sensor Fusion

    Department of Computer Science and Engineering, Shahjalal University of Science & Technology, Sylhet, Bangladesh · Computer Applications Department, Academic Bridge Program, Qatar Foundation, Doha, Qatar. · Intelligent Signal Processing (ISP) Research Lab, Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha, Kuwait +5

  19. Fusion-Poly: A Polyhedral Framework Based on Spatial-Temporal Fusion for 3D Multi-Object Tracking

    Mar 9, 2026Xian Wu, Yitao Wu, Xiaoyu Li +33D MOTCamera-LiDAR Fusion

    State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150006, China.

  20. Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

    Mar 9, 2026Jiaming Zhang, Mingxu Liu, Hongchao Shu +6Augmented RealitySurgical Instrument Tracking

    Johns Hopkins University · Johns Hopkins University, School of Medicine · University of Arkansas, Johns Hopkins University

  21. Toward Multimodal Industrial Fault Analysis: A Single-Speed Chain Conveyor Dataset with Audio and Vibration Signals

    Mar 7, 2026Zhang Chen, Yucong Zhang, Xiaoxiao Miao +1Industrial Fault DiagnosisFault Detection

    Digital Innovation Research Center, Duke Kunshan University, Kunshan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China +1

  22. AI-Driven Predictive Maintenance with Environmental Context Integration for Connected Vehicles: Simulation, Benchmarking, and Field Validation

    Mar 7, 2026Kushal Khemani, Anjum Nazir QureshiPredictive MaintenanceMultimodal Sensor Fusion

    Independent Researcher, India · Rajiv Gandhi College of Engineering Research and Technology

  23. SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics

    Mar 6, 2026Haoying Zhou, Chang Liu, Yimeng Wu +7Robotic Data CollectionSurgical Robotics

    Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, USA. · Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA. +2

  24. Bi Directional Feedback Fusion for Activity Aware Forecasting of Indoor CO2 and PM2.5

    Mar 6, 2026Harshala Gammulle, Lidia Morawska, Sridha Sridharan +1Air Quality PredictionMultivariate Time Series Forecasting

    Signal Processing, Artificial Intelligence and Vision Technologies (SAIVT) Lab, Queensland University of Technology, Brisbane, Australia. · International Laboratory for Air Quality and Health (ILAQH), Queensland University of Technology, Brisbane, Australia.

  25. TAPFormer: Robust Arbitrary Point Tracking via Transient Asynchronous Fusion of Frames and Events

    Mar 5, 2026Jiaxiong Liu, Zhen Tan, Jinpu Zhang +4Point TrackingMultimodal Sensor Fusion

    National University of Defense Technology · Hunan University

  26. Recognition of Daily Activities through Multi-Modal Deep Learning: A Video, Pose, and Object-Aware Approach for Ambient Assisted Living

    Mar 4, 2026Kooshan Hashemifard, Pau Climent-Pérez, Francisco Florez-RevueltaHuman Activity RecognitionMultimodal Sensor Fusion

    Research Group on Ambient Intelligence for Active and Healthy Ageing, Department of Computing Technology, University of Alicante, Ctra. San Vicente del Raspeig, s/n, 03690, San Vicente del Raspeig, Spain. · Alicante Institute for Health and Biomedical Research (ISABIAL), Avda. Pintor Baeza, 12, 03010, Alicante, Spain. · ValgrAI - Valencian Graduate School and Research Network of Artificial Intelligence, Camí de Vera s/n, 46022, Valencia, Spain.

  27. Real-time loosely coupled GNSS and IMU integration via Factor Graph Optimization

    Mar 3, 2026Radu-Andrei Cioaca, Cristian Rusu, Paul Irofti +3Factor Graph OptimizationMultimodal Sensor Fusion

    Three Tensors S.R.L., Romania · Navigation Systems Definition Section (TEC-SEN), European Space Agency · Romanian InSpace Engineering S.R.L. (RISE), Romania

  28. EIMC: Efficient Instance-aware Multi-modal Collaborative Perception

    Mar 3, 2026Kang Yang, Peng Wang, Lantao Li +4Cooperative PerceptionCommunication-Efficient Collaborative Perception

    School of Information, Renmin University of China, Beijing, China, 100872 · Sony Research and Development Center China, Beijing, China · National University of Defense Technology, Hunan, China, 410073

  29. ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop

    Mar 3, 2026Shuangzhi Li, Lei Ma, Xingyu Li3D Object DetectionMissing-Modality Learning

    University of Alberta, Canada · The University of Tokyo, Japan

  30. Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining

    Mar 2, 2026Yuxuan Li, Yuming Chen, Yunheng Li +3Remote Sensing Image UnderstandingMultimodal Pretraining

    PCA Lab & VCIP, CS, Nankai University. · NKIARI, Shenzhen Futian.