Multimodal Semantic Segmentation

Follow this topic Ask about this topic

Latest papers

34 papers9 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation
Nov 6, 2026Junbo Zhang, Hang Su, Zhaofan Li +2Audio-Visual Understanding Multimodal Semantic Segmentation
School of Computer Science, Wuhan University · Academy of Advanced Interdisciplinary Studies, Wuhan University
Quantum-Inspired Convolutional Networks for High-Resolution Pseudo-Label Generation from Multimodal Earth Observation Data
Oct 29, 2026Abhishek Singh, Giovanni Cuozzo, Alexander Jacob +1Pseudo-Labeling Weakly Supervised Semantic Segmentation
Institute for Earth Observation, Eurac Research, Bolzano, Italy
DriveOSS: Dataset and Benchmark for 3D Anomaly Segmentation and Open-set Semantic Segmentation in Autonomous Driving
Oct 23, 2026Zhiran Yan, Savankumar Bhanderi, Sourabh Lolge +4Autonomous Driving Datasets Autonomous Driving Benchmarks
Technische Hochschule Ingolstadt · UCLA
OpenEarth-OVS44: A Multimodal Remote Sensing Dataset for Open-Vocabulary Segmentation
Oct 21, 2026Yimin Wei, Hongruixuan Chen, Hao Zheng +5Open-Vocabulary Remote Sensing Image Segmentation Remote Sensing Image Segmentation
The University of Tokyo · RIKEN AIP · East China Normal University +1
RAHF-Net: Range-Aware Heterogeneous Fusion for LiDAR-Assisted RGB Semantic Segmentation
Oct 18, 2026R. D. Lin, Pengcheng Weng, Ge Wang +2Camera-LiDAR Fusion Multimodal Semantic Segmentation
School of Software Engineering, Xi’an Jiaotong University, China · School of Computer Science and Technology, Xi’an Jiaotong University, China
FloodVLM-Pix: Language-Guided Pixel-Level Flood Understanding
Oct 16, 2026Chengyang Zhang, Zhihao Wang, Zhiyu Long +4Remote Sensing VLMs Text-Guided Image Segmentation
Tsinghua University · National Disaster Reduction Center
Xiaomi-Lens-0 Technical Report
Oct 16, 2026Pei Fu, Yike Liu, Yue Sun +13Large Vision-Language Models Referring Image Segmentation
Xiaoai Plus, SeerRay Team
FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities
Oct 15, 2026Xin Zhang, Robby T. TanMultimodal Semantic Segmentation Multimodal Fusion
National University of Singapore
Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation
Oct 13, 2026Hu Cao, Yifeng Cheng, Mu He +3Multimodal Semantic Segmentation Multimodal Fusion
School of Automation, Southeast University, Sipailou 2, Nanjing, 210096, China. · Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Boltzmannstraße 15, Munich, 85748, Germany.
MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency
Oct 7, 2026Markus Gross, Andreas Greiner, Taehyoung Kim +11Multimodal Semantic Segmentation Remote Sensing Benchmarks
Autonomous Aerial Systems, Fraunhofer Institute IVI · Computer Vision Group, Technical University of Munich · Munich Center for Machine Learning (MCML) +2
DTFormer: Text-Guided Semantic Alignment for RGB-D Segmentation
Oct 4, 2026Ziang Wei, Yinlong Liu, Yan Xia +2Semantic Segmentation Multimodal Semantic Segmentation
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Germany · City University of Macau, China · University of Science and Technology of China, China +1
Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
Aug 3, 2026Jiazhen Liu, Mingkuan Feng, Long ChenInstance Segmentation Image Segmentation
The Hong Kong University of Science and Technology (HKUST), Hong Kong.
UMSS: Towards Unsupervised Multi-modal Semantic Segmentation
Jul 14, 2026Haitian Zhang, Thai Duy Nguyen, Xiangyuan Wang +2Cross-Modal Representation Learning Multimodal Semantic Segmentation
EmPACT Lab, School of EEE, Nanyang Technological University, Singapore · The University of Hong Kong, Hong Kong SAR, China
DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance
Jun 8, 2026Qiangqiang Zhou, Jiawei Xu, Yong Chen +3Image Segmentation Multimodal Semantic Segmentation
School of Artificial Intelligence, Jiangxi Normal University, Nanchang, China · Institute of AI Education, East China Normal University, Shanghai, China · Yale School of Medicine, Yale University, New Haven, CT, USA
Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities
May 19, 2026Irem Ulku, Ö. Özgür Tanrıöver, Erdem AkagündüzMissing-Modality Learning Remote Sensing Image Segmentation
Department of Computer Engineering, Ankara University, Ankara, Türkiye · Department of Modeling and Simulation, Graduate School of Informatics, METU, Ankara, Türkiye
MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation
May 11, 2026Ziyi Wang, Xianping Ma, Ziyao Wang +2Remote Sensing Image Understanding Remote Sensing Image Captioning
The Chinese University of Hong Kong (Shenzhen) · Southwest Jiaotong University
UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition
May 8, 2026Shuai Zhang, Zhecheng Shi, Zhuxiao Li +4Disentangled Representation Learning Multimodal Semantic Segmentation
1HKUST(GZ) · 2HKUST
Open-Vocabulary Semantic Segmentation Network Integrating Object-Level Label and Scene-Level Semantic Features for Multimodal Remote Sensing Images
Apr 27, 2026Jinkun Dai, Yuanxin Ye, Peng Tang +4Remote Sensing Image Segmentation Text-Guided Image Segmentation
Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · State-Province Joint Engineering Laboratory of Spatial Information Technology for High-Speed Railway Safety, Southwest Jiaotong University, Chengdu 611756, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China +1
Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Apr 17, 2026Irem Ulku, Erdem Akagündüz, Ömer Özgür TanrıöverMissing-Modality Learning Remote Sensing Image Segmentation
Department of Computer Engineering, Ankara University, Ankara, Türkiye · Department of Modeling and Simulation, Graduate School of Informatics, METU, Ankara, Türkiye
RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation
Apr 14, 2026Guoan Xu, Yang Xiao, Guangwei Gao +3Multimodal Semantic Segmentation Semantic Segmentation
Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW 2007, Australia · PCA Lab, Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Bionic Vision Systems Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200050, China +2
EDFNet: Early Fusion of Edge and Depth for Thin-Obstacle Segmentation in UAV Navigation
Apr 6, 2026Negar FathiThin-Structure Segmentation Multimodal Semantic Segmentation
University of Nebraska–Lincoln, Lincoln, NE, USA
Moondream Segmentation: From Words to Masks
Apr 3, 2026Ethan ReidReferring Image Segmentation Image Segmentation
M87 Labs, San Francisco, CA, USA.
SegRGB-X: General RGB-X Semantic Segmentation Model
Mar 30, 2026Jiong Liu, Yingjie Xu, Xingcheng Zhou +4Multimodal Semantic Segmentation Semantic Segmentation
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Munich, Germany
Weakly supervised multimodal segmentation of acoustic borehole images with depth-aware cross-attention
Mar 21, 2026Jose Luis Lima de Jesus SilvaMultimodal Semantic Segmentation Image Segmentation
Federal University of Bahia, Institute of Geosciences, Department of Geophysics, Salvador, BA 40170-115, Brazil · Grupo de Estudos e Aplicac¸ ˜ao de Inteligˆencia Artificial em Geof´ısica (GAIA), Federal University of Bahia, Salvador, BA 40170-115, Brazil
MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing
Mar 18, 2026Yimin Wei, Aoran Xiao, Hongruixuan Chen +2Remote Sensing Image Segmentation Open-Vocabulary Semantic Segmentation
The University of Tokyo · RIKEN AIP
DualSwinFusionSeg: Multimodal Martian Landslide Segmentation via Dual Swin Transformer with Multi-Scale Fusion and UNet++
Mar 14, 2026Shahriar Kabir, Abdullah Muhammed Amimul Ehsan, Istiak Ahmmed Rifti +1Multimodal Semantic Segmentation Image Segmentation
Bangladesh University of Engineering and Technology, Bangladesh · University of California, Riverside, USA
Bidirectional Cross-Attention Fusion of High-Resolution RGB and Low-Resolution Hyperspectral Inputs for Multimodal Semantic Segmentation
Mar 14, 2026Jonas V. Funk, Lukas Roming, Andreas Michel +4Multimodal Semantic Segmentation Semantic Segmentation
KIT, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany · Fraunhofer IOSB, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation, 76131 Karlsruhe, Germany
SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data
Mar 3, 2026Lekang Wen, Liang Liao, Jing Xiao +1Multimodal Semantic Segmentation Remote Sensing Image Segmentation
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University · Hangzhou Institute of Technology, Xidian University · School of Artificial Intelligence, Wuhan University
Enhancing 3D LiDAR Segmentation by Shaping Dense and Accurate 2D Semantic Predictions
Feb 21, 2026Xiaoyu Dong, Tiankui Xian, Wanshui Gan +1Multimodal Semantic Segmentation 3D Semantic Segmentation
The University of Tokyo, Tokyo, Japan · RIKEN AIP, Tokyo, Japan
Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation
Sep 23, 2025Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual Understanding Multimodal Semantic Segmentation
Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Nov 6, 2026J/K move · Enter open · S save

Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Junbo Zhang, Hang Su, Zhaofan Li, Hang Dong+1

School of Computer Science, Wuhan University · Academy of Advanced Interdisciplinary Studies, Wuhan University

Audio-Visual Understanding Multimodal Semantic Segmentation Prototype Learning Audio-Visual Alignment

Open paper AskPDF arXiv

Audio-Visual Segmentation (AVS) aims to accurately segment sounding objects in video frames by leveraging audio-visual correspondence cues. However, it remains challenging due to the intrinsic semantic incompleteness within a single modality and the semantic gap between audio and visual representations. Traditional feature-fusion decoding approaches struggle to suppress fusion noise effectively, while recent methods that incorporate data-dependent priors often increase the complexity of modeling audio-visual correlations, leading to poor cross-domain generalization. To address these issues, we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS. Specifically, we design a Self-Supervised Audio Enhancement (SSAE) module that introduces contrastive learning to adaptively align audio representations with gradient-blocked visual embeddings, thus narrowing the semantic gap between modalities. Furthermore, a Dynamic Prototype Constraint (DPC) module is developed to refine pixel-wise category perception via momentum-based prototype updating, while enhancing localization of sounding regions through cross-modal interaction. Extensive experiments demonstrate that our method achieves SOTA performance across two AVS benchmarks and six sub-tasks, exhibiting strong robustness and generalization ability.