Multimodal Classification

Ask about this topic

Overview

Multimodal classification studies how to combine different kinds of evidence—for example, medical images with clinical records, or meme images with text—to predict categories. Common approaches include feature or decision fusion, cross-attention and contrastive alignment, mixture-of-experts models, and large vision-language models; benchmarks range from medical diagnosis and sentiment analysis to document, time-series, and content moderation tasks. Current work also addresses missing or unreliable inputs through modality dropout, imputation, adaptive routing, and selective acquisition, while evaluating fairness, calibration, and interpretability. A central tension is that adding a modality can improve predictions when it contributes complementary information, but can also hurt when it is noisy, redundant, or unavailable at deployment.

Latest papers

CardsList

200 papers24 papers in the last four weeks, down 4% on the four weeks before. 0.1% of all new papers.

  1. EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models

    Apr 13, 2026Jincheng Xie, Xingchen Xiao, Runheng Liu +3Cross-Modal LearningMultimodal Embedding

    School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JD iCity, JD Technology, JD Intelligent Cities Research, Beijing, China

  2. MONETA: Multimodal Industry Classification through Geographic Information with Multi Agent Systems

    Apr 9, 2026Arda Yüksel, Gabriel Thiem, Susanne Walter +3ClassificationMultimodal Classification

    Trustworthy Human Language Technologies · Technical University of Darmstadt, Germany · Deutsche Bundesbank +2

  3. Detection of Hate and Threat in Digital Forensics: A Case-Driven Multimodal Approach

    Apr 8, 2026Ponkoj Chandra ShillHate Speech DetectionDigital Forensics

    Computer Science and Engineering, University of Nevada, Reno, Reno, NV, USA

  4. USCNet: Transformer-Based Multimodal Fusion with Segmentation Guidance for Urolithiasis Classification

    Apr 8, 2026Changmiao Wang, Songqi Zhang, Yongquan Zhang +9Medical DiagnosisMedical Image Classification

    C. Wang and X. Wan are with Shenzhen Research Institute of Big Data, Shenzhen 518172, China · Y. Zhang, S. Zhang, and Y. Wang are with Zhejiang University of Finance and Economics, Hangzhou 310018, China · L. Liu is with Anhui University of Finance and Economics, Anhui 233000, China +4

  5. MultiPress: A Multi-Agent Framework for Interpretable Multimodal News Classification

    Apr 4, 2026Tailong Luo, Hao Li, Rong Fu +7Multimodal ClassificationMultimodal Agents

    New York Institute of Technology · University of Arizona · University of Macau +3

  6. Multimodal Machine Learning for Early Prediction of Metastasis in a Swedish Multi-Cancer Cohort

    Mar 31, 2026Franco Rugolon, Korbinian Randl, Braslav Jovanovic +2Clinical Outcome PredictionClinical Risk Prediction

    Department of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden

  7. SACRED: A Faithful Annotated Multimedia Multimodal Multilingual Dataset for Classifying Connectedness Types in Online Spirituality

    Mar 28, 2026Qinghao Guan, Yuchen Pan, Donghao Li +6Multimodal DatasetMultimodal Classification

    Department of Communication and Media Research, University of Zurich, Zurich, Switzerland · Department of Computational Linguistics, University of Zurich, Zurich, Switzerland · Department of Informatics, University of Zurich, Zurich, Switzerland +1

  8. Material Identification using Multi-Modal Intrinsic Radiation and Radiography

    Mar 27, 2026Khoa Nguyen, Brendt Wohlberg, Oleg Korobkin +1Multimodal ClassificationMultimodal Sensor Fusion

    Los Alamos National Laboratory, Los Alamos, NM 87545, USA

  9. Multimodal Deep Learning for Diabetic Foot Ulcer Staging Using Integrated RGB and Thermal Imaging

    Mar 27, 2026Gulengul Mermer, Mustafa Furkan Aksu, Gozde Ozsezer +3Multimodal ClassificationThermal Imaging

  10. Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning

    Mar 26, 2026Yuqiao Zeng, Xu Wang, Tengfei Liang +3Active LearningMultimodal Classification

    Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education; State Key Laboratory of Advanced Rail Autonomous Operation; School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK

  11. A Neuro-Symbolic System for Interpretable Multimodal Physiological Signals Integration in Human Fatigue Detection

    Mar 25, 2026Mohammadreza Jamalifard, Yaxiong Lei, Parasto Azizinezhad +2Multimodal Model InterpretabilityNeuro-Symbolic Reasoning

    School of Computer Science and Electronic Engineering, University of Essex, United Kingdom

  12. From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics

    Mar 24, 2026Paolo Cupini, Francesco PierriLLM-Assisted AnnotationMultimodal Large Language Models

    Politecnico di Milano

  13. Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deployment

    Mar 23, 2026Zigui Wang, Minghui Sun, Jiang Shu +3Multimodal LearningMultimodal Classification

    Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC, USA · Duke Center for Autism and Brain Development, Duke University School of Medicine, Durham, NC, USA · Department of Psychiatry and Behavioral Sciences, Durham, NC, USA +1

  14. Improving Generalization on Cybersecurity Tasks with Multi-Modal Contrastive Learning

    Mar 20, 2026Jianan Huang, Rodolfo V. Valentim, Luca Vassio +4Cross-Modal Representation TransferCybersecurity

    Politecnico di Torino, Italy · Universit`a di Torino, Italy · Huawei Paris Research Center, France

  15. CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data

    Mar 20, 2026Tianling Liu, Hongying Liu, Fanhua Shang +3Cross-Modal LearningMultimodal Classification

    College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Medical School of Tianjin University, Tianjin 300072, China. · Peng Cheng Lab, Shenzhen 518055, China. +3

  16. A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data

    Mar 20, 2026Aram Ansary Ogholbake, Hannah Choi, Spencer Brandenburg +10Medical Image ClassificationMultimodal Classification

  17. Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media

    Mar 19, 2026Thi Huyen Nguyen, Koustav Rudra, Wolfgang NejdlClassificationMultimodal Classification

    L3S Research Center, Leibniz University Hannover, Hannover, Germany · Indian Institute of Technology Kharagpur, Kharagpur, India

  18. Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

    Mar 18, 2026Ahmed Sharshar, Hosam Elgendy, Saad El Dine Ahmed +2LLM Safety BenchmarksLanguage Model Safety Evaluation

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria

  19. Early Intervention for VFM-based Multimodal Medical Image Classification

    Mar 18, 2026Qijie Wei, Hailan Lin, Xirong LiCross-Modal Representation LearningVision Foundation Model Adaptation

    Renmin University of China, Beijing 100872, China

  20. 3D MRI-Based Alzheimer's Disease Classification Using Multi-Modal 3D CNN with Leakage-Aware Subject-Level Evaluation

    Mar 18, 2026Md Sifat, Sania Akter, Akif Islam +5Alzheimer's Disease ClassificationMultimodal Classification

    Department of Computer Science and Engineering, University of Rajshahi, Rajshahi, Bangladesh · School of Computer Science and Engineering, University of Aizu, Aizuwakamatsu, Japan · Department of Computer Science and Engineering, Pabna University of Science and Technology, Pabna, Bangladesh

  21. Tabular LLMs for Interpretable Few-Shot Alzheimer's Disease Prediction with Multimodal Biomedical Data

    Mar 17, 2026Sophie Kearney, Shu Yang, Zixuan Wen +8Alzheimer's Disease ClassificationTabular Foundation Models

    University of Pennsylvania, Philadelphia, PA, USA · Stony Brook University, Stony Brook, NY, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA +2

  22. Data-Local Autonomous LLM-Guided Neural Architecture Search for Multiclass Multimodal Time-Series Classification

    Mar 16, 2026Emil Hardarson, Luka Biedebach, Ómar Bessi Ómarsson +3Large Language Model-Guided NASMultivariate Time Series Classification

    Reykjavik University, Reykjavik, Iceland · University of Southampton, Southampton, United Kingdom

  23. DamageArbiter: A Multimodal Arbitration Framework for Disaster Damage Assessment from Street-View Imagery

    Mar 16, 2026Yifan Yang, Lei Zou, Wenjing Gong +6Disaster Damage AssessmentMultimodal Classification

    Department of Geography, Texas A&M University, College Station, USA · Department of Landscape Architecture & Urban Planning, Texas A&M University, College Station, USA · Department of Industrial and Systems Engineering, University of Florida, Gainesville, USA +2

  24. Team LEYA in 10th ABAW Competition: Multimodal Ambivalence/Hesitancy Recognition Approach

    Mar 13, 2026Elena Ryumina, Alexandr Axyonov, Dmitry Sysoev +4Ambivalence and Hesitancy RecognitionMultimodal Classification

    St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia · HSE University, St. Petersburg, Russia

  25. GLEAM: A Multimodal Imaging Dataset and HAMM for Glaucoma Classification

    Mar 13, 2026Jiao Wang, Chi Liu, Yiying Zhang +8Glaucoma DetectionOphthalmic Image Analysis

    College of the Information Science and Engineering, Northeastern University, Shenyang 110819, China · Department of Ophthalmology, Shenyang Fourth People’s Hospital, Shenyang 110000, China

  26. Multimodal classification of Radiation-Induced Contrast Enhancements and tumor recurrence using deep learning

    Mar 12, 2026Robin Peretzke, Marlin Hanstein, Maximilian Fischer +15Multimodal ClassificationClassification

    Deutsches Krebsforschungszentrum (DKFZ), Division of Medical Image Computing, Heidelberg, Germany · Medical Faculty, Heidelberg University, Heidelberg, Germany · Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany +8

  27. Multimodal Deep Learning for Dynamic and Static Neuroimaging: Integrating MRI and fMRI for Alzheimer Disease Analysis

    Mar 9, 2026Anima Kujur, Zahra MonfaredAlzheimer's Disease ClassificationNeuroimaging

    Interdisciplinary Centre for Scientific Computing (IWR), Department of Mathematics and Computer Science, Heidelberg University, Germany

  28. Multimodal Large Language Models as Image Classifiers

    Mar 6, 2026Nikita Kisel, Illia Volkov, Klara Janouskova +1Multimodal Large Language ModelsMultimodal Classification

    Visual Recognition Group, Czech Technical University in Prague

  29. XMACNet: An Explainable Lightweight Attention based CNN with Multi Modal Fusion for Chili Disease Classification

    Mar 6, 2026Tapon Kumer Ray, Rajkumar Y, Shalini R +3Plant Disease ClassificationLightweight CNNs

    Student, Vellore Institute of Technology, Amaravati, Andhra Pradesh, India. · Assistant Professor, Vellore Institute of Technology, Amaravati, Andhra Pradesh, India. · Student, Angel College of Engineering and Technology, Tiruppur, Tamilnadu, India +2

  30. An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classification

    Mar 3, 2026L. Julián Lechuga López, Farah E. Shamout, Tim G. J. RudnerSelective PredictionUncertainty Calibration

    New York University · New York University Abu Dhabi · University of Toronto