Overview

The field studies how AI can combine evidence from different kinds of input to answer questions, recognize events, retrieve information, and reason about scenes over time. Common approaches include vision-language models and multimodal large language models, trained with contrastive alignment, cross-attention, instruction tuning, and reinforcement learning; some systems add modality-specific experts, retrieval, or tools. Benchmarks test image and video question answering, document and chart understanding, audio-visual reasoning, grounding, and performance across languages and domains. A central challenge is ensuring that answers genuinely depend on relevant evidence from each modality, while handling long inputs efficiently and making capabilities measurable with reliable tests.

Latest papers

CardsList

489 papers13 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

  1. Leveraging Dual-Process Theory for Multimodal Multiparty Turn-Taking Prediction

    Nov 14, 2026Rutherford A. Patamia, Ming Liu, Wei Luo +4Turn-Taking PredictionMultimodal Understanding

  2. UniMore: Fine-Grained Supervision for Unified Understanding and Generation

    Nov 6, 2026Haoyue Bai, Yinyi Luo, Yicheng He +3Unified Multimodal ModelsMultimodal Learning

  3. Prism: Bidirectional Self-Teaching for Unified Multimodal Models

    Nov 4, 2026Peng Zhang, Qian Chen, Xianyin Zhang +6Unified Multimodal ModelsMultimodal Generation

  4. S2TDR: Structured Multimodal Reconstruction for Sticker-Aware Dialogue Understanding

    Oct 30, 2026Yuanchen Shi, Fang KongMultimodal UnderstandingMultimodal Generation

  5. Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception

    Oct 27, 2026Lai Wei, Yatai Li, Liangbo He +10Multimodal UnderstandingMultimodal Large Language Models

  6. JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces

    Oct 20, 2026Jiahan Li, Huangyuan Su, Shengbang Tong +9Unified Multimodal ModelsMultimodal Generation

  7. SenseAlign: Understanding Vision and Touch through Language at Test Time

    Oct 20, 2026Chuyang Ye, Qinting Jiang, Jingyan JiangMultimodal UnderstandingCross-Modal Alignment

  8. Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

    Oct 19, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5Unified Multimodal ModelsMultimodal Large Language Models

  9. VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

    Oct 16, 2026Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza +11Assistive Technology for Blind and Low-Vision PeopleVisually Impaired Navigation Assistance

  10. What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

    Oct 14, 2026Tengju Ru, Weitong Lian, Zecong Tang +6Multimodal UnderstandingUnified Multimodal Models

  11. GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

    Oct 10, 2026Zonghuan Li, Hayley Hung, Chirag RamanMultimodal Model EvaluationMultimodal Understanding

  12. UkrMeme: Multimodal Explainable Persuasion Detection in Ukrainian Memes

    Oct 9, 2026Nazar Andrushko, Vera Schmitt, Veronika SolopovaMultimodal Meme ClassificationVLM Evaluation

  13. Addressing Overcommitment in the Reasoning of Gendered Economic Memes under Multimodal Ambiguity

    Oct 8, 2026Kushal Kanwar, Dushyant Singh Chauhan, Kapil Rana +2Multimodal Meme ClassificationMultimodal Understanding

  14. Gestalt: Large Multimodal Interplay Model

    Sep 30, 2026Zequn Yang, Yu Miao, Haotian Ni +8Unified Multimodal ModelsCross-Modal Alignment

  15. Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

    Sep 29, 2026Shuyang Jiang, Fucheng Deng, Yuchuan Luo +1Unified Multimodal ModelsGradient Interference

  16. Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models

    Sep 28, 2026Wentao Zhou, Weijie Gan, Jiayun WangAdversarial TrainingUnified Multimodal Models

  17. Refinement Symmetry in Multimodal Transformers

    Sep 26, 2026Yuhao Du, Shunian ChenAttention MechanismsMultimodal Robustness

  18. Multimodal Voice Activity Projection for Social Robot Mediation: Expected Behavior and Deployment Constraints

    Sep 23, 2026Antonio Cano, Guillermo Perez, Luis Merino +1Turn-Taking PredictionHuman-Robot Interaction

  19. Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction

    Sep 18, 2026Qi Chen, Yunfei Chu, Haolin He +15Multimodal Intent RecognitionMultimodal Model Evaluation

  20. Efficient Unified Multimodal Understanding (EUMU): Winning Solution for the MUMU Track at the 8th LSVOS Challenge

    Sep 16, 2026Dayoung Kil, Seong-heum KimUnified Multimodal ModelsEfficient Multimodal Inference

  21. Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue

    Sep 14, 2026Leon Hammerla, Patrick Schrottenbacher, Alexander MehlerMultimodal Understanding