Multimodal Foundation Models

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 303

All topics

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Jun 23, 2026Lianghua Huang, Zhi-Fan Wu, Wei Wang +22Audio-Video Generation Unified Multimodal Models
Wan Team, Alibaba Group
Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning
Jun 23, 2026Julien Khlaut, Charles Corbière, Baptiste Callard +93D Foundation Models Medical Imaging Foundation Models
Raidium, Paris, 75014, France · Department of Vascular and Oncological Interventional Radiology,2026 Hôpital Européen Georges Pompidou, AP-HP, Paris, France · Faculté de Santé, Université Paris-Cité, Paris, France +3
S1-Omni-Image: A Unified Model for Scientific Image Understanding, Generation, and Editing
Jun 23, 2026Qingxiao Li, Zikai Wang, Qingli Wang +1AI for Science Unified Multimodal Models
XScience Lab, Wenge AI
Latent Visual States for Efficient Multimodal Reasoning
Jun 23, 2026Xiuwei Chen, Wentao Hu, Yongxin Wang +8Efficient Multimodal Inference Large Vision-Language Models
Sun Yat-sen University · The Hong Kong Polytechnic University · MBZUAI +2
Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Jun 22, 2026Davide Caffagni, Alberto Compagnoni, Federico Melis +5Cross-Modal Alignment Multimodal Large Language Models
University of Modena and Reggio Emilia · University of Pisa · AMD Silo AI
Foundation Models for Epileptogenic Zone Identification in Drug-Resistant Epilepsy
Jun 21, 2026Thi Kieu Khanh Ho, Thomas Lai, Petr Klimes +5Electroencephalography Multimodal Foundation Models
Department of Electrical and Computer Engineering, McGill University, Montr´eal, Quebec, Canada. · Mila-Quebec AI Institute, Montr´eal, Quebec, Canada. · Montreal Neurological Institute and Hospital, McGill University, Montr´eal, Quebec, Canada. +5
MMGist: A Comprehensive Multimodal Benchmark for 2027
Jun 21, 2026Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong +3VLM Evaluation Large Vision-Language Models
1Shanghai Jiao Tong University · 2Sichuan University
BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
Jun 20, 2026Qizhi Pei, Zhimeng Zhou, Yi Duan +9Unified Multimodal Models Multimodal Pretraining
1Gaoling School of Artificial Intelligence, Renmin University of China, 2OpenDataLab, Shanghai Artificial Intelligence Laboratory · 3Zhejiang University, 4Shanghai Innovation Institute · 2OpenDataLab, Shanghai Artificial Intelligence Laboratory, 3Zhejiang University, 4Shanghai Innovation Institute +5
Bridging the Age Gap: Towards Detecting Neural Audio Codec Synthesized Elderly Speech Deepfake
Jun 19, 2026Orchid Chetia Phukan, Girish, Mohd Mujtaba Akhtar +1Audio Deepfake Detection Deepfake Detection
BIIC Lab, NTHU, Taiwan
Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models
Jun 18, 2026Yifeng Wu, Huimin Huang, Ruiluo Wu +5Visual Spatial Reasoning Multimodal Large Language Models
Fudan University · Shenzhen University of Advanced Technology · Tencent Jarvis Lab +1
3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
Jun 18, 2026Jintang Xue, Xinyu Wang, Yixing Wu +23D Representation Learning Multimodal Foundation Models
University of Southern California Los Angeles, California, USA · Ohio State University Columbus, Ohio, USA
SAFE-Cascade: Cost-Adaptive Vision-Language Routing for Chart Question Answering
Jun 17, 2026Ayush Dwivedi, Qixin Wang, Ashvi Soni +5Visual Question Answering Efficient VLM Inference
Walmart USA · University of Arkansas USA
Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games
Jun 17, 2026Shengyuan Ding, Xilin Wei, Xinyu Fang +4Multimodal Memory Sequential Decision Making
1Fudan University · 2Shanghai Innovation Institute · 3Shanghai Artificial Intelligence Laboratory +2
Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning
Jun 17, 2026Pengyu Li, Zhitao Gao, Lingling Zhang +5Cross-Modal Knowledge Distillation Efficient Multimodal Inference
Xi’an Jiaotong University · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · MOE KLINNS Lab, Xi’an Jiaotong University +1
Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification
Jun 16, 2026Wujian Peng, Lingchen Meng, Yuxuan Cai +7Unified Multimodal Models Autoregressive Image Generation
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Qwen Team, Alibaba Inc.
MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias
Jun 16, 2026Xingming Li, Ao Cheng, Qiyao Sun +4VLM Robustness Vision-Language Models
National University of Defense Technology, changsha, China · 2Chinese University of Hong Kong, Shenzhen, China · 3Intelligent Game and Decision Lab, Beijing, China
Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
Jun 16, 2026Haoqi Yuan, Zhixuan Liang, Anzhe Chen +20Robot Skill Learning Robot Foundation Models
Qwen Team
Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations
Jun 15, 2026Zhiqiang Zhou, Junliang Dai, Xu lingVLM Interpretability Multimodal CoT Reasoning
Hunan Chemical Industry Vocational and Technical College, Zhuzhou, Hunan 412000, China
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
Jun 15, 2026Yuanteng Chen, Peisong Wang, Zhilei Liu +9Mixed-Precision Quantization LLM Quantization
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Zhongguancun Academy
Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis
Jun 15, 2026Jingyu Hu, Giuseppe Tripodi, Reed Naidoo +2Computational Pathology Multimodal Robustness
The Alan Turing Institute, London, United Kingdom · University of Manchester, Manchester, United Kingdom · The Institute of Cancer Research, London, United Kingdom +1
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
Jun 15, 2026Shuai Wang, Liang Li, Yang Chen +3Diffusion Transformer Unified Multimodal Models
Nanjing University · ByteDance Seed · University of Hong Kong
VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
Jun 15, 2026Young Rok Jang, Hyesoo Kong, Kyunghwan An +3VLM Evaluation Multimodal Large Language Models
LG AI Research
OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing
Jun 14, 2026Zebang Cheng, Shuimu Chen, Boxue Yang +7On-Policy Self-Distillation Multimodal Large Language Models
Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University +6
Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades
Jun 13, 2026Zhongye Liu, Yaopei Zeng, Yurui Chang +1Adversarial Attacks on VLMs Multimodal Robustness
Pennsylvania State University
Gaze Heads: How VLMs Look at What They Describe
Jun 12, 2026Rohit Gandikota, David BauImage Captioning Visual Attention
Northeastern University
OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models
Jun 11, 2026Ibrahim Gulluk, Max Van Puyvelde, Olivier GevaertMedical VQA Medical Image Classification
Department of Electrical Engineering, Stanford University · Department of Biomedical Data Science, Stanford University School of Medicine · Department of Mathematical Modelling, Statistics and Bioinformatics, Ghent University
GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models
Jun 10, 2026Garvita Allabadi, Matteo Sodano, Roberto Estevão +4Multimodal ICL Multimodal Large Language Models
University of Illinois Urbana Champaign · University of Bonn · Microsoft
Emerging Flexible Designs for Geospatial Multimodal Foundation Models
Jun 10, 2026Philipe Dias, Waqwoya Abebe, Abhishek Potnis +4Geospatial Foundation Models Multimodal Foundation Models
Oak Ridge National Laboratory USA
InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning
Jun 10, 2026Ziang Yan, Sheng Xia, Jiashuo Yu +10Efficient Multimodal Inference Multimodal Reasoning
Shanghai AI Laboratory · Shanghai Innovation Institute · Nanjing University
Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training
Jun 10, 2026Michal Chudoba, Sergey Alyaev, Petra Galuscakova +1LLM Fine-Tuning Multimodal Large Language Models
University of Stavanger · Stavanger, Norway · NORCE Research +1

Jun 23, 2026, cs.CVJ/K move · Enter open · S save

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

Lianghua Huang, Zhi-Fan Wu, Wei Wang, Yupeng Shi+21

Wan Team, Alibaba Group

Audio-Video Generation Unified Multimodal Models Multimodal Foundation Models Multimodal Agents

Open paper AskPDF arXiv

We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.

Momentum

Related topics

Latest papers 303