Multimodal Representations

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 44

All topics

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Multimodal Pretraining Multimodal Representations
Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics
Spherical Interpolation for Backward-Compatible Multimodal Representations
Sep 30, 2026Simone Ricci, Niccolò Biondi, Federico PerniciMultimodal Representations Cross-Modal
DINFO (Department of Information Engineering), University of Florence, Italy · MICC (Media Integration and Communication Center) · University of Trento, Italy
Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback
Sep 30, 2026Ho-min Park, Byungkon KangMultimodal Representations Multimodal Classification
Data Science Center, Texas Children’s Hospital Baylor College of Medicine Houston, Texas 77030, USA · Department of Computer Science SUNY Korea Incheon, Republic of Korea
Multi-Marginal Inverse Optimal Transport for Contrastive Learning Via Explicit Anchor-Positive-Negative Coupling
Sep 27, 2026Ngoc-Hai Nguyen, Thuan Nguyen, Prakash Ishwar +1Contrastive Learning Optimal Transport Approach
Department of Electrical and Computer Engineering Tufts University · Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Boston University
MAD-Guard: Controlled Study of Autoregressive Generation versus Direct Decision Interfaces for Closed Multimodal Forensic Tasks
Sep 27, 2026Hao ChenMultimodal Representations Autoregressive Generation
School of Cyber Security, Guangdong Polytechnic Normal University, Guangzhou, China
ICE: Task-Aligned Clifford Latent Fields for Multimodal Graph Foundation Models
Sep 24, 2026Xunkai Li, Xu Wang, Yinlin Zhu +4Graph Foundation Model Multimodal Representations
Department of Computer Science, Beijing Institute of Technology, Beijing, China · School of Airspace Science and Engineering, Shandong University, WeiHai, China · School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou, China
LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations
Sep 23, 2026Sandra Arcos-Holzinger, Debashish Chakraborty, Rohita Mocharla +7Multimodal Representations Fine-Grained Video Understanding
Johns Hopkins University · University of Melbourne · Human Language Technology Center of Excellence +2
PhyMo: A Physical-Field Modality for Multimodal AI4Physics
Sep 23, 2026Henan Sun, Haitao Hu, Jin Liu +4Multimodal Representations Modalities
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Noah’s Ark Lab · Tencent HY +1
Hub-Spectral Activation of Latent Multimodal Knowledge
Sep 15, 2026Ying Guo, Haidong Chen, Linrui Xu +7Multimodal Representations Cross-Modal
Beijing Key Laboratory of Key Technologies for AI+ Domain Applications, North China University of Technology, Beijing 100144, China · School of Geosciences and Info-Physics, Central South University, Changsha 410083, China · NExT++ Research Centre, National University of Singapore, Singapore +2
FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
Sep 15, 2026Guangyu Sun, Shlok Kumar Mishra, Wentao Bao +8Multimodal Representations Modality Alignment
Meta AI
Cross-Regional Grapevine Cold Hardiness Prediction via Learned Multimodal Latent Representations
Aug 31, 2026William Solow, Paola Pesantez-Cabrera, Markus Keller +3Agricultural Prediction Future Latent Representations
Oregon State University · Washington State University
Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgery
Aug 31, 2026Onur Izmitlioglu, Shervin Dehghani, Tarek Ghannoum +2Ophthalmology Optical Coherence Tomography
Computer Aided Medical Procedures, Technical University of Munich, Germany · SynthesEyes GmbH, Munich, Germany · Department of Ophthalmology, LMU University Hospital, Munich, Germany
Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information
Aug 9, 2026Xianghan Meng, Wei He, Zhiyuan Huang +1Multimodal Representations Modalities
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, P.R. China
GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System
Jul 31, 2026Jiping Liu, Zhongmin Zhang, Zisen Sang +7Multimodal Recommendation Model Multimodal Representations
Rajax Network Technology (Taobao Shangou of Alibaba) · Shanghai & Beijing, China · Central South University +1
OmniStyle-INR: Universal and Multimodal Style Transfer for INRs
Jul 17, 2026Rafał Kajca, Michał Miziołek, Kornel Howil +2Photorealistic Style Transfer Multimodal Representations
1Jagiellonian University, Faculty of Mathematics and Computer Science · 3IDEAS Research Institute · 2Jagiellonian University, Doctoral School of Exact and Natural Sciences
AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
Jul 16, 2026Sarthak Jain, Qiran Hu, Zhen Zhu +1Multimodal Representations Cross-Modal
University of Illinois Urbana-Champaign · Google DeepMind
The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model
Jul 15, 2026Zijie Yu, Gaowen Liu, Ramana Rao Kompella +2Contrastive Language-Image Pre-Training Model Spherical Latent Space
Tsinghua University · Cisco Research · University of Illinois Chicago
Multi-Agent Collaborative Reasoning with Tool-Augmented Evidence for Urban Region Profiling
Jul 15, 2026Xixuan Hao, Yutian Jiang, Jiabo Liu +4Multi-Agent Reasoning Urban Environments
The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · University of Washington Seattle, USA · Chang’an University Xi’an, China +1
MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search
Jul 13, 2026Zhen-Lin Chen, Maosen Sheng, Peng Lin +4Multimodal Recommendation Model Multimodal Representations
JD.COM Beijing, China
Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs
Jul 8, 2026Pengyu Zhang, Klim Zaporojets, Congfeng Cao +2Knowledge Graph Embeddings Multimodal Representations
University of Amsterdam Amsterdam, The Netherlands · Aarhus University Aarhus, Denmark
What Images Cannot Say: Language-Guided Olfactory Representation Learning
Jul 7, 2026Eleftherios Tsonis, Xi Wang, Vicky KalogeitonSmells Multimodal Representations
LIX, École Polytechnique, IP Paris, CNRS
Bumblebee: Interleaved Mixed-Layer Building Blocks for Large-Scale Recommendation Systems
Jul 6, 2026David Bauer, Cancan Zhang, Wenshun Liu +13Multimodal Representations Cross-Modal
Meta
Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detector
Jun 29, 2026Elys Allesiardo, Antoine Caubrière, Valentin VielzeufMultimodal Embeddings Multimodal Representations
Orange Research 4 Rue du Clos Courtel, 35510 Cesson-Sévigné, France
Meta-learning as a principle for human-like visual representations
Jun 24, 2026Can Demircan, Marcel Binz, Alireza Modirshanechi +1Meta-Learning Visual Representations
Helmholtz Munich, Munich, Germany · Max Planck Institute for Biological Cybernetics, Tübingen, Germany
Morphology-Aware Multimodal Representation Learning for Insect Phylogenetic Reconstruction
Jun 20, 2026Zixuan Liu, Kaijie Yu, Chun He +5Phylogenetic Inference Morphology
Zhejiang Key Lab of Accessible Perception and Intelligent Systems, Zhejiang University, Hangzhou 310027, China · State Key Laboratory of Rice Biology and Breeding & Ministry of Agricultural and Rural Affairs Key Laboratory of Molecular Biology of Crop Pathogens and Insects, Institute of Insect Sciences, Zhejiang University, Hangzhou 310058, China · Rural Development Academy, Zhejiang University, Hangzhou 310058, China +1
Generative-Model Predictive Planning for Navigation in Partially Observable Environments
Jun 17, 2026Thomas Quilter, Yifan Zhu, Guorui Quan +2Diffusion Planning Partially Observable Markov Decision Process
Department of Computer Science Aalto University Espoo, Finland
When to Align, When to Predict: A Phase Diagram for Multimodal Learning
Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Multimodal Learning Multimodal Representations
Technion · Genentech · Brown University
RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities
Jun 3, 2026Vasiliki Rizou, Pascal Frossard, Dorina ThanouMultimodal Representations Representation Learning
EPFL
Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals
Jun 1, 2026Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang +2Multimodal Fusion Multimodal Representations
Adelaide University Adelaide, Australia · Shandong University Jinan, China
MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment
May 28, 2026Dang Nguyen Hong, Nhi Ngoc-Yen Nguyen, Huy-Hieu PhamSubspace Multiscale
Hanoi University of Science and Technology, Hanoi, Vietnam · VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam +1
UniNote: A Unified Embedding Model for Multimodal Representation and Ranking
May 28, 2026Jinghan Zhao, Wenwei Jin, Anqi Li +5Multimodal Retrieval Multimodal Representations
Xiaohongshu Beijing, China · Shanghai Jiao Tong University Shanghai, China · Huazhong University of Science and Technology Wuhan, Hubei, China +1
Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals
May 26, 2026Phu X. Nguyen, Konstantinos Kontras, Wei Dai +5Electrocardiogram Foundation Models Multimodal Representations
1KU Leuven · 3MIT · 4DFKI +1
TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens
May 15, 2026Jianpeng Cheng, Xian Wu, Jiangfan Zhang +10Multimodal Embeddings Compact Latent Sensor Tokens
1Meta AI
The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?
May 10, 2026Zhaoyang Zhang, Run Shao, Dongyue Wu +4Multimodal Representations Attractors
1Central South University · 4Ant Group · 2Huazhong University of Science and Technology +1
Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
May 5, 2026Hahyeon Choi, Nojun KwakMultimodal Representations Sparsity
Seoul National University, Seoul, South Korea.
Learning Generalizable Multimodal Representations for Software Vulnerability Detection
Apr 28, 2026Zeming Dong, Yuejun Guo, Qiang Hu +5Multimodal Representations Software
University of Luxembourg · 2Luxembourg Institute of Science and Technology · 3Tianjin University
AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce
Apr 22, 2026Biao Zhang, Lixin Chen, Bin Zhang +3Multimodal Representations Multimodal Understanding
Taobao & Tmall Group of Alibaba
Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations
Apr 20, 2026Yunjia Xi, Menghui Zhu, Jianghao Lin +4Multimodal Representations Layer-Wise
Shanghai Jiao Tong University Shanghai, China · Huawei Noah’s Ark Lab Shanghai, China · Antai College of Economics and Management, Shanghai Jiao Tong University Shanghai, China
Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering
Apr 18, 2026Tianyi Chen, Haobo Wang, Kai Tang +5Hyperbolic Learning Multi-View
School of Software Technology, Zhejiang University, Zhejiang 310027, China · College of Computer Science and Technology, Zhejiang University, Zhejiang 310027, China · Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing 100044, China +2
Semantic Purification for Conditional Representation Learning
Feb 5, 2026Jiaquan Wang, Yan Lyu, Chen Li +1Representation Learning Multimodal Representations
School of Computer Science and Engineering, Southeast University, China · Biomedicine Discovery Institute, Department of Biochemistry and Molecular Biology, Monash University, Australia
Multimodal Feature Prototype Learning for Interpretable and Discriminative Cancer Survival Prediction
Oct 7, 2025Shuo Jiang, Zhuwen Chen, Liaoman Xu +6Survival Predictions Learnable Prototypes
Hangzhou Dianzi University, Hangzhou 310018, China · School of Information and Communication Technology, Griffith University, QLD 4215, Australia · Shenzhen Research Institute of Big Data, Shenzhen 518172, China +3
The Platonic Universe: Do Foundation Models See the Same Sky?
Sep 23, 2025UniverseTBD, :, Trinidad Borrell +11Astronomical Images Foundation Model
Independent Researcher · AstroAI · Harvard-Smithsonian CfA +4
A quantitative analysis of semantic information in deep representations of text and images
May 21, 2025Santiago Acevedo, Andrea Mascaretti, Riccardo Rende +3Semantic Representations Multimodal Representations
Scuola Internazionale Superiore di Studi Avanzati (SISSA) · Universitat Pompeu Fabra (UPF) · Catalan Institute of Research and Advanced Studies (ICREA) and Universitat Pompeu Fabra (UPF)
CLIP Embeddings for AI-Generated Image Detection: A Few-Shot Study with Lightweight Classifier
May 15, 2025Ziyang OuAi-Generated Image Detection Contrastive Language-Image Pre-Training Model
Department of Electrical and Computer Engineering University of Rochester Rochester, NY 14627, USA

Sep 30, 2026, cs.CVJ/K move · Enter open · S save

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

Hongyuan Tao, Xinggang Wang, Lianghui Zhu, Yongkang Li+6

Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics

Multimodal Pretraining Multimodal Representations Flow Models Generative Models

Open paper AskPDF arXiv

We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for multimodal pretraining. Multimodal Flow introduces a unified continuous architecture that integrates multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. The model predicts multiple target chunks in parallel during training and generates hyperchunks sequentially at inference. We instantiate MF-1 and pretrain it on multimodal data. Across 0.6B, 1.2B, and 1.6B scales, continued pretraining consistently improves multimodal modeling. With only 150B pretraining tokens, MF-1 achieves an average score of 82.8 across GenEval and DPG-Bench and 75.3 across VQAv2, MMBench, and POPE, remaining competitive with unified models trained on substantially more data. Under matched data, optimization, and parameter budgets, Multimodal Flow further outperforms representative hybrid and discrete models. These results establish continuous chunk-based embedding flow modeling as a new fully continuous paradigm for unified multimodal modeling. The related code and model are publicly released at https://github.com/hustvl/Multimodal-Flow.

Momentum

Related topics

Latest papers 44