Multimodal Transformer

Latest papers

22 papers2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning
Oct 10, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive Learning Multimodal Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University
Virtual Encoders in Multimodal Transformers
Sep 22, 2026Katsuya Ogata, Yuta NakashimaVisual Representation Learning Multimodal Large Language Models
The University of Osaka
M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction
Sep 11, 2026Wenzhe Jin, Haina TangMixture of Experts Vessel Trajectory Prediction
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
DeFiFusion: Combining Transaction Events with Smart Contracts to Detect Price Manipulation Attacks
Sep 10, 2026Rui Cao, Shaojing Fan, Liming Fang +3Financial Fraud Detection Decentralized Finance
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China · Department of Electrical and Computer Engineering (ECE), National University of Singapore, Singapore · School of Aerospace Engineering, Nanjing University of Aeronautics and Astronautics, China +2
NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
Aug 31, 2026Aurélien Lac, Tony WuMultimodal Foundation Models Multimodal Embedding
H Company
Twins: Learn to Predict Unified Representations with Focal Loss
Jul 24, 2026Kaixiong Gong, Xin Cai, Bin Lin +9Focal Loss Diffusion Transformer
The Chinese University of Hong Kong · Tencent, Hunyuan · City University of Hong Kong +2
Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models
Jul 16, 2026Yufeng Ji, Wenhao Tang, Haoyi Niu +3VLM Adaptation Multimodal Transformer
Shanghai Qizhi Institute · The Chinese University of Hong Kong · Hong Kong Embodied AI Lab +2
Route-MHT: Multimodal Transformer Guardrails for Thermal Visual Place Recognition
Jul 6, 2026Zhiyuan Lu, Kanji TanakaThermal Imaging Visual Place Recognition
Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
Jun 30, 2026Zhaoyang Luo, Runmin Dong, Miao Yang +4Efficient VLM Inference Multimodal Large Language Models
Tsinghua Shenzhen International Graduate School, Shenzhen, China · Sun Yat-sen University, Zhuhai, China · Tsinghua University, Beijing, China +1
GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention
Jun 4, 2026Giordano Cicchetti, Eleonora Grassucci, Danilo ComminielloCross-Attention Multimodal Transformer
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
May 30, 2026Ravil Mussabayev, Rustam MussabayevRepresentation Geometry Multimodal Large Language Models
Satbayev University
Toward Native Multimodal Modeling: A Roadmap
May 25, 2026Siyu An, Junru Lu, Junnan Dong +18Unified Multimodal Models Multimodal Generation
Tencent Youtu Lab
Resilient Vision-Tabular Multimodal Learning under Modality Missingness
May 12, 2026Camillo Maria Caruso, Valerio Guarrasi, Paolo SodaMissing-Modality Learning Multimodal Robustness
Research Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Roma, Italy, Europe · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden, Europe
LongCat-Next: Lexicalizing Modalities as Discrete Tokens
Mar 29, 2026Meituan LongCat Team, Bin Xiao, Chao Wang +86Unified Multimodal Models Multimodal Learning
Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model
Mar 23, 2026SII-GAIR, Sand. ai, : +43Multimodal Transformer Audio-Video Generation
Semantic Audio-Visual Navigation in Continuous Environments
Mar 20, 2026Yichen Zeng, Hebaixu Wang, Meng Liu +4Audio-Visual Navigation Multimodal Transformer
Wuhan University · Zhongguancun Academy · Shandong Jianzhu University +4
GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes
Mar 18, 2026Huajian Zeng, Abhishek Saroha, Daniel Cremers +1Multimodal Transformer Trajectory Generation
TU München · MBZUAI · MCML +1
Show Me When and Where: Towards Referring Video Object Segmentation in the Wild
Mar 15, 2026Mingqi Gao, Jinyu Yang, Jingnan Luo +4Multimodal Transformer Referring Video Object Segmentation
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · Warwick Manufacturing Group, University of Warwick, Coventry CV1 7AL, U.K. · Tapall.ai, Shenzhen 518055, China +3
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
Mar 13, 2026Yichen Zhang, Da Peng, Zonghao Guo +19Unified Multimodal Models Multimodal Generation
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Multimodal Understanding Feature Attribution
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Feb 27, 2026Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1Multimodal Learning Unified Multimodal Models
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning
Feb 17, 2026Hongxuan Wu, Yukun Zhang, Xueqing ZhouVision-Language Models VLM Interpretability
Duke kunshan University, Duke University · The Chinese University of Hong Kong, Hong Kong, China · Fudan University, Shanghai, China

Oct 10, 2026J/K move · Enter open · S save

Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

Tillmann Rheude, Roland Eils, Benjamin Wild

Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

Multimodal Contrastive Learning Multimodal Learning Multimodal Pretraining Multimodal Transformer

Open paper AskPDF arXiv

Multimodal contrastive learning typically relies on manually specified scoring functions, such as cosine similarity, to quantify the similarity between representations from different modalities. Several such functions have been recently proposed to extend contrastive learning beyond two modalities. In principle, however, any function that assigns high compatibility to matched samples and low compatibility to mismatched ones could serve as a scorer. Motivated by this broader view, we instantiate the scorer with a Transformer and find that even a randomly initialized model, when kept frozen, can outperform strong handcrafted multimodal contrastive baselines, whereas optimizing the scorer jointly with the encoders is less effective. Building on this observation, we introduce the Prior-Fitted Scorer (PFS): a Transformer scorer pre-trained on synthetic multimodal data generated from structural causal model (SCM) priors before transferring it to real-world data. The SCM exposes the scorer to diverse dependency structures, including shared, pairwise, and private information, providing a structured prior over higher-order cross-modal similarity. Across four real-world datasets, frozen Transformer scorers are competitive to contrastive baselines, and prior-fitting further improves the random initialization. More broadly, our results suggest that multimodal contrastive learning need not rely on manual similarity functions: Transformers can provide strong inductive biases, while causal pretraining can further shape these biases.