Multimodal Transformer

Follow this topic Ask about this topic

Latest papers

27 papers3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

TEAM: Topology-Aware, Semantic-Sensitive, General Transformer for Multimodal-Attributed Graphs
Nov 8, 2026Yushuo Li, Xunkai Li, Zhengyu Wu +2Multimodal Graph Learning Multimodal Transformer
Beijing Institute of Technology, China.
HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning
Nov 2, 2026Alice Driessen, Emil Malmsten, Steffen Wolf +11Multimodal Fusion Multimodal Learning
kaiko.ai, Zürich, Switzerland · Biomedical Data Science Center, Lausanne University Hospital, Lausanne, Switzerland · Swiss Institute of Bioinformatics (SIB), Lausanne, Switzerland +6
Grounding Long Financial Text with Semantically-Enriched Tables via Multimodal Learning
Oct 27, 2026Xiaoyu HuMultimodal Transformer Tabular Representation Learning
Rutgers University
Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning
Oct 10, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive Learning Multimodal Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University
Virtual Encoders in Multimodal Transformers
Sep 22, 2026Katsuya Ogata, Yuta NakashimaVisual Representation Learning Multimodal Large Language Models
The University of Osaka
M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction
Sep 11, 2026Wenzhe Jin, Haina TangMixture of Experts Vessel Trajectory Prediction
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
DeFiFusion: Combining Transaction Events with Smart Contracts to Detect Price Manipulation Attacks
Sep 10, 2026Rui Cao, Shaojing Fan, Liming Fang +3Financial Fraud Detection Decentralized Finance
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China · Department of Electrical and Computer Engineering (ECE), National University of Singapore, Singapore · School of Aerospace Engineering, Nanjing University of Aeronautics and Astronautics, China +2
NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
Aug 31, 2026Aurélien Lac, Tony WuMultimodal Foundation Models Multimodal Embedding
H Company
Twins: Learn to Predict Unified Representations with Focal Loss
Jul 24, 2026Kaixiong Gong, Xin Cai, Bin Lin +9Focal Loss Diffusion Transformer
The Chinese University of Hong Kong · Tencent, Hunyuan · City University of Hong Kong +2
Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models
Jul 16, 2026Yufeng Ji, Wenhao Tang, Haoyi Niu +3VLM Adaptation Multimodal Transformer
Shanghai Qizhi Institute · The Chinese University of Hong Kong · Hong Kong Embodied AI Lab +2
Route-MHT: Multimodal Transformer Guardrails for Thermal Visual Place Recognition
Jul 6, 2026Zhiyuan Lu, Kanji TanakaThermal Imaging Visual Place Recognition
Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
Jun 30, 2026Zhaoyang Luo, Runmin Dong, Miao Yang +4Efficient VLM Inference Multimodal Large Language Models
Tsinghua Shenzhen International Graduate School, Shenzhen, China · Sun Yat-sen University, Zhuhai, China · Tsinghua University, Beijing, China +1
GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention
Jun 4, 2026Giordano Cicchetti, Eleonora Grassucci, Danilo ComminielloCross-Attention Multimodal Transformer
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
May 30, 2026Ravil Mussabayev, Rustam MussabayevRepresentation Geometry Multimodal Large Language Models
Satbayev University
Toward Native Multimodal Modeling: A Roadmap
May 25, 2026Siyu An, Junru Lu, Junnan Dong +18Unified Multimodal Models Multimodal Generation
Tencent Youtu Lab
Resilient Vision-Tabular Multimodal Learning under Modality Missingness
May 12, 2026Camillo Maria Caruso, Valerio Guarrasi, Paolo SodaMissing-Modality Learning Multimodal Robustness
Research Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Roma, Italy, Europe · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden, Europe
LongCat-Next: Lexicalizing Modalities as Discrete Tokens
Mar 29, 2026Meituan LongCat Team, Bin Xiao, Chao Wang +86Discrete Visual Tokenization Unified Multimodal Models
Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model
Mar 23, 2026SII-GAIR, Sand. ai, : +43Multimodal Transformer Audio-Video Generation
Semantic Audio-Visual Navigation in Continuous Environments
Mar 20, 2026Yichen Zeng, Hebaixu Wang, Meng Liu +4Audio-Visual Navigation Multimodal Transformer
Wuhan University · Zhongguancun Academy · Shandong Jianzhu University +4
GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes
Mar 18, 2026Huajian Zeng, Abhishek Saroha, Daniel Cremers +1Multimodal Transformer Trajectory Generation
TU München · MBZUAI · MCML +1
Show Me When and Where: Towards Referring Video Object Segmentation in the Wild
Mar 15, 2026Mingqi Gao, Jinyu Yang, Jingnan Luo +4Multimodal Transformer Referring Video Object Segmentation
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · Warwick Manufacturing Group, University of Warwick, Coventry CV1 7AL, U.K. · Tapall.ai, Shenzhen 518055, China +3
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
Mar 13, 2026Yichen Zhang, Da Peng, Zonghao Guo +19Unified Multimodal Models Multimodal Generation
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Multimodal Understanding Feature Attribution
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Feb 27, 2026Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1Multimodal Learning Unified Multimodal Models
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning
Feb 17, 2026Hongxuan Wu, Yukun Zhang, Xueqing ZhouVision-Language Models VLM Interpretability
Duke kunshan University, Duke University · The Chinese University of Hong Kong, Hong Kong, China · Fudan University, Shanghai, China
Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learning
Feb 4, 2026Nicholas Barnfield, Subhabrata Sen, Pragya SurIn-Context Learning Multimodal ICL
Department of Statistics, Harvard University
Federated learning for unpaired multimodal data
Jan 25, 2026Anders EklundCentered Kernel Alignment Multimodal Federated Learning
Department of Biomedical Engineering, Linköping University, Sweden · Department of Computer and Information Science, Linköping University, Sweden · Center for Medical Image Science and Visualization (CMIV), Linköping University, Sweden

Nov 8, 2026J/K move · Enter open · S save

TEAM: Topology-Aware, Semantic-Sensitive, General Transformer for Multimodal-Attributed Graphs

Yushuo Li, Xunkai Li, Zhengyu Wu, Ronghua Li+1

Beijing Institute of Technology, China.

Multimodal Graph Learning Multimodal Transformer Cross-Modal Representation Learning

Open paper AskPDF arXiv

Multimodal-Attributed Graphs (MAGs) organize heterogeneous data in graphs whose nodes are characterized by modalities such as text and images rather than traditional unimodal attributes. However, many representative MAG methods organize propagation around node-level states and topology-defined neighborhoods, which can obscure fine-grained modality contributions: a bounded neighborhood may contain semantically weak evidence, while compatible context need not be confined to it. To address both limitations, we propose TEAM, a modality-centric framework that constructs and encodes graph context with node-modality tokens rather than treating each node as a single interaction unit. Concretely, TEAM unfolds the local topology into a Modality Encoding Tree and performs tree-anchored semantic evidence acquisition, organizing local structural context while expanding the accessible evidence space outside the constructed local Tree. We further introduce the Modality-Source Context Transformer (MSCT), which encodes the selected node-modality tokens and coordinates interactions among modality-source groups. By retaining textual and visual attributes as separate interaction units until readout, it supports fine-grained cross-node cross-modal exchange without first collapsing them into whole-node features. Across the evaluated baselines, TEAM attains the highest reported mean on all graph-learning metrics, all three G2Text metrics, and both G2Image metrics. On Toys, it surpasses the strongest baseline by 3.37 and 2.93 percentage points in Accuracy and Macro-F1, respectively.