Multimodal Alignment

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 52

All topics

From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment
Oct 1, 2026Liwei Lin, Gus XiaSymbolic Music Generation Sheet Music
Computer Science Department, NYU Shanghai · Machine Learning Department, MBZUAI
Prototype-guided Bilateral Alignment Multimodal Federated Learning
Sep 30, 2026Tianchi Liao Tianchi_Liao, Lele Fu, Sheng Huang +3Heterogeneous Federated Learning Multimodal Alignment
Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China · School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China
Binding Multiple Modalities via Multimodal Wasserstein Barycenter
Sep 27, 2026Xiaole Tang, Jiayi Xu, Xiang Gu +2Multimodal Alignment Complementary Modalities
Xi’an Jiaotong University
MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction
Sep 22, 2026Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2Multimodal Clinical Data Multimodal Pretraining
Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1
VoT: Vision-of-Thought for Unified Multimodal Representation Alignment
Sep 7, 2026Jingxiang Sun, Chao Liao, Zhengxiong Luo +6Modern Text-To-Image Models Diffusion Transformers
ByteDance Seed
Beyond Multimodal Alignment: Shared Physical Representations Across Sensors and Action Orders
Aug 19, 2026Kaizhen Tan, Xin Xu, Siru Tao +4Multimodal Alignment Sensors
New York University, New York, NY, USA · Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport
Aug 4, 2026Yixuan Florence Wu, Yilun Zhu, Naichen ShiMultimodal Alignment Unimodal Encoders
Department of Engineering Sciences and Applied Mathematics Northwestern University Evanston, IL 60208, USA · Deparment of Electrical Engineering and Computer Science University of Michigan Ann Arbor, MI 48109, USA · Department of Industrial Engineering and Management Sciences Department of Mechanical Engineering Northwestern University Evanston, IL 60208, USA
Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration
Aug 3, 2026Aoru Xue, Yujing Sun, Yiming Ren +3Motion Capture Motion Estimation
ShanghaiTech University, Shanghai, China · ShanghaiTech University, China · Digital Trust Centre, Nanyang Technological University, Singapore +1
Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer
Aug 3, 2026Tyler Ashoff, Jordan RoduLarge Language Model Alignment Cross-Lingual Transfer
Department of Statistics, University of Virginia. · Department of Statistics, University of Virginia
Persistent Convolution: A Topological Framework for AI Alignment Testing and Semantic Space Characterization
Jul 31, 2026Tyler Ashoff, Jordan RoduArtificial Intelligence Alignment Multimodal Alignment
Department of Statistics, University of Virginia. · Department of Statistics, University of Virginia
DAS-PMVC: A Framework for Partial Multi-View Clustering via Dual Alignment and Structure Enhancement
Jul 30, 2026Shubin Ma, Liang Zhao, Chuanye He +4Multi-View Multimodal Alignment
Dalian University of Technology Dalian, Liaoning, China · Inspur Group Co., Ltd. Jinan, Shandong, China · China University of Mining and Technology Xuzhou, Jiangsu, China +1
Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Multimodal Continual Instruction Tuning Multimodal Alignment
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
Multi-Sensor Alignment for Weather Simulations
Jul 28, 2026Samsad Alam, Devyani Lambhate, Aditya Mohan +2Multi-Sensor Fusion Multimodal Alignment
Department of Cyber-Physical Systems at the Indian Institute of Science, Bengaluru
GroupVideo: Multi-Identity Customized Text-to-Video Generation
Jul 23, 2026Xinyang Song, Libin Wang, Jianxin Sun +4Interactive Video Generation Text-To-Video Generation Model
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · New Laboratory of Pattern Recognition and the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Open Your Model's Eyes: Video and Context-Aware Multimodal Backchannel Prediction
Jul 22, 2026Min-Jae Kim, Jun-Yeong Moon, Mujeen Sung +1Multimodal Alignment Visual Cues
1Korea University, Seoul, Republic of Korea · 2Kyung Hee University, Yongin, Republic of Korea
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Jul 20, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive Learning Multimodal Alignment
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University
OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment
Jul 20, 2026Wenxiao Fan, Hang Yin, Kan LiSpatial Supervision Spatial Reasoning
School of Computer Science, Beijing Institute of Technology
Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
Jul 17, 2026Xunkai Li, Guohao Fu, Yuming Ai +4Federated Graph Learning Graph Foundation Model
Beijing Institute of Technology, Beijing, 100811, China
Local Multimodal Music Alignment from Global Supervision
Jul 10, 2026Irmak Bukey, Zachary Novack, Jongmin Jung +2Multimodal Alignment Multimodal Contrastive Learning
Carnegie Mellon University · University of California San Diego · Neutune +1
AeroVerse-SatAgent: UAV-Satellite Collaborative Spatial Reasoning Inspired by the Dual Visual Pathway Theory of Cognitive Neuroscience
Jun 30, 2026Wenyi Zhang, Fanglong Yao, Youzhi Liu +5Spatial Reasoning Multiple Satellite Systems
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100190, China · School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China +4
Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction
Jun 29, 2026Dominik Winter, Dominik Vonficht, Loïc Le Bescond +6Digital Pathology Biomarker
AstraZeneca Computational Pathology and Biomarkers, Munich, Germany · AstraZeneca, Early Oncology and Translation Medicine, Cambridge, UK
What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment
Jun 24, 2026Salini Yadav, Taveena Lotey, Pravendra Singh +1Electroencephalography Decoding Visual Perception
Department of CSE IIT Roorkee, India · Department of CSE IIT (ISM) Dhanbad, India
Morphology-Aware Multimodal Representation Learning for Insect Phylogenetic Reconstruction
Jun 20, 2026Zixuan Liu, Kaijie Yu, Chun He +5Phylogenetic Inference Morphology
Zhejiang Key Lab of Accessible Perception and Intelligent Systems, Zhejiang University, Hangzhou 310027, China · State Key Laboratory of Rice Biology and Breeding & Ministry of Agricultural and Rural Affairs Key Laboratory of Molecular Biology of Crop Pathogens and Insects, Institute of Insect Sciences, Zhejiang University, Hangzhou 310058, China · Rural Development Academy, Zhejiang University, Hangzhou 310058, China +1
SARLO-80: Worldwide Slant SAR Language Optic Dataset 80cm
Jun 18, 2026Solène Debuysère, Nicolas Trouvé, Nathan Letheule +2Synthetic Aperture Radar Multimodal Dataset
1DEMR-ONERA – The French Aerospace Lab, Université Paris-Saclay, Palaiseau, France · 2DTIS-ONERA – The French Aerospace Lab, Université Paris-Saclay, Palaiseau, France · 3Hugging Face, London
OTCHA: Optimal Transport-driven Confidence-aware Latent Hub Alignment for Multi-View Medical Image Classification
Jun 18, 2026Jiwoong Yang, Haejun Chung, Ikbeom JangMedical Image Classification Medical Imaging Datasets
Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea
GraphBEV++: Multi-Modal Feature Alignment for Autonomous Driving
Jun 15, 2026Ziying Song, Caiyan Jia, Lin Liu +3Multi-Sensor Fusion Autonomous Driving Perception
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University · School of Artificial Intelligence (School of Software), Yanshan University · University of Macau +2
Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection
Jun 1, 2026David J. Lerch, Livien Majer, Zeyun Zhong +3Multimodal Alignment Fine-Grained Video Understanding
Fraunhofer IOSB, Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognition
May 31, 2026Koby Aharonov, Oren Shrout, Ayellet TalMultimodal Alignment Contrastive Loss
Technion – Israel Institute of Technology
UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment
May 29, 2026Zheng Wang, Shuo Wang, Junhong WangEmotion Recognition Cross-Modal Fusion
Institute of Advanced Technology, University of Science and Technology of China, Hefei, 230026, China · Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230026, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230026, China
OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment
May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Multimodal Alignment Modality Alignment
Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3
MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
May 25, 2026Shristi Das Biswas, Kaushik RoyMultimodal Continual Instruction Tuning Multimodal Alignment
Purdue University
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?
May 22, 2026Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra +1Multimodal Alignment Multimodal Retrieval
University of Amsterdam Amsterdam, The Netherlands
What Semantics Survive the Connector? Diagnosing VLM-to-DiT Alignment in Video Editing
May 20, 2026Hangyu Lin, Chao Wen, Chengming Xu +4Video Editing Multimodal Alignment
HKUST · FDU · ZJU +1
CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook
May 18, 2026Zeyu Chen, Jie Li, Kai HanMultimodal Alignment Cross-Modal Conflicts
Visual AI Lab, The University of Hong Kong
Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
May 18, 2026Xinpeng Dong, Min Zhang, Kairong Han +3Recent Vision-Language Models Multimodal Large Language Models
Zhejiang University · East China Normal University · Zhejiang University of Science and Technology
Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Data
May 16, 2026Shiwon Kim, Yu Rang ParkMultimodal Alignment Multimodal Pretraining
Yonsei University
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
May 15, 2026Yishun Lu, Wes ArmourMultimodal Alignment Multimodal Model
University of Oxford, Oxford, United Kingdom
GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective
May 15, 2026Xu Wang, Xunkai Li, Yinlin Zhu +2Multimodal Alignment Unimodal Encoders
School of Airspace Science and Engineering, Shandong University, WeiHai, China · Department of Computer Science, Beijing Institute of Technology, Beijing, China · School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou, China
Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment
May 14, 2026Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim +1Patient Trajectories Electronic Health Records
Division of Intramural Research National Library of Medicine, National Institutes of Health Bethesda, MD, USA
SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning
May 13, 2026Truong Pham, Anay Majee, Rishabh IyerMultimodal Alignment Modality Alignment
The University of Texas at Dallas
Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation
May 12, 2026Roman Klypa, Alberto Bietti, Sergei GrudininMessenger Rna Protein Design
Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK · Grenoble, France · Center for Computational Mathematics, Flatiron Institute +1
Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models
May 3, 2026Junyuan Xiao, Dingkang Liang, Xin Zhou +9Multimodal Alignment Interactive Video Generation
Tsinghua University, China · Huazhong University of Science and Technology, China · CSU, China +4
ClipTBP: Clip-Pair based Temporal Boundary Prediction with Boundary-Aware Learning for Moment Retrieval
Apr 30, 2026Ji-Hyeon Kim, Ho-Joong Kim, Seong-Whan LeeMultimodal Alignment Representation-Alignment Loss
Dept. of Artificial Intelligence, Korea University, Seoul 02841, Republic of Korea
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
Apr 29, 2026Junwon You, Mihyun Jang, Sangwoo Mo +1Multimodal Alignment Recent Vision-Language Models
1KAIST · 2POSTECH
PRIMA: Pre-training with Risk-integrated Image-Metadata Alignment for Medical Diagnosis via LLM
Feb 26, 2026Yiqing Wang, Chunming He, Ming-Chen Lu +4Multimodal Clinical Data Multimodal Pretraining
Department of Biomedical Engineering, Duke University, Durham, NC, USA · Department of Ophthalmology and Visual Sciences, University of Michigan, Ann Arbor, MI, USA
MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models
Jan 28, 2026Wenbo Xu, Wei Lu, Xiangyang LuoDeepfake Detection Forgeries
School of Computer Science and Engineering, Sun Yat-sen University Guangzhou, Guangdong, China · State Key Laboratory of Mathematical Engineering and Advanced Computing Zhengzhou, Henan, China
Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis
Nov 22, 2025Wenjing Liu, Qin Ren, Wen Zhang +2Multimodal Alignment Survival Predictions
Stony Brook University, Stony Brook, NY, USA · Johns Hopkins University, Baltimore, MD, USA · Brookhaven National Laboratory, Upton, NY, USA
MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment
Sep 17, 2025Elena Camuffo, Francesco Barbato, Mete Ozay +2Object Detection Multimodal Alignment
Samsung R&D Institute UK, United Kingdom · University of Padova, Italy
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
Aug 17, 2025Xuhui Zhan, Tyler DerrMultimodal Alignment Multimodal Reasoning
Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.
Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models
Jun 12, 2025Konstantinos Vilouras, Ilias Stogiannidis, Junyu Yan +2Chest X-Ray Radiology Report Generation
School of Engineering, University of Edinburgh · Canon Medical Research Europe Ltd.
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaMultimodal Alignment Multimodal Retrieval
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
May 15, 2025Chunyu Ye, Yunhao Zhang, Jingyuan Sun +3Brain-Computer Interface Multimodal Alignment
1State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Department of Computer Science, The University of Manchester +1

Oct 1, 2026, cs.SDJ/K move · Enter open · S save

From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment

Liwei Lin, Gus Xia

Computer Science Department, NYU Shanghai · Machine Learning Department, MBZUAI

Symbolic Music Generation Sheet Music Multimodal Alignment Earth

Open paper AskPDF arXiv

How can we understand what a music foundation model has learned \textit{internally}? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as \textit{structured relations} rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, \textbf{we shift from feature identification to structure-based analysis}, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.

Momentum

Related topics

Latest papers 52