Cross-Modal Representation Transfer

Follow this topic Ask about this topic

Latest papers

34 papers12 papers in the last four weeks, up 140% on the four weeks before. 0.1% of all new papers.

MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization
Mar 13, 2026Chenyang Zhu, Hongxiang Li, Xiu Li +1Cross-Modal Representation Transfer T2I Generation
Tsinghua University · HKUST
Any2Any: Unified Arbitrary Modality Translation for Remote Sensing
Mar 4, 2026Haoyang Chen, Jing Zhang, Hebaixu Wang +7Remote Sensing Image Understanding Image-to-Image Translation
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University, and Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. 100094 · School of Electronic Information, Wuhan University, Wuhan, China +2
Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer
Feb 22, 2026Chenhang Cui, An Zhang, Yuxin Chen +5Cross-Modal Representation Transfer VLM Reasoning
National University of Singapore, Singapore · University of Science and Technology of China, Hefei, China · Nanyang Technological University, Singapore
Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
Feb 20, 2026Jingyang Qiao, Zhizhong Zhang, Xin Tan +3Cross-Modal Representation Transfer Multimodal Large Language Models
East China Normal University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +1

Mar 13, 2026J/K move · Enter open · S save

MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization

Chenyang Zhu, Hongxiang Li, Xiu Li, Long Chen

Tsinghua University · HKUST

Cross-Modal Representation Transfer T2I Generation Multimodal Generation

Open paper AskPDF arXiv

Concept customization typically binds rare tokens to a target concept. Unfortunately, these approaches often suffer from unstable performance as the pretraining data seldom contains these rare tokens. Meanwhile, these rare tokens fail to convey the inherent knowledge of the target concept. Consequently, we introduce Knowledge-aware Concept Customization, a novel task aiming at binding diverse textual knowledge to target visual concepts. This task requires the model to identify the knowledge within the text prompt to perform high-fidelity customized generation. Meanwhile, the model should efficiently bind all the textual knowledge to the target concept. Therefore, we propose MoKus, a novel framework for knowledge-aware concept customization. Our framework relies on a key observation: cross-modal knowledge transfer, where modifying knowledge within the text modality naturally transfers to the visual modality during generation. Inspired by this observation, MoKus contains two stages: (1) In visual concept learning, we first learn the anchor representation to store the visual information of the target concept. (2) In textual knowledge updating, we update the answer for the knowledge queries to the anchor representation, enabling high-fidelity customized generation. To further comprehensively evaluate our proposed MoKus on the new task, we introduce the first benchmark for knowledge-aware concept customization: KnowCusBench. Extensive evaluations have demonstrated that MoKus outperforms state-of-the-art methods. Moreover, the cross-model knowledge transfer allows MoKus to be easily extended to other knowledge-aware applications like virtual concept creation and concept erasure. We also demonstrate the capability of our method to achieve improvements on world knowledge benchmarks.

Latest papers

MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization

Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer

Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework