Cross-Modal Retrieval

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 250% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 134

All topics

SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding
Apr 17, 2026Keisuke Gomi, Keiji YanaiCross-Modal Learning Image-Text Retrieval
The University of Electro-Communications
MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation
Apr 4, 2026Sijun Dai, Qiang Huang, Xiaoxing You +1Cross-Modal Learning Multimodal RAG
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University
FusionBERT: Multi-View Image--3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder
Apr 2, 2026Wei Li, Yufan Ren, Hanqing Jiang +6Multi-View Learning Cross-Modal Retrieval
IROOTECH TECHNOLOGY, Hangzhou, Zhejiang, China · Wolf 1069 b Lab, Sany Group, Hangzhou, Zhejiang, China · Zhejiang University, Hangzhou, Zhejiang, China +4
Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction
Mar 10, 2026Yao Zhang, Zhuchenyang Liu, Yanlan He +2Representation Learning Feature Interaction Modeling
Aalto University, Espoo, Finland · Fudan University, Shanghai, China · Georgia Institute of Technology, Atlanta GA, USA
SleepLM: Natural-Language Intelligence for Human Sleep
Feb 27, 2026Zongzhe Xu, Zitao Shuai, Eideen Mozaffari +3Multimodal Pretraining Cross-Modal Retrieval
University of California, Los Angeles
Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
Feb 5, 2026Qi Li, Yanzhe Zhao, Yongxin Zhou +4Efficient Multimodal Inference Multimodal Large Language Models
Honor Device Co., Ltd., China
Generating a Paracosm for Training-Free Zero-Shot Composed Image Retrieval
Jan 31, 2026Tong Wang, Yunhan Zhao, Shu KongImage Matching Cross-Modal Retrieval
University of Macau, Macau 999078, China · Uiversity of California at Irvine, Irvine, CA 92697, USA · Institute of Collaborative Innovation, University of Macau, Macau 999078, China
ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search
Jan 30, 2026Tao Yu, Haopeng Jin, Hao Wang +19Temporal Video Grounding Cross-Modal Retrieval
CASIA · UCAS · Peking University +2
VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing
Dec 12, 2025Emanuel Sánchez Aimar, Gulnaz Zhambulova, Fahad Shahbaz Khan +2Cross-Modal Learning Multimodal Embedding
Linköping University, Sweden
GeoBridge++: Fact-Guided Geo-Semantic Bridging for Unified Cross-View Geo-Localization
Dec 2, 2025Zixuan Song, Jing Zhang, Di Wang +6Cross-View Geo-Localization Geospatial Representation Learning
College of Computer Science and Technology and Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China · Zhongguancun Academy, Beijing 100094, China · School of Computer Science, Wuhan University, Wuhan, China +1
MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
Oct 17, 2025Qiyu Wu, Shuyang Cui, Satoshi Hayakawa +3Multimodal Robustness Multimodal Large Language Models
Sony Group Corporation · Sony AI
Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity
Oct 17, 2025Naoki Yoshida, Satoshi Hayakawa, Yuhta Takida +3Cross-Modal Learning Kernel Mean Embeddings
The University of Tokyo · Sony Group Corporation · Sony AI
TCMA: Text-Conditioned Multi-granularity Alignment for Drone Cross-Modal Text-Video Retrieval
Oct 11, 2025Zixu Zhao, Yang Zhan, Yunhao Li +1Cross-Modal Retrieval Vision-Language Alignment
Carnegie Mellon University, Pittsburgh, PA 15213 USA · The Hong Kong Polytechnic University, Hong Kong, China · Wuhan University, Wuhan 430079, China
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg

Apr 17, 2026, cs.CVJ/K move · Enter open · S save

SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding

Keisuke Gomi, Keiji Yanai

The University of Electro-Communications

Cross-Modal Learning Image-Text Retrieval Multimodal Embedding Cross-Modal Retrieval

Open paper AskPDF arXiv

Cross-modal retrieval between food images and recipe texts is an important task with applications in nutritional management, dietary logging, and cooking assistance. Existing methods predominantly rely on dual-encoder architectures with separate image and text encoders, requiring complex alignment strategies and task-specific network designs to bridge the semantic gap between modalities. In this work, we propose SIMMER (Single Integrated Multimodal Model for Embedding Recipes), which applies Multimodal Large Language Model (MLLM)-based embedding models, specifically VLM2Vec, to this task, replacing the conventional dual-encoder paradigm with a single unified encoder that processes both food images and recipe texts. We design prompt templates tailored to the structured nature of recipes, which consist of a title, ingredients, and cooking instructions, enabling effective embedding generation by the MLLM. We further introduce a component-aware data augmentation strategy that trains the model on both complete and partial recipes, improving robustness to incomplete inputs. Experiments on the Recipe1M dataset demonstrate that SIMMER achieves state-of-the-art performance across both the 1k and 10k evaluation settings, substantially outperforming all prior methods. In particular, our best model improves the 1k image-to-recipe R@1 from 81.8% to 87.5% and the 10k image-to-recipe R@1 from 56.5% to 65.5% compared to the previous best method.

Momentum

Related topics

Latest papers 134