See & Sniff: Learning Visuo-Olfactory Representations
Authors: Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak
Organizations: Korea Advanced Institute of Science and Technology, Korea · Hankuk University of Foreign Studies, Korea · Ulsan National Institute of Science and Technology, Korea
Abstract
While modern multimodal models integrate vision with language, audio, or touch, olfaction remains largely unexplored due to the lack of paired visuo-olfactory data. We introduce SmellNet-V, a scalable visuo-olfactory dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category. This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory dataset into a cross-modal benchmark without costly co-collection. Building on this dataset, we propose See & Sniff, a self-supervised framework that learns joint visuo-olfactory representations via dense local alignment and naturally produces smell saliency maps for spatial grounding of odor sources. We further introduce pixel-level smell localization task and a benchmark for evaluation. Our method surpasses smell-only baselines by 7% in smell classification from smell alone and generalizes to cross-modal retrieval and smell localization, establishing visuo-olfactory learning as a new direction in multimodal perception.
Images tell us what a scene looks like, but rarely what it would feel like to be there. While recent datasets pair visual scenes with electronic-nose measurements, aligning smell signals with images remains challenging because many olfactory cues arise from contextual environmental factors that are not directly visible in pixels. We introduce SCENT, a multimodal framework that uses language guidance as a semantic bridge between vision and olfaction. Our approach leverages Vision-Language Models (VLMs) to generate scene descriptors capturing objects, environmental context, and plausible ambient smell cues suggested by the visual scene. These descriptors provide semantic guidance for learning olfactory representations. We train a smell encoder that maps electronic-nose signals into a shared embedding space aligned with both visual and textual representations, and introduce a languageguided latent decomposition that separates object-specific odors from contextual environmental contributions. Experiments on the New York Smells dataset demonstrate that SCENT significantly improves crossmodal retrieval compared to vision-only baselines, achieving state-of-theart performance on smell-to-image and smell-to-text retrieval tasks. In addition, our framework produces interpretable olfactory representations that enable the disentanglement of complex smell mixtures. Our results reveal the importance of contextual semantic information for grounding olfactory perception in multimodal learning and pave the way for future research in this area.
The Contrastive Olfaction-Language-Image Pre-training 2 (COLIP-2) model is a multimodal embeddings space that places olfaction as a first-class citizen among vision and language. Molecular structure, gas-sensor readings, odor-descriptor language, and images are all trained into a single shared representation space, so that a robot can localize a detected aroma to objects in a scene probabilistically. No ImageNet-scale datasets of paired image-scent examples exists which warrants the need for their collection. Our intent with the release of COLIP-2 is to demonstrate the limit of what can be built for robotics with open-sourced olfactory data in order to ground the argument for why new methodologies and datasets are necessary in order to enable advanced olfactory-oriented perception capabilities. We enumerate results from internal testing of the COLIP-2 architecture and make necessary optimizations to run the model at the edge for real-time robotics applications. While developed with robotics in mind, the design of COLIP-2 has been influenced by experts across many disciplines of science in academia and industry, and we hope that the model can be useful in any multimodal domain requiring olfactory intelligence.
Olfaction is important for emotion regulation because it acts as a non-intrusive and cognitively lightweight pathway that directly engages the brain s affective circuitry and achieves unobtrusive emotional modulation. This trait is essential for advancing practical affective computing in daily and attention-critical scenarios. However, current olfactory emotion research has two key limitations. First, it overemphasises the valence dimension while neglecting arousal. Second, it lacks multimodal datasets that synchronously capture central and peripheral physiological responses to olfactory stimuli. To address these issues, we construct a large-scale multimodal olfactory emotion dataset based on 111 subjects, in which odors are labeled in the 2D arousal-valence space and electroencephalogram (EEG), electrocardiogram (ECG), and photoplethysmography (PPG) signals synchronously recorded. Nevertheless, multimodal signals present challenges such as non-stationarity, differences in latency, and cross-modal heterogeneity. Thus, we propose a spatiotemporal-frequency hybrid fusion network (STF-HFNet), which integrates three core modules. Frequency aggregation processing learns adaptive frequency aggregation in order to model non-stationary dynamics. Reciprocal guided attention enables reciprocal bidirectional calibration for cross-modal temporal alignment without synchronisation priors. Hybrid collaborative fusion combines spatial and channel attention mechanisms to enhance cross-modal complementarity while suppressing redundant information. Extensive experiments show that STF-HFNet achieves state-of-the-art (SOTA) recognition accuracies of 88.34% on the AMIGOS dataset and 92.40% on our self-constructed dataset, and outperform the SOTA methods by 8.27% and 5.07%, respectively.