cs.CLMar 8, 2026
SaveBenchmark for Assessing Olfactory Perception of Large Language Models
Organizations: Department of Electrical Engineering, Yale University, New Haven, CT, USA · Human Nature Lab, Yale University, New Haven, CT, USA · Patina, New York, NY, USA · Department of Ecology, University of California, Santa Barbara, CA, USA
Abstract
Here we introduce the Olfactory Perception (OP) benchmark, designed to assess the capability of large language models (LLMs) to reason about smell. The benchmark contains 1,010 questions across eight task categories spanning odor classification, odor primary descriptor identification, intensity and pleasantness judgments, multi-descriptor prediction, mixture similarity, olfactory receptor activation, and smell identification from real-world odor sources. Each question is presented in two prompt formats, compound names and isomeric SMILES, to evaluate the effect of molecular representations. Evaluating 21 model configurations across major model families, we find that compound-name prompts consistently outperform isomeric SMILES, with gains ranging from +2.4 to +18.9 percentage points (mean approx +7 points), suggesting current LLMs access olfactory knowledge primarily through lexical associations rather than structural molecular reasoning. The best-performing model reaches 64.4% overall accuracy, which highlights both emerging capabilities and substantial remaining gaps in olfactory reasoning. We further evaluate a subset of the OP across 21 languages and find that aggregating predictions across languages improves olfactory prediction, with AUROC = 0.86 for the best performing language ensemble model. LLMs should be able to handle olfactory and not just visual or aural information.
Figures & tables
| Work | Input | Specific Tasks | What is Tested |
| Chemistry LLM benchmarks | |||
| ChemBench Mirza et al. (2025) | Chemistry questions | General, organic, analytical, toxicity chemistry (2,700 Q) | Structural & reaction knowledge |
| ChemIQ Runcie et al. (2025) | SMILES | Carbon counting, ring counting, SMILES-to-IUPAC, NMR elucidation, reaction prediction (816 Q) | Molecular comprehension |
| LlaSMol Yu et al. (2024) | SMILES/SELFIES | Name conversion, property prediction, synthesis (14 tasks, 3M samples) | Fine-tuned LLM chemistry |
| LLMs and Sensory Perception | |||
| Marjieh et al. Marjieh et al. (2024) | Sensory stimuli | Pairwise similarity judgments across pitch, loudness, color, consonants, taste, timbre | 6 modalities (olfaction excluded) |
Table 1: Comparison of related work on LLMs, chemistry, and olfaction.
| Question Category | Options | n | Note | Type | Reference |
|---|---|---|---|---|---|
| Odor Classification | Odorous/Odorless | 175 | Molecular Weight 350.0 50/50 | Pure | Mayhew et al. (2022) |
| Odor Descriptor | 1 answer, 3 distractors | 175 | Stratified to represent less dominant descriptors, 29 Descriptors total | Pure | https://d3t14p1xronwr0.cloudfront.net/docs/ifra-fragrance-ingredient-glossary-april-2020.pdf (2020) |
| Odor Intensity | High/Low | 175 | Above and below median, with score difference of 10 | Pure | Keller et al. (2017) |
| Odor Pleasantness | High/Low | 175 | Above and below median, with score difference of 10 | Pure | Keller et al. (2017) |
| Rate-all-that-apply | 2-5 answers from 138 distractors | 100 | 25 questions for 2,3,4, and 5 descriptors | Pure | Lee et al. (2023) |
| Odor Similarity of Mixtures | Rater scale | 100 | Mixtures with 2-10 molecules | Mixture | Snitz et al. (2013) ; Bushdid et al. (2014) ; Ravia et al. (2020) |
Table 2: Question categories and data sources used in the benchmark.
Figure 1: Olfactory Perception (OP) benchmark task description and output format across tasks.
| Simple ( =700) | Intermediate ( =200) | Hard ( =110) | ||||||||
| Model | Reasoning | OC | OPD | OIn | OPl | RATA | OS | ORA | SIT | Overall |
| Closed-Source | ||||||||||
| GPT-5 | high | 89.7 | 73.7 | 66.3 | 71.4 | 36.4 | 34.0 | 40.8 | 76.7 | 61.1 |
| GPT-5 | low | 90.3 | 72.0 | 70.9 | 71.4 | 30.8 | 28.0 | 40.4 | 73.3 | 59.6 |
| GPT-5 Pro | high | 92.0 | 73.1 | 71.4 | 70.9 | 36.1 | 29.0 | 42.5 | 80.0 | 61.9 |
| GPT-5.2 Pro | high | 88.6 | 76.0 | 72.6 | 72.0 | 34.6 | 28.0 | 52.8 | 73.3 | 62.2 |
Table 3: Olfactory Perception (OP) benchmark results using compound name prompts. Accuracy (%) is reported per task and overall (unweighted mean across tasks). Abbreviations: OC = Odor Classification ( =175), OPD = Odor Primary Descriptor ( =175), OIn = Odor Intensity ( =175), OPl = Odor Pleasantness ( =175), RATA = Rate-All-That-Apply ( =100), OS = Odor Similarity ( =100), ORA = Olfactory Receptor Activation ( =80), SIT = Smell Identification Test ( =30). Bold = best overall; underlined = second best.
Figure 2: Olfactory Perception (OP) benchmark performance (a) Two bars per model show overall mean accuracy across all categories for isomeric SMILES and compound name prompts. Multi - answer categories use multilabel F1; all others use any - overlap accuracy. Error bars indicate 95% confidence intervals of the cross - category mean. Models are grouped by LLM family with shaded bands and dashed boundaries; labels use shortened model names. (b) Eight subplots show per - category accuracy for each model, with separate isomeric SMILES and compound - name prompts. Horizontal error bars are bootstrap standard deviations of the mean accuracy. Each panel includes a category - specific chance baseline shown by gray dashed line.
Figure 3: Predictions correlations and performance difficulty across olfactory tasks. (a) Pearson correlations across models for three continuous - rating categories (odor intensity, odor pleasantness, and mixture similarity). Each row shows the distribution of model correlations for isomeric SMILES and Compound name prompts; gray points represent individual models, and the best - performing model per prompt/category is highlighted with a colored circle. State - of - the - art reference performance is indicated by red dashed lines. (b) Combined label - difficulty ranking for RATA (blue) and olfactory - receptor activation (red). For each label, per - model F1 scores are computed in a multilabel setting; the bar shows the mean F1 across models with an error bar (standard deviation), while points represent individual model values.
Figure 4: Multilingual RATA performance for compound name prompt. (a) Per-language distributions of model performance, summarized as mean per-question multilabel F1, colored points denote individual models and the black star denotes a cross-model ensemble. (b) DeepSeek (32K) AUROC by language family using per-label vote fractions across languages within each family (East Asian ans Others are not same language families), black curve represent an ensemble of all languages. (c) AUROC for each model using an all-language majority vote ensemble, plus a cross-model ensemble pooling all models from all languages. Dashed diagonal indicates chance.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Question difficulty distribution. Each ridge shows the density of questions at a given difficulty level, measured as the percentage of the 21 evaluated models answering correctly (compound name prompts); ridge heights are proportional to the number of questions per category and annotated numbers indicate bin counts. OC clusters at the right (easy), OS at the left (hard), and OIn/OPl exhibit bimodal distributions with mass at both extremes
Figure 6: RATA and ORA question-level F1 difficulty (compound name prompts, top 4 models per task). Left: Histograms of per-question F1 scores. RATA shows a roughly bell-shaped distribution peaking at 0.4–0.6, while ORA is bimodal with clusters at F1 = 0 and F1 > 0.6. Right: Box plots summarize the F1 distributions; diamonds indicate means. The bimodal ORA pattern indicates that models either possess the relevant receptor–ligand knowledge or lack it entirely.
Figure 7: Confusion matrices for odor similarity predictions (compound name prompts). Each matrix shows row-normalized percentages (true label predicted label). Models are ordered by overall accuracy (18–35%). Claude models exhibit extreme “Slightly Dissimilar” bias: Claude Opus 4.5 predicts this category for 72–96% of questions regardless of ground truth (third column). In contrast, DeepSeek (8K) achieves the highest accuracy by distributing predictions more broadly, including 44% correct on “Strongly Dissimilar.” No model exceeds 12% accuracy on either Similar category (top two rows, diagonal cells), indicating that predicting perceptual similarity remains beyond current LLM capabilities.
| Similar | Dissimilar | |||||
| Model | SS (/25) | SlS (/25) | Combined | SlD (/25) | SD (/25) | Overall |
| Closed-Source | ||||||
| GPT-5 (high) | 2 | 2 | 4/50 (8%) | 14 | 16 | 34% |
| GPT-5 (low) | 2 | 5 | 7/50 (14%) | 12 | 9 | 28% |
| GPT-5 Pro | 2 | 2 | 4/50 (8%) | 9 | 16 | 29% |
| GPT-5.2 Pro | 2 | 1 | 3/50 (6%) | 10 | 15 | 28% |
Table 4: Odor Similarity (OS) per-subcategory accuracy using compound name prompts. The ground truth is balanced with 25 questions per category (100 total). SS = Strongly Similar, SlS = Slightly Similar, SlD = Slightly Dissimilar, SD = Strongly Dissimilar. Combined Similar accuracy is computed as (SS + SlS correct) / 50. Chance level is 25% (random selection among four equiprobable categories). No model exceeds chance on the combined Similar categories.
Figure 8: Per-question model accuracy versus molecular overlap for Odor Similarity. Each point represents one of 100 mixture pairs, colored by ground-truth category (SS = Strongly Similar, SlS = Slightly Similar, SlD = Slightly Dissimilar, SD = Strongly Dissimilar). Dashed line indicates chance (25%). Models achieve above-chance accuracy on “Strongly Similar” only when mixtures share 9 molecules (top right). For mixtures with 0–2 shared molecules, accuracy on Similar categories (blue points) clusters near 0%, while Dissimilar categories (orange/red) achieve 30–80%. This reveals that models use molecular overlap as a proxy for perceptual similarity; they cannot recognize that perceptually similar mixtures may share few or no molecules.
Figure 9: Representative failure case on Odor Similarity (OS). Ground truth indicates “Strongly Similar” (perceptual distance = 0.23) despite only 15% molecular overlap. Claude Opus 4.6 predicts “Slightly Dissimilar” (distance = 0.58–0.62) on both prompt formats, revealing systematic reliance on molecular composition rather than perceptual integration.
Figure 10: Case study: D296N knowledge gap in olfactory receptor activation (ORA). GPT-5.2 Pro achieves perfect F1 on both prompts, while Claude Opus 4.6 max systematically excludes hOR2W1_D296N due to an incorrect loss-of-function assumption. Both models share the same initial misconception, but only GPT-5.2 Pro self-corrects during reasoning. Question compound: Methyl butanoate (Isomeric SMILES: CCCC(=O)OC ).
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,355 papers