TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods

Anna T. Thomas; Sohum Patnaik; Caroline Cotto; Benjamin Sanchez-Lengeling

cs.AIOct 1, 2026

TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods

Authors: Anna T. Thomas, Sohum Patnaik, Caroline Cotto, Benjamin Sanchez-Lengeling

Organizations: Stanford Computer Science · Food Intelligence Lab · NECTAR · University of Toronto Chemical Engineering

Abstract

Sustainable protein discovery lacks the fast computational proxies, analogous to molecular docking or density functional theory, that accelerate drug and materials discovery. Evaluating whether a novel food tastes like its animal-based target requires expensive human sensory panels, bottlenecking the design-build-test loop. We introduce TasteBench, a multimodal benchmark and privacy-preserving competition for sensory prediction, spanning two tasks: a food-level ranking task built on 21K+ human evaluations across 215 plant-based foods in 24 product categories, yielding 935 within-category ranking pairs, and a supporting molecular-level taste classification task over 15K flavor molecules. To enable rigorous interpretation of model performance, we characterize the ground truth: inter-rater agreement among panelists is low (Krippendorff's $α= .077$ ), and the split-half reliability ceiling of panel-aggregated rankings is .825, establishing the range within which ML systems on this benchmark should be assessed. We evaluate baselines across four input modalities; on the same pairs panelists rated, the best model achieves .661 pairwise accuracy, competitive with the median individual panelist (.650), and .683 across all within-category pairs. TasteBench provides the evaluation infrastructure and baselines for measuring progress on computational screening for sustainable protein discovery.

Figures & tables

Figure 1: Overview of TasteBench , a multimodal benchmark built on 21k+ human evaluations across 215 plant-based foods in 24 product categories, and 15K flavor molecules.

Figure 2: Models perform competitively with small human panels . Pairwise accuracy vs. panel size (left) and discrete comparison (right, Table 1 ).

Model	Pw. Acc.	$\rho_{\mathrm{S}}$	$\tau_{\mathrm{K}}$	R@1	R@2	R@3
Gemini 3.1 Pro	.654 † [.619,.713]	.370 † [.250,.509]	.298 † [.179,.409]	.271 † [.167,.424]	.514 [.476,.701]	.578 † [.491,.786]
Qwen 3.5 397B	.630 † [.591,.688]	.320 † [.203,.459]	.255 † [.141,.365]	.229 † [.132,.354]	.354 † [.215,.483]	.500 † [.375,.656]
MMRF (cosine)	.574 [.528,.635]	.212 † [.077,.380]	.151 † [.023,.278]	.208 † [.104,.323]	.292 [.135,.396]	.333 [.181,.379]
MMRF (L2)	.571 [.524,.632]	.202 † [.064,.358]	.153 † [.031,.277]	.222 † [.104,.335]	.403 † [.285,.576]	.542 † [.438,.729]
Ridge	.579 [.534,.632]	.160 [.028,.302]	.115 † [ $-.005$ ,.226]	.250 † [.181,.435]	.292 [.118,.396]	.542 † [.493,.749]
Bradley–Terry	.610 † [.571,.661]	.266 † [.156,.402]	.178 † [.064,.287]	.250 † [.181,.422]	.292 [.125,.396]	.375 [.187,.458]

Table 2: Ensembles of supervised models and LLMs outperform the median human panelist. Ranking metrics on 215 NECTAR plant-based products (935 within-category pairs; LOOCV) with 95% BCa CIs. Bold = best in column; † = CI overlaps the leader (no significant difference at 95%)

Model	Accuracy	Precision	Recall	F1	AUROC
Reported by Zimmermann et al. (2025)
XGBoost (fp)	.899	.817	.740	.766	.852
XGBoost (fp+desc)	.896	.884	.740	.778	.851
Balanced RF (fp)	.797	.585	.732	.601	.839
FART	.894 † [.880,.906]	.879 [.675,.897]	.739 † [.668,.869]	.774 † [.671,.875]	.974 † [.968,.979]
This work

Table 3: Taste classification on the FART test split (2,254 molecules). Baselines from Zimmermann et al. (2025) ; 95% BCa CIs (10,000 resamples). Bold = best; † = CI overlaps leader. Per-class breakdown in Table 13 .

Model	C $=$ FART	C $=$ GNN	C $=$ ChemBERTa-2	$\Delta_{\text{GNN}}$	$\Delta_{\text{Chem}}$
Unsupervised
MMRF (cosine)	.574 [.528,.635]	.566 [.518,.624]	.573 [.526,.633]	$-$ .008	$-$ .002
MMRF (L2)	.571 [.524,.632]	.563 [.518,.619]	.568 [.522,.628]	$-$ .008	$-$ .002
Supervised – linear
Ridge	.579 [.534,.632]	.561 [.513,.614]	.598 [.554,.655]	$-$ .018	$+$ .019
Supervised – pairwise

Table 4: Molecular-encoder quality does not transfer to product-level ranking. Pairwise accuracy (935 within-category pairs, LOOCV), compound block swapped, all else fixed. 95% BCa CIs (10,000 resamples).

\Delta

relative to FART; all lie inside their CIs.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Hierarchy of cost in sustainable protein evaluation.

Panel of Figure 4 — Figure 4: Prompt template for LLM-based models. Feature ablation removes the corresponding XML blocks.

Group	Features
Meat (14 cat.)	Total Fat, Sodium, Protein, Dietary Fiber
Non-Sweet Dairy (6 cat.)	Total Fat, Sodium
Cheese (2 cat.)	Total Fat, Sodium, Total Carbs
Sweet Dairy (2 cat.)	Total Fat, Total Sugars

Appendix

Table 5: Nutrition features by category group. All values normalized to per-100g serving.

Model	Micro	Macro	$\Delta$	Partial pooling (EB)
Ensemble (BT $+$ Gemini)
NNLS	.683	.641	$-$ .042	.682
Rank average	.671	.629	$-$ .041	.668
Mean	.623	.595	$-$ .028	.613
Zero-shot LLMs
Gemini 3.1 Pro	.654	.633	$-$ .021	.666

Appendix

Table 14: Category-balanced evaluation preserves the ranking and the best model. Pairwise accuracy on the 935 within-category pairs under three aggregation schemes. Micro is the paper’s reported metric (Table 2 ); macro weights each of the 24 categories equally; EB is DerSimonian–Laird partial pooling toward the precision-weighted random-effects grand mean, with each category’s sampling variance taken from its product count.

\Delta

=

macro

-

micro. Point estimates only: intervals for the macro and EB columns come from a within-category percentile bootstrap rather than the BCa procedure used elsewhere in the paper, so they are not comparable to the intervals in Table 2 and are omitted here. Spearman correlation between the macro and micro orderings is .881.

Min. products	Categories kept	Micro	Macro	Rank (micro / macro)
5	24	.683	.641	1 / 1
6	21	.689	.661	1 / 1
7	20	.694	.674	1 / 1
8	18	.694	.672	1 / 1
9	16	.701	.682	1 / 1
10	12	.700	.674	1 / 1

Appendix

Table 15: Dropping small categories does not change the best model. The best model (NNLS over Bradley–Terry and Gemini 3.1 Pro) as categories below each product-count threshold are excluded. Rank is among the twelve models of Table 14 , under micro and macro averaging respectively. The smallest category has 5 products, so the first row is the full benchmark.

Quantity	Value
Panel
Panelist vs. another single panelist, pairwise ordering	.547
Median panelist vs. panel mean, within block	.650 (IQR .500–.750, $n=627$ )
Krippendorff’s $\alpha$ (ordinal, per block)	.077 (mean)
Kendall’s $W$ (rank concordance, per block)	.108 (median)
Split-half reliability of the panel mean	.825

Appendix

Table 16: The model against individual panelists, on matched pairs. Agreement quantities for the panel and for the best model. The panelist rows are computed on the within-block pairs each panelist actually rated; the model is scored on those same pairs in the first model row, and on all 935 within-category pairs in the second. The category-balanced row is the partial-pooled figure from Table 14 . Rows are not mutually comparable: the panelist median is a per-panelist quantity on within-block pairs, not a category-macro one.

Signature	Prevalence	$\Delta$ accuracy (flagged $-$ clean)	Fisher $p$
Product identification	12.5%	$+$ .042	.403
Brand mention (mostly analogy)	10.2%	$+$ .013	.820
External-performance recall	2.4%	$-$ .026	.821
Any signature	14.2%	$+$ .032	.489

Appendix

Table 17: Recognition rarely surfaces, and it does not track accuracy. Prevalence of each signature in the saved reasoning for the ingredients-plus-image configuration, and the accuracy difference between the pairs where it appears and the rest. Scored on the 931 pairs for which a graded judgment and a Bradley–Terry comparison both exist: of the 935 within-category pairs, three have tied panel means and are not gradable, and one more is unmatched. Fisher’s exact test, two-sided; no signature reaches significance. Signatures are not mutually exclusive.

Split	$n$	Gemini $-$ Bradley–Terry
Recognized	116	$+$ .026
Not recognized	815	$+$ .052
Interaction		$-$ .026, 95% CI [ $-$ .159, $+$ .108] (n.s.)

Appendix

Table 18: The LLM’s margin is not concentrated where it recognized the product. Per-pair margin of Gemini 3.1 Pro over Bradley–Terry, split by whether the reasoning identified the product. Bradley–Terry sees only features, so it holds pair difficulty constant. Over these 931 pairs Gemini scores .662 and Bradley–Terry .613, a margin of

+

.048; both differ slightly from Table 2 , which scores the same models over all 935 pairs including the three ties. Bradley–Terry also improves on the recognized subset (.605 to .672), which is what identifies those pairs as easier rather than leaked. The interaction interval is a within-category bootstrap.

Arm	Original	Brand-blind	$\Delta$ acc.	Flip rate	McNemar $p$	Brand mentions
Gemini 3.1 Pro	.639	.634	$-$ .005	12.6%	.712	10.8% $\to$ 1.7%

Appendix

Table 19: Forbidding brand reasoning barely moves accuracy. Original versus brand-blind system prompt, both re-run fresh in the same session so that model drift cancels; the user prompt is byte-identical between arms, and 932 pairs are scored in both. Flip rate is the fraction of pairs whose predicted winner changes. The brand-mention rate is measured on the same reasoning scan as Table 17 .

Figure 5: Worked example of the ensemble pipeline for one plant-based burger (Product A) in the Burgers category ( $n=10$ ). All four modalities feed the supervised Bradley–Terry scorer through encoding and fusion; the non-molecular modalities are also read in raw form by the zero-shot LLM scorer (Gemini 3.1 Pro). The two scores are combined via non-negative least squares to produce the predicted within-category ranking.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 391 papers

FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images
Apr 15, 2026Sabab Ishraq, Aarushi Aarushi, Juncai Jiang +1VLM Evaluation
College of Engineering and Computer Science, University of Central Florida, Orlando, FL, USA · College of Business Administration, University of Central Florida, Orlando, FL, USA · Institute of Artificial Intelligence, University of Central Florida, Orlando, FL, USA
TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts
Oct 5, 2026Oliver Jaffe, Dane SherburnLLM Evaluation AI Agent Benchmarks
P-Zero Research
FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models
May 29, 2026Mingyang Mao, Bhargav Rishi Medisetti, Utkarsh Grover +4Healthcare Clinical Decision Support
Department of Electrical Engineering, University of South Florida · Muma College of Business, University of South Florida · Computer Science, University of Copenhagen

Apr 15, 2026J/K move · Enter open · S save

FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images

Sabab Ishraq, Aarushi Aarushi, Juncai Jiang, Chen Chen

College of Engineering and Computer Science, University of Central Florida, Orlando, FL, USA · College of Business Administration, University of Central Florida, Orlando, FL, USA · Institute of Artificial Intelligence, University of Central Florida, Orlando, FL, USA

VLM Evaluation

Open paper AskPDF arXiv

Humans routinely infer taste, smell, texture, and even sound from food images a phenomenon well studied in cognitive science. However, prior vision language research on food has focused primarily on recognition tasks such as meal identification, ingredient detection, and nutrition estimation. Image-based prediction of multisensory experience remains largely unexplored. We introduce FoodSense, a human-annotated dataset for cross-sensory inference containing 66,842 participant-image pairs across 2,987 unique food images. Each pair includes numeric ratings (1-5) and free-text descriptors for four sensory dimensions: taste, smell, texture, and sound. To enable models to both predict and explain sensory expectations, we expand short human annotations into image-grounded reasoning traces. A large language model generates visual justifications conditioned on the image, ratings, and descriptors. Using these annotations, we train FoodSense-VL, a vision language benchmark model to produce both multisensory ratings and grounded explanations directly from food images. This work connects cognitive science findings on cross-sensory perception with modern instruction tuning for multimodal models and shows that many popular evaluation metrics are insufficient for visually sensory inference.

Abstract

Benchmark Design

Explore similar work

FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images

TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models