FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images

Sabab Ishraq; Aarushi Aarushi; Juncai Jiang; Chen Chen

cs.CVApr 15, 2026

FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images

Authors: Sabab Ishraq, Aarushi Aarushi, Juncai Jiang, Chen Chen

Organizations: College of Engineering and Computer Science, University of Central Florida, Orlando, FL, USA · College of Business Administration, University of Central Florida, Orlando, FL, USA · Institute of Artificial Intelligence, University of Central Florida, Orlando, FL, USA

Abstract

Humans routinely infer taste, smell, texture, and even sound from food images a phenomenon well studied in cognitive science. However, prior vision language research on food has focused primarily on recognition tasks such as meal identification, ingredient detection, and nutrition estimation. Image-based prediction of multisensory experience remains largely unexplored. We introduce FoodSense, a human-annotated dataset for cross-sensory inference containing 66,842 participant-image pairs across 2,987 unique food images. Each pair includes numeric ratings (1-5) and free-text descriptors for four sensory dimensions: taste, smell, texture, and sound. To enable models to both predict and explain sensory expectations, we expand short human annotations into image-grounded reasoning traces. A large language model generates visual justifications conditioned on the image, ratings, and descriptors. Using these annotations, we train FoodSense-VL, a vision language benchmark model to produce both multisensory ratings and grounded explanations directly from food images. This work connects cognitive science findings on cross-sensory perception with modern instruction tuning for multimodal models and shows that many popular evaluation metrics are insufficient for visually sensory inference.

Figures & tables

Panel of Figure 1 — Figure 1: Annotation interface and example. Left: A food image as presented to participants (Taco, image 0005). Center: The structured rating task—participants rated each of four sensory dimensions on a 0–7 scale (0 = Can’t tell from picture; 1 = Very bad; 7 = Very good) and provided one to two free-text descriptors per sense. Right: Illustrative annotation for a taco image showing rescaled ratings (1–5) and representative free-text descriptors across all four sensory dimensions.

Figure 2: Dataset statistics for the Multisensory Food Dataset. (a) Distribution of annotator counts per image (mean $=22.38$ , SD $=2.02$ ). (b) Kernel density estimates of rescaled ratings (1–5) per sensory dimension, restricted to CanInfer $=1$ ratings. (c) Proportion of ratings where participants could infer each sensory property from the image. (d) Single-rater reliability ICC(1,1) with 95% confidence intervals; dotted line marks ICC $=0.20$ . (e) Aggregate label reliability ICC(1,k) with 95% confidence intervals for mean ratings across $\bar{k}\approx 21$ annotators; dotted line marks ICC $=0.70$ .

Dimension	$N$	Mean	SD	ICC(1,1) [95% CI]	ICC(1,k) [95% CI]	$\bar{k}$
Taste	63,978	3.89	1.08	0.169 [0.160, 0.178]	0.813 [0.803, 0.823]	21.42
Smell	63,259	3.83	1.05	0.189 [0.179, 0.199]	0.831 [0.822, 0.840]	21.18
Texture	64,986	3.74	1.06	0.166 [0.158, 0.175]	0.813 [0.803, 0.822]	21.76
Sound	61,972	3.44	0.99	0.101 [0.094, 0.108]	0.699 [0.683, 0.714]	20.75

Table 1: Descriptive statistics and inter-rater reliability for the full released dataset (2,987 images), computed on rescaled sensory ratings (1–5) with CanInfer

=1

per dimension. ICC(1,1): single-rater reliability; ICC(1,k): aggregate label reliability for mean of

\bar{k}

annotators per image. All ICC values significant at

p<.0001

Panel of Figure 3 — Figure 3: Pipeline. A Southern Scampi image with human sensory annotations is expanded by Gemma 3 27B IT into image-grounded rationales; Food-Llama judges and filters hallucinated content. FoodSense-VL predicts ratings and explanations from images alone. The output box shows an example texture prediction with visual justification.

	MAE $\downarrow$					RMSE $\downarrow$
Model	Taste	Smell	Tex.	Sound	Avg	Taste	Smell	Tex.	Sound	Avg
LLaVA-v1.6-34B	0.601	0.426	0.406	0.304	0.435	0.745	0.549	0.542	0.389	0.556
InternVL2.5-26B	0.635	0.423	0.663	0.305	0.507	0.787	0.543	0.802	0.390	0.630
Qwen2.5-VL-32B	0.528	0.432	0.667	0.730	0.589	0.673	0.569	0.808	0.850	0.725
Gemma 3 27B (base)	0.411	0.505	0.511	0.980	0.602	0.504	0.617	0.633	1.067	0.705
Food-LLaMA-11B	0.558	0.764	0.532	1.108	0.740	0.706	0.979	0.683	1.337	0.926

Table 2: MAE

\downarrow

and RMSE

\downarrow

by sensory dimension.

	Spearman $\rho\uparrow$					Lin’s CCC $\uparrow$
Model	Taste	Smell	Tex.	Sound	Avg	Taste	Smell	Tex.	Sound	Avg
InternVL2.5-26B	0.171	0.171	0.227	0.140	0.177	0.028	0.063	0.086	0.134	0.078
LLaVA-v1.6-34B	0.271	0.147	0.189	0.181	0.197	0.115	0.064	0.113	0.159	0.113
Qwen2.5-VL-32B	0.311	0.141	0.225	0.265	0.236	0.159	0.121	0.078	0.140	0.124
Gemma 3 27B (base)	0.196	0.178	0.140	0.208	0.181	0.159	0.151	0.181	0.053	0.136
Food-LLaMA-11B	0.085	0.036	0.108	0.127	0.089	0.074	0.010	0.097	0.040	0.055

Table 3: Spearman

\rho\uparrow

and Lin’s Concordance Correlation Coefficient (CCC)

\uparrow

by sensory dimension. Best in bold .

	Pearson $r\uparrow$					Ordinal Accuracy $\uparrow$
Model	Taste	Smell	Tex.	Sound	Avg	Taste	Smell	Tex.	Sound	Avg
InternVL2.5-26B	0.229	0.226	0.262	0.187	0.226	0.694	0.646	0.600	0.719	0.665
LLaVA-v1.6-34B	0.328	0.143	0.234	0.208	0.229	0.721	0.651	0.612	0.726	0.678
Qwen2.5-VL-32B	0.312	0.158	0.233	0.280	0.246	0.703	0.671	0.605	0.616	0.649
Gemma 3 27B (base)	0.217	0.178	0.184	0.263	0.211	0.696	0.489	0.564	0.600	0.587
Food-LLaMA-11B	0.075	0.013	0.123	0.109	0.080	0.561	0.415	0.618	0.300	0.474

Table 4: Pearson

r\uparrow

and Ordinal Accuracy

\uparrow

(3-class: Low/Med/High) by sensory dimension. Best in bold .

Setting	MAE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$	CCC $\uparrow$	$\sigma_{\text{pred}}$	Ord.Acc $\uparrow$	W/in 1 $\uparrow$
Single-Stage	0.589	0.254	0.227	0.169	0.367	0.593	0.834
Two-Stage (FoodSense-VL)	0.633	0.297	0.274	0.238	0.591	0.542	0.793
$\Delta$ (Two $-$ Single)	+0.044	+0.043	+0.047	+0.069	+0.224	$-$ 0.051	$-$ 0.041

Table 5: Ablation: Single-stage (flat) vs. two-stage (curriculum) fine-tuning.

Panel of Figure 4 — Figure 4: Qualitative sensory inferences for Steak Rice from four models. Human GT: Taste=4.3, Smell=4.3, Texture=4.4, Sound=4.1.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Value	Meaning
0	Can’t tell from picture
1	Very bad
2	Bad
3	Slightly bad
4	Neutral
5	Slightly good

Appendix

Table A1 : Seven-point Likert scale used for sensory ratings.

Figure A1 : Qualtrics survey interface as presented to participants, showing the food image display, seven-point rating scales for each sensory dimension, and free-text descriptor entry fields.

Statistic	Value
Mean annotations per image	22.38
Standard deviation	2.02
Minimum	3
Maximum	42
Total annotations (full dataset)	66,842
Total annotations (training set)	58,443

Appendix

Table B2 : Summary statistics for annotation counts across 2,987 images.

Figure B2 : Distribution of annotation counts per image across the full dataset of 2,987 images. The dashed line indicates the mean (22.38 annotations per image).

Split	Images	Annotations	% of Total
Train	2,185	43,758	74.9%
Validation	292	5,834	10.0%
Test	438	8,851	15.2%
Total	2,915	58,443	100%

Appendix

Table B3 : Dataset partitions with image and annotation counts.

Dimension	Unique Terms	Top 10 Descriptors
Taste	9,217	sweet, savory, salty, good, spicy, delicious, bland, fishy, fresh, meaty
Smell	9,161	sweet, fishy, savory, fresh, good, spicy, meaty, cheesy, fried, delicious
Texture	6,807	soft, crunchy, chewy, smooth, crispy, mushy, slimy, creamy, rough, hard
Sound	5,323	crunchy, quiet, soft, crunch, none, squishy, chewy, slurp, wet, no sound

Appendix

Table B4 : Descriptor vocabulary statistics and top-10 terms by sensory dimension.

Figure B3 : Top-10 descriptor frequencies per sensory dimension across all 2,987 images.

	Taste	Smell	Texture	Sound
Taste	—	0.893	0.931	0.759
Smell	0.893	—	0.856	0.726
Texture	0.931	0.856	—	0.840
Sound	0.759	0.726	0.840	—

Appendix

Table B5 : Pearson correlations between per-image mean sensory ratings (

N=2{,}987

; all

p<.001

Figure B4 : Heatmap of Pearson correlations between per-image mean sensory ratings ( $N=2{,}987$ ).

Dimension	Mean Unique	Mean TTR	Top-1 Coverage	Top-3 Coverage
Taste	16.04 (3.22)	0.725 (0.137)	0.218	0.396
Smell	16.75 (3.26)	0.760 (0.135)	0.200	0.367
Texture	14.60 (2.84)	0.658 (0.120)	0.245	0.457
Sound	14.43 (3.08)	0.663 (0.134)	0.232	0.446

Appendix

Table B6 : Per-image descriptor diversity and coverage statistics across 2,987 images. TTR = type-token ratio (higher = more diverse). Top-

k

coverage = fraction of annotations using one of the

k

most common descriptors for that image.

Dimension	Pearson $r$	$p$ -value
Taste	0.249	$<.0001$
Smell	$-$ 0.047	$.011$
Texture	0.141	$<.0001$
Sound	0.068	$.0002$

Appendix

Table B7 : Pearson correlations between per-image descriptor TTR and rating SD (

N=2{,}987

). Higher TTR indicates more diverse descriptors; higher rating SD indicates more numeric disagreement.

Figure C5 : Two annotation examples randomly sampled from the test set (seed = 42). Left: Image 2737 (mac and cheese). Right: Image 0768 (chocolate dessert).

Model	Taste	Smell	Texture	Sound
Human GT	0.502	0.518	0.498	0.385
FoodSense-VL (Two-Stage)	0.499	0.488	0.759	0.619
Single-Stage (lr= $2{\times}10^{-6}$ )	0.236	0.392	0.490	0.351
Gemma 3 27B (base)	0.252	0.382	0.461	0.3498
LLaVA-v1.6-34B	0.155	0.252	0.149	0.398
InternVL2.5-26B	0.083	0.075	0.245	0.181

Appendix

Table D8 : Prediction standard deviation (

\sigma_{\text{pred}}

) per sense and overall. Human GT row shows the spread of ground-truth mean ratings across the 438 test images. Bold = closest to GT spread.

Model	${\sim}1$	${\sim}2$	${\sim}3$	${\sim}4$	${\sim}5$	Entropy
Human GT	0.0%	1.5%	30.2%	63.5%	4.7%	1.239
FoodSense-VL (Two-Stage)	1.0%	8.9%	29.7%	48.5%	12.0%	1.769
Single-Stage (lr= $2{\times}10^{-6}$ )	0.1%	3.5%	30.6%	62.9%	3.0%	1.269
Gemma 3 27B (base)	0.1%	7.0%	26.3%	63.9%	2.7%	1.337
LLaVA-v1.6-34B	0.0%	0.6%	11.1%	63.9%	24.4%	1.307
InternVL2.5-26B	0.0%	0.2%	2.0%	64.3%	33.6%	1.066

Appendix

Table D9 : Distribution of predicted ratings across bins (percentage of all predictions). Entropy computed as

H=-\sum p_{i}\log_{2}p_{i}

over the 5 bins.

Model	Taste	Smell	Texture	Sound
Human GT	$3.88\pm 0.50$	$3.80\pm 0.52$	$3.74\pm 0.50$	$3.44\pm 0.39$
FoodSense-VL (Two-Stage)	$3.88\pm 0.50$	$3.48\pm 0.49$	$3.64\pm 0.76$	$2.62\pm 0.62$
Single-Stage	$3.82\pm 0.24$	$3.57\pm 0.39$	$3.81\pm 0.49$	$2.51\pm 0.35$
Gemma 3 27B (base)	$3.85\pm 0.25$	$3.56\pm 0.38$	$3.85\pm 0.46$	$2.38\pm 0.35$
LLaVA-v1.6-34B	$4.47\pm 0.16$	$3.90\pm 0.25$	$3.98\pm 0.15$	$3.20\pm 0.40$
InternVL2.5-26B	$4.49\pm 0.08$	$4.00\pm 0.07$	$4.17\pm 0.25$	$3.45\pm 0.18$

Appendix

Table D10 : Mean predicted rating

\pm

std per sensory dimension. Human GT shows the mean of annotator-averaged ratings.

	Ours	Single	Base	LLaVA	IntVL	Qwen	F-LLaMA
Ours (Two-Stage)	—	.148 ∗	.134 ∗	.312 ∗	.374 ∗	.338 ∗	.125 ∗
Single-Stage	—	—	.036	.460 ∗	.517 ∗	.487 ∗	.274 ∗
Base	—	—	—	.446 ∗	.502 ∗	.472 ∗	.259 ∗
LLaVA	—	—	—	—	.096 ∗	.071 ∗	.249 ∗
InternVL	—	—	—	—	—	.133 ∗	.345 ∗
Qwen	—	—	—	—	—	—	.213 ∗

Appendix

Table D11 : Pairwise KS

D

-statistics across all model rating distributions (all senses pooled). ∗

p<0.05

FoodSense-VL
	Sandwich				Steak Rice				Taco				Scampi				Ice Cream
	T	S	X	U	T	S	X	U	T	S	X	U	T	S	X	U	T	S	X	U
Human GT	4.1	4.1	4.3	3.6	4.3	4.3	4.4	4.1	4.5	4.4	4.5	3.7	3.7	3.5	3.5	2.7	4.1	3.3	3.7	1.8
FoodSense-VL (Stage 2)	3.5	3.0	2.5	2.0	4.0	3.5	3.0	2.5	4.2	4.0	4.5	3.8	3.5	3.0	2.5	2.0	4.5	4.0	4.5	3.5
Base (Gemma 3 27B)	3.5	3.0	2.5	1.5	3.8	3.5	3.2	2.5	4.2	3.8	4.5	2.5	3.8	3.5	3.2	2.0	4.2	3.8	4.5	2.5
Open-source baselines

Appendix

Table E12 : Sensory ratings on six representative food images (1–5 scale). Human GT (mean of

\sim

25 annotators) is shown for comparison. Best model prediction (closest to GT) per cell in bold . “–” = unavailable.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods
Oct 1, 2026Anna T. Thomas, Sohum Patnaik, Caroline Cotto +1Benchmark Design Learning to Rank
Stanford Computer Science · Food Intelligence Lab · NECTAR +1
DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain
Apr 12, 2026Song Jin, Juntian Zhang, Xun Zhang +6VLM Evaluation Fine-Grained Visual Perception
Gaoling School of Artificial Intelligence, Renmin University of China · Meituan · Wuhan University
CulturalMenuBench: Probing the Knowledge-Application Gap in Multimodal Culinary Reasoning
Sep 3, 2026Bo Zeng, Linfeng Gao, Peiqin Lin +9Multimodal Reasoning Multilingual VLM Evaluation
Alibaba Group · Xiamen University · University of Hamburg

Oct 1, 2026J/K move · Enter open · S save

TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods

Anna T. Thomas, Sohum Patnaik, Caroline Cotto, Benjamin Sanchez-Lengeling

Stanford Computer Science · Food Intelligence Lab · NECTAR+1

Benchmark Design Learning to Rank Multimodal Model Evaluation Molecular Property Prediction

Open paper AskPDF arXiv

Sustainable protein discovery lacks the fast computational proxies, analogous to molecular docking or density functional theory, that accelerate drug and materials discovery. Evaluating whether a novel food tastes like its animal-based target requires expensive human sensory panels, bottlenecking the design-build-test loop. We introduce TasteBench, a multimodal benchmark and privacy-preserving competition for sensory prediction, spanning two tasks: a food-level ranking task built on 21K+ human evaluations across 215 plant-based foods in 24 product categories, yielding 935 within-category ranking pairs, and a supporting molecular-level taste classification task over 15K flavor molecules. To enable rigorous interpretation of model performance, we characterize the ground truth: inter-rater agreement among panelists is low (Krippendorff's

α= .077

), and the split-half reliability ceiling of panel-aggregated rankings is .825, establishing the range within which ML systems on this benchmark should be assessed. We evaluate baselines across four input modalities; on the same pairs panelists rated, the best model achieves .661 pairwise accuracy, competitive with the median individual panelist (.650), and .683 across all within-category pairs. TasteBench provides the evaluation infrastructure and baselines for measuring progress on computational screening for sustainable protein discovery.

Abstract

VLM Evaluation

Explore similar work

TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods

DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain

CulturalMenuBench: Probing the Knowledge-Application Gap in Multimodal Culinary Reasoning