Single-Image Food Mass Estimators Do Not Transfer Their Gram Scale Across Datasets

Yongsheng Lian

cs.CVOct 21, 2026

Single-Image Food Mass Estimators Do Not Transfer Their Gram Scale Across Datasets

Authors: Yongsheng Lian

Organizations: University of Louisville, Louisville, KY, USA

Abstract

Estimating food mass from a photograph is central to image-based dietary assessment, but whether trained models transfer to new cameras and foods is unclear. We fine-tuned DINOv2-base on 2,253 weighed Nutrition5k dishes to predict mass from one RGB photograph and compared it with a zero-shot vision-language model, Qwen3-VL-32B, on Nutrition5k and three external datasets: ASA24 studio portions, ECUSTFD whole foods and Food Portion Benchmark meal trays. In domain, the fine-tuned model's mean absolute error was 24.8 g, less than half of Qwen's. On external datasets it did not beat Qwen. For a food it had never seen, it ranked portions by size, but no better than a count of food pixels. Its gram scale was wrong in two ways: it changed with how much of the scene the photograph showed, and it was off by a different factor for each food. Training on other foods or trays from the tested dataset cut the external error by 38 to 65%. DINOv2-large, a larger backbone trained on the same dishes, did not reduce the external error. One labeled photograph of the same food cut the model's ASA24 error to 28.3 g, better than that photograph achieved alone. Prediction intervals that held the true mass for 90% of Nutrition5k dishes held it for at most half of external photographs. Nutrition5k often photographed the same plate several times, so a random test split shared near-identical photographs with training and overstated accuracy. Single-image mass estimators therefore need labeled photographs from the setting where they are used.

Figures & tables

Dataset (part)	Images	Content	Mass label	Role
Nutrition5k training set	2,253	Overhead cafeteria plates	Weighed	Training (5-fold CV)
Nutrition5k validation	502	Same	Weighed	Method selection
Nutrition5k internal test	498	Same	Weighed	Supporting only; near-duplicates of training (Sec. IV-D )
ASA24 training set	5,099 (6,329)	NCI studio portion photographs	FNDDS standard weights	Training; CV grouped by food
ASA24 test	1,000	Same; 162 foods, none in the training set	FNDDS standard weights	External test; once per pre-declared model
ECUSTFD	1,429	Top views of 146 whole foods (19 types), coin for scale	Weighed per object	External test; never trained on

TABLE I: Datasets and their roles. CV, cross-validation.

Panel of Figure 2 — Fig. 1: Fine-tuning DINOv2 to predict food weight. (a) During training, DINOv2 and a small head predict the logarithm of the dish weight from the photograph. For Nutrition5k dishes, a second head predicts a depth-derived height-sum proxy; dashed elements are used only during training. Both errors are combined and all weights are updated. (b) For prediction, only the photograph is needed: it passes through the five copies trained in 5-fold cross-validation, and their log weights are averaged.

Method	N5k	ASA24	ECUSTFD	FPB
Qwen3-VL-32B	60.3	72.9	40.5	248.5
gpt-6-luna	56.5	81.6	37.3	225.6
Fine-tuned DINOv2-base	24.8	67.9	44.7	467.5
Constant prediction	116.4	121.2	69.2	373.3

TABLE II: MAE (g) on the Nutrition5k validation set and on the three external test sets without training on them. Constant, the median weight of the training set or, for ECUSTFD, of the test set itself.

Fig. 2: MAE on the four datasets; note the different scales. Each color is one model, named by the datasets whose training photographs it used. The test photographs of ASA24 and FPB show foods and trays absent from their training photographs, and ECUSTFD was never used in training.

Food	Scale ratio
Yogurt, fruit, 4-ounce container	0.21
White rice	0.36
Cream of tomato soup	0.37
Median of 162 foods	0.88
Pita bread	2.67
Saltine crackers	3.80

TABLE III: Scale ratio of the Nutrition5k-only model for single ASA24 test foods, over each food’s series of portions.

Test set	Method	Spearman	Scale ratio	MAE (g)	Rescaled MAE (g)
Nutrition5k (internal)	Qwen3-VL-32B, single prompt	0.88	1.10	54.3	54.0
	DINOv2-base, Nutrition5k only	0.98	1.00	19.7	19.7
	DINOv2-large, Nutrition5k+ASA24	0.98	1.02	17.5	17.7
ASA24	Qwen3-VL-32B, single prompt	0.76	0.72	72.9	69.3
	DINOv2-base, Nutrition5k only	0.79	0.96	67.9	68.4
	DINOv2-large, Nutrition5k+ASA24	0.90	1.02	42.0	42.2

TABLE IV: Ordering versus gram scale. Rescaled MAE, MAE after dividing every prediction by the scale ratio.

Fig. 3: A typical photograph from each dataset, with its true mass and the estimates of the Nutrition5k-only DINOv2 and Qwen. Each is the photograph whose ratio of DINOv2’s estimate to the true mass is closest to the median ratio on that dataset, so it shows a typical error rather than a failure.

Training data	Backbone	N5k	ASA24	ECUSTFD	FPB
N5k	Base	24.8	67.9	44.7	467.5
N5k + ASA24	Large	23.3	42.0 T	36.0	463.6
N5k + ASA24 + FPB	Base	24.6	–*	–	161.7 T

TABLE V: MAE (g) of fine-tuned DINOv2 by training data. T , the model was trained on photographs from that dataset; *, not scored on the test set.

Fig. 4: Per-food calibration on one ASA24 test series (brown rice, aerial view; Nutrition5k+ASA24 DINOv2-large). The middle portion (blue) serves as the labeled photograph: the model predicted 123 g for a label of 167 g, giving a correction factor $\exp(-0.75\log(123/167))=1.26$ , which is applied to the other five photographs. Labels are FNDDS standard-portion weights. Illustration: series with at least six photographs were ranked by how close their MAE reduction, with the middle portion as the labeled photograph, was to the median over series (56%); this series (58%) was the third closest and was chosen as a single, plainly visible food. In the evaluation (Table VI ) the labeled photograph is drawn at random.

	Validation		Test
$k$	Uncal.	Cal.	Uncal.	Cal.
1	37.7	28.7	42.6	28.3
2	37.7	25.4	42.6	27.3
3	37.7	25.2	43.0	26.7

TABLE VI: Per-food calibration on ASA24: MAE (g) with

k

labeled photographs per food.

Test set	Coverage (95% CI)	Width (g)	Coverage (95% CI)	Width (g)
	DINOv2-base, Nutrition5k		DINOv2-large, Nutrition5k+ASA24
Nutrition5k validation	89.8 (87.1–92.4)	101	–	–
Nutrition5k test	91.2 (88.6–93.6)	95	99.2 (98.4–99.8)	258
ASA24	32.1 (26.3–37.9)	58	88.7 (84.8–92.4)	156
ECUSTFD	49.3 (42.2–57.0)	71	91.3 (86.2–95.6)	199
FPB	0.9 (0.4–1.5)	28	7.0 (5.6–8.5)	69

TABLE VII: Coverage (%) of 90% prediction intervals, with 95% CIs, and width, the median of upper minus lower bound in grams. An interval that works as intended has coverage near 90%, with a CI that includes 90; observed coverage varies around 90 by chance. No validation value is given for the two-domain model, whose intervals were calibrated partly on the validation set.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 5: One ASA24 test series, angel food cake, shown at its native $160\times 160$ pixels. Each label is the FNDDS weight of the stated portion, not a weighing, so (d) and (e) receive the same label although they show different amounts.

Panel of Figure 13 — Fig. 6: ASA24 label construction. Agreement concerns successfully mapped positive weights; unmapped rows do not automatically exclude an image. Expanded training retains the test-food exclusions. Labels are standard-portion assignments, not physical measurements.

Setting	Value
Starting weights	DINOv2-base (86 M parameters) or DINOv2-large (300 M), as released by Meta AI
Input	RGB photograph, $448\times 336$ pixels, ImageNet normalization
Augmentation (training only)	Horizontal and vertical flips ( $p=0.5$ each); brightness $\times U(0.9,1.1)$ and $+U(-0.05,0.05)$ ; no cropping
Image features	[CLS] token concatenated with mean patch token
Prediction layers	512-unit GELU layer, dropout 0.2; linear outputs for log weight (and nutrients); linear depth output from the image features
Main loss	Smooth- $\ell_{1}$ on log weight, $\beta=0.1$

Appendix

TABLE VIII: Training settings for the DINOv2 models.

Model	MAE	$\Delta$ (95% CI)	MAE	$\Delta$ (95% CI)	$\rho_{\mathrm{food}}$	Scale	MAE	$\Delta$ (95% CI)	Scale	MAE	Scale
	Validation		ASA24				ECUSTFD			FPB
DINOv2, fine-tuned, depth (final)	24.8	—	67.9	—	0.86	0.96	44.7	—	1.02	468	0.14
DINOv2, fine-tuned, no depth	25.7	$+0.9$ (0.2, 1.6)	70.4	$+2.5$ (0.0, 5.2)	0.86	1.00	44.4	$-0.3$ ( $-1.7$ , 1.1)	0.99	468	0.14
DINOv2, frozen, depth	35.2	$+10.4$ (7.6, 13.2)	115.7	$+47.8$ (36.7, 58.5)	0.82	0.28	88.2	$+43.5$ (32.7, 54.1)	0.38	501	0.07
DINOv2, frozen, no depth	35.5	$+10.8$ (7.9, 13.6)	116.4	$+48.5$ (37.3, 59.2)	0.79	0.28	86.1	$+41.4$ (30.8, 51.8)	0.41	501	0.07
ResNet-50, depth, LR $10^{-3}$	35.2	$+10.4$ (8.0, 13.1)	78.3	$+10.4$ (3.4, 17.1)	0.80	0.82	60.0	$+15.3$ (10.6, 20.0)	1.14	471	0.14

Appendix

TABLE IX: Matched supervised baselines.

\Delta

, difference in MAE from the final model, with 95% CI;

\rho_{\mathrm{food}}

, Spearman correlation within each ASA24 food; Scale, scale ratio; LR, backbone learning rate.

Nutrition5k validation (502 dishes)
	FT, depth	FT, no depth	Frozen, depth
MAE (g)	24.8 (24.6–24.9)	25.6 (25.3–25.8)	35.0 (34.3–35.4)
ASA24 training photographs (5,099)
MAE (g)	56.3 (55.8–56.8)	57.1 (56.7–57.9)	99.9 (98.5–100.5)
Scale	0.99 (0.94–1.04)	1.03 (0.96–1.11)	0.30 (0.30–0.32)
$\rho$	0.80 (0.80–0.81)	0.80 (0.79–0.80)	0.79 (0.79–0.80)

Appendix

TABLE X: Seed controls: mean (range) over four seeds. FT, fine-tuned;

\rho

, Spearman correlation.

Method	MAE (g)	Median AE (g)	RMSE (g)	Bias (g)	Within 50 g	$\Delta$ MAE vs single-prompt Qwen (95% CI), g
Qwen3-VL-32B, single prompt	60.35	41.00	94.34	$-9.59$	60.4%	–
Qwen3-VL-32B, single prompt + kNN residual correction	47.69	28.66	73.57	$-0.50$	66.5%	$-12.66$ ( $-17.32$ , $-8.41$ )
Fine-tuned DINOv2-base, Nutrition5k only	24.78	16.02	39.35	$-4.33$	88.6%	$-35.57$ ( $-41.27$ , $-30.26$ )
Fine-tuned DINOv2-large, Nutrition5k+ASA24	23.29	14.08	38.49	$-0.48$	90.2%	$-37.06$ ( $-42.90$ , $-31.61$ )

Appendix

TABLE XI: Mass error on the Nutrition5k validation set.

\Delta

, difference in MAE from single-prompt Qwen, with 95% CI.

Available from the setting of use	Most accurate	Evidence
Nothing	A zero-shot VLM, or no clear difference	ASA24: Qwen 72.9, fine-tuned 67.9 (difference not significant). ECUSTFD: 40.5 and 44.7 (not significant). FPB: 248.5 and 467.5. Second VLM (gpt-6-luna): 81.6, 37.3 and 225.6
Labeled photographs of other foods or trays	Fine-tuned with those photographs	ASA24: 42.0 against 72.9 for Qwen. FPB: 161.7 against 248.5
The food’s identity and one labeled photograph of it	Fine-tuned with per-food correction	ASA24: 28.3, against 42.6 uncorrected and 48.0 from the labeled photograph and food area alone
Training data from the same camera and menu	Fine-tuned	Nutrition5k validation: 24.8 against 60.4 for Qwen

Appendix

TABLE XII: Most accurate estimator by the data available from the setting of use: MAE (g).

Study and method	Dishes	MAE (g)	Inputs	Coverage
DietAI24 [ 5 ]	1,000	45.1	Image; FNDDS retrieval	84.2% successful; failure handling unspecified
Open-KNEAD, Gemma-MoE [ 6 ]	Not stated	66.7	Image	Full test split
Liao and Li [ 16 ] , sum of per-food estimates	502	36.6	Image; LLM food detections	Official test split; trained per dataset
Nutrition5k RGB baseline [ 1 ]	507	40.4	Image (supervised)	Official test split
ViT pretrained on ImageNet-21k [ 9 ]	10% of plates	60.6	Side-video frames (supervised)	Own 90/10 split by plate
Swin-Nutrition [ 8 ] , as re-implemented in [ 4 ]	507	27.5	Image (supervised)	Official test split; original trained on side-angle frames and reported no grams

Appendix

TABLE XIII: Published Nutrition5k food-mass results, for context only; rows are not directly comparable.

Cross-Dataset Generalization

Cross-dataset generalization studies whether AI models retain accuracy, calibration, and usefulness on data from sources beyond their training set.

See all 142 papers

Oct 8, 2026

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.

Jingbo Yue, Bruce Coburn, Jinge Ma +2

Elmore Family School of Electrical and Computer Engineering, Purdue University West Lafayette, IN, USA

Visual Grounding

Ask PDF

Jul 17, 2026

Geometry-Enhanced Portion Estimation for Multimodal LLMs

Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker. Multimodal LLMs (MLLMs) recognize a wide range of foods zero-shot in uncontrolled photos, yet they are weak at portion estimation -- a gap we measure across the current frontier (Gemini, GPT, and Claude flagships alike). We present a method that enhances a frozen, commercial MLLM with an accurate portion head: a small geometry-enhanced network on a frozen DINOv2 backbone with a structured softmax-ownership volume, consuming the MLLM's per-food name, bounding box, and density range -- no depth sensor, no MLLM fine-tuning. Evaluated fully open-vocabulary on three real-world benchmarks, the head cuts per-food portion error by 33-41% relative to the MLLM alone, outperforms every flagship MLLM's direct estimates, and surpasses each benchmark's originally published image-only model at its own reported metric.

Lin Liao, Peng Li

Multimodal Large Language Models Multimodal Foundation Models

Ask PDF

Oct 19, 2026

Small Average Error, Wide Individual Error: General-Purpose LLMs Estimating Energy Intake from Meal Photos in a Feeding Trial

Photo-based estimation of food energy could ease dietary assessment, and general-purpose multimodal large language models (LLMs) make such estimates easy to obtain. Most evaluations use one photo taken before eating and compare with energy served, while dietary assessment needs energy eaten. We evaluated eight models from three vendors (Claude Haiku, Sonnet and Opus 5.5; GPT-6 luna, 6.1 sol and 6 astra; Gemini 3.8 Flash and 4 Argon) on 802 meals from ACETADA, a controlled feeding trial with weighed intake, with the before photo only, before and after photos in separate calls (subtraction), or both photos in one request. Three results ran against intuition. Subtraction under-estimated intake for six of eight models, because the models still report hundreds of kilocalories for a nearly empty tray. Scaling the before-only estimate by the average share eaten (0.87) matched much of the gain from the after photo; beyond that constant, the second photo lowered meal-level error by only 2.2 to 4.4 points, mainly in meals with food left. With one photo, two models had a near-zero intake bias only because they under-estimated the served energy and ignored uneaten food. In ablations, adding the after photo improved accuracy under both prompt formats for seven models, while prompt effects varied across models. Over 148 feeding days, five models kept the 90% confidence interval of their mean error within a pre-set 10% margin; one model also met the pre-set 30% limits of agreement, narrowly (-28% to +27%; limits outside 30% cannot be excluded), and the other seven did not. Protein was under-estimated by every model, most in protein-dense meals; repeated calls changed error by at most 1.4 points; a high reasoning effort helped only the smallest model. A low average bias is not evidence of accurate individual intake estimation; such estimates should support, not replace, assessment by a dietitian.

Xiaomeng Li

Link Heart Limited, Houston, TX, USA

LLM Reliability Multimodal Model Evaluation

Ask PDF

Abstract

Cross-Dataset Generalization

Explore similar work

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

Geometry-Enhanced Portion Estimation for Multimodal LLMs

Small Average Error, Wide Individual Error: General-Purpose LLMs Estimating Energy Intake from Meal Photos in a Feeding Trial