Photo-based estimation of food energy could ease dietary assessment, and general-purpose multimodal large language models (LLMs) make such estimates easy to obtain. Most evaluations use one photo taken before eating and compare with energy served, while dietary assessment needs energy eaten. We evaluated eight models from three vendors (Claude Haiku, Sonnet and Opus 5.5; GPT-6 luna, 6.1 sol and 6 astra; Gemini 3.8 Flash and 4 Argon) on 802 meals from ACETADA, a controlled feeding trial with weighed intake, with the before photo only, before and after photos in separate calls (subtraction), or both photos in one request. Three results ran against intuition. Subtraction under-estimated intake for six of eight models, because the models still report hundreds of kilocalories for a nearly empty tray. Scaling the before-only estimate by the average share eaten (0.87) matched much of the gain from the after photo; beyond that constant, the second photo lowered meal-level error by only 2.2 to 4.4 points, mainly in meals with food left. With one photo, two models had a near-zero intake bias only because they under-estimated the served energy and ignored uneaten food. In ablations, adding the after photo improved accuracy under both prompt formats for seven models, while prompt effects varied across models. Over 148 feeding days, five models kept the 90% confidence interval of their mean error within a pre-set 10% margin; one model also met the pre-set 30% limits of agreement, narrowly (-28% to +27%; limits outside 30% cannot be excluded), and the other seven did not. Protein was under-estimated by every model, most in protein-dense meals; repeated calls changed error by at most 1.4 points; a high reasoning effort helped only the smallest model. A low average bias is not evidence of accurate individual intake estimation; such estimates should support, not replace, assessment by a dietitian.
Figures & tables
Value
Meals (participants)
802 (152)
Energy eaten per meal, median (IQR), kcal
622 (463–857)
— breakfast / lunch / dinner (n)
524 (286) / 623 (257) / 763 (259)
Meals with energy served defined
764
Share of energy left per meal, median (IQR)
5.9% (1.7–16.5%)
Meals with ≥10% of energy left
292
Table 1
Model
Before only
Before × c
Subtraction
Both photos
Both photos, valid subset (n)
Δ vs before only
Δ vs before × c
Δ vs before × c, ≥10% left
Haiku 5.5
24.7 (−6.7)
27.3
36.5 (−33.5)
24.9 (−14.0)
24.1 (800)
+0.1 [−1.9, +2.3]
−2.5 [−4.2, −0.6]
−1.8 [−4.4, +0.6]
Sonnet 5.5
24.2 (+0.9)
24.9
25.1 (−18.4)
21.8 (−2.9)
19.3 (789)
−2.4 [−5.3, +0.2]
−3.0 [−5.5, −0.8]
−1.3 [−5.9, +2.7]
Opus 5.5
30.1 (+21.7)
22.7
21.2 (+1.8)
20.5 (+9.0)
20.5 (801)
−9.6 [−13.2, −6.5]
−2.2 [−4.8, +0.3]
−7.5 [−12.3, −3.3]
GPT-6 luna
23.9 (+4.3)
23.7
26.2 (−17.6)
21.0 (−6.1)
21.0 (802)
−2.8 [−5.0, −1.0]
−2.6 [−4.2, −1.1]
−1.3 [−5.1, +1.9]
GPT-6.1 sol
26.9 (+14.8)
22.3
23.7 (−10.7)
19.5 (+1.2)
19.1 (801)
−7.4 [−10.3, −4.6]
−2.8 [−4.8, −0.7]
−6.6 [−10.6, −3.1]
GPT-6 astra
27.6 (+16.4)
22.4
22.5 (−7.7)
19.0 (+5.3)
19.0 (802)
−8.6 [−11.3, −6.1]
−3.4 [−5.3, −1.5]
−8.8 [−12.7, −5.1]
Table 2
Model
(1) Before, simple prompt
(3b) Before, structured prompt
(3a) Both photos, simple prompt
(3) Both photos, structured prompt
Photo effect, structured prompt (3) − (3b)
Photo effect, simple prompt (3a) − (1)
Prompt effect, before only (3b) − (1)
Prompt effect, both photos (3) − (3a)
Haiku 5.5
24.7 (−6.7)
26.2 (−17.8)
25.0 (−14.3)
24.9 (−14.0)
−1.3 [−2.8, +0.4]
+0.2 [−1.9, +2.2]
+1.5 [−0.3, +3.2]
−0.1 [−1.6, +1.4]
Sonnet 5.5
24.2 (+0.9)
26.9 (−1.3)
18.9 (−8.0)
21.8 (−2.9)
−5.1 [−9.3, −1.3]
−5.3 [−8.1, −3.0]
+2.7 [−1.3, +6.9]
+2.9 [+1.2, +4.6]
Opus 5.5
30.1 (+21.7)
22.9 (+9.2)
20.3 (+6.3)
20.5 (+9.0)
−2.5 [−4.4, −0.6]
−9.7 [−13.4, −6.5]
−7.1 [−9.8, −4.9]
+0.1 [−0.7, +1.1]
GPT-6 luna
23.9 (+4.3)
23.5 (+3.7)
20.7 (−7.1)
21.0 (−6.1)
−2.5 [−4.1, −1.0]
−3.1 [−5.1, −1.4]
−0.4 [−1.5, +0.7]
+0.3 [−0.7, +1.3]
GPT-6.1 sol
26.9 (+14.8)
26.2 (+17.0)
18.7 (−2.2)
19.5 (+1.2)
−6.7 [−9.6, −3.8]
−8.2 [−11.1, −5.6]
−0.7 [−1.5, +0.2]
+0.8 [−0.2, +2.1]
GPT-6 astra
27.6 (+16.4)
27.2 (+19.1)
18.4 (+1.6)
19.0 (+5.3)
−8.1 [−10.9, −5.6]
−9.2 [−12.1, −6.6]
−0.5 [−1.4, +0.4]
+0.6 [+0.0, +1.2]
Table 3
Model
Estimate − served
Energy left
Estimate − eaten
Meals with ≥10% left where the two parts offset*
Median offset, kcal
Haiku 5.5
−156
+103
−53
178 / 292
153
Sonnet 5.5
−104
+103
−1
146 / 292
147
Opus 5.5
+38
+103
+141
61 / 292
126
GPT-6 luna
−80
+103
+23
141 / 292
153
GPT-6.1 sol
−8
+103
+95
84 / 292
141
GPT-6 astra
+4
+103
+107
88 / 292
126
Table 4
Model, input
Mean day error %
90% CI
Equivalence
95% LoA %
Days within 800 kJ %
Haiku 5.5, before only
−4.2 [−8.9, +0.7]
−8.1, −0.3
yes
−48 to +39
34
Haiku 5.5, both photos
−12.0 [−16.2, −7.2]
−15.7, −8.2
no
−53 to +29
29
Sonnet 5.5, before only
+5.5 [−0.7, +12.3]
+0.1, +11.0
no
−62 to +73
37
Sonnet 5.5, both photos
−0.6 [−4.2, +3.2]
−3.7, +2.5
yes
−41 to +40
41
Opus 5.5, before only
+25.4 [+19.4, +32.3]
+20.3, +31.0
no
−32 to +83
25
Opus 5.5, both photos
+12.0 [+8.7, +15.5]
+9.2, +14.8
no
−25 to +49
43
Table 5
Figure 1: Day-level Bland–Altman plots for energy eaten, both photos in one request. Each point is one participant’s feeding day. Solid and dashed orange lines: mean error and 95% limits of agreement. Grey lines: the pre-set ±10% (solid) and ±30% (dotted) limits. Black dashed line: local-linear smoother.
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker. Multimodal LLMs (MLLMs) recognize a wide range of foods zero-shot in uncontrolled photos, yet they are weak at portion estimation -- a gap we measure across the current frontier (Gemini, GPT, and Claude flagships alike). We present a method that enhances a frozen, commercial MLLM with an accurate portion head: a small geometry-enhanced network on a frozen DINOv2 backbone with a structured softmax-ownership volume, consuming the MLLM's per-food name, bounding box, and density range -- no depth sensor, no MLLM fine-tuning. Evaluated fully open-vocabulary on three real-world benchmarks, the head cuts per-food portion error by 33-41% relative to the MLLM alone, outperforms every flagship MLLM's direct estimates, and surpasses each benchmark's originally published image-only model at its own reported metric.
Multimodal Large Language Models (MLLMs) are increasingly used for dietary assessment from meal images, where retrieval-augmented grounding was shown to sharpen nutrition estimates. However, we find this premise no longer holds for current MLLMs. A modern MLLM's direct estimate now matches or surpasses the full retrieval pipeline. This raises a question: if retrieval no longer improves the overall estimate, can it still deliver the two things clinicians value, accurate portions and a traceable, item-by-item record? We pursue this while preserving what matters for clinical adoption: minimal user burden (a single, unannotated meal image), explainability (an auditable record), and privacy (locally hosted inference). We introduce Open-KNEAD, a knowledge-grounded agentic framework for meal nutrition estimation that is training-free and locally deployable. Each decomposed food item is grounded to a Food and Nutrient Database for Dietary Studies (FNDDS) code via selective, nutrient-aware retrieval, composing an auditable per-item record. Across two open MLLM families and three cuisines, Open-KNEAD improves portion estimates over both prior grounding methods and direct estimation in most backbone-dataset settings. An agent-internal recipe-prior step further recovers the invisible cooking-added energy that biases estimates on non-US cuisine. The advantage is largest on the dietitian-verified ACETADA dataset, where the local open agent surpasses the direct portion estimates of two frontier closed models by roughly 30% and 53%, all while keeping every meal image on local hardware. We release the Open-KNEAD framework and its agent-ready FNDDS knowledge base.
Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.
Jingbo Yue, Bruce Coburn, Jinge Ma +2
Elmore Family School of Electrical and Computer Engineering, Purdue University West Lafayette, IN, USA