Small Average Error, Wide Individual Error: General-Purpose LLMs Estimating Energy Intake from Meal Photos in a Feeding Trial
Organizations: Link Heart Limited, Houston, TX, USA
Abstract
Photo-based estimation of food energy could ease dietary assessment, and general-purpose multimodal large language models (LLMs) make such estimates easy to obtain. Most evaluations use one photo taken before eating and compare with energy served, while dietary assessment needs energy eaten. We evaluated eight models from three vendors (Claude Haiku, Sonnet and Opus 5.5; GPT-6 luna, 6.1 sol and 6 astra; Gemini 3.8 Flash and 4 Argon) on 802 meals from ACETADA, a controlled feeding trial with weighed intake, with the before photo only, before and after photos in separate calls (subtraction), or both photos in one request. Three results ran against intuition. Subtraction under-estimated intake for six of eight models, because the models still report hundreds of kilocalories for a nearly empty tray. Scaling the before-only estimate by the average share eaten (0.87) matched much of the gain from the after photo; beyond that constant, the second photo lowered meal-level error by only 2.2 to 4.4 points, mainly in meals with food left. With one photo, two models had a near-zero intake bias only because they under-estimated the served energy and ignored uneaten food. In ablations, adding the after photo improved accuracy under both prompt formats for seven models, while prompt effects varied across models. Over 148 feeding days, five models kept the 90% confidence interval of their mean error within a pre-set 10% margin; one model also met the pre-set 30% limits of agreement, narrowly (-28% to +27%; limits outside 30% cannot be excluded), and the other seven did not. Protein was under-estimated by every model, most in protein-dense meals; repeated calls changed error by at most 1.4 points; a high reasoning effort helped only the smallest model. A low average bias is not evidence of accurate individual intake estimation; such estimates should support, not replace, assessment by a dietitian.
Figures & tables
| Value | |
|---|---|
| Meals (participants) | 802 (152) |
| Energy eaten per meal, median (IQR), kcal | 622 (463–857) |
| — breakfast / lunch / dinner (n) | 524 (286) / 623 (257) / 763 (259) |
| Meals with energy served defined | 764 |
| Share of energy left per meal, median (IQR) | 5.9% (1.7–16.5%) |
| Meals with ≥10% of energy left | 292 |
| Model | Before only | Before × c | Subtraction | Both photos | Both photos, valid subset (n) | Δ vs before only | Δ vs before × c | Δ vs before × c, ≥10% left |
|---|---|---|---|---|---|---|---|---|
| Haiku 5.5 | 24.7 (−6.7) | 27.3 | 36.5 (−33.5) | 24.9 (−14.0) | 24.1 (800) | +0.1 [−1.9, +2.3] | −2.5 [−4.2, −0.6] | −1.8 [−4.4, +0.6] |
| Sonnet 5.5 | 24.2 (+0.9) | 24.9 | 25.1 (−18.4) | 21.8 (−2.9) | 19.3 (789) | −2.4 [−5.3, +0.2] | −3.0 [−5.5, −0.8] | −1.3 [−5.9, +2.7] |
| Opus 5.5 | 30.1 (+21.7) | 22.7 | 21.2 (+1.8) | 20.5 (+9.0) | 20.5 (801) | −9.6 [−13.2, −6.5] | −2.2 [−4.8, +0.3] | −7.5 [−12.3, −3.3] |
| GPT-6 luna | 23.9 (+4.3) | 23.7 | 26.2 (−17.6) | 21.0 (−6.1) | 21.0 (802) | −2.8 [−5.0, −1.0] | −2.6 [−4.2, −1.1] | −1.3 [−5.1, +1.9] |
| GPT-6.1 sol | 26.9 (+14.8) | 22.3 | 23.7 (−10.7) | 19.5 (+1.2) | 19.1 (801) | −7.4 [−10.3, −4.6] | −2.8 [−4.8, −0.7] | −6.6 [−10.6, −3.1] |
| GPT-6 astra | 27.6 (+16.4) | 22.4 | 22.5 (−7.7) | 19.0 (+5.3) | 19.0 (802) | −8.6 [−11.3, −6.1] | −3.4 [−5.3, −1.5] | −8.8 [−12.7, −5.1] |
| Model | (1) Before, simple prompt | (3b) Before, structured prompt | (3a) Both photos, simple prompt | (3) Both photos, structured prompt | Photo effect, structured prompt (3) − (3b) | Photo effect, simple prompt (3a) − (1) | Prompt effect, before only (3b) − (1) | Prompt effect, both photos (3) − (3a) |
|---|---|---|---|---|---|---|---|---|
| Haiku 5.5 | 24.7 (−6.7) | 26.2 (−17.8) | 25.0 (−14.3) | 24.9 (−14.0) | −1.3 [−2.8, +0.4] | +0.2 [−1.9, +2.2] | +1.5 [−0.3, +3.2] | −0.1 [−1.6, +1.4] |
| Sonnet 5.5 | 24.2 (+0.9) | 26.9 (−1.3) | 18.9 (−8.0) | 21.8 (−2.9) | −5.1 [−9.3, −1.3] | −5.3 [−8.1, −3.0] | +2.7 [−1.3, +6.9] | +2.9 [+1.2, +4.6] |
| Opus 5.5 | 30.1 (+21.7) | 22.9 (+9.2) | 20.3 (+6.3) | 20.5 (+9.0) | −2.5 [−4.4, −0.6] | −9.7 [−13.4, −6.5] | −7.1 [−9.8, −4.9] | +0.1 [−0.7, +1.1] |
| GPT-6 luna | 23.9 (+4.3) | 23.5 (+3.7) | 20.7 (−7.1) | 21.0 (−6.1) | −2.5 [−4.1, −1.0] | −3.1 [−5.1, −1.4] | −0.4 [−1.5, +0.7] | +0.3 [−0.7, +1.3] |
| GPT-6.1 sol | 26.9 (+14.8) | 26.2 (+17.0) | 18.7 (−2.2) | 19.5 (+1.2) | −6.7 [−9.6, −3.8] | −8.2 [−11.1, −5.6] | −0.7 [−1.5, +0.2] | +0.8 [−0.2, +2.1] |
| GPT-6 astra | 27.6 (+16.4) | 27.2 (+19.1) | 18.4 (+1.6) | 19.0 (+5.3) | −8.1 [−10.9, −5.6] | −9.2 [−12.1, −6.6] | −0.5 [−1.4, +0.4] | +0.6 [+0.0, +1.2] |
| Model | Estimate − served | Energy left | Estimate − eaten | Meals with ≥10% left where the two parts offset* | Median offset, kcal |
|---|---|---|---|---|---|
| Haiku 5.5 | −156 | +103 | −53 | 178 / 292 | 153 |
| Sonnet 5.5 | −104 | +103 | −1 | 146 / 292 | 147 |
| Opus 5.5 | +38 | +103 | +141 | 61 / 292 | 126 |
| GPT-6 luna | −80 | +103 | +23 | 141 / 292 | 153 |
| GPT-6.1 sol | −8 | +103 | +95 | 84 / 292 | 141 |
| GPT-6 astra | +4 | +103 | +107 | 88 / 292 | 126 |
| Model, input | Mean day error % | 90% CI | Equivalence | 95% LoA % | Days within 800 kJ % |
|---|---|---|---|---|---|
| Haiku 5.5, before only | −4.2 [−8.9, +0.7] | −8.1, −0.3 | yes | −48 to +39 | 34 |
| Haiku 5.5, both photos | −12.0 [−16.2, −7.2] | −15.7, −8.2 | no | −53 to +29 | 29 |
| Sonnet 5.5, before only | +5.5 [−0.7, +12.3] | +0.1, +11.0 | no | −62 to +73 | 37 |
| Sonnet 5.5, both photos | −0.6 [−4.2, +3.2] | −3.7, +2.5 | yes | −41 to +40 | 41 |
| Opus 5.5, before only | +25.4 [+19.4, +32.3] | +20.3, +31.0 | no | −32 to +83 | 25 |
| Opus 5.5, both photos | +12.0 [+8.7, +15.5] | +9.2, +14.8 | no | −25 to +49 | 43 |
| Method [3] | Feeding days | Estimated / true energy, mean (SD) % |
|---|---|---|
| ASA24-Australia (self-administered) | 143 | 105 (29) |
| Intake24-Australia (self-administered) | 150 | 102 (29) |
| Mobile food record, trained analyst | 148 | 101 (15) |
| Image-assisted interviewer recall | 145 | 115 (20) |
| Model | Protein | Carbohydrate | Fat |
|---|---|---|---|
| Haiku 5.5 | −29 (−70 to +12) | +1 (−40 to +43) | −12 (−52 to +29) |
| Sonnet 5.5 | −24 (−62 to +14) | +14 (−33 to +62) | +3 (−39 to +45) |
| Opus 5.5 | −8 (−52 to +37) | +25 (−24 to +75) | +18 (−33 to +70) |
| GPT-6 luna | −16 (−63 to +30) | +12 (−31 to +55) | +0 (−45 to +46) |
| GPT-6.1 sol | −13 (−57 to +30) | +16 (−26 to +58) | +7 (−36 to +51) |
| GPT-6 astra | −10 (−54 to +35) | +21 (−23 to +64) | +12 (−31 to +56) |
| Model | Before only | Subtraction | Both photos |
|---|---|---|---|
| Haiku 5.5 | −4.2 [−8.9, +0.7]; −48 to +39; 34 | −32.9 [−35.7, −30.3]; −62 to −4; 8 | −12.0 [−16.2, −7.2]; −53 to +29; 29 |
| Sonnet 5.5 | +5.5 [−0.7, +12.3]; −62 to +73; 37 | −16.1 [−18.8, −13.4]; −48 to +15; 28 | −0.6 [−4.2, +3.2]; −41 to +40; 41 |
| Opus 5.5 | +25.4 [+19.4, +32.3]; −32 to +83; 25 | +3.7 [+0.9, +6.5]; −29 to +37; 49 | +12.0 [+8.7, +15.5]; −25 to +49; 43 |
| GPT-6 luna | +6.7 [+1.9, +11.8]; −41 to +54; 39 | −16.5 [−19.4, −13.5]; −49 to +16; 22 | −4.3 [−7.4, −1.0]; −38 to +30; 43 |
| GPT-6.1 sol | +17.7 [+12.6, +23.4]; −33 to +69; 29 | −9.4 [−12.5, −6.3]; −43 to +24; 30 | +3.4 [+0.3, +6.5]; −31 to +38; 51 |
| GPT-6 astra | +20.2 [+15.0, +25.9]; −31 to +71; 24 | −5.5 [−8.5, −2.6]; −38 to +27; 42 | +7.6 [+4.4, +10.8]; −27 to +42; 45 |
| Model | Meal MAE%: none → cal | Day mean %: none → cal | Day 95% LoA: none → cal | Days within 800 kJ %: none → cal |
|---|---|---|---|---|
| Haiku 5.5 | 24.9 → 25.0 | −12.0 → −4.5 | −53..+29 → −50..+41 | 29 → 32 |
| Sonnet 5.5 | 21.8 → 23.2 | −0.7 → −0.9 | −41..+40 → −42..+41 | 41 → 39 |
| Opus 5.5 | 20.5 → 19.1 | +12.1 → +0.5 | −25..+49 → −33..+34 | 43 → 43 |
| GPT-6 luna | 21.0 → 22.3 | −4.2 → −1.9 | −38..+29 → −37..+34 | 43 → 40 |
| GPT-6.1 sol | 19.5 → 20.8 | +3.4 → −1.1 | −31..+38 → −36..+33 | 51 → 41 |
| GPT-6 astra | 19.1 → 19.3 | +7.7 → −1.1 | −27..+43 → −34..+32 | 45 → 42 |
| Model | Breakfast (n = 286) | Lunch (n = 257) | Dinner (n = 259) |
|---|---|---|---|
| Haiku 5.5 | 19.0 / −6.7 | 22.4 / −6.2 | 31.4 / −25.8 |
| Sonnet 5.5 | 18.3 / +3.1 | 18.8 / −1.6 | 26.9 / −8.6 |
| Opus 5.5 | 20.6 / +12.1 | 20.0 / +12.3 | 20.8 / +4.1 |
| GPT-6 luna | 19.1 / −3.5 | 19.5 / +1.8 | 23.8 / −14.3 |
| GPT-6.1 sol | 19.8 / +5.0 | 18.2 / +10.0 | 20.4 / −8.7 |
| GPT-6 astra | 20.3 / +8.8 | 17.9 / +11.7 | 19.0 / −2.4 |
| Model | MAE% before×c / both | Δ both − before×c, all [95% CI] | Δ, ≥10% left [95% CI] | Day mean %: before×c / both | Day LoA %: before×c / both | Days within 800 kJ %: before×c / both |
|---|---|---|---|---|---|---|
| Haiku 5.5 | 27.3 / 24.9 | −2.5 [−4.2, −0.6] | −1.8 [−4.4, +0.6] | −16.6 / −12.0 | −55..+22 / −53..+29 | 28 / 29 |
| Sonnet 5.5 | 24.9 / 21.8 | −3.0 [−5.5, −0.8] | −1.3 [−5.9, +2.7] | −8.2 / −0.6 | −67..+51 / −41..+40 | 32 / 41 |
| Opus 5.5 | 22.7 / 20.5 | −2.2 [−4.8, +0.3] | −7.5 [−12.3, −3.3] | +9.2 / +12.0 | −41..+60 / −25..+49 | 35 / 43 |
| GPT-6 luna | 23.7 / 21.0 | −2.6 [−4.2, −1.1] | −1.3 [−5.1, +1.9] | −7.1 / −4.3 | −48..+34 / −38..+30 | 27 / 43 |
| GPT-6.1 sol | 22.3 / 19.5 | −2.8 [−4.8, −0.7] | −6.6 [−10.6, −3.1] | +2.5 / +3.4 | −42..+47 / −31..+38 | 41 / 51 |
| GPT-6 astra | 22.4 / 19.0 | −3.4 [−5.3, −1.5] | −8.8 [−12.7, −5.1] | +4.7 / +7.6 | −40..+49 / −27..+42 | 40 / 45 |
| Model | ≥10% left | <10% left | Difference |
|---|---|---|---|
| Haiku 5.5 | −28 [−47, −8] | +20 [+4, +40] | −48 [−75, −22] |
| Sonnet 5.5 | −43 [−78, −9] | +2 [−21, +21] | −45 [−87, −3] |
| Opus 5.5 | −145 [−181, −110] | −8 [−29, +8] | −138 [−178, −97] |
| GPT-6 luna | −45 [−76, −14] | −1 [−10, +7] | −43 [−74, −12] |
| GPT-6.1 sol | −116 [−147, −84] | −3 [−16, +13] | −113 [−146, −79] |
| GPT-6 astra | −135 [−166, −102] | −8 [−16, −0] | −127 [−157, −93] |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 789 papers