Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

Jingbo Yue; Bruce Coburn; Jinge Ma; Jui-Feng Chi; Fengqing Zhu

cs.CVOct 8, 2026

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

Authors: Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu

Organizations: Elmore Family School of Electrical and Computer Engineering, Purdue University West Lafayette, IN, USA

Abstract

Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.

Figures & tables

Figure 1: One-round item-grounded recovery. Verified initial and recovery evidence are re-audited and reconciled into $F$ . Coverage assessment and nutrition estimation follow. The example shows fruit-salad geometry repair and meat-and-rice evidence recovery.

Model	Evidence state	$\rho_{\rm item}$ N5k ∗ /ACE	$\rho_{\rm sem}$ N5k ∗ /ACE	$\rho_{\rm geo}$ N5k ∗ /ACE
G26B	Initial	35.38/93.01	27.08/77.86	26.50/76.14
G26B	Post-recovery (ours)	35.44/93.07 (+0.06/+0.07 pp)	31.41/86.01 (+4.33/+8.15 pp)	30.85/84.24 (+4.36/+8.11 pp)
Q35B	Initial	35.80/90.98	27.70/80.67	27.52/80.58
Q35B	Post-recovery (ours)	35.94/91.00 (+0.14/+0.02 pp)	32.77/87.60 (+5.07/+6.93 pp)	32.45/87.49 (+4.93/+6.91 pp)

Table 1: Pre-reconciliation recovery on N5k ∗ /ACE: reference-normalized rates (%); parentheses show changes from Initial (pp).

	G26B		Q35B
Representation	$P_{\rm micro}$	$R_{\rm micro}$	$P_{\rm micro}$	$R_{\rm micro}$
Initial inventory	98.35	93.02	90.54	87.19
Final- $F$ (Ours)	98.62 ↑0.27	94.34 ↑1.32	93.73 ↑3.19	87.87 ↑0.68

Table 2: Manual food-identity audit on 150 ACETADA C1 meals per model. Values are micro precision/recall (%); superscripts show changes from Initial (pp).

Model	Direct	Adapted DietAI24	Ours (C1)	Ours (C1+C2)
Nutrition5k (N5k)
G26B	81.1/119.3	123.7/164.9	74.3 / 107.4	80.8/111.2
Q35B	114.3/138.8	150.8/189.1	74.2 / 96.9	77.1/97.1
ACETADA (ACE)
G26B	217.8/ 168.1	368.7/250.6	140.5 /177.1	159.4/180.4
Q35B	192.5/ 166.2	438.4/262.9	144.8 /172.5	148.1/173.8

Table 3: Meal-level MAE (mass g / energy kcal; lower is better). Direct, Adapted DietAI24, and Ours (C1) use the same matched C1 samples; Ours (C1+C2) reports our broader-cohort result.

Figure 2: Q35B/ACETADA C1 mass and energy predictions without baseline-intersection filtering, grouped by

\mathcal{R}_{\rm full}

and

\mathcal{R}_{\rm gap}

Explore similar work

Geometry-Enhanced Portion Estimation for Multimodal LLMs
Jul 17, 2026Lin Liao, Peng LiMultimodal Large Language Models Multimodal Foundation Models
Open-KNEAD: Knowledge-grounded Nutrition Estimation via Agentic Decomposition
Jul 14, 2026Bruce Coburn, Jingbo Yue, Jinge Ma +3Retrieval-Augmented Generation Large Vision-Language Models
Purdue University, USA
NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis
Jun 8, 2026Runze Yan, Minxiao Wang, Jiaying Lu +3Multimodal Large Language Models
Center for Data Science, Nell Hodgson Woodruff School of Nursing, Emory University, 1520 Clifton Rd NE, Atlanta, 30322, GA, United States. · Rollins School of Public Health, Emory University, 1518 Clifton Rd2026 NE, Atlanta, 30322, GA, United States.

Jul 17, 2026, cs.CVJ/K move · Enter open · S save

Geometry-Enhanced Portion Estimation for Multimodal LLMs

Lin Liao, Peng Li

Multimodal Large Language Models Multimodal Foundation Models

Open paper AskPDF arXiv

Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker. Multimodal LLMs (MLLMs) recognize a wide range of foods zero-shot in uncontrolled photos, yet they are weak at portion estimation -- a gap we measure across the current frontier (Gemini, GPT, and Claude flagships alike). We present a method that enhances a frozen, commercial MLLM with an accurate portion head: a small geometry-enhanced network on a frozen DINOv2 backbone with a structured softmax-ownership volume, consuming the MLLM's per-food name, bounding box, and density range -- no depth sensor, no MLLM fine-tuning. Evaluated fully open-vocabulary on three real-world benchmarks, the head cuts per-food portion error by 33-41% relative to the MLLM alone, outperforms every flagship MLLM's direct estimates, and surpasses each benchmark's originally published image-only model at its own reported metric.

Abstract

Explore similar work

Geometry-Enhanced Portion Estimation for Multimodal LLMs

Open-KNEAD: Knowledge-grounded Nutrition Estimation via Agentic Decomposition

NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis