NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation
Authors: Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol, Timothy Liu, Zhengkui Wang
Organizations: Singapore Institute of Technology, Singapore · Indraprastha Institute of Information Technology Delhi (IIIT-Delhi), New Delhi, India · Rochester Institute of Technology, Rochester, NY, USA · NVIDIA AI Technology Centre, Singapore
Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM), which uses textual guidance to reweight and align RGB-depth frequency components; (2) an Ingredient-Aware Mask-based Prediction Head (IA-MPH), whose gating and channel masks are conditioned on food identity; and (3) modality-specific Internal Semantic Modeling (ISM) blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of 13.60±0.10%, outperforming our IGSMNet implementation by 0.89 percentage points and OmniFood8k by 2.90 percentage points (both p<0.001). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by 1.50±0.17 percentage points (p<0.001). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware
Figures & tables
Fig. 1: Overview of the proposed ingredient-aware RGB–depth fusion architecture for food nutrition estimation. A scale-shift residual adapter (SSRA) calibrates the depth input, while dual ConvNeXt-B backbones extract multi-scale RGB and depth features. An Ingredient-aware Spatial Mixer (ISM) captures spatial dependencies, followed by an Ingredient-Conditioned Hierarchical Feature Attention Module (IC-FAFM) that integrates ingredient semantics with multi-scale visual features. An Ingredient-Aware Prediction Head (IA-MPH) further refines the fused representation through ingredient-guided cross-attention to jointly predict calories, mass, protein, carbohydrates, and fat.
Fig. 2: Detailed structure of the three NutriVision modules. (A) The ISM block: windowed multi-head self-attention (7×7, 4 heads) with a DPE-derived positional bias B=MLP(Δx,Δy) in a pre-norm residual layout ( ⊕ ), applied twice per scale independently to each modality. (B) IC-FAFM at one scale: per-modality 2-D FFTs are split at the low-pass radius τ⋅rmax(τ=0.20) , the low and high bands are summed across modalities and merged by a 3×3 convolution into Oi , which is then modulated by ingredient cross-attention (q = We) with a broadcast residual to yield O~i . (C) IA-MPH: the concatenation x=[f;cr;cd]∈R1280 is gated by σ(We) , masked to the top-75% channels by magnitude during training only (dashed), and regressed via a second ingredient cross-attention and a three-layer MLP (1280→512→256→5)
Fig. 3: What the FFT split inside IC-FAFM separates. (a) the input dish. (b) its log-magnitude 2-D FFT, with the low-pass mask of radius τrmax=0.20rmax overlaid. (c) inverse FFT of the low band, which preserves plate boundary, portion footprint, and overall layout. (d) inverse FFT of the high band, which preserves edges, textures, and fine-grained ingredient appearance.
Fig. 4: Ingredient cross-attention inside IC-FAFM (Eq. 10 ) on the same dish, queried with three different ingredient strings. The hot regions shift meaningfully: “chicken thighs” attends to the seared chicken on the right, “broccoli florets” migrates upward toward the green florets, and “brown rice” redistributes again. The attention is a soft prior, not a segmentation; this is qualitative evidence of ingredient–visual grounding , not causal evidence of a headline accuracy gain—the corresponding retraining ablation (A7, Δ=+0.18 PMAE) is at the per-run noise floor.
TABLE II: Compute budget for NutriVision. Forward-pass FLOPs are estimated from per-block published numbers (ConvNeXt-Base ∼ 15.4 GFLOPs at 2242 [ 17 ] ); training and inference times are measured on a single RTX A6000 (48 GB). End-to-end inference includes the DepthAnything-V3 forward and the CLIP text encoding for the ingredient string.
Fig. 5: Training trajectory on Nutrition5k (representative seed). Train loss (purple, left) decays smoothly; test-set mean PMAE (blue, right) plateaus near 13.5% ; we report the best-on-test checkpoint (§ IV ). The 3-seed mean best-mean-PMAE is 13.60±0.10% .
Parameter
Value
Optimizer
Adam
Base learning rate
5e−5
Backbone learning rate
1e−5
LR schedule
Exp. decay, γ=0.99 /epoch
Weight decay
1e−5
Gradient clipping
∥⋅∥2=1.0
TABLE III: Training hyperparameters.
Method
Venue
Input (test)
Cal
Mass
Fat
Carb
Prot
Mean
Δ vs. ours
Google-Nutrition-rgbd [ 2 ]
CVPR’21
RGB-D (real)
18.8
18.9
18.1
23.8
20.9
20.1
+6.50
Swin-Nutrition [ 8 ]
Foods’22
RGB
15.3
12.5
22.1
20.8
15.4
17.2
+3.60
DPF-Nutrition [ 12 ]
Foods’23
RGB (est. depth)
14.7
10.6
22.6
20.7
20.2
17.8
+4.20
OmniFood8k [ 3 ]
CVPR’26
RGB
14.1
10.2
21.0
18.9
18.4
16.5
+2.90
Kwan et al. [ 11 ] (NuNet)
TMM’25
RGB-D (real)
12.80
8.72
19.67
18.66
18.42
15.65
+2.05
IGSMNet [ 4 ]
Foods’25
RGB-D (real)+Ingr.
12.2
9.4
19.1
18.3
16.0
15.0
+1.40
TABLE IV: Per-nutrient and mean PMAE (%) on the Nutrition5k 5:1 dish-level split. Lower is better. Best in bold. † The “IGSMNet (our reimpl.)” row is our re-implementation of the IGSMNet architecture from [ 4 ] (the full training code was not publicly available at the time of writing); we follow the paper’s architectural description and train under identical conditions to NutriVision. ‡ Per-nutrient columns for the NutriVision row are from a representative seed=1 run (best-mean-PMAE 13.50% ); the Mean column reports the 3-seed mean ± std, and the Δ column uses the 3-seed mean. All baselines are as published.
Fig. 6: Four representative Nutrition5k test predictions, spanning low-, mid-, and high-calorie dishes. For each plate we show the input photograph, the comma-separated ingredient list used at inference, and a compact ground-truth-vs-prediction table for the five nutrients. Green cells indicate the prediction is within ± 13.5% of GT (one PMAE band); orange cells mark larger misses. Errors are largest on dishes with heavily-stacked or occluded ingredients.
#
Configuration
Cal
Mass
Fat
Carb
Prot
Mean
Δ Mean
A1
Full model (RGB + Depth + CLIP)
12.65
10.37
16.45
14.76
13.35
13.52
–
A2
No CLIP (zero ingredient embedding)
24.87
15.94
35.43
26.35
34.59
27.44
+13.92
A3
No depth (zero depth)
12.88
10.42
16.56
14.74
13.67
13.65
+0.13
A4
No CLIP and no depth (RGB only)
25.70
16.29
35.79
26.40
36.18
28.07
+14.55
A5
No RGB (sanity check)
68.04
59.37
73.81
65.98
64.50
66.34
+52.82
A6
No SSRA at inference ( α=1,β=0 , residual =0 )
12.65
10.37
16.45
14.76
13.36
13.52
+0.00
TABLE V: Input-level ablation on the Nutrition5k test split (3,096 samples). Inputs are zeroed at test time only; the model is the representative seed=1 NutriVision checkpoint (best mean PMAE 13.52% ; 3-seed mean 13.60±0.10% , Tab. IV ).
Fig. 7: Input-level ablation visualised as radar plots over the five nutrients (lower is better; the polygon shrinks toward the centre as PMAE drops). (a) All four ablation variants overlaid: removing RGB (red) explodes the polygon, removing ingredient text (orange) roughly doubles it, while the full model (blue) and the no-depth variant (green) are visually indistinguishable. (b) Zoomed in to 0–20 % PMAE, the full and no-depth polygons coincide almost perfectly, confirming the +0.13 pp mean delta in Table V .
#
Configuration
Cal
Mass
Fat
Carb
Prot
Mean
Δ Mean
A7
No ingredient cross-attention in IC-FAFM
12.9
10.5
16.6
15.1
13.5
13.7
+0.10
A8
Element-wise add fusion (no FFT split)
12.8
10.3
16.8
14.8
13.4
13.6
+0.00
A9
No ISM (window-MHSA + DPE → identity)
12.6
10.3
16.4
14.6
13.1
13.4
−0.20
A10 ‡
Plain MLP head (IA-MPH → concat + 3-layer MLP)
13.4
10.9
17.4
17.9
14.6
15.10±0.26
+1.50±0.17
A11
No inter-modal alignment loss ( λalign=0 )
12.9
10.5
16.8
14.7
13.6
13.7
+0.10
A12
Smooth-L1 loss (replacing normalised L1)
12.6
10.3
20.6
16.0
14.6
14.8
+1.20
TABLE VI: Module-level retraining ablation on Nutrition5k. Each row re-trains the full architecture from scratch with the indicated module replaced or removed; everything else is byte-identical to the headline NutriVision (A1 in Tab. V , 3-seed mean PMAE 13.60±0.10% ). Selection protocol: every row reports the best-mean-PMAE-on-the-test-split checkpoint over its 100-epoch run, matching the protocol used for A1 and disclosed in § IV . Statistical protocol: rows marked ‡ (A10) are retrained across 3 seeds; the Mean column reports mean ± std and Δ is a paired t -test against NutriVision (Sec. V , p<0.001 ). A7–A9, A11, A12 remain single-seed and their Δ is computed against the 3-seed mean of A1 ( 13.60% ). Lower is better.
Accurate estimation of food nutrition plays a vital role in promoting healthy dietary habits and personalized diet management. Most existing food datasets primarily focus on Western cuisines and lack sufficient coverage of Chinese dishes, which restricts accurate nutritional estimation for Chinese meals. Moreover, many state-of-the-art nutrition prediction methods rely on depth sensors, restricting their applicability in daily scenarios. To address these limitations, we introduce OmniFood8K, a comprehensive multimodal dataset comprising 8,036 food samples, each with detailed nutritional annotations and multi-view images. In addition, to enhance models' capability in nutritional prediction, we construct NutritionSynth-115K, a large-scale synthetic dataset that introduces compositional variations while preserving precise nutritional labels. Moreover, we propose an end-to-end framework for nutritional prediction from a single RGB image. First, we predict a depth map from a single RGB image and design the Scale-Shift Residual Adapter (SSRA) to refine it for global scale consistency and local structural preservation. Second, we propose the Frequency-Aligned Fusion Module (FAFM) to hierarchically align and fuse RGB and depth features in the frequency domain. Finally, we design a Mask-based Prediction Head (MPH) to emphasize key ingredient regions via dynamic channel selection for more accurate prediction. Extensive experiments on multiple datasets demonstrate the superiority of our method over existing approaches. Project homepage: https://yudongjian.github.io/OmniFood8K-food/
Dongjian Yu, Weiqing Min, Qian Jiang +3
School of Software, Yunnan University, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China
Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.
Jingbo Yue, Bruce Coburn, Jinge Ma +2
Elmore Family School of Electrical and Computer Engineering, Purdue University West Lafayette, IN, USA
Accurate dietary assessment is critical for precision nutrition, yet most image-based methods rely on a single pre-consumption image and provide only coarse, meal-level estimates. These approaches cannot determine what was actually consumed and often require restrictive inputs such as depth sensing, multi-view imagery, or explicit segmentation. In this paper, we propose a simple vision-language framework for food-item-level nutritional analysis using paired before-and-after eating images. Instead of relying on rigid segmentation masks, our method leverages natural language prompts to localize specific food items and estimate their weight directly from a single RGB image. We further estimate food consumption by predicting weight differences between paired images using a two-stage training strategy. We evaluate our method on three publicly available datasets and demonstrate consistent improvements over existing approaches, establishing a strong baseline for before-and-after dietary image analysis.
Gautham Vinod, Siddeshwar Raghavan, Bruce Coburn +1
Purdue University, West Lafayette, Indiana, U.S.A.