NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

Avinash Anand; Aman Kumar; Chaitanya Lakhchaura; Ashutosh Kumar; Akshita Abrol; Timothy Liu; Zhengkui Wang

cs.CVSep 27, 2026

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

Authors: Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol, Timothy Liu, Zhengkui Wang

Organizations: Singapore Institute of Technology, Singapore · Indraprastha Institute of Information Technology Delhi (IIIT-Delhi), New Delhi, India · Rochester Institute of Technology, Rochester, NY, USA · NVIDIA AI Technology Centre, Singapore

Abstract

Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM), which uses textual guidance to reweight and align RGB-depth frequency components; (2) an Ingredient-Aware Mask-based Prediction Head (IA-MPH), whose gating and channel masks are conditioned on food identity; and (3) modality-specific Internal Semantic Modeling (ISM) blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$ , outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$ ). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ( $p<0.001$ ). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware

Figures & tables

Fig. 1: Overview of the proposed ingredient-aware RGB–depth fusion architecture for food nutrition estimation. A scale-shift residual adapter (SSRA) calibrates the depth input, while dual ConvNeXt-B backbones extract multi-scale RGB and depth features. An Ingredient-aware Spatial Mixer (ISM) captures spatial dependencies, followed by an Ingredient-Conditioned Hierarchical Feature Attention Module (IC-FAFM) that integrates ingredient semantics with multi-scale visual features. An Ingredient-Aware Prediction Head (IA-MPH) further refines the fused representation through ingredient-guided cross-attention to jointly predict calories, mass, protein, carbohydrates, and fat.

Fig. 2: Detailed structure of the three NutriVision modules. (A) The ISM block: windowed multi-head self-attention (7×7, 4 heads) with a DPE-derived positional bias

\text{B}=\text{MLP}(\Delta x,\Delta y)

in a pre-norm residual layout (

\oplus

), applied twice per scale independently to each modality. (B) IC-FAFM at one scale: per-modality 2-D FFTs are split at the low-pass radius

\tau\cdot r_{max}(\tau=0.20)

, the low and high bands are summed across modalities and merged by a 3×3 convolution into

O_{i}

, which is then modulated by ingredient cross-attention (q = We) with a broadcast residual to yield

\tilde{O}_{i}

. (C) IA-MPH: the concatenation

x=[f;c_{r};c_{d}]\in\mathbb{R}^{1280}

is gated by

\sigma(We)

, masked to the top-75% channels by magnitude during training only (dashed), and regressed via a second ingredient cross-attention and a three-layer MLP

(1280\rightarrow 512\rightarrow 256\rightarrow 5)

Fig. 3: What the FFT split inside IC-FAFM separates. (a) the input dish. (b) its log-magnitude 2-D FFT, with the low-pass mask of radius $\tau\,r_{\max}=0.20\,r_{\max}$ overlaid. (c) inverse FFT of the low band, which preserves plate boundary, portion footprint, and overall layout. (d) inverse FFT of the high band, which preserves edges, textures, and fine-grained ingredient appearance.

Fig. 4: Ingredient cross-attention inside IC-FAFM (Eq. 10 ) on the same dish, queried with three different ingredient strings. The hot regions shift meaningfully: “chicken thighs” attends to the seared chicken on the right, “broccoli florets” migrates upward toward the green florets, and “brown rice” redistributes again. The attention is a soft prior, not a segmentation; this is qualitative evidence of ingredient–visual grounding , not causal evidence of a headline accuracy gain—the corresponding retraining ablation (A7, $\Delta\!=\!+0.18$ PMAE) is at the per-run noise floor.

Component	Params	Trainable	Notes
ConvNeXt-Base RGB	88.6 M	44 M	Stages 2–3 unfrozen
ConvNeXt-Base Depth	88.6 M	44 M	Stages 2–3 unfrozen
Depth adapter	6	6	$1\!\to\!3$ Conv $2\times 2$
Scale projections (8)	790 K	790 K	$1{\times}1$ to 256-ch
CLS projections (2)	1.1 M	1.1 M	Linear $1024{\to}512$
SSRA	100 K	100 K	$\alpha,\beta$ + CNN refiner

TABLE I: Per-component parameter budget.

Quantity	Value	Notes
Total parameters	$\sim$ 196 M	measured (Tab. I )
Trainable parameters	$\sim$ 105 M	stages 2–3 of both backbones unfrozen
Forward FLOPs (est.)	$\sim$ 35 GFLOPs	$2{\times}15.4$ (backbones) $+$ ISM/IC-FAFM/IA-MPH/SSRA overhead
Train time / epoch	$\sim$ 925 s	batch 8 on RTX A6000, 1923 batches/epoch
Total train time	$\sim$ 38.5 h	150 epochs
End-to-end inference	$\sim$ 3 s / image	RGB $\to$ depth $\to$ CLIP $\to$ NutriVision

TABLE II: Compute budget for NutriVision. Forward-pass FLOPs are estimated from per-block published numbers (ConvNeXt-Base

\sim

15.4 GFLOPs at

224^{2}

[ 17 ] ); training and inference times are measured on a single RTX A6000 (48 GB). End-to-end inference includes the DepthAnything-V3 forward and the CLIP text encoding for the ingredient string.

Fig. 5: Training trajectory on Nutrition5k (representative seed). Train loss (purple, left) decays smoothly; test-set mean PMAE (blue, right) plateaus near $13.5\%$ ; we report the best-on-test checkpoint (§ IV ). The 3-seed mean best-mean-PMAE is $13.60\pm 0.10\%$ .

Parameter	Value
Optimizer	Adam
Base learning rate	$5{\rm e}{-5}$
Backbone learning rate	$1{\rm e}{-5}$
LR schedule	Exp. decay, $\gamma=0.99$ /epoch
Weight decay	$1{\rm e}{-5}$
Gradient clipping	$\\|\cdot\\|_{2}=1.0$

TABLE III: Training hyperparameters.

Method	Venue	Input (test)	Cal	Mass	Fat	Carb	Prot	Mean	$\Delta$ vs. ours
Google-Nutrition-rgbd [ 2 ]	CVPR’21	RGB-D (real)	18.8	18.9	18.1	23.8	20.9	20.1	$+6.50$
Swin-Nutrition [ 8 ]	Foods’22	RGB	15.3	12.5	22.1	20.8	15.4	17.2	$+3.60$
DPF-Nutrition [ 12 ]	Foods’23	RGB (est. depth)	14.7	10.6	22.6	20.7	20.2	17.8	$+4.20$
OmniFood8k [ 3 ]	CVPR’26	RGB	14.1	10.2	21.0	18.9	18.4	16.5	$+2.90$
Kwan et al. [ 11 ] (NuNet)	TMM’25	RGB-D (real)	12.80	8.72	19.67	18.66	18.42	15.65	$+2.05$
IGSMNet [ 4 ]	Foods’25	RGB-D (real)+Ingr.	12.2	9.4	19.1	18.3	16.0	15.0	$+1.40$

TABLE IV: Per-nutrient and mean PMAE (%) on the Nutrition5k 5:1 dish-level split. Lower is better. Best in bold. † The “IGSMNet (our reimpl.)” row is our re-implementation of the IGSMNet architecture from [ 4 ] (the full training code was not publicly available at the time of writing); we follow the paper’s architectural description and train under identical conditions to NutriVision. ‡ Per-nutrient columns for the NutriVision row are from a representative seed=1 run (best-mean-PMAE

13.50\%

); the Mean column reports the 3-seed mean

\pm

std, and the

\Delta

column uses the 3-seed mean. All baselines are as published.

Fig. 6: Four representative Nutrition5k test predictions, spanning low-, mid-, and high-calorie dishes. For each plate we show the input photograph, the comma-separated ingredient list used at inference, and a compact ground-truth-vs-prediction table for the five nutrients. Green cells indicate the prediction is within $\pm$ 13.5% of GT (one PMAE band); orange cells mark larger misses. Errors are largest on dishes with heavily-stacked or occluded ingredients.

#	Configuration	Cal	Mass	Fat	Carb	Prot	Mean	$\Delta$ Mean
A1	Full model (RGB + Depth + CLIP)	12.65	10.37	16.45	14.76	13.35	13.52	–
A2	No CLIP (zero ingredient embedding)	24.87	15.94	35.43	26.35	34.59	27.44	$+13.92$
A3	No depth (zero depth)	12.88	10.42	16.56	14.74	13.67	13.65	$+0.13$
A4	No CLIP and no depth (RGB only)	25.70	16.29	35.79	26.40	36.18	28.07	$+14.55$
A5	No RGB (sanity check)	68.04	59.37	73.81	65.98	64.50	66.34	$+52.82$
A6	No SSRA at inference ( $\alpha{=}1,\beta{=}0$ , residual ${=}0$ )	12.65	10.37	16.45	14.76	13.36	13.52	$+0.00$

TABLE V: Input-level ablation on the Nutrition5k test split (3,096 samples). Inputs are zeroed at test time only; the model is the representative seed=1 NutriVision checkpoint (best mean PMAE

13.52\%

; 3-seed mean

13.60\pm 0.10\%

, Tab. IV ).

Fig. 7: Input-level ablation visualised as radar plots over the five nutrients (lower is better; the polygon shrinks toward the centre as PMAE drops). (a) All four ablation variants overlaid: removing RGB (red) explodes the polygon, removing ingredient text (orange) roughly doubles it, while the full model (blue) and the no-depth variant (green) are visually indistinguishable. (b) Zoomed in to 0–20 % PMAE, the full and no-depth polygons coincide almost perfectly, confirming the $+0.13$ pp mean delta in Table V .

#	Configuration	Cal	Mass	Fat	Carb	Prot	Mean	$\Delta$ Mean
A7	No ingredient cross-attention in IC-FAFM	12.9	10.5	16.6	15.1	13.5	13.7	$+0.10$
A8	Element-wise add fusion (no FFT split)	12.8	10.3	16.8	14.8	13.4	13.6	$+0.00$
A9	No ISM (window-MHSA + DPE $\rightarrow$ identity)	12.6	10.3	16.4	14.6	13.1	13.4	$-0.20$
A10 ‡	Plain MLP head (IA-MPH $\rightarrow$ concat $+$ 3-layer MLP)	13.4	10.9	17.4	17.9	14.6	$\mathbf{15.10\!\pm\!0.26}$	$\mathbf{+1.50\!\pm\!0.17}$
A11	No inter-modal alignment loss ( $\lambda_{\text{align}}{=}0$ )	12.9	10.5	16.8	14.7	13.6	13.7	$+0.10$
A12	Smooth-L1 loss (replacing normalised L1)	12.6	10.3	20.6	16.0	14.6	14.8	$\mathbf{+1.20}$

TABLE VI: Module-level retraining ablation on Nutrition5k. Each row re-trains the full architecture from scratch with the indicated module replaced or removed; everything else is byte-identical to the headline NutriVision (A1 in Tab. V , 3-seed mean PMAE

\mathbf{13.60\pm 0.10\%}

). Selection protocol: every row reports the best-mean-PMAE-on-the-test-split checkpoint over its 100-epoch run, matching the protocol used for A1 and disclosed in § IV . Statistical protocol: rows marked ‡ (A10) are retrained across 3 seeds; the Mean column reports mean

\pm

std and

\Delta

is a paired

t

-test against NutriVision (Sec. V ,

p\!<\!0.001

). A7–A9, A11, A12 remain single-seed and their

\Delta

is computed against the 3-seed mean of A1 (

13.60\%

). Lower is better.

Feature Fusion

Feature fusion combines complementary representations across modalities, scales, layers, or feature types to improve prediction and perception.

See all 119 papers

OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion
Apr 14, 2026Dongjian Yu, Weiqing Min, Qian Jiang +3Multimodal Fusion
School of Software, Yunnan University, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China
Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery
Oct 8, 2026Jingbo Yue, Bruce Coburn, Jinge Ma +2Visual Grounding
Elmore Family School of Electrical and Computer Engineering, Purdue University West Lafayette, IN, USA
DietDelta: A Vision-Language Approach for Dietary Assessment via Before-and-After Images
Apr 7, 2026Gautham Vinod, Siddeshwar Raghavan, Bruce Coburn +1
Purdue University, West Lafayette, Indiana, U.S.A.

Apr 14, 2026J/K move · Enter open · S save

OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion

Dongjian Yu, Weiqing Min, Qian Jiang, Xing Lin+2

School of Software, Yunnan University, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China

Multimodal Fusion

Open paper AskPDF arXiv

Accurate estimation of food nutrition plays a vital role in promoting healthy dietary habits and personalized diet management. Most existing food datasets primarily focus on Western cuisines and lack sufficient coverage of Chinese dishes, which restricts accurate nutritional estimation for Chinese meals. Moreover, many state-of-the-art nutrition prediction methods rely on depth sensors, restricting their applicability in daily scenarios. To address these limitations, we introduce OmniFood8K, a comprehensive multimodal dataset comprising 8,036 food samples, each with detailed nutritional annotations and multi-view images. In addition, to enhance models' capability in nutritional prediction, we construct NutritionSynth-115K, a large-scale synthetic dataset that introduces compositional variations while preserving precise nutritional labels. Moreover, we propose an end-to-end framework for nutritional prediction from a single RGB image. First, we predict a depth map from a single RGB image and design the Scale-Shift Residual Adapter (SSRA) to refine it for global scale consistency and local structural preservation. Second, we propose the Frequency-Aligned Fusion Module (FAFM) to hierarchically align and fuse RGB and depth features in the frequency domain. Finally, we design a Mask-based Prediction Head (MPH) to emphasize key ingredient regions via dynamic channel selection for more accurate prediction. Extensive experiments on multiple datasets demonstrate the superiority of our method over existing approaches. Project homepage: https://yudongjian.github.io/OmniFood8K-food/

Abstract

Feature Fusion

Explore similar work

OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

DietDelta: A Vision-Language Approach for Dietary Assessment via Before-and-After Images