V-Nutri: Dish-Level Nutrition Estimation from Egocentric Cooking Videos

Chengkun Yue; Chuanzhi Xu; Jiangpeng He

cs.CVApr 13, 2026

V-Nutri: Dish-Level Nutrition Estimation from Egocentric Cooking Videos

Authors: Chengkun Yue, Chuanzhi Xu, Jiangpeng He

Organizations: Indiana University · The University of Sydney

Abstract

Nutrition estimation of meals from visual data is an important problem for dietary monitoring and computational health, but existing approaches largely rely on single images of the finally completed dish. This setting is fundamentally limited because many nutritionally relevant ingredients and transformations, such as oils, sauces, and mixed components, become visually ambiguous after cooking, making accurate calorie and macronutrient estimation difficult. In this paper, we investigate whether the cooking process information from egocentric cooking videos can contribute to dish-level nutrition estimation. First, we further manually annotated the HD-EPIC dataset and established the first benchmark for video-based nutrition estimation. Most importantly, we propose V-Nutri, a staged framework that combines Nutrition5K-pretrained visual backbones with a lightweight fusion module that aggregates features from the final dish frame and cooking process keyframes extracted from the egocentric videos. V-Nutri also includes a cooking keyframes selection module, a VideoMamba-based event-detection model that targets ingredient-addition moments. Experiments on the HD-EPIC dataset show that process cues can provide complementary nutritional evidence, improving nutrition estimation under controlled conditions. Our results further indicate that the benefit of process keyframes depends strongly on backbone representation capacity and event detection quality. Our code and annotated dataset is available at https://github.com/K624-YCK/V-Nutri.

Figures & tables

Figure 1: Traditional nutrition estimation is process-blind, relying only on the final dish image, where important ingredients may be hidden after cooking. V-Nutri combines process keyframes from egocentric cooking videos with the final dish frame for a process-aware estimation of calories and macronutrients.

Figure 2: Overview of V-Nutri (Inference). Given an egocentric cooking video, the process keyframes and final dish frame are selected by the cooking keyframe selector and final dish frame selector. A frozen Nutrition5K-pretrained backbone (ResNet-101, ViT-B/16, or ViT-L/16) encodes each process keyframe into an embedding $z_{1},\ldots,z_{K}$ and the final dish image into $z_{d}$ . Learned attention weights $\alpha_{1},\ldots,\alpha_{K}$ aggregate the process embeddings into a single representation $z_{p}$ through weighted pooling. The dish and process representations are fused and fed to an MLP regressor that predicts four nutritional values: calories, protein, fat, and carbohydrates.

Tier	Count	Definition
All instances	80	Recipe instances extracted from HD-EPIC (69 unique recipes)
Regression-ready	52	All ingredients have nutritional labels $\to$ dish-level target available
Fully complete	22	All ingredients have both nutritional labels and add-event timestamps

Table 1: Three-tier structure of the HD-EPIC nutrition estimation benchmark. Each tier is a subset of the one above.

Figure 3: Nutrition distribution of 52 benchmark instances. Violin plots show density with individual points; dashed and dotted lines denote the mean and median. All nutrients are right-skewed, and four instances have zero values across all nutrients.

Model	Epoch	Val Loss	Cal	Fat	Carb	Prot
ResNet101	48	2.308	73.2	5.60	6.63	5.14
ViT-B/16	4	2.559	88.0	7.00	7.95	6.52
ViT-L/16	31	2.203	77.7	5.76	6.81	5.19

Table 2: Backbone comparison on Nutrition5K (training with z-score normalization and augmentation). The best result per column is highlighted in bold . (Cal in kcal; Prot/Fat/Carb in g)

	ResNet101				ViT-B/16				ViT-L/16
Method	Cal	Pr	Fat	Cb	Cal	Pr	Fat	Cb	Cal	Pr	Fat	Cb
D(GT)-only	698	27.8	37.7	101	672	26.5	39.5	88.5	711	32.5	45.9	89.9
D+P (mean)	572	24.6	32.9	84.3	569	22.2	33.7	80.3	596	28.3	43.0	81.8
D+P (wtd.)	582	22.3	35.9	73.9	571	23.1	30.9	91.1	551	22.5	39.4	74.4

Table 3: Validation of Process Keyframes Utility with ground-truth process keyframes. With final dish frame is marked as D; with process keyframes is marked as P. The best result per column is highlighted in bold . (Cal in kcal; Prot/Fat/Carb in g)

	ResNet101				ViT-B/16				ViT-L/16
Method	Cal	Pr	Fat	Cb	Cal	Pr	Fat	Cb	Cal	Pr	Fat	Cb
GT	582	22.3	35.9	73.9	571	23.1	30.9	91.1	551	22.5	39.4	74.4
Dish-only	797	29.8	46.1	92.2	666	26.3	37.4	94.8	729	31.1	43.4	86.1
Pred-20	763	28.3	44.3	86.7	659	22.8	35.7	91.4	716	29.4	44.5	76.6
Pred-50	765	28.5	44.2	86.9	663	24.1	36.8	88.9	693	29.0	43.5	74.7
Pred-200	767	28.6	44.3	86.9	657	24.8	36.7	87.5	682	28.9	44.2	73.1

Table 4: 5-fold cross-validation results (mean MAE) for ablation study on sampling strategies. The best result per column is highlighted in bold . (Cal in kcal; Prot/Fat/Carb in g)

Figure 4: Predicted vs. ground-truth nutrient values for the ViT-B/16 with Pred-20 strategy. Dashed line: perfect prediction.

r

: Pearson correlation.

Backbone	Fusion	Dish-only	Ground Truth	Pred top-20
ResNet101	Concat	785	568	768
ResNet101	Gated	797	582	763
ViT-B/16	Concat	674	565	674
ViT-B/16	Gated	666	571	659
ViT-L/16	Concat	707	573	682
ViT-L/16	Gated	729	551	716

Table 5: Fusion architecture ablation (Calories MAE, 5-fold mean). The best result per column is highlighted in bold .

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation
Sep 27, 2026Avinash Anand, Aman Kumar, Chaitanya Lakhchaura +4Feature Fusion Multi-Source Image Fusion
Singapore Institute of Technology, Singapore · Indraprastha Institute of Information Technology Delhi (IIIT-Delhi), New Delhi, India · Rochester Institute of Technology, Rochester, NY, USA +1
NutriBench-Kitchen: Benchmarking Embodied AI for Nutrition Management
Sep 7, 2026Yulin Wei, Xiangchen Wang, Jianhui Pan +5Embodied QA AI Agent Benchmarks
Georgia Institute of Technology, Atlanta, USA · Southern University of Science and Technology, Shenzhen, China · SpatialTemporal AI, Shenzhen, China
Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery
Oct 8, 2026Jingbo Yue, Bruce Coburn, Jinge Ma +2Visual Grounding
Elmore Family School of Electrical and Computer Engineering, Purdue University West Lafayette, IN, USA

Sep 27, 2026J/K move · Enter open · S save

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar+3

Singapore Institute of Technology, Singapore · Indraprastha Institute of Information Technology Delhi (IIIT-Delhi), New Delhi, India · Rochester Institute of Technology, Rochester, NY, USA+1

Feature Fusion Multi-Source Image Fusion

Open paper AskPDF arXiv

Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM), which uses textual guidance to reweight and align RGB-depth frequency components; (2) an Ingredient-Aware Mask-based Prediction Head (IA-MPH), whose gating and channel masks are conditioned on food identity; and (3) modality-specific Internal Semantic Modeling (ISM) blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of

\mathbf{13.60\pm0.10\%}

, outperforming our IGSMNet implementation by

0.89

percentage points and OmniFood8k by

2.90

percentage points (both

p<0.001

). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by

1.50\pm0.17

percentage points (

p<0.001

). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware

Abstract

Explore similar work

NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

NutriBench-Kitchen: Benchmarking Embodied AI for Nutrition Management

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery