cs.CVNov 5, 2026

NutriVisRAG: Multimodal Exemplar Grounding for Nutrient Estimation with Vision-Language Models

Authors: Lubnaa Abdur Rahman, Ioannis Papathanail, Maria Panagiotou, Nefeli Dimitropoulou, Margret Harðardóttir, Elisabeth J. den Brok, Bastiaan E. de Galan, Stavroula Georgia Mougiakakou

Organizations: Graduate School for Cellular and Biomedical Sciences, University of Bern, 3012 Bern, Switzerland · ARTORG Center for Biomedical Engineering Research, University of Bern, 3008 Bern, Switzerland · Department of Endocrinology, Maastricht University Medical Centre+, and Cardiovascular Research Institute Maastricht (CARIM), Maastricht University, Maastricht, the Netherlands

Abstract

Diet is a major determinant of health, yet accurate dietary assessment from free-living meal images remains challenging. Existing retrieval-based Vision-Language Model (VLM) approaches to nutrient estimation typically rely on textual food representations, while, to our knowledge, direct grounding on visually similar annotated meals has not been investigated. We present NutriVisRAG, a retrieval-augmented generation framework that uses a vision encoder to retrieve similar meals with known nutritional annotations and provide them as multimodal in-context exemplars for VLM nutrient estimation at inference time, without task-specific fine-tuning. We evaluate NutriVisRAG across four datasets, four VLMs, three retrieval encoders, and two prompting strategies, including free-living meals from adults with insulin-treated diabetes, where carbohydrate estimates are compared against two dietitians' annotations. Across all 96 configurations (4 datasets x 4 VLMs x 3 retrieval encoders x 2 prompting strategies), NutriVisRAG reduced the primary estimation error in 56, with improvements more frequent under whole-meal (35/48) than per-item prompting (21/48). Across the evaluated configurations, the lowest NutriVisRAG error was below the best Vanilla error in 13 of 16 VLM--dataset pairs. For GPT-5.4-mini, under whole-meal prompting, the best-performing retrieval encoder for each dataset reduced the primary error on all four datasets relative to Vanilla, with reductions of 1.7--35.7%. Overall, our results show that grounding in multimodal exemplars comprising images and respective annotations can improve nutrient estimation at inference time without task-specific fine-tuning, although the benefits remain configuration-dependent.

Figures & tables

Explore similar work

CardsList
  1. NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation

    Sep 27, 2026Avinash Anand, Aman Kumar, Chaitanya Lakhchaura +4Feature FusionMulti-Source Image Fusion

  2. NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis

    Jun 8, 2026Runze Yan, Minxiao Wang, Jiaying Lu +3Multimodal Large Language Models

  3. Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

    Oct 8, 2026Jingbo Yue, Bruce Coburn, Jinge Ma +2Visual Grounding