NutriVisRAG: Multimodal Exemplar Grounding for Nutrient Estimation with Vision-Language Models
Organizations: Graduate School for Cellular and Biomedical Sciences, University of Bern, 3012 Bern, Switzerland · ARTORG Center for Biomedical Engineering Research, University of Bern, 3008 Bern, Switzerland · Department of Endocrinology, Maastricht University Medical Centre+, and Cardiovascular Research Institute Maastricht (CARIM), Maastricht University, Maastricht, the Netherlands
Abstract
Diet is a major determinant of health, yet accurate dietary assessment from free-living meal images remains challenging. Existing retrieval-based Vision-Language Model (VLM) approaches to nutrient estimation typically rely on textual food representations, while, to our knowledge, direct grounding on visually similar annotated meals has not been investigated. We present NutriVisRAG, a retrieval-augmented generation framework that uses a vision encoder to retrieve similar meals with known nutritional annotations and provide them as multimodal in-context exemplars for VLM nutrient estimation at inference time, without task-specific fine-tuning. We evaluate NutriVisRAG across four datasets, four VLMs, three retrieval encoders, and two prompting strategies, including free-living meals from adults with insulin-treated diabetes, where carbohydrate estimates are compared against two dietitians' annotations. Across all 96 configurations (4 datasets x 4 VLMs x 3 retrieval encoders x 2 prompting strategies), NutriVisRAG reduced the primary estimation error in 56, with improvements more frequent under whole-meal (35/48) than per-item prompting (21/48). Across the evaluated configurations, the lowest NutriVisRAG error was below the best Vanilla error in 13 of 16 VLM--dataset pairs. For GPT-5.4-mini, under whole-meal prompting, the best-performing retrieval encoder for each dataset reduced the primary error on all four datasets relative to Vanilla, with reductions of 1.7--35.7%. Overall, our results show that grounding in multimodal exemplars comprising images and respective annotations can improve nutrient estimation at inference time without task-specific fine-tuning, although the benefits remain configuration-dependent.
Figures & tables
| Dataset | Evaluation queries | Indexed references | Evaluation targets |
|---|---|---|---|
| Nutrition5k [ 51 ] | 815 | 2,445 | Weight, energy, CHO, protein, fat |
| SNAPMe [ 24 ] | 323 | 1,149 | Energy, CHO, protein, fat |
| MADiMa [ 7 ] | 160 (80 meals each at 40 and 60 cm) | – | Weight, energy, CHO, protein, fat |
| EDUCATE [ 12 ] | 247 | – | CHO |
| Per-item | Whole-meal | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| NutriVisRAG | NutriVisRAG | ||||||||
| Query set | Model | Vanilla | PE-Core | SigLIP2 | Nomic Vision | Vanilla | PE-Core | SigLIP2 | Nomic Vision |
| Nutrition5k | Gemma-4-31B | 53.6 | 53.7 | 54.6 | 59.8 | 63.5 | 51.0 ⋆ | 50.8 ⋆ | 55.8 ⋆ |
| GPT-5.4-mini | 58.0 | 42.7 ⋆ | 43.4 ⋆ | 47.1 ⋆ | 58.1 | 43.4 ⋆ | 44.2 ⋆ | 48.0 ⋆ | |
| Mistral-Small-3.2 | 72.1 | 111.2 ▼ | 119.5 ▼ | 145.1 ▼ | 78.7 | 62.3 ⋆ | 63.3 ⋆ | 74.3 ⋆ | |
| GLM-5.3-Flash | 54.0 | 54.1 ▼ | 53.4 ⋆ | 58.6 ▼ | 56.2 | 58.8 ▼ | 56.8 ▼ | 62.2 ▼ | |
| Per-item | Whole-meal | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Query set | Model | Encoder | Def. | Def. | Def. | Def. | Def. | Def. | ||||||||||||
| Nutrition5k | Gemma-4-31B | PE-Core | 68.6 | 64.0 | 68.4 | 53.7 | 50.1 | 53.0 | 60.1 | 60.4 | 61.8 | 66.9 ▼ | 50.7 | 70.5 | 52.8 ▼ | 44.0 | 57.2 | 57.4 ▼ | 50.0 | 65.4 |
| SigLIP2 | 70.8 ▼ | 45.7 | 75.7 | 54.2 ▼ | 44.8 | 60.4 | 61.1 ▼ | 49.8 | 73.1 ▼ | 64.0 ▼ | 47.9 | 53.3 | 55.3 ▼ | 44.3 | 50.5 | 58.8 ▼ | 48.4 | 55.4 | ||
| Nomic Vision | 66.6 | 64.4 | 66.7 | 79.7 | 72.2 | 103.9 | 87.8 | 80.7 | 102.8 | 70.5 ▼ | 52.5 | 64.9 | 75.0 | 55.7 | 70.4 | 85.5 | 62.5 | 74.5 | ||
| GPT-5.4-mini | PE-Core | 31.1 | 38.5 | 34.9 | 28.6 | 32.8 | 32.8 | 32.9 | 40.0 | 36.1 | 39.3 | 35.8 | 31.1 | 34.0 | 34.7 | 30.6 | 41.2 | 40.4 | 34.4 | |
| Per-item | Whole-meal | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Query set | Model | Setting | Weight | Energy | CHO | Protein | Fat | Macro | Micro | Weight | Energy | CHO | Protein | Fat | Macro | Micro |
| Nutrition5k | Gemma-4-31B | Vanilla | 36.3 | 47.7 | 56.0 | 44.3 | 63.3 | 54.5 | 53.6 | 49.2 | 58.4 | 69.6 | 46.4 | 78.4 | 64.8 | 63.5 |
| NutriVisRAG (PE-Core) | 48.4 ▼ | 46.0 | 61.0 | 45.8 | 52.9 ⋆ | 53.2 | 53.7 | 47.4 ⋆ | 44.9 ⋆ | 55.9 ⋆ | 43.9 ⋆ | 53.4 ⋆ | 51.0 | 51.0 ⋆ | ||
| NutriVisRAG (SigLIP2) | 47.6 ▼ | 47.8 | 58.6 | 47.1 | 58.8 ⋆ | 54.9 | 54.6 | 46.7 ⋆ | 45.3 ⋆ | 54.6 ⋆ | 43.5 ⋆ | 55.1 ⋆ | 51.1 | 50.8 ⋆ | ||
| NutriVisRAG (Nomic Vision) | 46.3 ▼ | 51.4 | 59.8 | 55.5 | 66.0 | 60.5 | 59.8 | 46.5 ⋆ | 49.6 ⋆ | 56.5 ⋆ | 51.2 | 61.4 ⋆ | 56.4 | 55.8 ⋆ | ||
| GPT-5.4-mini | Vanilla | 35.8 | 47.2 | 66.2 | 45.8 | 62.2 | 58.0 | 58.0 | 40.1 | 48.1 | 60.8 | 47.2 | 69.9 | 59.3 | 58.1 | |
| Per-item | Whole-meal | |||||||||||||||
| Query set | Model | Setting | Weight | Energy | CHO | Protein | Fat | Macro | Micro | Weight | Energy | CHO | Protein | Fat | Macro | Micro |
| MADiMa | Gemma-4-31B | Vanilla | 51.2 | 49.1 | 60.1 | 38.9 | 64.2 | 54.4 | 54.7 | 76.5 | 69.5 | 76.4 | 48.5 | 89.1 | 71.3 | 70.7 |
| NutriVisRAG (PE-Core) | 38.8 ⋆ | 40.5 ⋆ | 54.6 ⋆ | 41.7 | 52.7 ⋆ | 49.7 | 50.5 ⋆ | 71.9 ⋆ | 62.4 ⋆ | 64.9 ⋆ | 43.1 ⋆ | 85.8 | 64.6 | 62.7 ⋆ | ||
| NutriVisRAG (SigLIP2) | 38.3 ⋆ | 38.8 ⋆ | 51.1 ⋆ | 37.5 | 53.3 ⋆ | 47.3 | 47.5 ⋆ | 69.7 ⋆ | 64.6 ⋆ | 62.2 ⋆ | 45.1 | 93.8 | 67.0 | 63.5 ⋆ | ||
| NutriVisRAG (Nomic Vision) | 41.5 ⋆ | 40.1 ⋆ | 53.8 ⋆ | 40.8 | 58.3 | 51.0 | 50.9 ⋆ | 74.3 | 57.5 ⋆ | 64.1 ⋆ | 40.8 ⋆ | 79.2 ⋆ | 61.4 | 60.3 ⋆ | ||
| GPT-5.4-mini | Vanilla | 53.8 | 45.0 | 75.8 | 32.8 | 46.9 | 51.8 | 57.4 | 93.0 | 76.4 | 89.2 | 38.5 | 96.2 | 74.6 | 75.7 | |
| VLM | Prompt | Vanilla | PE-Core | SigLIP 2 | Nomic Vision |
|---|---|---|---|---|---|
| Gemma-4-31B | Per-item | 49.8 | 50.1 | 48.7 | 51.9 |
| Whole-meal | 59.3 | 52.1 | 51.6 | 52.6 | |
| GPT-5.4-mini | Per-item | 52.2 | 43.0 | 44.0 | 45.4 |
| Whole-meal | 58.5 | 45.0 | 46.0 | 47.5 | |
| Mistral-Small-3.2 | Per-item | 67.5 | 94.1 | 109.5 | 112.2 |
| Whole-meal | 62.2 | 60.8 | 64.7 | 69.9 |
| Energy | Micro / CHO | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Query set | Model | Encoder | NutriVisRAG | Lexical | Qwen3 | Gemma | Nomic Text | NutriVisRAG | Lexical | Qwen3 | Gemma | Nomic Text |
| Nutrition5k | GPT-5.4-mini | PE-Core | 32.8 | 96.0 ▼ | 57.9 | 90.6 ▼ | 78.3 ▼ | 40.0 | 131.9 ▼ | 61.5 | 132.9 ▼ | 83.4 ▼ |
| SigLIP2 | 30.2 | 147.3 ▼ | 48.2 | 82.7 ▼ | 77.7 ▼ | 35.6 | 172.7 ▼ | 54.3 | 126.8 ▼ | 86.0 ▼ | ||
| Nomic Vision | 29.9 | 127.7 ▼ | 54.5 ▼ | 90.4 ▼ | 80.8 ▼ | 37.6 | 161.2 ▼ | 57.6 ▼ | 137.4 ▼ | 81.5 ▼ | ||
| Gemma-4-31B | PE-Core | 50.1 | 144.4 ▼ | 70.1 | 117.9 ▼ | 82.9 ▼ | 60.4 | 176.2 ▼ | 84.5 | 164.5 ▼ | 96.4 ▼ | |
| SigLIP2 | 44.8 | 128.7 ▼ | 59.0 | 80.5 ▼ | 74.2 ▼ | 49.8 | 147.6 ▼ | 64.0 | 106.5 ▼ | 82.2 ▼ | ||
| Per-item | Whole-meal | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Energy | Micro / CHO | Energy | Micro / CHO | |||||||||||||||
| Query set | Model | Encoder | Default | Jina | Qwen3 | Nem. | Default | Jina | Qwen3 | Nem. | Default | Jina | Qwen3 | Nem. | Default | Jina | Qwen3 | Nem. |
| Nutrition5k | GPT-5.4-mini | PE-Core | 32.8 | 29.9 | 28.7 | 38.5 | 40.0 | 35.2 | 35.6 | 43.5 | 34.7 | 31.5 | 36.8 | 47.4 | 40.4 | 36.9 | 40.9 | 49.8 |
| SigLIP2 | 30.2 | 33.0 | 30.6 | 30.2 | 35.6 | 41.7 | 34.2 | 36.7 | 36.6 | 33.1 | 34.6 | 37.6 | 39.9 | 39.7 | 40.5 | 41.6 | ||
| Nomic Vision | 29.9 | 30.5 | 27.6 | 37.6 | 37.6 | 38.2 | 36.6 | 45.1 | 40.1 | 38.5 | 36.6 | 39.6 | 46.8 | 44.1 | 40.3 | 47.6 | ||
| Gemma-4-31B | PE-Core | 50.1 | 57.1 | 51.5 | 71.9 ▼ | 60.4 | 63.9 | 55.3 | 77.8 ▼ | 44.0 | 50.4 | 50.5 | 65.5 ▼ | 50.0 | 57.1 | 57.3 | 68.5 ▼ | |
In-Context Learning
In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.
See all 418 papers