cs.AIMar 3, 2026

FEAST: Retrieval-Augmented Multi-Hierarchical Food Classification for the FoodEx2 System

Authors: Lorenzo Molfetta, Alessio Cocchieri, Stefano Fantazzini, Giacomo Frisoni, Luca Ragazzi, Gianluca Moro

Organizations: Department of Computer Science and Engineering, University of Bologna, Cesena Campus, Via dell’Università 50, I-47522 Cesena, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Hierarchical text classification (HTC) and extreme multi-label classification (XML) tasks face compounded challenges from complex label interdependencies, data sparsity, and extreme output dimensions. These challenges are exemplified in the European Food Safety Authority's FoodEx2 system-a standardized food classification framework essential for food consumption monitoring and contaminant exposure assessment across Europe. FoodEx2 coding transforms natural language food descriptions into a set of codes from multiple standardized hierarchies, but faces implementation barriers due to its complex structure. Given a food description (e.g., "organic yogurt''), the system identifies its base term ("yogurt''), all the applicable facet categories (e.g., "production method''), and then, every relevant facet descriptors to each category (e.g., "organic production''). While existing models perform adequately on well-balanced and semantically dense hierarchies, no work has been applied on the practical constraints imposed by the FoodEx2 system. The limited literature addressing such real-world scenarios further compounds these challenges. We propose FEAST (Food Embedding And Semantic Taxonomy), a novel retrieval-augmented framework that decomposes FoodEx2 classification into a three-stage approach: (1) base term identification, (2) multi-label facet prediction, and (3) facet descriptor assignment. By leveraging the system's hierarchical structure to guide training and performing deep metric learning, FEASTlearns discriminative embeddings that mitigate data sparsity and improve generalization on rare and fine-grained labels. Evaluated on the multilingual FoodEx2 benchmark, FEAST outperforms the prior European's CNN baseline F1 scores by 12-38 % on rare classes.

Figures & tables

Figure 1: Illustration of the FoodEx2 coding system. An input food item is represented by a base term and facet descriptors detailing its key attributes, such as ingredients, processing, and packaging.
Figure 2: Overview of the proposed FoodEx2 food classification pipeline, which operates in three main stages: (1) Base Term Classification , where top- KK candidate base terms are retrieved and reranked to produce a top-1 base term; (2) Facet Category Classification , which identifies relevant facet categories using either an LLM, a multi-label classifier, or a bi-encoder approach; and (3) Facet Descriptor Classification , which retrieves and reranks descriptor candidates for the previously selected facet categories. The system combines encoder-based retrieval with flexible reranking strategies (LLM instruct or cross-encoder) to map free-text food descriptions into structured FoodEx2 codes. ⊕\oplus denotes concatenation and fp(⋅)f_{p}(\cdot) represents a prompt template function.
Figure 3: Prompt templates used for LLM’s multitask instruction tuning across base term, category, and descriptor classification tasks.
Figure 4: Comparison of F1-scores between LLaMA-3.1-8B LLaMA-3.1-8B ( AI, 2024 ) and DeBERTa-v3-small DeBERTa-v3-small ( He et al., 2021 ) across different facet categories (Task II).
More in this topic

Hierarchical Classification

Hierarchical classification uses relationships among labels to predict categories at multiple levels, from broad groups to fine-grained classes.

See all 89 papers

Explore similar work

CardsList
May 6, 2026

FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis

The widespread adoption of camera-equipped mobile devices and wearables has enabled convenient capture of meal images, making food recognition a key component for real time dietary monitoring. However, real-world food images present challenges due to high intra-class similarity and the frequent presence of multiple food items within a single image. While deep learning models achieve strong performance in coarse grained classification, they often struggle to capture fine-grained attributes such as cooking style. Moreover, open-ended generation in modern vision-language models can produce non-canonical labels, limiting their practical deployment. We propose FoodCHA, a multimodal agentic framework that reformulates food recognition as a hierarchical decision-making process. By progressively anchoring predictions, FoodCHA guides subcategory identification using high-level categories and guides cooking style recognition using subcategories, improving semantic consistency and attribute-level discrimination. To ensure practical deployability, FoodCHA utilizes the compact Moondream-2B vision language model, which provides strong reasoning capability while maintaining lower computational and memory overhead. Experiments on FoodNExTDB show that FoodCHA outperforms Food-Llama-3.2-11B by 13.8% and 38.2% in category and subcategory recognition precision, respectively, and achieves a striking 153.2% improvement in cooking style classification precision.

Woojin Lee, Pranav Mekkoth, Ye Tian +2

Department of Computer Science and Engineering University of California, San Diego (UCSD)

Hierarchical ClassificationFine-Grained Image ClassificationVLM ReasoningFine-Grained Visual PerceptionImage Classification
AskPDF
Mar 10, 2026

Beyond Fine-Tuning: Robust Food Entity Linking under Ontology Drift with FoodOntoRAG

Standardizing food terms from product labels and menus into ontology concepts is a prerequisite for trustworthy dietary assessment and safety reporting. The dominant approach to Named Entity Linking (NEL) in the food and nutrition domains fine-tunes Large Language Models (LLMs) on task-specific corpora. Although effective, fine-tuning incurs substantial computational cost, ties models to a particular ontology snapshot (i.e., version), and degrades under ontology drift. This paper presents FoodOntoRAG, a model- and ontology-agnostic pipeline that performs few-shot NEL by retrieving candidate entities from domain ontologies and conditioning an LLM on structured evidence (food labels, synonyms, definitions, and relations). A hybrid lexical--semantic retriever enumerates candidates; a selector agent chooses a best match with rationale; a separate scorer agent calibrates confidence; and, when confidence falls below a threshold, a synonym generator agent proposes reformulations to re-enter the loop. The pipeline approaches state-of-the-art accuracy while revealing gaps and inconsistencies in existing annotations. The design avoids fine-tuning, improves robustness to ontology evolution, and yields interpretable decisions through grounded justifications.

Jan Drole, Ana Gjorgjevikj, Barbara Korouši'c Seljak +1

Computer Systems Department, Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · Computer Systems Department, Jožef Stefan Institute, Ljubljana, Slovenia

Entity LinkingLanguage Model RobustnessRetrieval-Augmented Generation
AskPDF
Jul 25, 2026

DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding

Food segmentation is essential for applications such as intelligent catering, dietary assessment, and recommendation. However, existing benchmarks fail to capture the complexity of real-world dining scenes. The challenges of dense inter-dish overlap, fine-grained class similarity, and extreme long-tail class distributions exceed the fidelity of current datasets. To fill this gap, we introduce DishSeg24k, a large-scale dish-level segmentation benchmark with 24,096 images, 112,281 instances, and 278 fine-grained categories in real-world dining environments. Based on DishSeg24k, we further propose Food Expert-Adaptive Segmentation Transformers (FEAST) to address these challenges. FEAST models query-based decoding as a Markov Decision Process (MDP), where each decoder layer update is treated as a sequential decision step that explores uncertainty along dish boundaries. We further redesign the decoder with a reinforcement learning (RL)-guided Mixture-of-Experts (MoE) module, in which a dual-critic decoupled optimization scheme separates task-oriented query refinement from structure-aware expert routing. This design promotes expert specialization and prevents expert collapse under long-tail category distributions. Finally, extensive experiments on DishSeg24k demonstrate the state-of-the-art performance of FEAST, which outperforms previous methods by {+3.21%} mIoU, {+3.68%} mDice, and {+4.00%} mAcc, respectively. We further validate the effectiveness of FEAST on FoodSeg103. The dataset and code will be publicly released.

Yilin Wang, Haochen Shi, Guanyu Chen +4

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Hangzhou Dianzi University, Hangzhou, China +3

Image SegmentationInstance SegmentationMixture of Experts
AskPDF