Dual-Imbalance Continual Learning for Real-World Food Recognition

Xiaoyan Zhang; Jiangpeng He

cs.CVMar 31, 2026

Dual-Imbalance Continual Learning for Real-World Food Recognition

Authors: Xiaoyan Zhang, Jiangpeng He

Organizations: University of Michigan, Ann Arbor, Michigan, U.S.A · Indiana University, Bloomington, Indiana, U.S.A

Abstract

Visual food recognition in real-world dietary logging scenarios naturally exhibits severe data imbalance, where a small number of food categories appear frequently while many others occur rarely, resulting in long-tailed class distributions. In practice, food recognition systems often operate in a continual learning setting, where new categories are introduced sequentially over time. However, existing studies typically assume that each incremental step introduces a similar number of new food classes, which rarely happens in real world where the number of newly observed categories can vary significantly across steps, leading to highly uneven learning dynamics. As a result, continual food recognition exhibits a dual imbalance: imbalanced samples within each food class and imbalanced numbers of new food classes to learn at each incremental learning step. In this work, we introduce DIME, a Dual-Imbalance-aware Adapter Merging framework for continual food recognition. DIME learns lightweight adapters for each task using parameter-efficient fine-tuning and progressively integrates them through a class-count guided spectral merging strategy. A rank-wise threshold modulation mechanism further stabilizes the merging process by preserving dominant knowledge while allowing adaptive updates. The resulting model maintains a single merged adapter for inference, enabling efficient deployment without accumulating task-specific modules. Experiments on realistic long-tailed food benchmarks under our step-imbalanced setup show that the proposed method consistently improves by more than 3% over the strongest existing continual learning baselines. Code is available at https://github.com/xiaoyanzhang1/DIME.

Figures & tables

Figure 1: Illustration of the dual-imbalanced continual food recognition scenario. Tasks contain different numbers of classes (step imbalance), and each class has varying numbers of samples (long-tailed class distribution [ 32 ] ).

Figure 2: Overview of DIME. (a) Sequential food datasets are learned with lightweight adapters on a frozen backbone using Balanced Softmax. After each step, the new adapter is merged with the accumulated base adapter to produce a unified model. (b) The merging module performs spectral alignment via SVD, class-count guided weighting, and rank-wise threshold modulation.

Datasets	VFN186-LT			VFN186-Insulin			VFN186-T2D			Food101-LT
Evaluation Metrics	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$
SLCA [ 57 ]	63.07	70.94	70.30	63.10	73.11	71.59	63.74	73.19	72.15	72.35	80.25	79.53
InfLoRA [ 29 ]	51.49	62.10	60.99	51.83	62.64	62.02	52.58	62.95	61.91	57.80	72.23	70.84
EASE [ 61 ]	64.86	73.87	72.75	65.11	74.08	72.92	65.62	75.50	74.25	68.00	78.10	77.01
SEMA [ 48 ]	46.22	58.78	57.68	43.45	58.50	57.97	45.17	60.16	59.69	43.88	60.88	60.11
CL-LoRA [ 18 ]	60.40	72.53	71.68	60.57	74.09	72.89	58.84	72.83	71.19	55.82	72.48	71.47

Table 1: Last task accuracy (

A_{T}

), average accuracy (

\bar{A}

), and weighted average accuracy (

\mathrm{w}\bar{A}

) on VFN186-LT, VFN186-Insulin, VFN186-T2D, and Food101-LT with

\rho=0.01

. Results are averaged over 5 runs. The best results are in bold and the second-best results are underlined .

Figure 3: Performance comparison under different step-imbalance ratios $\rho$ on VFN186-LT and Food101-LT with $T=10$ incremental steps. Our method consistently achieves higher and more stable performance across tasks under all imbalance settings.

Datasets	VFN186-T2D			Food101-LT
Evaluation Metrics	Large	Middle	Small	Large	Middle	Small
SLCA [ 57 ]	57.47	69.45	67.64	73.32	74.01	70.9
InfLoRA [ 29 ]	57.57	43.65	22.91	67.55	52.39	16.00
EASE [ 61 ]	64.75	68.00	71.64	59.96	77.89	81.4
SEMA [ 48 ]	42.87	44.76	31.27	46.51	59.06	25.30
CL-LoRA [ 18 ]	49.85	57.93	76.36	46.45	70.67	72.7

Table 2: Task-wise accuracy on large, middle, and small tasks on VFN186-T2D and Food101-LT. The best results are in bold and the second-best results are underlined .

Variant	SM	CCW	RTM	BSM	VFN186-Insulin
					$A_{T}$	$\mathrm{w}\bar{A}$
Base	(Direct Merge)	(Equal Weight)		(CE)	66.73	74.90
+ SM	$\checkmark$	(Equal Weight)		(CE)	67.20	74.95
+ CCW	$\checkmark$	$\checkmark$		(CE)	67.95	76.68
+ RTM	$\checkmark$	$\checkmark$	$\checkmark$	(CE)	68.68	77.67
+ BSM	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	69.31	78.07

Table 3: Step-by-step ablation study on VFN186-Insulin. The best results are in bold .

Method	Inference Time (s) $\downarrow$	FLOPs (G) $\downarrow$	Accuracy (%) $\uparrow$
MOS	89.15	372.27	65.70
ACMap	10.01	33.73	64.52
EASE	74.67	335.64	65.35
SEMA	9.16	33.84	39.61
CL-LoRA	80.90	335.82	49.87
DIME (Ours)	9.50	33.73	68.49

Table 4: Efficiency comparison between adapter-based continual learning methods. The best results are in bold and the second-best results are underlined .

Panel of Figure 4 — Figure 4: Sensitivity analysis of rank-wise threshold parameters.

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 162 papers

Mar 20, 2026

Continual Learning for Food Category Classification Dataset: Enhancing Model Adaptability and Performance

Conventional machine learning pipelines often struggle to recognize categories absent from the original trainingset. This gap typically reduces accuracy, as fixed datasets rarely capture the full diversity of a domain. To address this, we propose a continual learning framework for text-guided food classification. Unlike approaches that require retraining from scratch, our method enables incremental updates, allowing new categories to be integrated without degrading prior knowledge. For example, a model trained on Western cuisines could later learn to classify dishes such as dosa or kimchi. Although further refinements are needed, this design shows promise for adaptive food recognition, with applications in dietary monitoring and personalized nutrition planning.

Piyush Kaushik Bhattacharyya, Devansh Tomar, Shubham Mishra +6

School of Computer Engineering, KIIT University, Bhubaneshwar, India

Continual Learning Class-Incremental Learning Image Classification

Ask PDF

May 6, 2026

FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis

The widespread adoption of camera-equipped mobile devices and wearables has enabled convenient capture of meal images, making food recognition a key component for real time dietary monitoring. However, real-world food images present challenges due to high intra-class similarity and the frequent presence of multiple food items within a single image. While deep learning models achieve strong performance in coarse grained classification, they often struggle to capture fine-grained attributes such as cooking style. Moreover, open-ended generation in modern vision-language models can produce non-canonical labels, limiting their practical deployment. We propose FoodCHA, a multimodal agentic framework that reformulates food recognition as a hierarchical decision-making process. By progressively anchoring predictions, FoodCHA guides subcategory identification using high-level categories and guides cooking style recognition using subcategories, improving semantic consistency and attribute-level discrimination. To ensure practical deployability, FoodCHA utilizes the compact Moondream-2B vision language model, which provides strong reasoning capability while maintaining lower computational and memory overhead. Experiments on FoodNExTDB show that FoodCHA outperforms Food-Llama-3.2-11B by 13.8% and 38.2% in category and subcategory recognition precision, respectively, and achieves a striking 153.2% improvement in cooking style classification precision.

Woojin Lee, Pranav Mekkoth, Ye Tian +2

Department of Computer Science and Engineering University of California, San Diego (UCSD)

Hierarchical Classification Fine-Grained Image Classification VLM Reasoning Fine-Grained Visual Perception Image Classification

Ask PDF

Jul 25, 2026

DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding

Food segmentation is essential for applications such as intelligent catering, dietary assessment, and recommendation. However, existing benchmarks fail to capture the complexity of real-world dining scenes. The challenges of dense inter-dish overlap, fine-grained class similarity, and extreme long-tail class distributions exceed the fidelity of current datasets. To fill this gap, we introduce DishSeg24k, a large-scale dish-level segmentation benchmark with 24,096 images, 112,281 instances, and 278 fine-grained categories in real-world dining environments. Based on DishSeg24k, we further propose Food Expert-Adaptive Segmentation Transformers (FEAST) to address these challenges. FEAST models query-based decoding as a Markov Decision Process (MDP), where each decoder layer update is treated as a sequential decision step that explores uncertainty along dish boundaries. We further redesign the decoder with a reinforcement learning (RL)-guided Mixture-of-Experts (MoE) module, in which a dual-critic decoupled optimization scheme separates task-oriented query refinement from structure-aware expert routing. This design promotes expert specialization and prevents expert collapse under long-tail category distributions. Finally, extensive experiments on DishSeg24k demonstrate the state-of-the-art performance of FEAST, which outperforms previous methods by {+3.21%} mIoU, {+3.68%} mDice, and {+4.00%} mAcc, respectively. We further validate the effectiveness of FEAST on FoodSeg103. The dataset and code will be publicly released.

Yilin Wang, Haochen Shi, Guanyu Chen +4

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Hangzhou Dianzi University, Hangzhou, China +3

Image Segmentation Instance Segmentation Mixture of Experts

Ask PDF

Abstract

Class-Incremental Learning

Explore similar work

Continual Learning for Food Category Classification Dataset: Enhancing Model Adaptability and Performance

FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis

DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding