Dual-Imbalance Continual Learning for Real-World Food Recognition

Xiaoyan Zhang; Jiangpeng He

cs.CVMar 31, 2026

Dual-Imbalance Continual Learning for Real-World Food Recognition

Authors: Xiaoyan Zhang, Jiangpeng He

Organizations: University of Michigan, Ann Arbor, Michigan, U.S.A · Indiana University, Bloomington, Indiana, U.S.A

Abstract

Visual food recognition in real-world dietary logging scenarios naturally exhibits severe data imbalance, where a small number of food categories appear frequently while many others occur rarely, resulting in long-tailed class distributions. In practice, food recognition systems often operate in a continual learning setting, where new categories are introduced sequentially over time. However, existing studies typically assume that each incremental step introduces a similar number of new food classes, which rarely happens in real world where the number of newly observed categories can vary significantly across steps, leading to highly uneven learning dynamics. As a result, continual food recognition exhibits a dual imbalance: imbalanced samples within each food class and imbalanced numbers of new food classes to learn at each incremental learning step. In this work, we introduce DIME, a Dual-Imbalance-aware Adapter Merging framework for continual food recognition. DIME learns lightweight adapters for each task using parameter-efficient fine-tuning and progressively integrates them through a class-count guided spectral merging strategy. A rank-wise threshold modulation mechanism further stabilizes the merging process by preserving dominant knowledge while allowing adaptive updates. The resulting model maintains a single merged adapter for inference, enabling efficient deployment without accumulating task-specific modules. Experiments on realistic long-tailed food benchmarks under our step-imbalanced setup show that the proposed method consistently improves by more than 3% over the strongest existing continual learning baselines. Code is available at https://github.com/xiaoyanzhang1/DIME.

Figures & tables

Figure 1: Illustration of the dual-imbalanced continual food recognition scenario. Tasks contain different numbers of classes (step imbalance), and each class has varying numbers of samples (long-tailed class distribution [ 32 ] ).

Figure 2: Overview of DIME. (a) Sequential food datasets are learned with lightweight adapters on a frozen backbone using Balanced Softmax. After each step, the new adapter is merged with the accumulated base adapter to produce a unified model. (b) The merging module performs spectral alignment via SVD, class-count guided weighting, and rank-wise threshold modulation.

Datasets	VFN186-LT			VFN186-Insulin			VFN186-T2D			Food101-LT
Evaluation Metrics	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$	$A_{T}$	$\bar{A}$	$\mathrm{w}\bar{A}$
SLCA [ 57 ]	63.07	70.94	70.30	63.10	73.11	71.59	63.74	73.19	72.15	72.35	80.25	79.53
InfLoRA [ 29 ]	51.49	62.10	60.99	51.83	62.64	62.02	52.58	62.95	61.91	57.80	72.23	70.84
EASE [ 61 ]	64.86	73.87	72.75	65.11	74.08	72.92	65.62	75.50	74.25	68.00	78.10	77.01
SEMA [ 48 ]	46.22	58.78	57.68	43.45	58.50	57.97	45.17	60.16	59.69	43.88	60.88	60.11
CL-LoRA [ 18 ]	60.40	72.53	71.68	60.57	74.09	72.89	58.84	72.83	71.19	55.82	72.48	71.47

Table 1: Last task accuracy (

A_{T}

), average accuracy (

\bar{A}

), and weighted average accuracy (

\mathrm{w}\bar{A}

) on VFN186-LT, VFN186-Insulin, VFN186-T2D, and Food101-LT with

\rho=0.01

. Results are averaged over 5 runs. The best results are in bold and the second-best results are underlined .

Figure 3: Performance comparison under different step-imbalance ratios $\rho$ on VFN186-LT and Food101-LT with $T=10$ incremental steps. Our method consistently achieves higher and more stable performance across tasks under all imbalance settings.

Datasets	VFN186-T2D			Food101-LT
Evaluation Metrics	Large	Middle	Small	Large	Middle	Small
SLCA [ 57 ]	57.47	69.45	67.64	73.32	74.01	70.9
InfLoRA [ 29 ]	57.57	43.65	22.91	67.55	52.39	16.00
EASE [ 61 ]	64.75	68.00	71.64	59.96	77.89	81.4
SEMA [ 48 ]	42.87	44.76	31.27	46.51	59.06	25.30
CL-LoRA [ 18 ]	49.85	57.93	76.36	46.45	70.67	72.7

Table 2: Task-wise accuracy on large, middle, and small tasks on VFN186-T2D and Food101-LT. The best results are in bold and the second-best results are underlined .

Variant	SM	CCW	RTM	BSM	VFN186-Insulin
					$A_{T}$	$\mathrm{w}\bar{A}$
Base	(Direct Merge)	(Equal Weight)		(CE)	66.73	74.90
+ SM	$\checkmark$	(Equal Weight)		(CE)	67.20	74.95
+ CCW	$\checkmark$	$\checkmark$		(CE)	67.95	76.68
+ RTM	$\checkmark$	$\checkmark$	$\checkmark$	(CE)	68.68	77.67
+ BSM	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	69.31	78.07

Table 3: Step-by-step ablation study on VFN186-Insulin. The best results are in bold .

Method	Inference Time (s) $\downarrow$	FLOPs (G) $\downarrow$	Accuracy (%) $\uparrow$
MOS	89.15	372.27	65.70
ACMap	10.01	33.73	64.52
EASE	74.67	335.64	65.35
SEMA	9.16	33.84	39.61
CL-LoRA	80.90	335.82	49.87
DIME (Ours)	9.50	33.73	68.49

Table 4: Efficiency comparison between adapter-based continual learning methods. The best results are in bold and the second-best results are underlined .

Panel of Figure 4 — Figure 4: Sensitivity analysis of rank-wise threshold parameters.

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 162 papers

Continual Learning for Food Category Classification Dataset: Enhancing Model Adaptability and Performance
Mar 20, 2026Piyush Kaushik Bhattacharyya, Devansh Tomar, Shubham Mishra +6Continual Learning Class-Incremental Learning
School of Computer Engineering, KIIT University, Bhubaneshwar, India
FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis
May 6, 2026Woojin Lee, Pranav Mekkoth, Ye Tian +2Hierarchical Classification Fine-Grained Image Classification
Department of Computer Science and Engineering University of California, San Diego (UCSD)
DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding
Jul 25, 2026Yilin Wang, Haochen Shi, Guanyu Chen +4Image Segmentation Instance Segmentation
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Hangzhou Dianzi University, Hangzhou, China +3

Mar 20, 2026J/K move · Enter open · S save

Continual Learning for Food Category Classification Dataset: Enhancing Model Adaptability and Performance

Piyush Kaushik Bhattacharyya, Devansh Tomar, Shubham Mishra, Divyanshu Rai+5

School of Computer Engineering, KIIT University, Bhubaneshwar, India

Continual Learning Class-Incremental Learning Image Classification

Open paper AskPDF arXiv

Conventional machine learning pipelines often struggle to recognize categories absent from the original trainingset. This gap typically reduces accuracy, as fixed datasets rarely capture the full diversity of a domain. To address this, we propose a continual learning framework for text-guided food classification. Unlike approaches that require retraining from scratch, our method enables incremental updates, allowing new categories to be integrated without degrading prior knowledge. For example, a model trained on Western cuisines could later learn to classify dishes such as dosa or kimchi. Although further refinements are needed, this design shows promise for adaptive food recognition, with applications in dietary monitoring and personalized nutrition planning.

Abstract

Class-Incremental Learning

Explore similar work

Continual Learning for Food Category Classification Dataset: Enhancing Model Adaptability and Performance

FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis

DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding