ChimeraLoRA: Multi-Head LoRA-Guided Synthetic Datasets

Hoyoung Kim; Minwoo Jang; Jabin Koo; Sangdoo Yun; Jungseul Ok

cs.CVFeb 23, 2026

ChimeraLoRA: Multi-Head LoRA-Guided Synthetic Datasets

Authors: Hoyoung Kim, Minwoo Jang, Jabin Koo, Sangdoo Yun, Jungseul Ok

Organizations: Graduate School of AI, POSTECH · Dept. of CSE, POSTECH · NAVER AI Lab

Abstract

Beyond general recognition tasks, specialized domains and fine-grained settings often encounter data scarcity, especially for tail classes. To obtain less biased and more reliable models under such scarcity, practitioners leverage diffusion models to supplement underrepresented regions of real data. Specifically, recent studies fine-tune pretrained diffusion models with LoRA on few-shot real sets to synthesize additional images. While an image-wise LoRA trained on a single image captures fine-grained details yet offers limited diversity, a class-wise LoRA trained over all shots produces diverse images as it encodes class priors yet tends to overlook fine details. To combine both benefits, we separate the adapter into a class-shared LoRA~ $A$ for class priors and per-image LoRAs~ $\mathcal{B}$ for image-specific characteristics. To expose coherent class semantics in the shared LoRA~ $A$ , we propose a semantic boosting by preserving class bounding boxes during training. For generation, we compose $A$ with a mixture of $\mathcal{B}$ using coefficients drawn from a Dirichlet distribution. Across diverse datasets, our synthesized images are both diverse and detail-rich while closely aligning with the few-shot real distribution, yielding robust gains in downstream classification accuracy.

Figures & tables

Figure 1: An overview of the proposed method. (left) We synthesize images with a multi-head LoRA that integrates the strengths of image-wise LoRA (LoFT [ 20 ] ) and class-wise LoRA (DataDream [ 21 ] ). The blue and red regions indicate where LoRA is applied during generation. (center) Given few-shot images, we fine-tune the multi-head LoRA while preserving bounding boxes obtained from Grounded-SAM [ 26 , 22 ] . (right) We merge LoRA heads using weights sampled from a Dirichlet distribution to obtain diverse synthetic images.

Figure 2: Qualitative results of synthetic images. (top) Four real images per class. (bottom) Synthetic images generated with LoRA based methods. For the camera class, LoFT (image-wise LoRA) shows low diversity with near duplicate single viewpoint shots, while DataDream (class-wise LoRA) increases diversity but lowers fidelity, often failing to render a camera. Our multi-head LoRA produces accurate cameras across varied viewpoints. Here, Avg merges heads with uniform weights and Dir uses Dirichlet sampled weights.

Figure 3: Robust generation using semantic boosting (SB). Without SB, a LoRA trained on a one-shot image often fails to render a car even when prompted with “a photo of a car”. With SB, repeated exposure to the car region during training robustly generates complete cars.

Figure 4: Effect of semantic boosting. (a) The input image is used to train a LoRA under varying cropping methods. (b) Without cropping, the generated images exhibit a distorted aspect ratio of the primary object. (c, d) Conventional random and center cropping methods result in outputs where the object is consistently truncated. (e) In contrast, our semantic boosting preserves the object’s structural integrity and details, leading to a robust generation.

Methods	AIR	CAL	CAR	DTD	EUR	FLO	FOD	PET	ISIC	AVG
CLIP (0-shot)	24.7	93.0	65.2	44.4	47.6	71.4	86.1	89.2	21.1	60.3
CLIP (4-shots)	$41.3_{\pm 0.3}$	$95.5_{\pm 0.1}$	$74.3_{\pm 0.2}$	$62.0_{\pm 0.7}$	$83.5_{\pm 0.6}$	$89.9_{\pm 0.4}$	$86.5_{\pm 0.1}$	$93.3_{\pm 0.4}$	$19.6_{\pm 1.5}$	$71.8_{\pm 0.4}$
IsSynth [ 14 ]	$39.9_{\pm 0.1}$	$95.5_{\pm 0.1}$	$71.5_{\pm 0.6}$	60.1 ±0.3	$73.4_{\pm 0.6}$	$89.0_{\pm 1.0}$	$85.6_{\pm 0.1}$	$91.6_{\pm 0.2}$	$23.8_{\pm 1.4}$	$70.1_{\pm 0.4}$
DataDream [ 21 ]	44.3 ±0.4	96.1 ±0.1	81.7 ±0.3	$56.0_{\pm 0.6}$	$72.2_{\pm 0.7}$	92.9 ±0.5	86.0 ±0.1	$92.2_{\pm 0.1}$	$20.7_{\pm 1.1}$	$71.3_{\pm 0.3}$
LoFT [ 20 ]	$41.7_{\pm 0.6}$	$95.7_{\pm 0.1}$	$78.0_{\pm 0.3}$	$58.0_{\pm 1.5}$	85.0 ±0.7	$91.3_{\pm 0.2}$	$85.1_{\pm 0.1}$	92.4 ±0.3	25.6 ±0.3	72.5 ±0.4
ChimeraLoRA	46.0 ±0.7	96.1 ±0.1	79.6 ±0.5	61.6 ±0.5	86.3 ±0.5	93.4 ±0.4	85.7 ±0.1	$\bm{93.4}_{\pm 0.1}$	$\bm{29.2}_{\pm 0.6}$	74.6 ±0.2

Table 1: Downstream performance with synthetic datasets under 4-shot scenarios. Starting with 4-shot labels, we generate 500 additional images per class to train on 504 per class in total, improving accuracy by 2.1 percentage points on average over state-of-the-art baselines across nine datasets. We mark the best in bold and the second best with underlines.

Methods	CIFAR10			ImageNet100			DTD			EuroSAT			Flowers102
Methods	Head	Tail	Avg.	Head	Tail	Avg.	Head	Tail	Avg.	Head	Tail	Avg.	Head	Tail	Avg.
Real	98.8	70.1	84.5	78.8	79.1	79.0	86.5	46.7	66.6	99.2	13.6	56.4	92.8	94.2	93.5
DataDream [ 21 ]	98.1	76.3	87.2	88.7	91.3	90.0	78.8	55.5	67.2	98.8	48.7	73.9	93.3	95.5	94.4
LoFT [ 20 ]	98.4	76.5	87.4	88.9	91.2	90.0	84.3	51.6	67.9	98.7	47.2	73.0	91.5	96.0	93.7
ChimeraLoRA	98.3	81.0	89.6	88.8	91.6	90.2	80.2	56.6	68.4	98.0	51.3	74.5	93.9	96.9	95.4

Table 2: Downstream performance with synthetic datasets under long-tail scenarios. When training with only 4 samples per tail class, the classifier’s decision boundary skews toward the long classes, yielding poor tail performance. After adding 500 synthetic images per tail class with ChimeraLoRA, accuracy improves by 7.62 percentage points on average relative to the real-only baseline, with a 14.74 percentage-point gain on the tail classes specifically.

Figure 5: t-SNE for real and synthetic images. ChimeraLoRA generates mainly inside the region spanned by the real anchors marked with crosses and attains the highest coverage across methods, with Cov $(\mathcal{R};\mathcal{S})=0.93$ and Cov $(\mathcal{S};\mathcal{R})=0.90$ .

Figure 6: Real-manifold coverage. Our ChimeraLoRA samples (yellow rectangles) fall within the region spanned by the four real anchors (red stars), indicating coverage of the real manifold, whereas the baselines drift outside.

Methods	FID@4 $\downarrow$	CLIP score $\uparrow$	Centroid Sim. $\uparrow$
Real (4-shots)	0.00	29.48	100.0
DataDream [ 21 ]	0.23	29.67	87.8
LoFT [ 20 ]	0.22	30.04	90.1
ChimeraLoRA	0.20	30.31	90.5

Table 3: Synthetic-to-real gap analyses. Across nine datasets, ChimeraLoRA produces synthetic images that most closely match the 4-shot real reference on average, with the lowest FID@4 and the highest CLIP score and centroid similarity.

Panel of Figure 7 — Figure 7: Robustness under scaling the synthetic budget. Accuracy rises with more synthetic images per class for ChimeraLoRA. Shaded regions indicate variability across seeds.

Figure 8: Synthetic images with sharing LoRA $B$ . For each class, the left three images are from our ChimeraLoRA with a shared LoRA $A$ , and the right three are from a variant with a shared LoRA $B$ . The right images look more diverse but often miss the target object or fine details such as a motorcycle’s wheel.

Multi-Head LoRA	Semantic Boosting	AIR	FLO
✗	✗	41.7	91.3
✓	✗	43.9	93.1
✗	✓	44.4	92.2
✓	✓	46.0	93.4

Table 4: Component ablation. Each proposed components contribute to performance gains.

Method	Rank	# parameters	Training time
DataDream	16	3.2M	95s
LoFT	4	0.8M	356s
Ours	4	1.9M	212s

Table 6: Trade-off between # parameters and training time.

Method	Wall-Time	Accuracy
Uniform $w$	4.6h	59.37
Reuse $w$	5.8h	61.31
New $w$	6.1h	61.51

Table 7: Trade-off between wall-time and accuracy.

Metric	$\alpha=0.2$	$\alpha=1.0$	$\alpha=5.0$
FID (Real vs. Synthetic, $\downarrow$ )	0.098	0.101	0.146
Distance (Inter-class, $\uparrow$ )	0.142	0.153	0.199

Table 8: Comparison metrics under different

\alpha

values.

Method	AIR	FLO
DataDream (+SB)	44.3 $\rightarrow$ 44.2	92.9 $\rightarrow$ 93.1
LoFT (+SB)	41.7 $\rightarrow$ 44.4	91.3 $\rightarrow$ 92.2
Ours	43.9 $\rightarrow$ 46.0	93.1 $\rightarrow$ 93.4

Table 9: Effect of semantic boosting on baselines.

Figure 9: Additional Qualitative Results. (a) LoFT, (b) DataDream, (c) ChimeraLoRA. From top to bottom, the classes are 1991 Volkswagen Golf Hatchback and 2007 BMW X5 SUV from Cars, followed by annualcrop and highway from EuroSAT.

Figure 10: Additional Qualitative Results. (a) LoFT, (b) DataDream, (c) ChimeraLoRA. From top to bottom, the classes are banded and bubbly from DTD, followed by basal cell carcinoma and vascular lesion from ISIC.

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 453 papers

Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks
May 12, 2026Daniil Dushenev, Nazariy Karpov, Daniil Zinovjev +2Class-Imbalanced Learning Synthetic Data Augmentation
National University of Science and Technology MISIS, Moscow, Russia
A LoRA is Worth a Thousand Pictures
Dec 16, 2024Chenxi Liu, Towaki Takikawa, Alec JacobsonWeight-Space Analysis Low-Rank Adaptation
University of Toronto · Outerport · Adobe Research
Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation
Mar 28, 2026Guohuan Xie, Xin He, Dingying Fan +3Few-Shot Semantic Segmentation Semantic Segmentation
College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China. +2

May 12, 2026J/K move · Enter open · S save

Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks

Daniil Dushenev, Nazariy Karpov, Daniil Zinovjev, Alexander Gorin+1

National University of Science and Technology MISIS, Moscow, Russia

Class-Imbalanced Learning Synthetic Data Augmentation Diffusion Model Fine-Tuning Diffusion-Based Data Augmentation

Open paper AskPDF arXiv

Class imbalance is a persistent challenge in visual recognition, particularly in safety-critical domains where collecting positive examples is expensive and rare events are inherently underrepresented. We propose a lightweight synthetic data augmentation pipeline that fine-tunes a LoRA adapter on as few as 20-50 real images of a rare class and uses a pretrained diffusion model to generate synthetic samples for training. We systematically vary the synthetic-to-real ratio and evaluate the approach across two structurally different domains: chest X-ray pathology classification (NIH ChestX-ray14) and industrial surface crack detection (Magnetic Tile Defect dataset). All evaluations are performed on held-out sets of real images only. Across both domains, synthetic augmentation consistently improves rare-class recall and F1 compared to training with real data alone. Performance improves with moderate synthetic augmentation and shows diminishing returns as the synthetic ratio increases. These results suggest that LoRA-adapted diffusion models provide a simple and scalable mechanism for augmenting rare classes, enabling effective learning in data-scarce scenarios across heterogeneous visual domains.

Abstract

Synthetic Data Augmentation

Explore similar work

Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks

A LoRA is Worth a Thousand Pictures

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation