TreemapMix: Dirichlet-Controlled Multi-Image Augmentation for Probability and Ordinal Supervision

Ejafa Bassam; Konstantin Garbers; Yingsheng Geng; Dalton Jens; Jiaxu Qian; Guyue Liu

cs.CVOct 28, 2026

TreemapMix: Dirichlet-Controlled Multi-Image Augmentation for Probability and Ordinal Supervision

Authors: Ejafa Bassam, Konstantin Garbers, Yingsheng Geng, Dalton Jens, Jiaxu Qian, Guyue Liu

Organizations: Peking University · Beijing University of Posts and Telecommunications

Abstract

Data augmentation improves visual recognition by exposing models to synthetic training examples that encourage generalization beyond the observed data. Mixed-sample methods extend this idea by combining multiple images. However, many common approaches either mix only pairs of images or rely on fixed small layouts, limiting control over source count, visible area proportions, and induced supervision. We propose TreemapMix, a distribution-controlled multi-image augmentation that samples source weights from a Dirichlet distribution and assigns them to area-proportional regions using a treemap partition. The resulting known region areas support two complementary objectives. TreemapMix with soft cross-entropy (SCE) treats areas as soft class-probability targets, while TreemapMix with a Plackett-Luce (PL) objective converts areas into ordinal supervision. On ImageNet-1K under matched training recipes, TreemapMix-SCE achieves substantially lower calibration error than evaluated mixed-sample baselines, while TreemapMix-PL achieves the strongest Top-1 accuracy. Transfer experiments on object detection and instance segmentation suggest that TreemapMix-pretrained features remain useful beyond classification. These results show that controlled multi-image composition can expose the same region-area information as either probability-matching or rank-matching supervision. Code is available at https://github.com/Ejafa/treemapmix.

Figures & tables

Panel of Figure 1 — Figure 1: Qualitative comparison of five augmentation strategies on matched source images using resize-and-center-crop preprocessing only. Mixup blends full images, CutMix pastes a random region, RICAP composes four tiles, and TreemapMix composes $K$ rescaled source images within a randomized layout.

Figure 2: ImageNet-1K accuracy and calibration for all 16 configurations under the matched 110-epoch protocol. Points show means over three seeds; horizontal intervals denote

\pm 1

sample standard deviation. Marker shape and color identify the backbone. ECE uses 15 bins and is reported in percentage points. Average ranks are computed within each backbone (lower is better).

Method	ViT-Wee $\uparrow$	ViT-Little $\uparrow$	ViT-Medium $\uparrow$	ViT-Betwixt $\uparrow$
No Aug	$75.10\pm 0.24$	$76.05\pm 0.14$	$76.59\pm 0.02$	$77.18\pm 0.14$
Single-Image Aug	$77.94\pm 0.15$	$79.43\pm 0.22$	$79.80\pm 0.04$	$80.28\pm 0.16$
Mixup+CutMix	$76.20\pm 0.16$	$78.84\pm 0.05$	$80.04\pm 0.20$	$81.09\pm 0.13$
Mixup	$76.22\pm 0.12$	$78.64\pm 0.10$	$79.80\pm 0.11$	$80.84\pm 0.09$
CutMix	$76.13\pm 0.07$	$79.12\pm 0.07$	$80.23\pm 0.09$	$81.23\pm 0.12$
RICAP	$77.05\pm 0.15$	$78.41\pm 0.14$	$78.74\pm 0.07$	$79.13\pm 0.11$

Table 1: ImageNet-1K Top-1 accuracy (%) under the matched 110-epoch protocol. Mean

\pm

sample standard deviation over three seeds; bold marks the best mean per backbone.

Method	ViT-Wee $\downarrow$	ViT-Little $\downarrow$	ViT-Medium $\downarrow$	ViT-Betwixt $\downarrow$
No Aug	$3.62\pm 0.13$	$4.53\pm 0.36$	$4.29\pm 0.20$	$4.42\pm 0.02$
Single-Image Aug	$6.29\pm 0.13$	$5.19\pm 0.12$	$5.53\pm 0.21$	$5.17\pm 0.40$
Mixup+CutMix	$9.05\pm 0.47$	$7.53\pm 0.10$	$8.28\pm 0.51$	$7.91\pm 0.35$
Mixup	$8.69\pm 0.20$	$7.54\pm 0.15$	$7.92\pm 0.20$	$7.70\pm 0.11$
CutMix	$7.32\pm 0.84$	$5.56\pm 0.54$	$7.26\pm 0.14$	$5.63\pm 0.20$
RICAP	$3.11\pm 0.02$	$4.43\pm 0.11$	$5.96\pm 0.43$	$6.44\pm 0.11$

Table 2: ImageNet-1K ECE (15 bins, percentage points). Same runs as Table 1 ; bold marks the lowest mean per backbone.

	ConvNeXtV2-Nano		DeiT-Tiny		Swin-Tiny
Method	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
Mixup+CutMix	$79.79\pm 0.11$	$9.48\pm 0.13$	$68.86\pm 0.28$	$12.34\pm 0.27$	$79.62\pm 0.06$	$7.50\pm 0.13$
RICAP	$79.08\pm 0.10$	$5.95\pm 0.17$	$71.71\pm 0.04$	$2.04\pm 0.28$	$78.68\pm 0.07$	$7.78\pm 0.05$
TreemapMix-SCE	$80.01\pm 0.09$	$\mathbf{2.74\pm 0.27}$	$71.78\pm 0.26$	$\mathbf{1.32\pm 0.26}$	$80.55\pm 0.09$	$\mathbf{2.52\pm 0.11}$
TreemapMix-PL	$\mathbf{80.45\pm 0.10}$	$6.82\pm 0.07$	$\mathbf{71.81\pm 0.07}$	$2.71\pm 0.15$	$\mathbf{80.57\pm 0.12}$	$6.58\pm 0.06$

Table 3: Cross-architecture ImageNet-1K results under 110 epochs. Mean

\pm

sample standard deviation over three seeds; accuracy in percent and ECE in percentage points (15 bins). Bold marks the best mean per metric.

Method	ViT-Wee	ViT-Little	ViT-Medium	ViT-Betwixt
Top-1 accuracy (%) $\uparrow$
Mixup+CutMix	$41.72\pm 0.08$	$41.67\pm 0.43$	$40.95\pm 0.07$	$39.78\pm 0.21$
RICAP	$35.70\pm 0.15$	$35.39\pm 0.00$	$34.35\pm 0.02$	$31.44\pm 3.37$
TreemapMix-SCE	$41.44\pm 0.40$	$41.09\pm 0.40$	$39.48\pm 0.13$	$38.67\pm 0.27$
TreemapMix-PL	$\mathbf{43.12\pm 0.06}$	$\mathbf{42.69\pm 0.07}$	$\mathbf{42.26\pm 0.07}$	$\mathbf{41.60\pm 0.17}$
ECE (percentage points) $\downarrow$

Table 4: ImageNet-LT accuracy and ECE under the matched 110-epoch protocol. Mean

\pm

sample standard deviation over three seeds; ECE uses 15 bins. Bold marks the best mean per metric.

Method	$K$	Throughput (images/s)	Throughput reduction	Total loader time (s)
No Aug	1	$1670.4\pm 30.2$	$-30.2\%$	$11.43\pm 0.13$
Single-Image Aug	1	$1282.8\pm 35.1$	$0.0\%$	$14.35\pm 0.27$
Mixup+CutMix	2	$1223.4\pm 29.7$	$4.6\%$	$15.13\pm 0.36$
TreemapMix-SCE	4	$1039.4\pm 37.0$	$19.0\%$	$19.83\pm 0.46$
TreemapMix-PL	6	$954.3\pm 33.0$	$25.6\%$	$21.30\pm 0.36$
RICAP	4	$364.8\pm 5.9$	$71.6\%$	$44.00\pm 0.47$

Table 5: Complete loader benchmark: five runs (seeds 42–46), batch size 128, eight workers, first batch, 20 warmup and 100 measured batches. Reduction is

100(1-T/T_{\mathrm{single}})

;

\Delta

RSS uses unrounded means. Memory is GiB (

2^{30}

bytes). Total time covers all consumed batches; throughput uses measured batches only.

	ViT-Wee		ViT-Little
Method	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
Single-Image Aug	$80.37\pm 0.08$	$3.89\pm 0.16$	$81.12\pm 0.04$	$3.39\pm 0.06$
Mixup+CutMix	$79.73\pm 0.15$	$6.87\pm 0.64$	$81.74\pm 0.09$	$5.47\pm 0.52$
TreemapMix-SCE	$80.41\pm 0.04$	$\mathbf{3.23\pm 0.49}$	$81.58\pm 0.33$	$\mathbf{2.77\pm 1.42}$
TreemapMix-PL	$\mathbf{80.64\pm 0.16}$	$6.94\pm 0.14$	$\mathbf{81.95\pm 0.34}$	$7.44\pm 0.54$

Table 6: ImageNet-1K long-horizon results for ViT-Wee and ViT-Little under a 320-epoch schedule. Accuracy is reported in percent, and ECE in percentage points using 15 bins. Values are mean

\pm

one sample standard deviation over seeds 42, 43, and 44. Bold marks the best mean for each model and metric.

Figure 3: Area-logit alignment diagnostics on the held-out composed diagnostic set for ViT-Betwixt checkpoints from the main ImageNet-1K study. Lines show means over three seeds and shaded bands denote

\pm 1

sample standard deviation: pairwise rank accuracy (left), area-logit Spearman correlation (center), and largest-area top-class accuracy (right).

	ViT-Wee		ViT-Betwixt
Method	Box AP $\uparrow$	Mask AP $\uparrow$	Box AP $\uparrow$	Mask AP $\uparrow$
Mixup+CutMix	23.66 $\pm$ 0.01	20.34 $\pm$ 0.10	29.04 $\pm$ 0.07	25.25 $\pm$ 0.10
RICAP	23.08 $\pm$ 0.17	19.82 $\pm$ 0.07	27.86 $\pm$ 0.08	24.21 $\pm$ 0.06
TreemapMix-SCE	24.00 $\pm$ 0.17	20.55 $\pm$ 0.20	29.44 $\pm$ 0.16	25.56 $\pm$ 0.14
TreemapMix-PL	24.32 $\pm$ 0.12	20.71 $\pm$ 0.04	29.71 $\pm$ 0.09	25.75 $\pm$ 0.10

Table 7: COCO transfer results after ViTDet-style Mask R-CNN fine-tuning from the main ImageNet-1K experiment checkpoints with three seeds per method. Values are mean

\pm

standard deviation over these three transferred seeds; higher is better. Box AP evaluates bounding boxes and mask AP evaluates instance masks. AP is averaged over IoU thresholds from 0.50 to 0.95 in increments of 0.05.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Setting	Original	New
ViT-Betwixt
	dtype	float16	bfloat16
	aug_repeats	3.0	0.0
	warmup_epochs	20	10
	epochs	600	110
	lr	1.2e-3	1.5e-3

Appendix

Table 8: Adaptations to the reference timm ImageNet-1K recipes used for the main 110-epoch comparison. We shorten the original long schedules, disable augmentation repeats, switch to bfloat16 , and retune the learning rate; all other recipe settings are kept unchanged unless listed.

Method	Mixing settings
No Aug	Single-image transforms off; Mixup+CutMix with $\alpha=0.8/1.0$ , $p=1$
Single-Image Aug	No mixing
Mixup	$\alpha=0.8$ , $p=1$
CutMix	$\alpha=1.0$ , $p=1$
Mixup+CutMix	$\alpha=0.8/1.0$ , $p=1$ ; choose either method with probability $0.5$
RICAP	$p=0.5$ ; close_epochs =10; center $[0.5,1.5]$ ; scale $[0.8,1.2]$

Appendix

Table 9: ImageNet-1K augmentation settings. All recipes except No Aug include single-image transforms (cropping, flipping, color/AutoAugment-style transforms, and random erasing).

p

is the application probability;

\alpha

denotes the method’s mixing parameter. Settings were checked against archived run configurations and the historical training implementation. OpenMixup options beyond

\alpha

and

p

are implementation defaults without recorded overrides. The archived No Aug label disables single-image transforms but retains Mixup+CutMix with equal selection probabilities. RICAP records the configured close parameter; its effective cutoff depends on the configured training horizon. All TreemapMix variants use aspect-ratio filtering with maximum ratio 20.

Model	Opt.	Base LR	LR Decay	WD	DropPath	EMA
ViT-Wee	NAdamW	1.5e-3	cosine	0.06	0.2	yes
ViT-Little	NAdamW	1.5e-3	cosine	0.06	0.2	no
ViT-Medium	NAdamW	1.5e-3	cosine	0.06	0.2	no
ViT-Betwixt	NAdamW	1.5e-3	cosine	0.08	0.2	no

Appendix

Table 10: ViT baseline optimizer and training hyperparameters for the main comparison. Opt. denotes optimizer, LR learning rate, WD weight decay, and EMA exponential moving average of model weights. NAdamW is Nesterov-accelerated Adam with decoupled weight decay. DropPath is the stochastic-depth probability.

Model	Embed Dim	Depth	Heads	MLP Ratio	Drop Path	LR	Layer Decay
ViT-Wee	256	14	4	5.0	0.1	6.0e-4	0.7
ViT-Betwixt	640	12	10	4.0	0.1	8.0e-4	0.7

Appendix

Table 11: Hyperparameters for the object-detection transfer experiments with a ViTDet-style adapter and Mask R-CNN on COCO. Embed Dim is the embedding dimension, Heads is the attention-head count, MLP Ratio is the multilayer perceptron expansion ratio, and LR is the learning rate.

	ViT-Wee		ViT-Betwixt
Method	Box AP50 $\uparrow$	Mask AP50 $\uparrow$	Box AP50 $\uparrow$	Mask AP50 $\uparrow$
Mixup+CutMix	39.67 $\pm$ 0.01	35.64 $\pm$ 0.09	46.46 $\pm$ 0.19	42.59 $\pm$ 0.30
RICAP	38.68 $\pm$ 0.24	34.82 $\pm$ 0.20	44.91 $\pm$ 0.04	41.05 $\pm$ 0.20
TreemapMix-SCE	40.37 $\pm$ 0.19	36.18 $\pm$ 0.19	47.06 $\pm$ 0.15	43.12 $\pm$ 0.13
TreemapMix-PL	40.90 $\pm$ 0.01	36.59 $\pm$ 0.13	47.58 $\pm$ 0.27	43.48 $\pm$ 0.14

Appendix

Table 12: COCO transfer AP50 after ViTDet-style Mask R-CNN fine-tuning from ImageNet-1K checkpoints. In this transfer setting, TreemapMix-PL obtains the best box and mask AP50 for both ViT-Wee and ViT-Betwixt, while TreemapMix-SCE also improves over Mixup+CutMix and RICAP. Values are mean

\pm

standard deviation over three seeds; higher is better, and bold marks the best mean in each column.

Method	ViT-Wee $\downarrow$	ViT-Little $\downarrow$	ViT-Medium $\downarrow$	ViT-Betwixt $\downarrow$
Single-Image Aug	$0.923\pm 0.006$	$0.869\pm 0.008$	$0.864\pm 0.001$	$0.855\pm 0.001$
No Aug	$1.120\pm 0.009$	$1.137\pm 0.005$	$1.154\pm 0.006$	$1.134\pm 0.008$
Mixup+CutMix	$1.015\pm 0.011$	$0.896\pm 0.002$	$0.854\pm 0.008$	$0.811\pm 0.004$
Mixup	$1.012\pm 0.004$	$0.904\pm 0.002$	$0.862\pm 0.002$	$0.820\pm 0.002$
CutMix	$0.994\pm 0.011$	$0.870\pm 0.004$	$0.840\pm 0.005$	$0.787\pm 0.005$
RICAP	$0.893\pm 0.004$	$0.846\pm 0.007$	$0.870\pm 0.008$	$0.857\pm 0.003$

Appendix

Table 13: ImageNet-1K negative log-likelihood for all 16 configurations under the matched 110-epoch protocol. Values are mean

\pm

one sample standard deviation over three seeds. Bold marks the lowest mean in each backbone.

Method	ViT-Wee $\downarrow$	ViT-Little $\downarrow$	ViT-Medium $\downarrow$	ViT-Betwixt $\downarrow$
Single-Image Aug	$0.314\pm 0.002$	$0.294\pm 0.003$	$0.291\pm 0.001$	$0.285\pm 0.001$
No Aug	$0.355\pm 0.003$	$0.349\pm 0.002$	$0.344\pm 0.001$	$0.335\pm 0.001$
Mixup+CutMix	$0.342\pm 0.003$	$0.305\pm 0.001$	$0.292\pm 0.003$	$0.277\pm 0.002$
Mixup	$0.340\pm 0.002$	$0.308\pm 0.001$	$0.294\pm 0.001$	$0.280\pm 0.001$
CutMix	$0.338\pm 0.003$	$0.298\pm 0.002$	$0.287\pm 0.002$	$0.270\pm 0.001$
RICAP	$0.319\pm 0.002$	$0.303\pm 0.002$	$0.304\pm 0.002$	$0.299\pm 0.001$

Appendix

Table 14: ImageNet-1K Brier score for all 16 configurations under the matched 110-epoch protocol. Values are mean

\pm

one sample standard deviation over three seeds. Bold marks the lowest mean in each backbone.

Method	ViT-Wee $\uparrow$	ViT-Little $\uparrow$	ViT-Medium $\uparrow$	ViT-Betwixt $\uparrow$
No Aug	$65.05\pm 0.44$	$63.69\pm 0.81$	$60.72\pm 0.54$	$60.70\pm 0.35$
Single-Image Aug	$71.50\pm 0.09$	$71.97\pm 0.45$	$72.56\pm 0.35$	$72.16\pm 0.06$
Mixup+CutMix	$74.15\pm 0.47$	$74.96\pm 0.64$	$75.81\pm 0.24$	$75.83\pm 0.16$
Mixup	$73.13\pm 0.10$	$73.46\pm 0.34$	$74.18\pm 0.18$	$74.02\pm 0.18$
CutMix	$73.39\pm 0.51$	$74.05\pm 0.16$	$75.21\pm 0.22$	$74.92\pm 0.83$
FMix	$74.58\pm 0.37$	$74.84\pm 0.22$	$75.52\pm 0.22$	$75.40\pm 0.07$

Appendix

Table 15: CIFAR-100 Top-1 accuracy (%) at 256 pixels, approximately 210 epochs including 10 warmup epochs. Mean

\pm

sample standard deviation over three seeds; bold marks the best mean per backbone.

Method	ViT-Wee $\downarrow$	ViT-Little $\downarrow$	ViT-Medium $\downarrow$	ViT-Betwixt $\downarrow$
No Aug	$7.64\pm 0.20$	$6.19\pm 0.39$	$4.61\pm 0.32$	$5.01\pm 0.08$
Single-Image Aug	$3.16\pm 0.17$	$3.75\pm 0.24$	$3.44\pm 0.22$	$3.90\pm 0.30$
Mixup+CutMix	$14.43\pm 0.28$	$14.16\pm 0.50$	$14.07\pm 0.19$	$13.93\pm 0.22$
Mixup	$12.67\pm 0.18$	$12.24\pm 0.24$	$12.23\pm 0.05$	$12.14\pm 0.24$
CutMix	$11.74\pm 0.02$	$10.85\pm 0.37$	$10.35\pm 0.09$	$10.49\pm 0.28$
FMix	$5.93\pm 0.14$	$5.57\pm 0.18$	$5.19\pm 0.14$	$4.86\pm 0.22$

Appendix

Table 16: CIFAR-100 ECE (15 bins, percentage points). Same runs as Table 15 ; bold marks the lowest mean per backbone.

Figure 7: CIFAR-100 comparison at 256-pixel resolution under an approximately 210-epoch schedule, including 10 warmup epochs. Points show the mean over seeds 42, 43, and 44; intervals denote

\pm

one sample standard deviation. ECE uses 15 bins and is reported in percentage points. Average ranks summarize within-backbone ranks, with rank 1 best. Complete values appear in Tables 15 and 16 .

		Mixup+CutMix		TreemapMix-SCE		TreemapMix-PL
Model	Optimizer	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$	Acc. $\uparrow$	ECE $\downarrow$
ViT-Wee	AdamW	75.76	8.99	77.18	0.43	77.75	5.41
	Muon	77.92	9.88	78.80	2.04	79.05	5.93
	NAdamW	76.30	9.29	77.48	0.94	78.16	5.44
ViT-Little	AdamW	78.77	8.11	79.50	1.36	79.93	5.96
	Muon	80.27	8.61	80.72	3.05	80.87	6.57

Appendix

Table 17: ImageNet-1K optimizer comparison under the settings in Appendix F.3 , using one run with seed 42 per configuration. Accuracy is reported in percent and ECE in percentage points; bold marks the best value for each metric within each row.

Panel of Figure 8 — Figure 8: Appendix heatmaps for the ImageNet-1K TreemapMix hyperparameter sweep on ViT-Wee. ECE is measured using 15 bins. The top row shows Top-1 over the available $(K,\alpha)$ grid ( $K=3,\ldots,10$ ), and the bottom row shows ECE over the same grid. The sweep uses a simplified protocol: we do not apply aspect filtering and we do not use $\alpha$ or $K$ scheduling, so these absolute values are expected to be somewhat worse than the tuned main results. The heatmaps show that for best results in terms of ECE and accuracy, small $K$ values together with moderately small $\alpha$ values should be used.

Panel of Figure 9 — Figure 9: TreemapMix randomness for fixed sources, $\alpha=0.5$ , and $K=5$ . Independent Dirichlet draws produce different compositions while preserving sources and settings.

Panel of Figure 10 — Figure 10: Effect of source count for fixed images, $\alpha=2.0$ , and maximum aspect ratio 15.0. Increasing $K$ from 2 to 9 gives finer-grained mixtures.

Panel of Figure 11 — Figure 11: Effect of Dirichlet concentration for fixed images, $K=5$ , and maximum aspect ratio 15.0. Smaller $\alpha$ favors more unequal region areas.

Panel of Figure 12 — Figure 12: Gradient $\times$ Input attribution for three ImageNet-1K classes. Each row shows the boxed input and overlays for Mixup+CutMix, RICAP, SCE, and PL. Red boxes identify target patches; brighter regions indicate larger attribution. Labels give target probabilities and patch areas. TreemapMix often concentrates attribution on target regions; PL probabilities are sharper than SCE.

Data Augmentation

Data augmentation expands or transforms training examples to improve learning, data efficiency, and robustness to distribution shifts.

See all 255 papers

InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation
Jul 21, 2026Khawar Islam, Arif Mahmood, Xin Jin +1Synthetic Data Augmentation Data Augmentation
The University of Melbourne, Melbourne, Australia · The Information Technology University, Lahore, Pakistan · Westlake University, China
$S^{2}$ -FracMix: Label-Preserving Self-Saliency Mixup Augmentation
Jun 24, 2026Khawar Islam, Arif Mahmood, Xin Jin +1Data Augmentation Mixup
The University of Melbourne, Australia · Information Technology University, Pakistan · Westlake University, China
OA-CutMix: Correcting the Label Bias of CutMix
Jun 3, 2026Tobias Christian Nauen, Stanislav Frolov, Federico Raue +2Data Augmentation Image Classification
RPTU University Kaiserslautern-Landau, Kaiserslautern, Germany · German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Jul 21, 2026J/K move · Enter open · S save

InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation

Khawar Islam, Arif Mahmood, Xin Jin, Naveed Akhtar

The University of Melbourne, Melbourne, Australia · The Information Technology University, Lahore, Pakistan · Westlake University, China

Synthetic Data Augmentation Data Augmentation

Open paper AskPDF arXiv

In image and video technologies, data augmentation is widely used to improve the generalization of deep visual models, and mixup-based strategies that interpolate between samples have become the dominant approach. However, computing informative mixing regions adds substantial overhead, and blending content across different images frequently disrupts the semantic integrity of the resulting sample. We propose \our{}, a data augmentation method that constructs challenging yet label-consistent training samples entirely within a single visual sample. \our{} first extracts multi-scale salient patches from the sample using a lightweight saliency detector, refines each patch with an instruction-guided generative model, and blends the edited patch back into the non-salient regions of the same sample; because the generative edits are computed once and cached offline, this step adds negligible training cost. To further diversify the learned representation, \our{} injects self-similar fractal structure into the same salient regions at an adaptive ratio, so each training sample carries both fractal and non-fractal structure. We derive a second-order approximation of the resulting vicinal risk, showing that the method simultaneously enforces invariance to the generative edit and suppresses curvature along the perturbed salient directions, and we verify both predictions empirically. We evaluate on small to large backbones for instance Convolutional Neural Networks (CNNs), Vision Transformers (ViTs) and Vision-Language Foundational Models (VLMs) across seven benchmarks covering coarse- and fine-grained classification, robustness to corruption and occlusion, calibration, and transfer and self-supervised learning, InstructMixup outperforms nine competing augmentation methods, surpassing the strongest baseline across all benchmarks.

Abstract

Data Augmentation

Explore similar work

InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation

S2S^{2}S2-FracMix: Label-Preserving Self-Saliency Mixup Augmentation

OA-CutMix: Correcting the Label Bias of CutMix

$S^{2}$ -FracMix: Label-Preserving Self-Saliency Mixup Augmentation