In test and evaluation (T&E) of AI models, evaluators typically have access only to a held-out test dataset and no knowledge of the training data or its semantic composition. If critical input conditions---such as time-of-day, weather, or scene location---are underrepresented in this test dataset, T&E results can be misleadingly due to this semantic coverage gap. We propose a three-stage generative AI pipeline to detect and fill such semantic gaps without requiring ground-truth labels of these semantics. First, vision-language models (VLMs) perform zero-shot binary labeling of semantic factors, and an embedding-space clustering step identifies which semantics most affect downstream model behavior. Second, diffusion-based image-to-image models transform existing covered images into the underrepresented semantic domain, preserving scene content while augmenting high-level appearance. Third, statistical analysis of the augmented image embeddings verifies that the synthetic imagery improves dataset coverage before it is incorporated. Crucially, the synthetic data is used only to enrich the test set and no model retraining is performed for T&E. We validate the full pipeline on both image classification and object detection under day/night semantic gaps, showing that the augmented test datasets better distinguish models with and without exposure to the missing semantic, thereby yielding T&E conclusions that more closely match those of an ideal, fully covered dataset. Code available at https://github.com/Hashe037/DEDUCE
Figures & tables
Figure 1: Overview of test and evaluation (T&E) and proposed improvement to test dataset. The goal of T&E is to use a test dataset to properly evaluate the real-life performance of models without access to their training datasets. However, the test dataset might have underrepresented semantic factors (such as time-of-day) which could bias the evaluations and lead to improper model selection. We denote these underrepresented semantic factors as “uncovered,” and this paper proposes a pipeline to identify and fill in gaps in semantic coverage with synthetic augmented data to improve overall T&E.
Figure 2: Overall semantic pipeline. The first module, “Identify,” labels and identifies impactful uncovered semantics utilizing image-text matching. The second module, “Augment,” creates new images with the uncovered semantic present utilizing diffusion-based image augmentations. While we offer two routes to augment imagery, we currently only implement the pretrained image-to-image models. The last module, “Evaluate,” evaluates whether the generated imagery improves the dataset statistics. If it improves the dataset sufficiently, the new synthetic data is then appended to the T&E dataset.
Figure 3: Overall semantic identification module (purple module in Figure 2 ). In (a), image-text matching with semantic-rich textual descriptors and the dataset imagery lends a set of binary semantic labels for each image. In (b), these images are then processed by the downstream model to get their feature embeddings. If any of the semantic labels generated in part (a) cluster together in the embedding space with their associated images, we can conclude that this semantic has a large impact on feature embeddings and therefore is important to the downstream task.
Figure 4: Augmentation module used in paper (blue module in Figure 2 ). In (a), we separate the T&E dataset via the semantic labels generated in the previous “Identify” module. The covered semantic factor images—which correspond to “day” images in this paper—are then passed along an image-to-image diffusion model (e.g., CycleGAN-Turbo [ 26 ] ) which converts them to the uncovered “night” semantic in (b). These synthetic images are then combined with uncovered and covered images to constitute the final improved T&E dataset in (c).
Figure 5: Statistical embedding evaluation metrics used in the “Evaluate” module (yellow module in Figure 2 ). In (a), good “coverage” results in a small hypersphere radii that encompasses the nearest image embeddings. In (b), low drift means the embeddings are more similar to the target semantic. In (c), high completeness means that the overall embedding space is explored sufficiently.
Figure 6: Semantics with highest cluster recall from the semantic identification module in Section 3.1 ordered left to right, where high cluster recall suggests high semantic “importance.” (a) represents the CCT trailcam dataset while (b) is the BDD100k driving dataset. High cluster recall indicates a significant effect on the feature embeddings and therefore on the downstream model. In both (a) and (b) the most clustered semantic is the expected uncovered “night” semantic.
Figure 7: Example images from the datasets compared to augmented imagery. (a) contains day/night examples from the CCT dataset while (b) contains day/night examples from the BDD100k driving dataset. The real night imagery in (a) are from the same scenes as the day images, while in (b) we just chose similar looking images from the dataset for comparison.
(a) CCT Trailcam Dataset
Drift ↓ (vs. Night)
Coverage ↓ (of Night)
Completeness ↑ (Overall)
FID ↓ (vs. Night)
Just Night
NA
3.465
0.077
NA
Night + Day
0.290
3.342
0.259
299.73
Night + Synth. Night
0.104
3.115
0.335
168.28
Night + Day + Synth. Night
NA
3.113
0.392
NA
Table 1: Dataset embeddings evaluation of the day/night semantic for both (a) CCT and (b) BDD100k datasets, where night is the uncovered and underrepresented semantic. Bolded is best performer for each metric.
(a) F1 Score for CCT
(b) mAP@50 Score for BDD100k
Real Day
Day to Synth. Night
Real Night
Real Day
Day to Synth. Night
Real Night
Model #1 (Trained on Day)
0.711
0.100
0.073
Model #1 (Trained on Day)
0.526
0.372
0.354
Model #2 (Trained on Day + Night)
0.694
0.452
0.445
Model #2 (Trained on Day + Night)
0.517
0.411
0.411
Table 2: T&E evaluation on image semantic subsets for both (a) CCT and (b) BDD100k datasets. Bolded is best metric for the subset.
(a) F1 Score for CCT (Higher is Better)
(b) mAP@50 Score for BDD100k (Higher is Better)
Original
Improved (Original + Synth. Night)
Ideal (Original + Real Night)
Original
Improved (Original + Synth. Night)
Ideal (Original + Real Night)
Model #1 (Trained on Day)
0.705
0.591
0.603
Model #1 (Trained on Day)
0.467
0.403
0.427
Model #2 (Trained on Day + Night)
0.694
0.642
0.645
Model #2 (Trained on Day + Night)
0.462
0.417
0.443
Table 3: T&E evaluation on toy datasets for both (a) CCT and (b) BDD100k datasets. Bolded is the best metric for the dataset. Since Model #2 is a more diverse training set, we expect the dataset to have the best metrics for that model.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: T-SNE cluster visualizations of the T&E dataset for the trailcam imagery using a downstream EfficientNet model. (a) shows the clusters according to k-means with k=6. (b) depicts the labels for a semantic which clusters poorly, which in this case is the “damage” semantic. Most of the labels do not lie in the same k-means cluster, suggesting the semantic has a somewhat random effect on the embedding. (c) shows a well-clustered semantic label which is “night.” Most of these lie in the same cluster, which suggest it has a large effect and bias on the embeddings.
Figure 9: Example images from the datasets compared to augmented imagery with well- and poor-aligned CycleGAN-Turbo models. The well-aligned model was trained on the given dataset, while the poor-aligned model was trained on the opposite dataset. This results in different manifestations of the day/night semantic.
(a) CCT Trailcam Dataset
Drift ↓ (vs. Night)
Coverage ↓ (of Night)
FID ↓ (vs. Night)
Just Night
NA
3.465
NA
Night + Day
0.290
3.342
299.73
Night + CCT Synth. Night
0.104
3.115
168.28
Night + BDD100k Synth. Night
0.240
3.343
301.00
Appendix
Table 4: Dataset embeddings evaluation of the day/night semantic for both (a) CCT and (b) BDD100k datasets, where night is the uncovered and underrepresented semantic. We compare against synthetic data trained on different datasets. Bolded is the best performer for each metric.
(a) F1 Score for CCT
(b) mAP@50 Score for BDD100k
Real Day
BDD100k Synth. Night
Real Night
Real Day
CCT Synth. Night
Real Night
Model #1 (Trained on Day)
0.711
0.282
0.073
Model #1 (Trained on Day)
0.526
0.366
0.354
Model #2 (Trained on Day + Night)
0.694
0.341
0.445
Model #2 (Trained on Day + Night)
0.517
0.409
0.411
Appendix
Table 5: T&E evaluation on image semantic subsets for both (a) CCT and (b) BDD100k datasets. The synthetic data is generated by a model trained on the opposite dataset. Compare to Table 2 in the paper.
Figure 10: Visualization of BDD100k weather images and proposed CycleGAN-Turbo [ 26 ] synthetic augmentations.
Figure 11: Results of the semantic identification and clustering on the BDD100k weather semantics. The embeddings used for clustering are from a ResNet [ 13 ] model designed for weather tagging for BDD100k.
(a) BDD100k Weather Clear vs. Rainy
Drift ↓ (vs. Rainy)
Coverage ↓ (of Rainy)
Completeness ↑ (Overall)
FID ↓ (vs. Rainy)
Just Rainy
NA
0.805
0.086
NA
Rainy + Clear
0.400
0.739
0.572
109.73
Rainy + Synth. Rainy
0.062
0.628
0.569
98.31
Rainy + Clear + Synth. Rainy
NA
0.623
0.739
NA
Appendix
Table 6: Results of the statistical evaluation of the synthetic imagery in the embedding space for BDD100k weather semantics. Downstream model is a weather-tagging ResNet [ 13 ] model.
Generalized few-shot semantic segmentation (GFSS) is fundamentally limited by the coverage of novel-class appearances under scarce annotations. While diffusion models can synthesize novel-class images at scale, practical gains are often hindered by insufficient coverage and noisy supervision when masks are unavailable or unreliable. We propose Syn4Seg, a generation-enhanced GFSS framework designed to expand novel-class coverage while improving pseudo-label quality. Syn4Seg first maximizes prompt-space coverage by constructing an embedding-deduplicated prompt bank for each novel class, yielding diverse yet class-consistent synthetic images. It then performs support-guided pseudo-label estimation via a two-stage refinement that i) filters low-consistency regions to obtain high-precision seeds and ii) relabels uncertain pixels with image-adaptive prototypes that combine global (support) and local (image) statistics. Finally, we refine only boundary-band and unlabeled pixels using a constrained SAM-based update to improve contour fidelity without overwriting high-confidence interiors. Extensive experiments on PASCAL-5i and COCO-20i demonstrate consistent improvements in both 1-shot and 5-shot settings, highlighting synthetic data as a scalable path for GFSS with reliable masks and precise boundaries.
Guohuan Xie, Xin He, Dingying Fan +3
College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China. +2
Semantic segmentation models struggle with data sparsity and rare or visually diverse regions, e.g., dense regions or small objects in aerial or autonomous mobility data. While synthetic augmentation is an appealing solution, directly generating new labeled data risks misalignment of labels and generated pixels. Existing solutions to this problem often rely on external models, or employ coarse heuristics such as indiscriminately augmenting all foreground objects or entire backgrounds, which wastes capacity on uninformative pixels. To address this, we propose an uncertainty-guided synthetic context augmentation strategy that strictly preserves label validity and efficiently maximizes pixel informativeness per synthetic sample - no external guardrails required. Using a baseline segmenter's predictive entropy, we identify uncertain semantic regions and inpaint only the complementary visual context. When fine-tuning the segmenter on this synthetic data, we compute the loss only over the original pixels, excluding inpainted regions. This focuses learning on the unmodified, uncertain regions while presenting them in novel contexts. We demonstrate substantial mIoU gains on Cityscapes, UAVID, and BDD100K with the largest gains on rare and difficult classes such as buses, trains, or (from the aerial perspective) cars. Our results demonstrate that uncertainty-guided context augmentation is a highly effective lever to improve segmentation performance on complex datasets, with code provided at https://github.com/XITASO/Preserve-the-Hard-Regenerate-the-Rest.
Nikolai Röhrich, Julian Gleißner, Ahmed H. A. Ibrahim +2
1XITASO GmbH · 2Zuse School of Excellence in Reliable AI · 3Technische Hochschule Ingolstadt +1
Recent text-to-image (T2I) diffusion models produce visually stunning images and demonstrate excellent prompt following. But do they perform well as synthetic vision data generators? In this work, we revisit the promise of synthetic data as a scalable substitute for real training sets and uncover a surprising performance regression. We generate large-scale synthetic datasets using state-of-the-art T2I models released between 2022 and 2025, train standard classifiers solely on this synthetic data, and evaluate them on real test data. Despite observable advances in visual fidelity and prompt adherence, classification accuracy on real test data consistently declines with newer T2I models as training data generators. Our analysis reveals a hidden trend: These models collapse to a narrow, aesthetic-centric distribution that undermines diversity and real data distribution coverage. Overall, our findings challenge a growing assumption in vision research, namely that progress in generative realism implies progress in data realism. We thus highlight an urgent need to rethink the capabilities of modern T2I models as reliable training data generators.
Krzysztof Adamkiewicz, Brian Bernhard Moser, Stanislav Frolov +3
1RPTU University Kaiserslautern-Landau, Kaiserslautern, Germany · 2German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany