Improving Semantic Coverage in AI Test & Evaluation through Generative Augmentation

Connor Hashemi; Bharadwaj Ravichandran; Connor Anderson; Shitij Govil; Brandon RichardWebster; Brian Hu; Keith Fieldhouse

cs.CVOct 9, 2026

Improving Semantic Coverage in AI Test & Evaluation through Generative Augmentation

Authors: Connor Hashemi, Bharadwaj Ravichandran, Connor Anderson, Shitij Govil, Brandon RichardWebster, Brian Hu, Keith Fieldhouse

Organizations: Kitware Inc, 1712 Route 9 Suite 300, Clifton Park, NY USA

Abstract

In test and evaluation (T&E) of AI models, evaluators typically have access only to a held-out test dataset and no knowledge of the training data or its semantic composition. If critical input conditions---such as time-of-day, weather, or scene location---are underrepresented in this test dataset, T&E results can be misleadingly due to this semantic coverage gap. We propose a three-stage generative AI pipeline to detect and fill such semantic gaps without requiring ground-truth labels of these semantics. First, vision-language models (VLMs) perform zero-shot binary labeling of semantic factors, and an embedding-space clustering step identifies which semantics most affect downstream model behavior. Second, diffusion-based image-to-image models transform existing covered images into the underrepresented semantic domain, preserving scene content while augmenting high-level appearance. Third, statistical analysis of the augmented image embeddings verifies that the synthetic imagery improves dataset coverage before it is incorporated. Crucially, the synthetic data is used only to enrich the test set and no model retraining is performed for T&E. We validate the full pipeline on both image classification and object detection under day/night semantic gaps, showing that the augmented test datasets better distinguish models with and without exposure to the missing semantic, thereby yielding T&E conclusions that more closely match those of an ideal, fully covered dataset. Code available at https://github.com/Hashe037/DEDUCE

Figures & tables

Figure 1: Overview of test and evaluation (T&E) and proposed improvement to test dataset. The goal of T&E is to use a test dataset to properly evaluate the real-life performance of models without access to their training datasets. However, the test dataset might have underrepresented semantic factors (such as time-of-day) which could bias the evaluations and lead to improper model selection. We denote these underrepresented semantic factors as “uncovered,” and this paper proposes a pipeline to identify and fill in gaps in semantic coverage with synthetic augmented data to improve overall T&E.

Figure 2: Overall semantic pipeline. The first module, “Identify,” labels and identifies impactful uncovered semantics utilizing image-text matching. The second module, “Augment,” creates new images with the uncovered semantic present utilizing diffusion-based image augmentations. While we offer two routes to augment imagery, we currently only implement the pretrained image-to-image models. The last module, “Evaluate,” evaluates whether the generated imagery improves the dataset statistics. If it improves the dataset sufficiently, the new synthetic data is then appended to the T&E dataset.

Figure 3: Overall semantic identification module (purple module in Figure 2 ). In (a), image-text matching with semantic-rich textual descriptors and the dataset imagery lends a set of binary semantic labels for each image. In (b), these images are then processed by the downstream model to get their feature embeddings. If any of the semantic labels generated in part (a) cluster together in the embedding space with their associated images, we can conclude that this semantic has a large impact on feature embeddings and therefore is important to the downstream task.

Figure 4: Augmentation module used in paper (blue module in Figure 2 ). In (a), we separate the T&E dataset via the semantic labels generated in the previous “Identify” module. The covered semantic factor images—which correspond to “day” images in this paper—are then passed along an image-to-image diffusion model (e.g., CycleGAN-Turbo [ 26 ] ) which converts them to the uncovered “night” semantic in (b). These synthetic images are then combined with uncovered and covered images to constitute the final improved T&E dataset in (c).

Figure 5: Statistical embedding evaluation metrics used in the “Evaluate” module (yellow module in Figure 2 ). In (a), good “coverage” results in a small hypersphere radii that encompasses the nearest image embeddings. In (b), low drift means the embeddings are more similar to the target semantic. In (c), high completeness means that the overall embedding space is explored sufficiently.

Figure 6: Semantics with highest cluster recall from the semantic identification module in Section 3.1 ordered left to right, where high cluster recall suggests high semantic “importance.” (a) represents the CCT trailcam dataset while (b) is the BDD100k driving dataset. High cluster recall indicates a significant effect on the feature embeddings and therefore on the downstream model. In both (a) and (b) the most clustered semantic is the expected uncovered “night” semantic.

Figure 7: Example images from the datasets compared to augmented imagery. (a) contains day/night examples from the CCT dataset while (b) contains day/night examples from the BDD100k driving dataset. The real night imagery in (a) are from the same scenes as the day images, while in (b) we just chose similar looking images from the dataset for comparison.

(a) CCT Trailcam Dataset
	Drift $\downarrow$ (vs. Night)	Coverage $\downarrow$ (of Night)	Completeness $\uparrow$ (Overall)	FID $\downarrow$ (vs. Night)
Just Night	NA	3.465	0.077	NA
Night + Day	0.290	3.342	0.259	299.73
Night + Synth. Night	0.104	3.115	0.335	168.28
Night + Day + Synth. Night	NA	3.113	0.392	NA

Table 1: Dataset embeddings evaluation of the day/night semantic for both (a) CCT and (b) BDD100k datasets, where night is the uncovered and underrepresented semantic. Bolded is best performer for each metric.

(a) F1 Score for CCT				(b) mAP@50 Score for BDD100k
	Real Day	Day to Synth. Night	Real Night		Real Day	Day to Synth. Night	Real Night
Model #1 (Trained on Day)	0.711	0.100	0.073	Model #1 (Trained on Day)	0.526	0.372	0.354
Model #2 (Trained on Day + Night)	0.694	0.452	0.445	Model #2 (Trained on Day + Night)	0.517	0.411	0.411

Table 2: T&E evaluation on image semantic subsets for both (a) CCT and (b) BDD100k datasets. Bolded is best metric for the subset.

(a) F1 Score for CCT (Higher is Better)				(b) mAP@50 Score for BDD100k (Higher is Better)
	Original	Improved (Original + Synth. Night)	Ideal (Original + Real Night)		Original	Improved (Original + Synth. Night)	Ideal (Original + Real Night)
Model #1 (Trained on Day)	0.705	0.591	0.603	Model #1 (Trained on Day)	0.467	0.403	0.427
Model #2 (Trained on Day + Night)	0.694	0.642	0.645	Model #2 (Trained on Day + Night)	0.462	0.417	0.443

Table 3: T&E evaluation on toy datasets for both (a) CCT and (b) BDD100k datasets. Bolded is the best metric for the dataset. Since Model #2 is a more diverse training set, we expect the dataset to have the best metrics for that model.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: T-SNE cluster visualizations of the T&E dataset for the trailcam imagery using a downstream EfficientNet model. (a) shows the clusters according to k-means with k=6. (b) depicts the labels for a semantic which clusters poorly, which in this case is the “damage” semantic. Most of the labels do not lie in the same k-means cluster, suggesting the semantic has a somewhat random effect on the embedding. (c) shows a well-clustered semantic label which is “night.” Most of these lie in the same cluster, which suggest it has a large effect and bias on the embeddings.

Figure 9: Example images from the datasets compared to augmented imagery with well- and poor-aligned CycleGAN-Turbo models. The well-aligned model was trained on the given dataset, while the poor-aligned model was trained on the opposite dataset. This results in different manifestations of the day/night semantic.

(a) CCT Trailcam Dataset
	Drift $\downarrow$ (vs. Night)	Coverage $\downarrow$ (of Night)	FID $\downarrow$ (vs. Night)
Just Night	NA	3.465	NA
Night + Day	0.290	3.342	299.73
Night + CCT Synth. Night	0.104	3.115	168.28
Night + BDD100k Synth. Night	0.240	3.343	301.00

Appendix

Table 4: Dataset embeddings evaluation of the day/night semantic for both (a) CCT and (b) BDD100k datasets, where night is the uncovered and underrepresented semantic. We compare against synthetic data trained on different datasets. Bolded is the best performer for each metric.

(a) F1 Score for CCT				(b) mAP@50 Score for BDD100k
	Real Day	BDD100k Synth. Night	Real Night		Real Day	CCT Synth. Night	Real Night
Model #1 (Trained on Day)	0.711	0.282	0.073	Model #1 (Trained on Day)	0.526	0.366	0.354
Model #2 (Trained on Day + Night)	0.694	0.341	0.445	Model #2 (Trained on Day + Night)	0.517	0.409	0.411

Appendix

Table 5: T&E evaluation on image semantic subsets for both (a) CCT and (b) BDD100k datasets. The synthetic data is generated by a model trained on the opposite dataset. Compare to Table 2 in the paper.

Figure 10: Visualization of BDD100k weather images and proposed CycleGAN-Turbo [ 26 ] synthetic augmentations.

Figure 11: Results of the semantic identification and clustering on the BDD100k weather semantics. The embeddings used for clustering are from a ResNet [ 13 ] model designed for weather tagging for BDD100k.

(a) BDD100k Weather Clear vs. Rainy
	Drift $\downarrow$ (vs. Rainy)	Coverage $\downarrow$ (of Rainy)	Completeness $\uparrow$ (Overall)	FID $\downarrow$ (vs. Rainy)
Just Rainy	NA	0.805	0.086	NA
Rainy + Clear	0.400	0.739	0.572	109.73
Rainy + Synth. Rainy	0.062	0.628	0.569	98.31
Rainy + Clear + Synth. Rainy	NA	0.623	0.739	NA

Appendix

Table 6: Results of the statistical evaluation of the synthetic imagery in the embedding space for BDD100k weather semantics. Downstream model is a weather-tagging ResNet [ 13 ] model.

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 456 papers

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation
Mar 28, 2026Guohuan Xie, Xin He, Dingying Fan +3Few-Shot Semantic Segmentation Semantic Segmentation
College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China. +2
Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models
Jun 30, 2026Nikolai Röhrich, Julian Gleißner, Ahmed H. A. Ibrahim +2Synthetic Data Augmentation Semantic Segmentation
1XITASO GmbH · 2Zuse School of Excellence in Reliable AI · 3Technische Hochschule Ingolstadt +1
When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators
Feb 23, 2026Krzysztof Adamkiewicz, Brian Bernhard Moser, Stanislav Frolov +3Synthetic-to-Real Domain Adaptation Distribution Shift
1RPTU University Kaiserslautern-Landau, Kaiserslautern, Germany · 2German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Mar 28, 2026J/K move · Enter open · S save

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation

Guohuan Xie, Xin He, Dingying Fan, Le Zhang+2

College of Software, Nankai University, Tianjin 300350, China. · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Information and Communication Engineering, UESTC, Chengdu 611731, China.+2

Few-Shot Semantic Segmentation Semantic Segmentation Synthetic Data Augmentation Pseudo-Labeling

Open paper AskPDF arXiv

Generalized few-shot semantic segmentation (GFSS) is fundamentally limited by the coverage of novel-class appearances under scarce annotations. While diffusion models can synthesize novel-class images at scale, practical gains are often hindered by insufficient coverage and noisy supervision when masks are unavailable or unreliable. We propose Syn4Seg, a generation-enhanced GFSS framework designed to expand novel-class coverage while improving pseudo-label quality. Syn4Seg first maximizes prompt-space coverage by constructing an embedding-deduplicated prompt bank for each novel class, yielding diverse yet class-consistent synthetic images. It then performs support-guided pseudo-label estimation via a two-stage refinement that i) filters low-consistency regions to obtain high-precision seeds and ii) relabels uncertain pixels with image-adaptive prototypes that combine global (support) and local (image) statistics. Finally, we refine only boundary-band and unlabeled pixels using a constrained SAM-based update to improve contour fidelity without overwriting high-confidence interiors. Extensive experiments on PASCAL-

5^i

and COCO-

20^i

demonstrate consistent improvements in both 1-shot and 5-shot settings, highlighting synthetic data as a scalable path for GFSS with reliable masks and precise boundaries.

Abstract

Synthetic Data Augmentation

Explore similar work

Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation

Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models

When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators