Scalable Evaluation of the Realism of Synthetic Environmental Augmentations in Images

Damian J. Ruck; Paul Vautravers; Oliver Chalkley; Jake Thomas

cs.CVMar 4, 2026

Scalable Evaluation of the Realism of Synthetic Environmental Augmentations in Images

Authors: Damian J. Ruck, Paul Vautravers, Oliver Chalkley, Jake Thomas

Organizations: Advai Ltd, 20–22 Wenlock Road, London, N1 7GU, United Kingdom

Abstract

Evaluation of AI systems often requires synthetic test cases, particularly for rare or safety-critical conditions that are difficult to observe in operational data. Generative AI offers a promising approach for producing such data through controllable image editing, but its usefulness depends on whether the resulting images are sufficiently realistic to support meaningful evaluation. We present a scalable framework for assessing the realism of synthetic image-editing methods and apply it to the task of adding environmental conditions-fog, rain, snow, and nighttime-to car-mounted camera images. Using 40 clear-day images, we compare rule-based augmentation libraries with generative AI image-editing models. Realism is evaluated using two complementary automated metrics: a vision-language model (VLM) jury for perceptual realism assessment, and embedding-based distributional analysis to measure similarity to genuine adverse-condition imagery. Generative AI methods substantially outperform rule-based approaches, with the best generative method achieving approximately 3.6 times the acceptance rate of the best rule-based method. Performance varies across conditions: fog proves easiest to simulate, while nighttime transformations remain challenging. Notably, the VLM jury assigns imperfect acceptance even to real adverse-condition imagery, establishing practical ceilings against which synthetic methods can be judged. By this standard, leading generative methods match or exceed real-image performance for most conditions. These results suggest that modern generative image-editing models can enable scalable generation of realistic adverse-condition imagery for evaluation pipelines. Our framework therefore provides a practical approach for scalable realism evaluation, though validation against human studies remains an important direction for future work.

Figures & tables

Figure 1: Overview of the experimental framework. Clear-day images are transformed using rule-based or generative AI methods to simulate adverse environmental conditions. Realism is assessed via VLM jury evaluation and embedding-based distributional analysis using CLIP and DINOv3.

Figure 2: Example augmentations across methods and target conditions. Top row shows real ACDC images. Subsequent rows show outputs from Rule-based Methods (imgaug, albumentations) and generative AI methods (OpenAI, Gemini, Qwen, Flux).

Figure 3: Realism scores by augmentation method and target environmental condition. (A) VLM jury acceptance rates. (B) Negative relative Mahalanobis distance in CLIP embedding space. Grey diamonds indicate baselines from real adverse-condition images. Error bars show 95% bootstrap confidence intervals. Higher values indicate greater realism.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Judge	Model	Max Tokens	Temperature	Reasoning Mode
GPT-4o	gpt-4o	2048	Default	None
Claude	claude-sonnet-4-20250514	2048	Default	Extended (1024 tokens)
Gemini	gemini-2.5-pro	2048	Default	Dynamic

Appendix

Table 1: VLM judge model configurations.

Judge Pair	Augmented Images	Real Images
Claude–GPT-4o	0.69	0.19
Claude–Gemini	0.37	0.33
GPT-4o–Gemini	0.45	0.13

Appendix

Table 2: Inter-judge agreement (Cohen’s

\kappa

) for augmented and real adverse-condition images.

Judge	Augmented Images	Real Images	N
Claude	0.759 [0.732, 0.787]	0.969 [0.944, 0.994]	956/160
GPT-4o	0.722 [0.692, 0.751]	0.919 [0.875, 0.956]	957/160
Gemini	0.438 [0.407, 0.471]	0.994 [0.981, 1.000]	960/160

Appendix

Table 3: Overall judge acceptance rates with 95% bootstrap confidence intervals.

Condition	Acceptance Rate [95% CI]
Night	1.000 [1.000, 1.000]
Fog	0.992 [0.975, 1.000]
Rain	0.933 [0.883, 0.975]
Snow	0.917 [0.858, 0.958]

Appendix

Table 4: Baseline acceptance rates for real adverse-condition images by condition (N=120 per condition).

Condition	GPT-4o Score	Overall Score
Fog	0.925	0.916
Night	0.850	0.840
Rain	0.825	0.817
Snow	0.850	0.858

Appendix

Table 5: OpenAI augmentation scores: GPT-4o judge vs. overall average.

Condition	Gemini Score [95% CI]	Overall Score
Fog	0.800 [0.675, 0.925]	0.924
Night	0.875 [0.750, 0.975]	0.958
Rain	0.650 [0.500, 0.800]	0.867
Snow	0.650 [0.500, 0.775]	0.858

Appendix

Table 6: Gemini augmentation scores: Gemini judge vs. overall average.

Method	Gemini-only	Non-Gemini	Full Jury
Qwen	0.850 (1)	0.997 (1)	0.948 (1)
Gemini	0.744 (2)	0.981 (2)	0.902 (2)
OpenAI	0.625 (3)	0.975 (3)	0.858 (3)
Flux	0.381 (4)	0.748 (4)	0.626 (4)
imgaug	0.013 (5)	0.389 (5)	0.263 (5)
albumentations	0.013 (5)	0.356 (6)	0.242 (6)

Appendix

Table 7: Method acceptance rates by jury composition. Rankings shown in parentheses.

Method	CLIP	DINOv3	CLIP+DINOv3
OpenAI	46.5	96.8	178.1
Qwen	53.7	85.4	188.9
Gemini	57.8	105.9	231.8
Flux	99.9	144.5	295.2
imgaug	201.5	307.1	689.1
albumentations	299.6	379.3	886.8

Appendix

Table 8: Method performance by embedding space (average relative Mahalanobis distance across conditions; lower is better).

Condition	CLIP	DINOv3	CLIP+DINOv3
Fog	3.2 [-0.2, 8.8]	2.0 [0.2, 4.2]	8.2 [4.1, 14.3]
Night	2.0 [0.3, 4.1]	2.4 [1.1, 3.8]	9.8 [5.6, 15.8]
Rain	0.2 [-0.6, 1.2]	2.0 [1.1, 3.0]	7.2 [4.8, 9.9]
Snow	1.6 [-0.3, 3.9]	3.3 [1.5, 5.3]	9.4 [6.2, 13.3]

Appendix

Table 9: Baseline relative Mahalanobis distances for real adverse-condition images [95% CI].

Condition	Best Method	Distance	Baseline	Ratio
Snow	Qwen	6.3	1.6	3.9 $\times$
Rain	Qwen	9.1	0.2	45.5 $\times$
Fog	Qwen	14.5	3.2	4.5 $\times$
Night	OpenAI	130.3	2.0	65.2 $\times$

Appendix

Table 10: Best augmentation method distances versus baseline (CLIP embeddings).

Method	CLIP	DINOv3	CLIP+DINOv3
OpenAI	130.3	223.8	444.5
Gemini	135.0	179.8	558.9
Qwen	185.0	216.5	596.7
Flux	219.8	188.6	499.8
Baseline	2.0	2.4	9.8

Appendix

Table 11: Nighttime augmentation distances by method and embedding space.

Judge	Model	Configuration
Claude	claude-sonnet-4-20250514	thinking_budget=1024
Gemini	gemini-2.5-pro	thinking_budget=-1 (dynamic)
GPT	gpt-4o	default

Appendix

Table 12: LLM judge configurations for failure classification.

Failure Type	Count	Percentage
Realism only	2486	80.0%
Both semantic and realism	311	10.0%
Semantic only	267	8.6%
Neither (ambiguous)	44	1.4%
Total	3108	100.0%

Appendix

Table 13: Distribution of failure types across all rejected augmentations (N=3,108 classifications).

Rule-based Methods
Method	N	Both	Semantic Only	Realism Only	Neither
imgaug	1059	1.7%	0.0%	97.5%	0.8%
albumentations	1092	16.6%	0.7%	82.1%	0.5%
Generative AI methods
OpenAI	204	2.9%	74.0%	17.6%	5.4%
Qwen	75	1.3%	66.7%	28.0%	4.0%

Appendix

Table 14: Failure type distribution by augmentation method. Percentages are row-wise.

Condition	N	Both	Semantic Only	Realism Only	Neither
Fog	426	0.0%	10.1%	88.3%	1.6%
Rain	849	4.8%	8.8%	85.5%	0.8%
Night	999	14.1%	4.6%	79.9%	1.4%
Snow	834	15.5%	12.4%	70.3%	1.9%

Appendix

Table 15: Failure type distribution by target condition. Percentages are row-wise.

Method	Condition	N	Both	Sem.	Real.	Neither
albumentations	snow	348	32.5%	0.3%	66.4%	0.9%
flux	night	294	34.7%	0.3%	63.3%	1.7%
imgaug	snow	330	3.9%	0.0%	94.8%	1.2%
imgaug	night	303	0.7%	0.0%	98.7%	0.7%
openai	rain	66	0.0%	93.9%	0.0%	6.1%
qwen	snow	27	0.0%	96.3%	0.0%	3.7%

Appendix

Table 16: Failure type distribution for selected method-condition combinations with highest failure counts or notable patterns.

Classification	Fleiss’ $\kappa$	Unanimous (3/3)	Majority (2/3)
Realism	0.904 (almost perfect)	97.4%	2.6%
Semantic	0.749 (substantial)	88.6%	11.4%

Appendix

Table 17: Inter-judge agreement for failure classification.

LLM Judge	Semantic Failure Rate	Realism Failure Rate
Gemini	22.2%	90.8%
GPT	19.1%	88.9%
Claude	14.5%	90.3%
Spread (max–min)	7.7%	1.9%

Appendix

Table 18: Classification rates by LLM judge, showing inter-judge variation.

Figure 4: Cases of maximal disagreement between VLM jury and Mahalanobis distance. Left (green): images unanimously accepted by VLMs despite high Mahalanobis distance. Right (red): images unanimously rejected by VLMs despite low Mahalanobis distance. Visual inspection suggests VLM judgments better capture perceptual realism.

Condition	Prompt
Fog	Task: In the image changes the weather so it is a foggy day. Criteria: $\bullet$ The image should be identical in every way to the provided image except that it should be foggy. $\bullet$ The camera angle and perspective must be the same. $\bullet$ The fog should be light and volumetric, getting denser with distance.
Rain	Task: In the image changes the weather so it is a rainy day. Criteria: $\bullet$ The image should be identical in every way to the provided image except that it should be rainy. $\bullet$ The camera angle and perspective must be the same. $\bullet$ The rain should be visible with natural rain streaks and wet surfaces.
Snow	Task: In the image changes the weather so it is a snowy day. Criteria: $\bullet$ The image should be identical in every way to the provided image except that it should be snowy. $\bullet$ The camera angle and perspective must be the same. $\bullet$ The snow should be falling and accumulated on horizontal surfaces.
Night	Task: In the image changes the time of day so it is nighttime. Criteria: $\bullet$ The image should be identical in every way to the provided image except that it should be nighttime. $\bullet$ The camera angle and perspective must be the same. $\bullet$ The lighting should be darker with evening/twilight atmosphere but objects should remain clearly visible.

Appendix

Table 19: Prompts used for generative AI augmentation methods.

Explore similar work

Mar 29, 2026

ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks

Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce ImagenWorld, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation.

Samin Mahdizadeh Sani, Max Ku, Nima Jamali +23

University of Waterloo · G-G-G · Independent +1

Image Generation Evaluation T2I Generation Evaluation Image Editing Evaluation

Ask PDF

Sep 26, 2026

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or processed images. Existing datasets provide limited support for evaluating these challenges jointly across diverse visual content. We introduce REALIS, a dataset of 1.43 million real and synthetic images generated by 42 modern text-to-image models, including the latest proprietary systems such as Nano Banana 2. REALIS combines prompts derived from real images, quality filtering, and stratified sampling to reduce class-specific shortcuts while preserving content diversity. We further introduce REALIS-Expert, a stress-test subset for high-quality synthetic images, where real and generated samples are selected with closely matched semantic and visual characteristics. We also propose a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing. Based on REALIS, our benchmark evaluates pretrained detectors, fine-tuned models, and zero-shot vision-language models under generator and post-processing shifts. On the hardest processed split, the best pretrained conventional detector achieves 0.550 ROC-AUC, compared with 0.752 for the best REALIS-trained detector. REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use.

Aleksandr Gushchin, Khaled Abud, Georgii Bychkov +5

MSU Institute for Artificial Intelligence · Trusted AI Research Center RAS · Lomonosov Moscow State University

AI-Generated Image Detection

Ask PDF

Apr 3, 2026

Gram-MMD: A Texture-Aware Metric for Image Realism Assessment

Evaluating the realism of generated images remains a fundamental challenge in generative modeling. Existing distributional metrics such as the Frechet Inception Distance (FID) and CLIP-MMD (CMMD) compare feature distributions at a semantic level but may overlook fine-grained textural information that can be relevant for distinguishing real from generated images. We introduce Gram-MMD (GMMD), a realism metric that leverages Gram matrices computed from intermediate activations of pretrained backbone networks to capture correlations between feature maps. By extracting the upper-triangular part of these symmetric Gram matrices and measuring the Maximum Mean Discrepancy (MMD) between an anchor distribution of real images and an evaluation distribution, GMMD produces a representation that encodes textural and structural characteristics at a finer granularity than global embeddings. To select the hyperparameters of the metric, we employ a meta-metric protocol based on controlled degradations applied to MS-COCO images, measuring monotonicity via Spearman's rank correlation and Kendall's tau. We conduct experiments on both the KADID-10k database and the RAISE realness assessment dataset using various backbone architectures, including DINOv2, DC-AE, Stable Diffusion's VAE encoder, VGG19, and the AlexNet backbone from LPIPS, among others. We also demonstrate on a cross-domain driving scenario (KITTI / Virtual KITTI / Stanford Cars) that CMMD can incorrectly rank real images as less realistic than synthetic ones due to its semantic bias, while GMMD preserves the correct ordering. Our results suggest that GMMD captures complementary information to existing semantic-level metrics.

Joé Napolitano, Pascal Nguyen

AMIAD

Image Generation Evaluation Image Quality Assessment

Ask PDF