Criticality in Dissimilar Decomposition and Undersampling of Random Datasets with Anomalies

Ghurumuruhan Ganesan

cs.LGAug 14, 2026

Criticality in Dissimilar Decomposition and Undersampling of Random Datasets with Anomalies

Authors: Ghurumuruhan Ganesan

Organizations: University of Bristol, UK

Abstract

Training datasets for upcoming LLMs would include a significant amount of AI text/image data generated from current LLMs. In such a scenario, it is important to understand how this affects batch decompositions and thereby, the performance of the resultant new LLM. In this paper, we consider AI generated data as anomalies linked" to main data points and study decomposition and undersampling properties of the overall random dataset. We use redundancy graphs and iteration techniques to obtain bounds for the minimum size of a strongly dissimilar (SD) decomposition and demonstrate a phase transition phenomena, wherein the minimum size is essentially determined by the *main* data points when the number of anomalies is small and is taken" over by the anomalies above a certain threshold. We also establish a size criticality result for the strong similarity of a randomly undersampled dataset and illustrate our results with examples involving categorical datasets, whose overall space size is much larger than the size of the dataset.

Figures & tables

Symbol	Meaning
$n$	size of the overall dataset
$v_{0}$	number of anomalous data points
$A_{v}$	$v^{th}$ data point
$\lambda$	similarity threshold
$\zeta_{up},\zeta_{low}$	extremal similarity probabilities of main data points
$p_{up},p_{low}$	extremal similarity probabilities of anomalous data points

Table 1: Notation.

Data Contamination in Language Models

Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.

See all 64 papers

LLM Benchmark Datasets Should Be Contamination-Resistant
May 19, 2026Ali Al-Lawati, Jason Lucas, Dongwon Lee +1LLM Evaluation Benchmark Contamination
The Pennsylvania State University, University Park, PA, USA.
Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
May 19, 2026Jingwen Liu, Ezra Edelman, Surbhi Goel +1Deep Learning Optimization Statistical Learning Theory
Columbia University · University of Pennsylvania · Kempner Institute, Harvard University
Synthetic Data Characterization via Training Dynamics
Sep 30, 2026Irene Lago, Ana Ezquerro, David VilaresSynthetic Data Generation for Language Models Synthetic Data Evaluation
Universidade da Coruña, CITIC, Spain · Graz University of Technology, IML, Austria

May 19, 2026J/K move · Enter open · S save

LLM Benchmark Datasets Should Be Contamination-Resistant

Ali Al-Lawati, Jason Lucas, Dongwon Lee, Suhang Wang

The Pennsylvania State University, University Park, PA, USA.

LLM Evaluation Benchmark Contamination LLM Reliability Data Contamination in Language Models Benchmark Validity

Open paper AskPDF arXiv

Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark datasets are included in pretraining corpora, i.e.,

\textit{contaminated}

, which diminishes their value as reliable measures of model generalization. In this paper, we argue that benchmark datasets should be

\textit{contamination-resistant}

, i.e.,

\textit{unlearnable}

, but support

\textit{inference}

. To accomplish this, we first highlight the wide prevalence of benchmark dataset contamination and outline the properties of contamination-resistant datasets. Second, we highlight how the asymmetry between the inference and training pipelines in the Transformer architecture can be leveraged to support contamination-resistance. Third, we outline mathematical advancements to make these datasets interoperable across various LLM architectures. Based on the above, we call on the community to ensure the reliability of LLM benchmarking by: (i) advancing novel contamination-resistant methodologies, (ii) developing supporting methods and platforms, and (iii) adopting contamination-resistant benchmarks into existing evaluation pipelines.

Abstract

Data Contamination in Language Models

Explore similar work

LLM Benchmark Datasets Should Be Contamination-Resistant

Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

Synthetic Data Characterization via Training Dynamics