Quantifying Error Tolerance in Synthetic Data: An Atomic-level Operand vs. Operator Perturbation Study

Jiaxiang Liu; Chenhao Yuan; Shuwen Xu; Boxuan Xing; Xiusheng Huang; Yinhao Xu; Hao Liu; Wenhao Teng

cs.CLAug 29, 2026

Quantifying Error Tolerance in Synthetic Data: An Atomic-level Operand vs. Operator Perturbation Study

Authors: Jiaxiang Liu, Chenhao Yuan, Shuwen Xu, Boxuan Xing, Xiusheng Huang, Yinhao Xu, Hao Liu, Wenhao Teng,

Organizations: C2DL, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences · Department of Gastrointestinal Surgery Fujian Provincial Cancer Hospital · College of Computer and Data Science, Fuzhou University

Abstract

Synthetic data generation has become a cornerstone for advancing large language models. However, the lack of the quantitative analysis for error tolerance became a critical bottleneck. Consequently, current filtering strategies fluctuate between two extremes: they are either overly aggressive, risking the exclusion of potentially valuable samples, or overly permissive, failing to eliminate erroneous samples effectively. To bridge this gap, this paper introduces Atomic Tree Operation Modeling (ATOM), a framework that decomposes data into functional units ( $f(x)\rightarrow y$ ). ATOM distinguishes benign Operand $x$ perturbations from fatal Operator $f$ perturbations. The former are needlessly discarded by aggressive filtering, while the latter slip through permissive filtering. Our experiments reveal a double dissociation: models are robust to operand perturbations but collapse under operator perturbations. By prioritizing operator over aggressive operand precision, our ATOM-synthesized data outperforms rigorous baselines (e.g., +3.1% gain over LIMA), suggesting that operator diversity matters more than operand precision. Our code is available at https://github.com/Lut-hub/ATOM.

Figures & tables

Figure 1: The illustration of extract atomic functions and two types of perturbations: Operand Perturbations modify $x$ while updating $y$ to keep $f$ valid, e.g., $Contains(\text{xk7q})$ $\rightarrow$ m3p / n8s / t4v . And Operator Perturbations keep $x$ fixed but modify $y$ to violate $f$ , e.g., $Contains(\text{primary colors})\rightarrow\text{{\color[rgb]{0.75,0,0.25}purple}/{\color[rgb]{0.5,0.5,0.5}gray}/{\color[rgb]{0,1,0}green}}$ .

Figure 2: The pipeline of our A tomic T ree O peration M odeling ( ATOM ) framework. ATOM consists of two phases: 1) Atomic Tree Construction, which builds an operator hierarchy where leaf nodes define concrete atomic operators, and 2) Atomic Data Synthesis, which supports both Atomic-QA and Atomic-X.

Figure 3: Performance comparison of LLaMA3.1 across Atomic-QA, Operand-Perturbed Atomic-QA, and Operator-Perturbed Atomic-QA. Dashed lines denote the average performance for each setting.

Figure 4: MCQ accuracy vs. training data scale (main plot), with model agreement under replaced vs. original entity settings. Agreement Rate is the proportion of questions both models answer correctly.

Figure 5: CKA similarity of the output hidden states between different trained models. The lower part represents the CKA similarity of Qwen 2.5-7B, while the upper part represents LLaMA 3.1-8B.

Model Family		ARC- Challenge	ARC- Easy	BoolQ	Hotpot- QA	Open- BookQA	PIQA	RACE	SIQA	SQuADv2	Wino- Grande	Avg
LLaMA3.1-8B	Base	0.549	0.847	0.820	0.106	0.366	0.807	0.433	0.529	0.375	0.765	0.560
	All Average	0.564	0.850	0.855	0.127	0.387	0.812	0.464	0.548	0.381	0.768	0.576
	Instruct	0.558	0.854	0.870	0.209	0.362	0.811	0.464	0.548	0.420	0.764	0.586
	LIMA	0.552	0.848	0.828	0.134	0.368	0.808	0.432	0.529	0.377	0.773	0.565
	Condor	0.563	0.854	0.855	0.117	0.382	0.811	0.470	0.546	0.378	0.770	0.575
	Magpie	0.575	0.845	0.859	0.122	0.380	0.808	0.464	0.563	0.374	0.761	0.575

Table 1: Overall performance across all training datasets on LLaMA3.1-8B and Qwen3-8B. Color-coded cells ( green / red ) are used to indicate performance above or below the mean across methods (All Average), with darker colors representing better or worse performance. The results for Qwen2.5-7B and Qwen2.5-14B are in Table 3 .

Figure 6: Radar plot of corpus diversity.

Figure 7: Average accuracy for LLaMA 3.1-8B and Qwen 2.5-7B on Atomic-Alpaca with operator perturbations at controlled rates (§ 7 ). Dashed lines: upper bound from training only on correct samples, truncated at 60% to approximate the solid lines’ data scale. See Figure 17 for Atomic-MetaMath.

Figure 8: Cohen’s Kappa between filtering methods (IFD, Length, Style) and ground-truth error labels as a function of sample error rate, evaluated on Atomic-Alpaca and Atomic-MetaMath .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: The initial operator tree of Atomic-QA. Internal nodes represent abstract operator categories, and leaf nodes define concrete atomic operators.

Figure 10: The initial operator tree of Alpaca.

Figure 11: The initial operator tree of MetaMath.

Category	Benchmark	Shots	Metric
Text Comprehension and Information Integration	BoolQ	3	Accuracy
	LongBench-HotpotQA	0	F1
	RACE	3	Accuracy
	SQuADv2	3	F1
Knowledge-Driven Reasoning	ARC-Challenge	3	Accuracy
	ARC-Easy	3	Accuracy

Appendix

Table 2: Evaluation benchmarks and configurations.

Model Family		ARC- Challenge	ARC- Easy	BoolQ	Hotpot- QA	Open- BookQA	PIQA	RACE	SIQA	SQuADv2	Wino- Grande	Avg
Qwen2.5-7B	Base	0.590	0.860	0.872	0.216	0.390	0.803	0.459	0.576	0.421	0.754	0.594
	All Average	0.586	0.859	0.875	0.190	0.399	0.800	0.474	0.572	0.364	0.747	0.587
	Instruct	0.616	0.873	0.865	0.143	0.404	0.786	0.512	0.566	0.324	0.735	0.582
	LIMA	0.586	0.861	0.871	0.207	0.396	0.801	0.459	0.574	0.424	0.757	0.594
	Condor	0.597	0.864	0.874	0.185	0.392	0.801	0.471	0.580	0.396	0.755	0.591
	Magpie	0.582	0.857	0.876	0.241	0.364	0.795	0.455	0.572	0.412	0.740	0.589

Appendix

Table 3: Overall performance across all training datasets on Qwen2.5-7B and Qwen2.5-14B. Color-coded cells ( green / red ) are used to indicate performance above or below the mean across methods (All Average), with darker colors representing better or worse performance.

Figure 12: Perplexity trends of LLaMA and Qwen models as a function of training data volume. The curve displays average PPL for both the original and entity-replaced variants, with an inset showing a zoomed region for the 30K–500K data range.

Figure 13: CKA variance across layers versus training data volume for LLaMA3.1-8B and Qwen2.5-7B, showing mean values (lines) and min–max ranges (shaded areas).

Dataset	Training Tokens
LIMA	627,993
Condor	21,131,550
Magpie	203,256,865
WizardLM	27,673,025
Alpaca	3,766,455
Self-Instruct	4,536,225

Appendix

Table 4: The number of training tokens in different datasets.

Figure 14: Performance comparison of Qwen2.5-7B across the Atomic-QA, Operand-Perturbed Atomic-QA, and Operator-Perturbed Atomic-QA settings. Each bar shows the model’s accuracy on individual tasks, while the dashed lines denote the average performance for each setting.

Figure 15: Performance comparison of Qwen2.5-14B across the Atomic-QA, Operand-Perturbed Atomic-QA, and Operator-Perturbed Atomic-QA settings. Each bar shows the model’s accuracy on individual tasks, while the dashed lines denote the average performance for each setting.

Figure 16: Performance comparison of Qwen3-8B across the Atomic-QA, Operand-Perturbed Atomic-QA, and Operator-Perturbed Atomic-QA settings. Each bar shows the model’s accuracy on individual tasks, while the dashed lines denote the average performance for each setting.

Model	Perturbation	Early	Middle	Late	Last layer
LLaMA3.1-8B	Operand	0.988	0.964	0.961	0.935
	Operator	0.932	0.865	0.888	0.832
	Base Model	0.942	0.743	0.668	0.745
Qwen2.5-7B	Operand	0.998	0.999	0.994	0.950
	Operator	0.998	0.999	0.988	0.899
	Base Model	0.997	0.999	0.973	0.773

Appendix

Table 5: Layer-wise CKA similarity against the model trained on clean Atomic-QA, averaged over the early, middle, and late thirds of the layers.

Model	Atomic-QA	Operator-Pert.	Operand-Pert.
Model	Atomic-QA	Operator-Pert.	w/ r.s.	w/ s.e.
LLaMA3.1-8B	35.80	36.53	99.76	35.82
Qwen3-8B	36.00	33.04	90.78	32.98
Qwen2.5-7B	43.33	42.17	136.24	42.32
Qwen2.5-14B	52.25	39.48	131.47	39.91

Appendix

Table 6: Average perplexity of training corpora evaluated by pretrained checkpoints. Operand-Pert. is reported for both the random-string construction (w/ r.s., used in

\lx@sectionsign

5 ) and the LLM-rewritten synthetic-entity construction (w/ s.e., Appendix D.6 ).

Model	Perturbation	ARC- Challenge	ARC- Easy	BoolQ	Hotpot- QA	Open- BookQA	PIQA	RACE	SIQA	SQuADv2	Wino- Grande
LLaMA3.1-8B	w/ r.s.	0.587	0.856	0.858	0.159	0.413	0.814	0.478	0.549	0.394	0.770
LLaMA3.1-8B	w/ s.e.	0.572 ( $-$ 0.015)	0.862 ( $+$ 0.006)	0.860 ( $+$ 0.002)	0.158 ( $-$ 0.001)	0.408 ( $-$ 0.005)	0.825 ( $+$ 0.011)	0.476 ( $-$ 0.002)	0.552 ( $+$ 0.003)	0.401 ( $+$ 0.007)	0.770 ( $+$ 0.000)
Qwen2.5-7B	w/ r.s.	0.588	0.854	0.877	0.306	0.412	0.799	0.483	0.570	0.386	0.746
Qwen2.5-7B	w/ s.e.	0.609 ( $+$ 0.021)	0.842 ( $-$ 0.012)	0.884 ( $+$ 0.007)	0.313 ( $+$ 0.007)	0.408 ( $-$ 0.004)	0.801 ( $+$ 0.002)	0.476 ( $-$ 0.007)	0.570 ( $+$ 0.000)	0.381 ( $-$ 0.005)	0.746 ( $+$ 0.000)
Qwen2.5-14B	w/ r.s.	0.659	0.884	0.876	0.364	0.452	0.817	0.512	0.576	0.406	0.795
Qwen2.5-14B	w/ s.e.	0.659 ( $+$ 0.000)	0.877 ( $-$ 0.007)	0.880 ( $+$ 0.004)	0.362 ( $-$ 0.002)	0.455 ( $+$ 0.003)	0.809 ( $-$ 0.008)	0.527 ( $+$ 0.015)	0.576 ( $+$ 0.000)	0.408 ( $+$ 0.002)	0.793 ( $-$ 0.002)

Appendix

Table 7: Downstream performance of the two operand-perturbation constructions. w/ r.s. : random strings (

\lx@sectionsign

5 ); w/ s.e. : plausible but synthetic entities generated with the same LLM-rewriting protocol as the operator perturbation. Parenthesised values are deltas relative to w/ r.s.

Model	Training Data	Atomic-QA	w/ r.s.	w/ s.e.
LLaMA3.1-8B	Atomic-QA	1.0	–	–
	w/ r.s.	0.935	1.0	–
	w/ s.e.	0.942	0.956	1.0
Qwen2.5-7B	Atomic-QA	1.0	–	–
	w/ r.s.	0.950	1.0	–
	w/ s.e.	0.945	0.967	1.0

Appendix

Table 8: Last-layer CKA similarity among models trained on clean Atomic-QA and the two operand-perturbation constructions.

Training Data	LLaMA3.1-8B	Qwen3-8B
Naive	0.457	0.534
Operand-Perturbed	0.439	0.558
Operator-Perturbed	0.298	0.392

Appendix

Table 9: Performance on the closed-world evaluation set for models trained on 20k closed-world data over synthetic entities with artificial relations.

Training Data	Tokens	Model	Avg. Perf.
Atomic-LIFD-20K	535,706	LLaMA3.1-8B	0.583 ( $\uparrow$ 0.018 )
		Qwen3-8B	0.627 ( $\uparrow$ 0.025 )
		Qwen2.5-7B	0.623 ( $\uparrow$ 0.029 )
		Qwen2.5-14B	0.638 ( $\uparrow$ 0.036 )
LIMA	627,993	LLaMA3.1-8B	0.565
		Qwen3-8B	0.602

Appendix

Table 10: Performance comparison between Atomic-LIFD-20K and LIMA at matched data scale.

Dataset	TTR	MTLD	MSTTR	MATTR	Vendi	LogDet
Atomic-QA	0.2688	226.97	0.8689	0.8695	4.2840	$-$ 1187.36
Atomic-LIFD-20K	0.2671	263.11	0.8817	0.8817	4.2963	$-$ 1169.71
Atomic-LIFD-100K	0.2616	253.22	0.8820	0.8806	4.2854	$-$ 1182.44

Appendix

Table 11: Diversity metrics of Atomic-QA and its LIFD-filtered subsets.

Model	Epoch 2	Epoch 3	Epoch 4	Epoch 5
LLaMA-3.1-8B	0.9995	0.9984	0.9961	0.9922
Qwen2.5-7B	0.9996	0.9986	0.9970	0.9944

Appendix

Table 12: CKA similarity between LIMA-finetuned and base models across training epochs.

Model	Epoch 2	Epoch 3	Epoch 4	Epoch 5
LLaMA-3.1-8B	0.565	0.563	0.566	0.564
Qwen2.5-7B	0.594	0.592	0.601	0.594

Appendix

Table 13: Average performance of LIMA-finetuned models across training epochs.

Figure 17: Accuracy on GSM8K, MATH-500, and MathQA for LLaMA 3.1-8B and Qwen 2.5-7B fine-tuned on Atomic-MetaMath with operator-level perturbations at controlled rates. Corresponds to Figure 7 , which reports Atomic-Alpaca on general benchmarks.

Panel of Figure 18 — Figure 18: Accuracy degradation under increasing atomic function error rates for (a) LLaMA and (b) Qwen. The x-axis denotes the proportion of erroneous atomic functions in the training set, while the y-axis represents the performance of the trained model.

Figure 19: Relationship between the atomic function error rate and the resulting percentage of wrong samples.

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 629 papers

Training-Aware Target Coverage for Synthetic Data Selection
Sep 30, 2026Yang Ba, Michelle V. Mancenido, Rong PanLLM Fine-Tuning Data Selection
School of Computing and Augmented Intelligence, Arizona State University · School of Mathematical and Natural Sciences, Arizona State University
SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data
Sep 11, 2026Praveen Kumar Myakala, Ravichandra Namburi, Sowmya Keragodu Jayaramu +1AI-Generated Text Detection Training Data Curation
Independent AI Researcher Melissa, USA · Independent Researcher Frisco, USA · Independent Researcher Austin, USA
Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection
May 13, 2026Anjir Ahmed Chowdhury, Syed Zawad, Feng YanSynthetic Data Generation LLM Inference Acceleration
Department of Computer Science University of Houston · IBM Research

Sep 30, 2026J/K move · Enter open · S save

Training-Aware Target Coverage for Synthetic Data Selection

Yang Ba, Michelle V. Mancenido, Rong Pan

School of Computing and Augmented Intelligence, Arizona State University · School of Mathematical and Natural Sciences, Arizona State University

LLM Fine-Tuning Data Selection Synthetic Data Selection

Open paper AskPDF arXiv

Synthetic data are increasingly used to scale LLM training, yet more synthetic data do not necessarily produce better models. Useful synthetic data must add information relevant to the target task without introducing errors that offset their benefit, and the value of an example can change as the training set grows. We develop a linear theory that characterizes this tradeoff and determines where synthetic data are useful, how much should be added, and the marginal value of adding one example to an existing set. The analysis shows the conditions when input coverage alone is sufficient and when synthetic errors must also be considered. Guided by these results, we introduce Training-Aware Target Coverage (TATC), a synthetic data selection method for LLM fine-tuning. TATC identifies candidates whose training effects are beneficial to the target task and selects among them to expand coverage of target-relevant directions not already represented by the available data. Experiments on text and image data verify the linear theory. With mathematical reasoning tasks, TATC selects synthetic solutions for fine-tuning Qwen2.5-Math-1.5B-Instruct and outperforms alternative synthetic-data selection methods on GSM8K across selection budgets. In summary, we provide a principled approach to synthetic data selection by quantifying and maximizing its value to the target task.

Abstract

Synthetic Data Generation

Explore similar work

Training-Aware Target Coverage for Synthetic Data Selection

SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data

Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection