In-Context Bias Propagation in LLM-Based Tabular Data Generation

Pol G. Recasens; Alberto Gutierrez; Jordi Torres; Josep. Ll Berral; Javier Carnerero-Cano; Anisa Halimi; Kieran Fraser

cs.LGJun 11, 2025

In-Context Bias Propagation in LLM-Based Tabular Data Generation

Authors: Pol G. Recasens, Alberto Gutierrez, Jordi Torres, Josep. Ll Berral, Javier Carnerero-Cano, Anisa Halimi, Kieran Fraser

Organizations: Barcelona Supercomputing Center (BSC), Spain · Universitat Politècnica de Catalunya (UPC), Spain · IBM Research Ireland

Abstract

Large Language Models (LLMs) are increasingly used for synthetic tabular data generation through in-context learning (ICL), offering a practical solution for data augmentation in data scarce scenarios. While prior work has shown the potential of LLMs to improve downstream task performance through augmenting underrepresented groups, these benefits often assume access to a subset of unbiased in-context examples, representative of the real dataset. In real-world settings, however, data is frequently noisy and demographically skewed. In this paper, we systematically study how statistical biases within in-context examples propagate to synthetic tabular data, showing that even mild in-context biases lead to global statistical distortions. We further introduce an adversarial scenario where a malicious contributor can inject bias into the synthetic dataset via a subset of in-context examples, ultimately compromising the fairness of downstream classifiers for a targeted and protected subgroup. Finally, we evaluate prompt-side mitigation strategies based on preprocessing in-context examples and instruction-based defenses, demonstrating that while such techniques can attenuate disparity, the inherent sensitivity of LLMs to adversarial prompts remains a persistent challenge. Our findings highlight a critical new vulnerability in LLM-based data generation pipelines within sensitive domains.

Figures & tables

Figure 1: End-to-end bias propagation. An adversarial user injects biased examples into the prompt for a targeted group. The LLM replicates these statistical patterns via in-context learning, propagating the bias into the synthetic tabular data, and further compromising downstream fairness.

Dataset	$\boldsymbol{\pi}$	$\boldsymbol{\mathrm{SPD}_{P}}$	Null 95% CI	$\boldsymbol{p_{\mathrm{perm}}}$
Adult	0.3	-0.325	[0.043, 0.369]	0.002
	0.6	-0.609	[0.021, 0.349]	0.002
Compas	0.3	-0.395	[-0.350, 0.112]	0.024
	0.6	-0.358	[-0.352, 0.091]	0.042
Diabetes	0.3	-0.256	[0.076, 0.513]	0.002
	0.6	-0.318	[0.079, 0.484]	0.002

Table 1: Statistical validation of bias in the in-context examples (

k=80

\mathrm{SPD}_{P}

denotes the prompt-level statistical parity difference. The null interval is obtained from

B=1000

clean prompts of the same size sampled from the real training data.

p_{\mathrm{perm}}

is the two-sided empirical permutation

p

-value.

Figure 2: Marginal in-context bias propagation ( Mixtral-8x7B , $k=80$ ). ( Left ) Drift in the generated distribution $D_{f}(\mathcal{D}_{G},\mathcal{D}_{0})$ versus drift in the prompt distribution $D_{f}(\mathcal{D}_{P},\mathcal{D}_{0})$ . The linear relationship shows that distributional shifts in prompts are proportionally transferred to synthetic outputs. ( Right ) Probability of the target group (Race/Gender) in generated samples increases linearly with prompt bias intensity $\pi$ , confirming that LLMs propagate univariate demographic skews from in-context examples.

Figure 3: Marginal bias propagation across model families. ( Left ) Probability of the target group in generated samples increases linearly with prompt bias intensity

\pi

for all models at

k=80

. The dashed line shows the anchor baseline

p_{\mathcal{D}_{0}}(\text{Target})

. ( Right ) Propagation coefficient

\beta_{k}

increases monotonically with context size

k

across all models.

Figure 4: Bias propagation strength increases with context size. Each subfigure shows results for a different context size

k\in\{20,40,60,80\}

. The red curve shows the target group probability in generated samples

p_{\mathcal{D}_{G}}(\text{Target})

, the blue curve shows the target group probability in prompt examples

p_{\mathcal{D}_{P}}(\text{Target})

, and the dashed line shows anchor baseline

p_{\mathcal{D}_{0}}(\text{Target})

. As

k

increases, the red curve converges toward the blue curve, indicating that larger context windows lead to stronger replication of prompt-level demographic distributions.

Figure 5: Conditional bias propagation for targeted and non-targeted subgroups. ( Left ) Conditional probability of generating High Income examples for each gender subgroup as the bias intensity

\pi

increases. The targeted subgroup exhibits strong linear sensitivity to

\pi

, while the non-targeted subgroup remains near 0.5. ( Right ) Statistical parity difference (SPD) between subgroups increases more steeply at

k=80

than

k=20

, demonstrating that larger context windows increase sensitivity to conditional bias propagation.

Figure 6: Model-specific sensitivity to conditional bias propagation. ( k=80, Adult ) The slope

\beta_{k}

quantifies how strongly the predicted positive rate for each gender subgroup changes with bias intensity

\pi

. We estimate

\beta_{k}

by fitting a linear regression to

P(Y=1\mid\text{subgroup})

across values of

\pi

. All models show substantially steeper slopes for the targeted ( Female ) subgroup compared to the non-targeted ( Male ) subgroup, indicating systematic propagation of the injected correlation. The magnitude varies across architectures, with larger and mixture models exhibiting stronger conditional bias transfer.

Figure 7: Multivariate intersectional probabilities. ( Granite-8b ) Conditional probability of high income for each gender–race cell in the generated data as a function of the intersectional bias strength

\pi

, for different context sizes

k

. As

\pi

and

k

increase, the LLM increasingly reproduces the intersectional pattern injected in the prompt.

Figure 8: Intersectional high–income probability heatmap. ( Granite-8b ) Conditional probability of high income $P(\text{HI}\mid\text{gender},\text{race})$ in the generated data for Adult under the intersectional bias. The left figure corresponds to an unbiased prompt ( $\pi=0$ ), while the right figure shows a fully biased prompt ( $\pi=0.6$ ) in which (Female, Black) and (Male, White) are high-income cells. The model reproduces this injected intersectional pattern, raising $P(\text{HI})$ for the high income subgroups and lowering it for the other subgroups.

Dataset	Protected Attr	Target Label	Feature-Aligned Values
Compas	Race: African-American	Recidivism (1)	Priors: 3–8 Age: 18–45 Juv. Felonies: 0 Charge Degree: M or F (Random)
Adult	Gender: Female Marital-status: Married-civ-spouse Native-country: United-States	Income: $>$ 50K	Age: 30–55 Education: HS-grad or Some-college Hours/Week: 38–41 Capital Loss: 0–50
Diabetes	Age: 21–30	Diabetes (1)	Glucose: 120–155 BMI: 27.0–33.0 Insulin: 80–180 Pedigree: 0.6–1.2
Thyroid	Age: 20–35	High Risk (2)	Goiter: 0 or 1 (Random) Family History: 0 or 1 (Random) Fatigue: 0 or 1 (Random) Gender: 0 (Female)

Table 2: Attributes used to construct adversarial feature-aligned examples for each dataset.

Model	$\pi$	Adult			Compas			Diabetes			Thyroid
		$\text{F1}_{\text{R}}$ ↑	$\text{SPD}_{\text{S}}$ ↓	$\text{SPD}_{\text{D}}$ ↓	$\text{F1}_{\text{R}}$ ↑	$\text{SPD}_{\text{S}}$ ↓	$\text{SPD}_{\text{D}}$ ↓	$\text{F1}_{\text{R}}$ ↑	$\text{SPD}_{\text{S}}$ ↓	$\text{SPD}_{\text{D}}$ ↓	$\text{F1}_{\text{R}}$ ↑	$\text{SPD}_{\text{S}}$ ↓	$\text{SPD}_{\text{D}}$ ↓
Real oracle	–	$0.85$	$-$	$0.13_{0.00}$	$0.62$	$-$	$-0.13_{0.00}$	$0.73$	$-$	$0.29_{0.00}$	$0.86$	$-$	$-0.06_{0.00}$
TVAE	0.0	$0.45$	$-0.04_{0.00}$	$-0.04_{0.00}$	$0.53$	$-0.04_{0.00}$	$0.01_{0.00}$	$0.66$	$-0.04_{0.00}$	$0.06_{0.00}$	$0.49$	$-0.04_{0.00}$	$-0.11_{0.00}$
CTGAN	0.0	$0.51$	$0.78_{0.00}$	$0.07_{0.00}$	$0.58$	$0.52_{0.00}$	$-0.04_{0.00}$	$0.67$	$0.74_{0.00}$	$0.42_{0.00}$	$0.46$	$0.56_{0.00}$	$-0.13_{0.00}$
NFLOW	0.0	$0.57$	$0.51_{0.00}$	$0.10_{0.00}$	$0.57$	$0.64_{0.00}$	$-0.03_{0.00}$	$0.50$	$0.51_{0.00}$	$0.02_{0.00}$	$0.44$	$0.44_{0.00}$	$0.02_{0.00}$
DDPM	0.0	$0.51$	$0.01_{0.02}$	$0.01_{0.02}$	$0.52$	$-0.62_{0.01}$	$-0.17_{0.07}$	$0.39$	$0.09_{0.02}$	$-0.01_{0.01}$	$0.58$	$0.37_{0.04}$	$-0.04_{0.01}$

Table 3: Utility and fairness metrics (F1-Score, SPD) across four datasets (

k{=}80

). For each dataset, we train the Random Forest downstream classifier with the synthetic dataset, and report F1 score on the real dataset (

\text{F1}_{\text{R}}

), SPD on the synthetic data (

\text{SPD}_{\text{S}}

), and SPD of the downstream model (

\text{SPD}_{\text{D}}

Figure 9: Normalized density of aligned features. (Mixtral-8

\times

7B,

k{=}80

ICL examples, Compas) Synthetic univariate distributions of key recidivism predictors for the protected subgroup (African-American) and the complementary group, as the fraction of biased in-context examples

\pi

increases. The attack steers African-American records towards the plausible values set by the attacker (e.g., low prior counts and fewer juvenile offenses) while leaving the non-targeted group closer to the baseline, illustrating how a number of adversarial examples can selectively reshape the synthetic feature profile of a protected subgroup.

Figure 10: Downstream fairness and race importance vs in-context bias

\pi

. (Left)

\Delta\lvert\text{SPD}\rvert

of downstream model predictions on the real dataset when fitted on the synthetic dataset, including and excluding race features. (Right) Race feature MDI importance in the Random Forest classifier trained on synthetic data generated with various LLMs.

Model	$\pi$	Adult		Compas		Diabetes	Thyroid
		JSD ↓	TVC ↑	JSD ↓	TVC ↑	JSD ↓	JSD ↓	TVC ↑
Granite-8B	0.0	0.096	0.790	0.258	0.895	0.340	0.183	0.971
	0.3	0.122	0.754	0.366	0.890	0.342	0.221	0.911
	0.6	0.201	0.724	0.484	0.824	0.377	0.269	0.819
Mistral-7B	0.0	0.161	0.716	0.318	0.871	0.263	0.232	0.927
	0.3	0.186	0.622	0.411	0.905	0.297	0.242	0.873

Table 4: Marginal fidelity of the synthetic dataset to the real dataset for different models and bias rates. We show Jensen–Shannon Divergence (JSD) for numerical variables and Total Variation (TVC) for categorical variables. The Diabetes dataset has only numerical features, so we report only JSD.

Model	$\pi$	Adult				Compas
		Asc. $\downarrow$		MMD $\downarrow$		Asc. $\downarrow$		MMD $\downarrow$
		D	T	D	T	D	T	D	T
Granite-8B	0.0	0.054	0.052	0.012	0.001	0.058	0.025	0.079	0.007
	0.3	0.082	0.249	0.013	0.023	0.061	0.062	0.076	0.004
	0.6	0.094	0.434	0.025	0.091	0.064	0.191	0.075	0.012
Mistral-7B	0.0	0.146	0.038	0.028	0.021	0.094	0.102	0.093	0.011

Table 5: Structural fidelity of the synthetic dataset to the real dataset for different models and bias rates. We show mixed-type association matrix distance (Asc.) and RBF maximum mean discrepancy (MMD) for D, excluding the protected attribute and target, and T, containing only the protected attribute and target.

Figure 11: Adversarial in-context bias injection and utility–fairness trade-off. ( Compas ) Effect of the in-context bias rate

\pi

on downstream performance on real data and fairness for four LLM generators. Each figure shows the macro F1 score of a Random Forest trained on the synthetic data and evaluated on the real test set (blue), the absolute SPD of the synthetic data (red), and the absolute SPD of the downstream predictions on real data (grey). While F1 on real data remains nearly constant as

\pi

increases, both synthetic and downstream SPDs grow, showing that small in-context biases can substantially degrade fairness while preserving downstream accuracy.

Figure 12: (Granite-8B, Compas). Equalized odds (left) and equal opportunity (right) of downstream model predictions on the real dataset across data generated with various LLMs and in-context bias

\pi

	$\pi=0.0$		$\pi=0.6$
Classifier	F1 R ↑	SPD D ↓	F1 R ↑	SPD D ↓
Random Forest	0.58	-0.52	0.61	-0.41
Logistic Regression	0.53	-0.43	0.55	-0.53
Gradient Boosting	0.58	-0.34	0.57	-0.38
TabPFN [ 10 ]	0.61	-0.15	0.62	-0.25

Table 6: Effect of downstream classifier on bias propagation for Compas (Granite-8B,

k=80

). For each classifier, we report macro F1 on the real test set (F1 R ) and SPD of the downstream predictions on the real test set (

\text{SPD}_{\mathrm{D}}

) at

\pi=0

and

\pi=0.6

Dataset	$\boldsymbol{\mathrm{SPD}_{P}}$ before	$\boldsymbol{\mathrm{SPD}_{P}}$ after	Biased drop	Clean drop
Adult	-0.839	-0.944	0.938	0.438
Compas	-0.358	-0.133	1.000	0.469
Diabetes	-0.318	-0.188	0.771	0.531
Thyroid	-0.517	-0.433	0.875	0.500

Table 7: Outlier-filter robustness at

\pi=0.6

and

k=80

\mathrm{SPD}_{P}

compares the unfiltered attacked prompt with the prompt after filtering with Isolation Forest. Biased drop is the fraction of injected adversarial examples removed by the filter, while Clean drop is the fraction of non-injected examples removed.

Model	Mitigation	Adult					Compas
		F1↑	$\text{SPD}_{\text{S}}$	$\text{SPD}_{\text{D}}$	$\text{EO}_{\text{D}}$ ↓	$\text{EOD}_{\text{D}}$ ↓	F1↑	$\text{SPD}_{\text{S}}$	$\text{SPD}_{\text{D}}$	$\text{EO}_{\text{D}}$ ↓	$\text{EOD}_{\text{D}}$ ↓
Granite-8B	No mitigation	$0.73$	$-0.48_{0.02}$	$-0.20_{0.02}$	$0.13_{0.03}$	$0.19_{0.03}$	$0.59$	$-0.25_{0.01}$	$-0.32_{0.03}$	$0.29_{0.09}$	$0.30_{0.08}$
	Random subset	$0.71$	$-0.43_{0.06}$	$-0.20_{0.02}$	$0.06_{0.03}$	$0.17_{0.04}$	$0.60$	$-0.26_{0.05}$	$-0.16_{0.04}$	$0.21_{0.07}$	$0.25_{0.07}$
	Group-balanced	$0.74$	$-0.36_{0.02}$	$-0.17_{0.01}$	$0.04_{0.02}$	$0.11_{0.01}$	$0.57$	$-0.21_{0.01}$	$-0.25_{0.04}$	$0.34_{0.06}$	$0.31_{0.05}$
	Fair-SPD	$0.75$	$-0.41_{0.01}$	$-0.15_{0.01}$	$0.09_{0.03}$	$0.13_{0.01}$	$0.60$	$-0.19_{0.02}$	$-0.12_{0.03}$	$0.14_{0.07}$	$0.15_{0.06}$
	Correlation filtering	$0.73$	$-0.44_{0.02}$	$-0.20_{0.01}$	$0.06_{0.02}$	$0.16_{0.02}$	$0.59$	$-0.22_{0.01}$	$-0.18_{0.03}$	$0.26_{0.06}$	$0.27_{0.06}$

Table 8: Change in utility (F1↑), Statistical Parity Difference on the synthetic data (

\text{SPD}_{\text{S}}

↓) and downstream model (

\text{SPD}_{\text{D}}

↓) after mitigation (

k{=}80

\pi=0.3,\epsilon=0.02

). The cardinality of group-balanced and random subset mitigation subsets are set to the Fair-SPD subset for comparison.

Figure 13: Synthetic and downstream fairness for each preprocessing approach. ( Granite-8B, Compas ) SPD of the synthetic dataset generated by the LLM (left) and downstream classifier predictions on the real test dataset (right) for increasing bias rates across preprocessing mitigations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	Attributes	Target	Protected group
Adult	Income prediction	age, workclass, fnlwgt, education, educational-num, marital-status, occupation, relationship, race, gender, capital-gain, capital-loss, hours-per-week, and native-country.	income: income $>50$ K	Female, married-civ-spouse, United-States.
COMPAS	Two-year recidivism prediction	sex, age, juv_fel_count, juv_misd_count, juv_other_count, priors_count, age-category indicators, race indicators, and charge-degree indicators. The prepared dataset also contains id as an identifier.	y: recidivism	African-American
Diabetes	Diabetes diagnosis	Pregnancies, Glucose, BloodPressure, SkinThickness, Insulin, BMI, DiabetesPedigreeFunction, and Age.	Outcome: diabetes diagnosis	Age between 21 and 30
Thyroid	Thyroid-risk classification	Age, Gender, Pregnancy, Family_History_of_Thyroid, Goiter, Fatigue, Weight_Change, Hair_Loss, Heart_Rate_Changes, Sensitivity_to_Cold_or_Heat, Increased_Sweating, Muscle_Weakness, Constipation_or_More_Bowel_Movements, Depression_or_Anxiety, Difficulty_Concentrating_or_Memory_Problems, and Dry_or_Itchy_Skin.	Thyroid_Risk_Level: no, moderate, or high risk	Age between 20 and 35
Bank	Term-deposit subscription prediction	age, job, marital, education, default, balance, housing, loan, contact, day, month, duration, campaign, pdays, previous, and poutcome.	y: subscribed to term deposit	Age between 18 and 30

Appendix

Table 9: Summary of the datasets used in the evaluation. The protected group is the subgroup targeted in the adversarial experiments.

Panel of Figure 14 — Figure 14: Bias propagation across different models. ( Left : Granite-8B; Center : Mistral-7B; Right : Qwen-70B). In all cases, as the bias in the prompt increases, the bias in the generated output scales up linearly.

Figure 15: Bias propagation strength vs. context size

k

. ( Top : Granite-8B; Middle : Mistral-7B; Bottom : Qwen-70B). The red curves (

D_{G}

) align closer to the blue curves (

D_{P}

) as

k

increases for all models.

Figure 16: Conditional bias propagation on Adult (target: Female). We systematically vary the in-context Income rate for the Female subgroup while fixing the non-targeted Male subgroup at

0.5

(

k=80

). While all models correctly propagate the targeted shift, they differ in their ability to isolate this bias. Larger models within the same family (Mixtral-8x7B) demonstrate conditional propagation with minimal spillover to the other subgroup compared to smaller models (Mistral-7B).

Figure 17: Normalized density of aligned features under feature-aligned bias injection. ( Compas ,

k{=}80

) Synthetic univariate distributions of key recidivism predictors for the protected subgroup (African-American) and the complementary group as the fraction of adversarial in-context examples

\pi

increases. Left: Mistral-7B; right: Qwen-70B. Increasing

\pi

steers the protected-group feature profiles toward attacker-chosen plausible values (e.g., lower priors_count and fewer juvenile offenses) while leaving the non-targeted group closer to baseline, illustrating the selective subgroup manipulation.

Figure 18: ( Compas ) Effect of the injected in-context bias rate

\pi

on downstream utility and fairness for four LLM-based tabular generators. Top: Logistic Regression; bottom: TabPFN. Curves report macro F1 of the downstream classifier trained on synthetic data and evaluated on the real test set (blue), absolute statistical parity difference (SPD) in the synthetic data (red), and absolute SPD of the classifier’s predictions on the real test set (grey). Across generators and both downstream models, macro F1 remains largely stable as

\pi

increases, whereas both synthetic and downstream SPD increase substantially.

Figure 19: Generalization to Bank Marketing. Effect of the injected in-context bias rate

\pi

on downstream utility and fairness for Granite-8B and Mixtral-8x7B (

k=80

). The blue curve reports the macro-F1 of a Random Forest classifier trained on the synthetic data and evaluated on the real test set. The red and grey curves report the absolute SPD of the synthetic data and downstream predictions, respectively. Both generators exhibit a clear increase in synthetic-data disparity while maintaining broadly stable utility. The downstream effect is smaller and model-dependent, indicating that prompt-level bias reliably alters the generated distribution, while its transfer to downstream predictions depends on the generator and task.

Model	$\boldsymbol{\pi}$	Adult			Compas			Diabetes			Thyroid
		$\boldsymbol{\Delta\mathrm{SPD}_{S}}$	95% CI	$\boldsymbol{p}$	$\boldsymbol{\Delta\mathrm{SPD}_{S}}$	95% CI	$\boldsymbol{p}$	$\boldsymbol{\Delta\mathrm{SPD}_{S}}$	95% CI	$\boldsymbol{p}$	$\boldsymbol{\Delta\mathrm{SPD}_{S}}$	95% CI	$\boldsymbol{p}$
Granite-8B	0.3	-0.677	[-0.770, -0.549]	0.002	-0.038	[-0.079, -0.001]	0.060	-0.119	[-0.158, -0.076]	0.002	-0.278	[-0.330, -0.221]	0.002
Granite-8B	0.6	-0.728	[-0.807, -0.600]	0.002	-0.174	[-0.213, -0.136]	0.002	-0.244	[-0.293, -0.202]	0.002	-0.322	[-0.374, -0.271]	0.002
Mistral-7B	0.3	-0.276	[-0.412, -0.117]	0.005	-0.191	[-0.226, -0.152]	0.002	-0.122	[-0.166, -0.075]	0.002	-0.217	[-0.278, -0.152]	0.002
Mistral-7B	0.6	-0.378	[-0.491, -0.263]	0.002	-0.219	[-0.255, -0.181]	0.002	-0.263	[-0.307, -0.214]	0.002	-0.303	[-0.357, -0.250]	0.002
Mixtral-8x7B	0.3	-0.195	[-0.381, -0.000]	0.017	-0.098	[-0.136, -0.056]	0.002	-0.130	[-0.173, -0.085]	0.002	-0.153	[-0.207, -0.100]	0.002

Appendix

Table 10: Statistical validation of bias propagation in the generated synthetic data.

\Delta\mathrm{SPD}_{S}=\mathrm{SPD}_{S}(\pi)-\mathrm{SPD}_{S}(0)

compares each biased condition against the same model and dataset at

\pi=0

. Confidence intervals are bootstrap intervals over generated rows with

B=1{,}000

resamples, and

p_{\mathrm{perm}}

is a two-sided permutation

p

-value.

Figure 20: Bias propagation persists without explicit mirroring instructions. ( Compas, k=80 ) Downstream utility and fairness on real test data for Logistic Regression (left) and TabPFN (right) classifiers trained on synthetic data. Mistral-7B and Granite-8B both show increasing SPD with bias rate

\pi

despite stable F1, demonstrating that bias propagation occurs even when prompts contain no instruction to replicate in-context statistics.

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 623 papers

Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text
Aug 31, 2026Minkyung Cho, Jihyo Kim, SeungWoo Song +4Social Bias in Language Models LLM Security
KAIST · KAIST InnoCORE PRISM-AI Center · Department of Artificial Intelligence, Dankook University
SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation
Apr 27, 2026Shuo Yang, Zheyu Zhang, Bardh Prenkaj +1Synthetic Tabular Data Generation
Technical University of Munich
Adaptive Generation of Bias-Eliciting Questions for LLMs
Oct 14, 2025Robin Staab, Jasper Dekoninck, Maximilian Baader +1Social Bias in Language Models Language Model Bias Evaluation
1ETH Zurich, Switzerland · 2Snyk, Zurich, Switzerland

Aug 31, 2026J/K move · Enter open · S save

Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text

Minkyung Cho, Jihyo Kim, SeungWoo Song, Junghun Yuk+3

KAIST · KAIST InnoCORE PRISM-AI Center · Department of Artificial Intelligence, Dankook University

Social Bias in Language Models LLM Security Data Poisoning Attacks

Open paper AskPDF arXiv

Synthetic data is increasingly used to train large language models (LLMs), yet its security implications remain poorly understood. Prior work on subliminal learning suggests that models can inherit behavioral traits from seemingly unrelated training data. In this work, we investigate whether such mechanisms can be exploited to inject targeted social biases into aligned models through semantically benign synthetic data. We construct a pipeline in which a misaligned teacher model generates filtered synthetic datasets across domains such as creative writing and code generation, which are then used to fine-tune aligned student models. Our experiments show that benign-looking synthetic data can act as a covert channel for transmitting targeted biases while largely preserving the student model's general task capabilities. These results reveal a previously underexplored security risk in synthetic data-driven LLM training pipelines and highlight the need for improved safeguards. As one possible step toward this goal, we suggest that log-linearity-based scoring may provide a useful signal for screening seemingly benign synthetic data.

Abstract

Synthetic Data Generation

Explore similar work

Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text

SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation

Adaptive Generation of Bias-Eliciting Questions for LLMs