Resource-Adaptive Federated Text Generation with Differential Privacy

Jiayi Wang; John Gounley; Heidi Hanson

cs.LGMar 7, 2026

Resource-Adaptive Federated Text Generation with Differential Privacy

Authors: Jiayi Wang, John Gounley, Heidi Hanson

Organizations: Oak Ridge National Laboratory Oak Ridge, TN, USA

Abstract

In cross-silo federated learning (FL), sensitive text datasets remain confined to local organizations due to privacy regulations, making repeated training for each downstream task both communication-intensive and privacy-demanding. A promising alternative is to generate differentially private (DP) synthetic datasets that approximate the global distribution and can be reused across tasks. However, pretrained large language models (LLMs) often fail under domain shift, and federated finetuning is hindered by computational heterogeneity: only resource-rich clients can update the model, while weaker clients are excluded, amplifying data skew and the adverse effects of DP noise. We propose a flexible participation framework that adapts to client capacities. Strong clients perform DP federated finetuning, while weak clients contribute through a lightweight DP voting mechanism that refines synthetic text. To ensure the synthetic data mirrors the global dataset, we apply control codes (e.g., labels, topics, metadata) that represent each client's data proportions and constrain voting to semantically coherent subsets. This two-phase approach requires only a single round of communication for weak clients and integrates contributions from all participants. Experiments show that our framework improves distribution alignment and downstream robustness under DP and heterogeneity.

Figures & tables

Figure 1: To perform DP synthetic text generation in cross-silo FL, we aim to address two challenges: heterogeneity in computational resources and data distributions. Our approach enables flexible participation through DP federated finetuning of the generator model on well-resourced clients and DP voting on generated synthetic text on the remaining clients.

$\mathcal{C}_{s}\%$	$\varepsilon=\infty$				$\varepsilon=8$ $\downarrow$				$\varepsilon=8$ with refinement $\uparrow$
$\mathcal{C}_{s}\%$	Acc.-1	F1-1	Acc.-2	F1-2	Acc.-1	F1-1	Acc.-2	F1-2	Acc.-1	F1-1	Acc.-2	F1-2
Pre.	0.7044	0.6240	0.4414	0.2704	—	—	—	—	0.7356	0.6818	0.4632	0.3295
1%	0.7367	0.7129	0.6541	0.6289	0.6815	0.6729	0.5113	0.3755	0.7126	0.6981	0.6149	0.5755
5%	0.7485	0.7299	0.6644	0.6455	0.6968	0.6842	0.6145	0.5656	0.7110	0.6978	0.6277	0.5942
10%	0.7487	0.7288	0.6661	0.6460	0.7123	0.6959	0.6280	0.5819	0.7252	0.7068	0.6326	0.6002
20%	0.7469	0.7280	0.6707	0.6519	0.7158	0.6941	0.6328	0.5937	0.7247	0.7060	0.6464	0.6285

Table 1: Experimental results for downstream tasks using Yelp synthetic data with IID setting. The results are partitioned into three parts according the conditions of DP and refinement. For each part, differenet percentages of

\mathcal{C}_{s}

client are considered. Pre. means the synthetic data are directly generated from a pretrained model. Acc.-1 and F1-1 represent the accuracy and F1 score for category classification. Acc.-2 and F1-2 represent the accuracy and F1 score for rating classification.

$\mathcal{C}_{s}\%$	$\varepsilon=\infty$				$\varepsilon=8$ $\downarrow$				$\varepsilon=8$ with refinement $\uparrow$
$\mathcal{C}_{s}\%$	Acc.(D)	F1(D)	Acc.(N)	F1(N)	Acc.(D)	F1(D)	Acc.(N)	F1(N)	Acc.(D)	F1(D)	Acc.(N)	F1(N)
Pre.	0.6456	0.5881	0.5364	0.5179	—	—	—	—	0.6464	0.5472	0.5864	0.5696
5%	0.6672	0.6113	0.6996	0.6996	0.6220	0.5019	0.5908	0.5854	0.8028	0.8024	0.7368	0.7326
10%	0.6904	0.6460	0.7164	0.7159	0.6332	0.5203	0.6076	0.6086	0.8100	0.8074	0.7348	0.7341
20%	0.7968	0.7913	0.7140	0.7144	0.6304	0.5346	0.6220	0.6179	0.8268	0.8280	0.7368	0.7374
30%	0.8612	0.8605	0.7432	0.7438	0.6380	0.5577	0.6284	0.6187	0.8312	0.8318	0.7400	0.7406

Table 2: Experimental results for downstream tasks using PubMed synthetic data with IID setting. Classification results for Chemicals and Drugs (D) and Healthcare (N) are provided. Additional results with

\varepsilon=4

and results using LLaMA as the generator can be found in Section D .

Panel of Figure 2 — Figure 2: MAUVE score for Yelp IID results and NER macro F1 score for PubMed IID results.

Rating	Category	$\varepsilon=\infty$		$\varepsilon=\infty$ with refinement		$\varepsilon=8$		$\varepsilon=8$ with refinement
classes	classes	Acc.-2	F1-2	Acc.-2	F1-2	Acc.-2	F1-2	Acc.-2	F1-2
1 & 3 stars	All	0.5394	0.4007	0.5898	0.4899	0.5266	0.3895	0.5790	0.4778
1 & 5 stars	All	0.5701	0.4971	0.6234	0.6017	0.5658	0.4853	0.6008	0.5876
3 & 5 stars	All	0.5748	0.5394	0.6023	0.6116	0.5632	0.5155	0.5904	0.5828
1, 3, & 5 stars	All	0.6475	0.6174	0.6563	0.6321	0.6084	0.5740	0.6232	0.6078
Rating	Category	$\varepsilon=\infty$		$\varepsilon=\infty$ with refinement		$\varepsilon=8$		$\varepsilon=8$ with refinement

Table 3: Experimental results for downstream tasks using Yelp synthetic data with non-IID setting. Two classes of experiments are presented; one where the rating classes between clients are varied and one where the business category classes between clients are varied.

Bias	$\varepsilon=\infty$		$\varepsilon=\infty$ with refinement		$\varepsilon=8$		$\varepsilon=8$ with refinement
Bias	Acc.(A)	F1(A)	Acc.(A)	F1(A)	Acc.(A)	F1(A)	Acc.(A)	F1(A)
M, N	0.5588	0.5526	0.6324	0.6222	0.4827	0.4719	0.5246	0.5123
C, M, N	0.5812	0.5757	0.6536	0.6536	0.5128	0.5035	0.5835	0.5732
D, C, M, N	0.6420	0.6405	0.6724	0.6725	0.5729	0.5617	0.6139	0.6044
Bias	$\varepsilon=\infty$		$\varepsilon=\infty$ with refinement		$\varepsilon=8$		$\varepsilon=8$ with refinement
Bias	Acc.(D)	F1(D)	Acc.(D)	F1(D)	Acc.(D)	F1(D)	Acc.(D)	F1(D)

Table 4: Experimental results for downstream tasks using PubMed synthetic data in non-IID setting. Classification results for Anatomy (A), Chemicals and Drugs (D) and Diseases (C) are provided in three settings where only the subset of classes list in the Bias column are represented on clients participating in federated finetuning. More classification results can be found in Section D .

Rating	Category	$\varepsilon=\infty$		$\varepsilon=\infty$ with refinement		$\varepsilon=8$		$\varepsilon=8$ with refinement
classes	classes	GPT-2	GPT-2-l	GPT-2	GPT-2-l	GPT-2	GPT-2-l	GPT-2	GPT-2-l
1 & 3 stars	All	0.1404	0.1430	0.1907	0.1969	0.1105	0.1180	0.1493	0.1552
1 & 5 stars	All	0.1617	0.2204	0.2128	0.2675	0.1444	0.1988	0.1915	0.2374
3 & 5 stars	All	0.1634	0.2312	0.2258	0.2785	0.1501	0.1921	0.1987	0.2372
1, 3, & 5 stars	All	0.1924	0.3465	0.2463	0.3658	0.1975	0.2425	0.1982	0.2600
All	2 Categories	0.1808	0.2912	0.2414	0.2808	0.1414	0.2016	0.1911	0.2174

Table 5: MAUVE score for Yelp non-IID setting evaluated with embeddings generated from pretrained GPT-2 and GPT-2-large. The experimental setting is the same as the setting in Table 3 .

Setting	$\varepsilon=\infty$			$\varepsilon=8$
Setting	M,N	C,M,N	D,C,M,N	M,N	C,M,N	D,C,M,N
No refine / Refine	0.5783 / 0.6529	0.8344 / 0.8402	0.8641 / 0.8739	0.8035 / 0.8521	0.8052 / 0.8513	0.8199 / 0.8523

Table 6: NER macro F1 score for PubMed non-IID results.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure A.3: The vote distributions of Yelp synthetic data generated for restaurant and rating stars 3. The left two figures are results only using the pretrained model. The right two figures are results using finetuned model with $10\%$ $\mathcal{C}_{s}$ clients. Before finetuning, most samples receive similar numbers of votes, limiting the effect of refinement. After DP finetuning, however, a long-tail distribution emerges: certain samples receive significantly more votes than others (Figure 3(d) ), reflecting the model’s improved ability to generate text that aligns with the original data distribution. This concentration of votes on high-quality samples is precisely what our refinement step exploits to enhance synthetic data quality.

Panel of Figure 10 — Figure A.4: The vote distributions of PubMed synthetic data generated for A, D. The left two figures are results only using the pretrained model. The right two figures are results using finetuned model with $10\%$ $\mathcal{C}_{s}$ clients. Similar observations as those from Figure A.3 can be obtained.

	Acc.-1	F1-1	Acc.-2	F1-2
$r=0.5$	0.6878	0.6807	0.5627	0.5389
$r=0.25$	0.7046	0.6948	0.5713	0.5561
$r=0.17$	0.7127	0.7020	0.5904	0.5678

Appendix

Table A.7: Downstream results with DP Yelp synthetic texts with refinement for different sampling rates

r

while the number of final synthetic texts is fixed. It can be seen that lower sampling rate, equivalent to higher number of initial synthetic texts, can bring better accuracy and F1 score.

	Acc.-1	F1-1	Acc.-2	F1-2
$K=1$	0.7099	0.6989	0.5721	0.5496
$K=3$	0.6940	0.6891	0.5786	0.5600
$K=5$	0.7046	0.6948	0.5713	0.5561

Appendix

Table A.8: Downstream results with DP Yelp synthetic texts with refinement for different sampling rates

K

while other hyperparameters are fixed. Different from

r

, there is no consistent pattern as

K

increases. One possible reason could be when the number of votes is sufficiently large, increasing votes might not be able to improve the results further.

Temp.	Acc.-1	F1-1	Acc.-2	F1-2
0.8	0.7096	0.6968	0.5609	0.5486
1.0	0.7046	0.6948	0.5713	0.5561
1.2	0.6958	0.6891	0.5611	0.5518

Appendix

Table A.9: Downstream results with DP Yelp synthetic texts with refinement for different temperature while other hyperparameters are fixed.

Part.	$\varepsilon=\infty$
Part.	Acc.(A)	F1(A)	Acc.(C)	F1(C)	Acc.(M)	F1(M)
Pre.	0.5280	0.3738	0.5604	0.4978	0.5708	0.5226
5%	0.6000	0.5965	0.6384	0.6375	0.7204	0.7173
10%	0.5960	0.5742	0.6760	0.6704	0.7796	0.7804
20%	0.6468	0.6468	0.7528	0.7513	0.7900	0.7910
30%	0.7180	0.7166	0.7688	0.7691	0.8056	0.8060

Appendix

Table A.10: Experimental results for downstream tasks using PubMed synthetic data with IID setting. Classification results for Anatomy (A), Disease (C) and Persons (M) are provided.

Bias	$\varepsilon=\infty$		$\varepsilon=\infty$ with refinement		$\varepsilon=8$		$\varepsilon=8$ with refinement
Bias	Acc.(M)	F1(M)	Acc.(M)	F1(M)	Acc.(M)	F1(M)	Acc.(M)	F1(M)
M, N	0.7336	0.7307	0.7408	0.7315	0.6927	0.6831	0.7134	0.7046
C, M, N	0.7316	0.7327	0.8124	0.8125	0.6932	0.6918	0.7521	0.7432
D, C, M, N	0.7744	0.7742	0.8152	0.8155	0.7326	0.7245	0.7731	0.7638
Bias	$\varepsilon=\infty$		$\varepsilon=\infty$ with refinement		$\varepsilon=8$		$\varepsilon=8$ with refinement
Bias	Acc.(N)	F1(N)	Acc.(N)	F1(N)	Acc.(N)	F1(N)	Acc.(N)	F1(N)

Appendix

Table A.11: Experimental results for downstream tasks using PubMed synthetic data in non-IID setting. Classification results for Persons (M), and Healthcare (N) are provided in three settings where only the subset of classes list in the Bias column are represented on clients participating in federated finetuning.

Rating	Category	$\varepsilon=4$		$\varepsilon=4$ with refinement
classes	classes	Acc.-2	F1-2	Acc.-2	F1-2
All	All	$0.6008$	$0.5460$	$0.6147$	$0.6160$
1 & 3 stars	All	$0.5124$	$0.3758$	$0.5869$	$0.5153$
1 & 5 stars	All	$0.4725$	$0.3324$	$0.5558$	$0.5558$
3 & 5 stars	All	$0.4460$	$0.2963$	$0.5014$	$0.4784$
1, 3, & 5 stars	All	$0.6003$	$0.5487$	$0.6047$	$0.6117$

Appendix

Table A.12: Experimental results for downstream tasks using Yelp synthetic data with non-IID setting for

\varepsilon=4

using GPT-2 as the generator.

\mathcal{C}_{s}

clients takes up

10\%

. It can be seen that the refinement can still improve the results under

\varepsilon=4

Rating	Category	$\varepsilon=8$		$\varepsilon=8$ with refinement
classes	classes	Acc.-2	F1-2	Acc.-2	F1-2
All	All	$0.6682$	$0.6489$	$0.6772$	$0.6509$
1 & 3 stars	All	$0.6596$	$0.6220$	$0.6631$	$0.6483$
1 & 5 stars	All	$0.6346$	$0.5692$	$0.6410$	$0.6188$
3 & 5 stars	All	$0.6520$	$0.6248$	$0.6619$	$0.6310$
1, 3, & 5 stars	All	$0.6595$	$0.6399$	$0.6613$	$0.6444$

Appendix

Table A.13: Experimental results for downstream tasks using Yelp synthetic data with non-IID setting using LLaMA as the generator with LoRA.

\mathcal{C}_{s}

clients takes up

10\%

Rating	Category	$\varepsilon=8$		$\varepsilon=8$ with uniform sampling		$\varepsilon=8$ with refinement
classes	classes	Acc.-2	F1-2	Acc.-2	F1-2	Acc.-2	F1-2
All	All	$0.6280$	$0.5819$	$0.6076$	$0.5566$	$0.6326$	$0.6002$
1 & 3 stars	All	$0.5266$	$0.3895$	$0.5158$	$0.3800$	$0.6326$	$0.6002$
1 & 5 stars	All	$0.5658$	$0.4853$	$0.5623$	$0.4662$	$0.6008$	$0.5876$
3 & 5 stars	All	$0.6280$	$0.5819$	$0.6076$	$0.5566$	$0.6326$	$0.6002$
1, 3, & 5 stars	All	$0.6084$	$0.5740$	$0.5996$	$0.5544$	$0.6232$	$0.6078$

Appendix

Table A.14: Experimental results for downstream tasks using Yelp synthetic data with non-IID setting for

\varepsilon=8

with uniform sampling. The number of generated samples is

50000

. Then

10000

samples are uniformly sampled to do downstream tasks. Results without sampling and results with refinement are included for the ease of comparison.

\mathcal{C}_{s}

clients takes up

10\%

. It can be seen that the uniform sampling cannot improve the results without refinement.

Differentially Private Federated Learning

Differentially private federated learning trains shared models across data-holding clients while limiting what updates reveal about individual records or users.

See all 64 papers

Concordia: Self-Improving Synthetic Tables for Federated LLMs
May 11, 2026Jimin Huang, Duanyu Feng, Nuo Chen +8LLM Fine-Tuning Synthetic Data Generation
University of Manchester · The Fin AI · National University of Singapore +5
Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning
Jul 8, 2026Maximilian Andreas Hoefler, Karsten Mueller, Wojciech SamekNon-IID Federated Learning Federated Learning
Fraunhofer Heinrich Hertz Institute · BIFOLD Technical University Berlin
DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
May 11, 2026Haaris Mehmood, Jie Xu, Karthikeyan Saravanan +2Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent
Samsung R&D Institute UK (SRUK) · Samsung AI Centre Cambridge

May 11, 2026J/K move · Enter open · S save

Concordia: Self-Improving Synthetic Tables for Federated LLMs

Jimin Huang, Duanyu Feng, Nuo Chen, Xiaoyu Wang+7

University of Manchester · The Fin AI · National University of Singapore+5

LLM Fine-Tuning Synthetic Data Generation Synthetic Tabular Data Generation Non-IID Federated Learning Federated Fine-Tuning Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) enables training large language models (LLMs) without sharing raw data, but adapting LLMs under strict data isolation and non-IID client distributions remains challenging in practice. Synthetic data offers a natural privacy-preserving surrogate for local training, yet existing federated pipelines typically treat synthetic generation as static or loosely coupled with downstream optimization, leading to rapidly diminishing utility under heterogeneous clients. We study federated adaptation of LLMs on tabular tasks where raw records and validation data cannot be shared, and local training must rely entirely on synthetic tables. We propose Concordia, a tri-level optimization framework that aligns synthetic data generation with federated validation utility despite these constraints. At the client level, models are adapted via parameter-efficient LoRA training on synthetic tables. Clients additionally learn lightweight utility scorers from private validation feedback to reweight synthetic samples during local training. At the outer level, each client refines its own synthetic table generator using group-relative policy optimization (GRPO), guided by an ensemble of heterogeneous scorers shared across clients, without aggregating generator parameters or exposing validation data. Experiments on privacy-sensitive tabular benchmarks from finance and healthcare demonstrate that Concordia consistently improves federated performance, cross-client stability, and robustness to distribution shift compared to static and decoupled synthetic-data baselines.

Abstract

Differentially Private Federated Learning

Explore similar work

Concordia: Self-Improving Synthetic Tables for Federated LLMs

Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning

DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models