Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation

Ashima Sood; Bryan Gardiner; Joan Condell

cs.CLAug 16, 2026

Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation

Authors: Ashima Sood, Bryan Gardiner, Joan Condell

Organizations: School of Computing, Engineering and Intelligent Systems, Ulster University Londonderry, Northern Ireland, United Kingdom

Abstract

Jointly fine-tuning an LLM on meeting-summarisation corpora of widely varying size raises a question that prior work leaves confounded: when a domain-balanced training mixture helps, is the gain due to the distribution of tokens across domains, or merely to the volume of data seen? We disentangle these factors by constructing balanced and natural (native-proportional) token mixtures at matched token budgets (2-32M) over five English meeting corpora, fine-tuning Mistral-7B with QLoRA, and evaluating per domain. Balancing redistributes quality, improving the data-scarce minority domains at a low cost to the data-rich ones. The trade favours balancing whenever the minority domains matter: their share under proportional allocation is fixed at 1-2% regardless of budget, so matching balanced quality on those domains requires far more total data. We further find that pruning low-value transcript lines removes ~15% of tokens from the conversational corpora at no measurable cost, and that balancing by tokens is not the same as balancing by examples. Fine-tuning one model per domain is competitive only on the data-rich domains and falls below the zero-shot model on the data-scarce ones. A two-annotator study of 741 judge-labelled facts validates our fact-level evaluation. Together these results give practitioners a basis for deciding when to balance an imbalanced multi-domain mixture, and on what unit.

Figures & tables

Figure 1: Pipeline. (A) Transcripts are pruned of conversational filler with Gemma-3-27B-it. (B) Training mixtures are built under three allocation schemes (balanced, natural, example) at a fixed token budget. (C) Zero-shot screening over six candidate LLMs selects Mistral-7B-Instruct-v0.3; Llama-3.2-3B is retained as a scaling control. (D) Mistral-7B is fine-tuned with QLoRA. (E) Generated minutes are scored with automatic metrics and an LLM judge.

Scheme		2M	4M	8M	16M	32M
Balanced	tokens	397K	794K	1.6M	3.2M	6.4M
Balanced	os	$1.0\times$	$1.4\times$	$2.8\times$	$5.6\times$	$11.1\times$
Natural	tokens	26K	51K	105K	215K	436K
Natural	os	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$	$1.0\times$

Table 1: Token allocation and oversampling (os) for the smallest corpus (ICSI). Balanced holds ICSI at a

{\sim}20

% budget share throughout; natural gives it

{\sim}1.4

%. Per-corpus breakdowns in Appendix C .

ROUGE-Lsum
Scheme	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
	20	6	38	242	862
Balanced	0.495	0.445	0.367	0.547	0.648	0.500	0.615
Natural	0.393	0.281	0.204	0.534	0.694	0.421	0.637
BERTScore-F1
Balanced	0.875	0.852	0.846	0.887	0.928	0.877	0.915

Table 2: RQ1: balanced vs. natural allocation per domain at the matched 32M budget (pruned, seed 42). Bold marks the higher-scoring scheme, italics the test-set meeting counts. Macro weights domains equally, micro weights meetings. Full grid in Table 22 ; three-seed means in Table 27 .

Macro
System	Completeness	Faithfulness	Conciseness
balanced 2M	$\mathbf{0.276_{[.25,.30]}}$	$\mathbf{0.765_{[.74,.79]}}$	$\mathbf{0.515_{[.48,.54]}}$
natural 2M	$0.259_{[.24,.28]}$	$0.739_{[.70,.78]}$	$0.467_{[.43,.51]}$
balanced 32M	$\mathbf{0.373_{[.35,.40]}}$	$\mathbf{0.742_{[.73,.76]}}$	$0.472_{[.45,.49]}$
natural 32M	$0.332_{[.31,.36]}$	$0.728_{[.70,.75]}$	$\mathbf{0.506_{[.46,.56]}}$
Micro

Table 3: Fact-level LLM-judge scores at the smallest and largest budgets, with bootstrap 95% CIs (10,000 resamples). Balanced leads on macro completeness and trails on micro, the same sign flip the automatic metrics show. Per-domain breakdown in Table 25 .

Figure 2: RQ2: Per-dataset quality across the token budget ladder (2

\rightarrow

32M) for balanced (blue) and natural (orange) allocation, on ROUGE-Lsum (top) and BERTScore-F1 (bottom). Shading marks balanced’s advantage on the minority panels. Quality rises with budget and flattens; the minority gap is open at nearly every budget and does not close by 32M.

Figure 3: RQ3: Pruned vs. unpruned training quality per dataset across the budget ladder (balanced allocation), on ROUGE-Lsum (top) and BERTScore-F1 (bottom). The two conditions are near-identical on BERTScore-F1 and converge on ROUGE-Lsum as the budget grows; the visible gaps are confined to the smallest budgets.

ROUGE-Lsum
Scheme	AMI	ICSI	ELITR	EPM	MB	Macro
	20	6	38	242	862
Balanced	.376	.258	.152	.222	.562	.314
Natural	.331	.129	.184	.274	.586	.301
Example	.328 ±.044	.292 ±.036	.229 ±.014	.271 ±.044	.505 ±.008	.325 ±.017
BERTScore-F1

Table 4: RQ4: Example-level balancing at

\sim

2M. Example-level (43 meetings per corpus) is mean

\pm

std over seeds {1,2,15}; Balanced and Natural are seed 42. Equal meeting counts give ICSI

\sim

578K tokens but MB only

\sim

137K. Full results in Appendix Table 21 .

	ROUGE-Lsum		BERTScore-F1
Dataset	M-7B	L-3B	M-7B	L-3B
AMI	$+0.122$	$+0.155$	$+0.018$	$+0.034$
ICSI	$+0.202$	$+0.141$	$+0.027$	$+0.040$
ELITR	$+0.089$	$+0.072$	$+0.010$	$+0.032$
EPM	$-0.039$	$-0.178$	$-0.006$	$-0.031$
MB	$-0.039$	$-0.061$	$-0.007$	$-0.014$

Table 5: RQ5: per-dataset mean balanced

-

natural gap for M-7B (Mistral-7B) and L-3B (Llama-3.2-3B), averaged over the three shared budgets (2M/8M/32M; pruned, seed 42). Positive favours balancing. Per-budget numbers in Table 26 .

ROUGE-Lsum
System	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
	20	6	38	242	862
Per-domain	0.382	0.210	0.126	0.564	0.703	0.397	0.647
Balanced 32M	0.495	0.445	0.367	0.547	0.648	0.500	0.615
BERTScore-F1
Per-domain	0.868	0.817	0.772	0.897	0.939	0.858	0.923

Table 6: RQ6: per-domain fine-tuned baselines vs. balanced allocation at 32M (pruned, seed 42). Each per-domain adapter is trained on the full training set of one corpus under the protocol of Table 18 and evaluated on its own domain. Bold marks the higher system per column; italics are test-set meeting counts. Macro weights domains equally, micro weights meetings. Bootstrap CIs in Table 14 .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Transcripts	Tokens (pre)	Reduction	Guard-saved
AMI	118	863,292	15.6%	3,030
ICSI	52	771,303	14.7%	2,586
ELITR	94	1,164,265	14.4%	3,770
MB	6,030	28,616,375	9.8%	69,493
EPM	2,252	23,878,795	2.3%	146,312
Total	8,546	55,294,030	6.8%	225,191

Appendix

Table 7: Transcript pruning on the training data (train

{+}

dev; test is unpruned), Gemma-3-27B-it tokenizer. Reduction is the fraction of tokens removed; Guard-saved is the number of lines force-retained by the content guard.

Figure 4: Annotator ratification of the pruner’s KEEP/DELETE decisions, pooled over 500 sampled lines (250 KEEP, 250 DELETE; 100 per corpus). Bars show how many of the model’s KEEP and DELETE calls each annotator judged correct vs. wrong.

Corpus		2M	4M	8M	16M	32M
AMI	tokens	399K	800K	1.6M	3.2M	6.4M
AMI	oversample	$1.0$	$1.19$	$2.45$	$4.86$	$9.73$
ICSI	tokens	397K	794K	1.6M	3.2M	6.4M
ICSI	oversample	$1.0$	$1.37$	$2.79$	$5.56$	$11.09$
ELITR	tokens	399K	800K	1.6M	3.2M	6.4M
ELITR	oversample	$1.0$	$1.0$	$1.70$	$3.38$	$6.82$

Appendix

Table 8: Balanced scheme : per-corpus token allocation and oversampling ratio across budgets. Each corpus receives an equal

{\sim}B/5

share, so the minority corpora (AMI, ICSI, ELITR) are increasingly oversampled to fill it while the majority corpora (EPM, MB) always draw distinct meetings (

1.0\times

Corpus		2M	4M	8M	16M	32M
AMI	tokens	31K	61K	125K	249K	500K
AMI	meetings	5	10	20	37	76
ICSI	tokens	26K	51K	105K	215K	436K
ICSI	meetings	3	4	8	15	33
ELITR	tokens	43K	87K	176K	353K	710K
ELITR	meetings	5	9	16	34	64

Appendix

Table 9: Natural scheme : per-corpus token allocation and unique meeting count across budgets. Shares follow native corpus size (AMI 1.6%, ICSI 1.4%, ELITR 2.2%, EPM 46.6%, MB 48.3%), so the majority corpora dominate every budget. The scheme never oversamples: even at 32M, ICSI draws only 33 of its 43 meetings.

Corpus	Meetings	s1	s2	s15
AMI	43	276K	283K	292K
ICSI	43	578K	578K	578K
ELITR	43	464K	473K	457K
EPM	43	417K	402K	417K
MB	43	137K	155K	167K
Total	215	1.87M	1.89M	1.91M

Appendix

Table 10: Example-level scheme : per-corpus token counts across three seeds (pruned). Each corpus contributes 43 meetings, but token counts differ because transcripts vary in length: ICSI (

\sim

\times

MB). ICSI is identical across seeds as it has exactly 43 meetings; other corpora vary with the random draw.

Split	Corpus	$n$	#Trunc.	%	Median (tok)	Kept (%)
Train	AMI	98	0	0.0	7,558	—
	ICSI	43	22	51.2	15,667	81.9
	ELITR	84	28	33.3	12,105	71.3
	EPM	2065	529	25.6	9,077	64.7
	MB	5169	397	7.7	1,890	62.3
	All	7459	976	13.1	3,261	—

Appendix

Table 11: Transcript lengths and truncation per corpus and split (Mistral-7B tokeniser, unpruned transcripts). #Trunc. counts transcripts exceeding the available budget; Kept is the mean percentage of a transcript retained, computed over truncated transcripts only. Train and development use the training budget (window minus template, summary, and end-of-turn token); test uses the inference budget (window minus template). Truncation is far heavier in training (

12.3\%

of train

{+}

dev) than at test (

3.3\%

), and rate and severity diverge: ICSI is truncated most often but retains the most, EuroParlMin least often of the three affected corpora but retains the least.

Figure 5: Multi-seed training trajectories (pruned, seeds 2 and 15): per-epoch training loss (faint) and development loss (bold, with markers) for the balanced (top) and natural (bottom) schemes at the 2M, 8M, and 32M budgets. The star marks the selected checkpoint, at the lowest development loss. As in the seed-42 grid (Figure 7 ), development loss bottoms out and then rises while training loss keeps falling, and each run stops two epochs after its minimum under patience-2 early stopping.

Corpus	Pairwise overlap	Coverage across seeds
EPM	0% (fully distinct)	—
MB	0% (fully distinct)	—
AMI	(arithmetic)	83/98
ELITR	(arithmetic)	74/84
ICSI	100% (forced)	43/43

Appendix

Table 12: Example-level seed selection {1,2,15}. Large corpora are fully distinct across seeds (genuine sampling variance); small-corpus overlap is arithmetically unavoidable (pool

\leq 3\times 43

) but offset by near-complete coverage of the minority data. For ICSI (

|{\rm pool}|=43

) all meetings are used, so only training order varies across seeds.

Model	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
	20	6	38	242	862
ROUGE-1
Llama-3.2-3B	0.462	0.360	0.369	0.318	0.211	0.344	0.243
Llama-3.1-8B	0.468	0.370	0.370	0.329	0.217	0.351	0.250
Qwen3-8B	0.402	0.398	0.344	0.294	0.179	0.324	0.213
Mistral-7B	0.454	0.343	0.401	0.343	0.245	0.357	0.274

Appendix

Table 13: Zero-shot summarisation quality across five meeting corpora. Macro is the unweighted mean over corpora; Micro is weighted by test-set size (counts in italics). ELITR scores take the maximum over its multiple references. Best per column in bold .

Figure 6: Per-domain baselines: per-epoch training loss (faint) and development loss (bold, with markers) for each adapter, measured on the development set of the same corpus (pruned, seed 42). The star marks the selected checkpoint. The data-rich adapters (EPM, MB) reach a development-loss minimum and are terminated by patience-2 early stopping, as in Figure 7 ; the data-scarce adapters (AMI, ICSI, ELITR) are still improving at the five-epoch ceiling.

Metric	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
	20	6	38	242	862
ROUGE-1	0.394 [0.30,0.48]	0.224 [0.14,0.30]	0.130 [0.09,0.17]	0.570 [0.54,0.60]	0.720 [0.70,0.74]	0.408 [0.38,0.43]	0.662 [0.65,0.68]
ROUGE-2	0.168 [0.12,0.22]	0.049 [0.03,0.07]	0.041 [0.03,0.06]	0.432 [0.41,0.46]	0.632 [0.61,0.65]	0.264 [0.25,0.28]	0.561 [0.54,0.58]
ROUGE-L	0.234 [0.18,0.29]	0.138 [0.10,0.18]	0.100 [0.07,0.13]	0.511 [0.48,0.54]	0.696 [0.68,0.71]	0.336 [0.32,0.35]	0.628 [0.61,0.64]
ROUGE-Lsum	0.382 [0.29,0.47]	0.210 [0.13,0.29]	0.126 [0.09,0.17]	0.564 [0.53,0.59]	0.703 [0.69,0.72]	0.397 [0.37,0.42]	0.647 [0.63,0.66]
BERTScore	0.868 [0.86,0.88]	0.817 [0.79,0.84]	0.772 [0.70,0.82]	0.897 [0.89,0.90]	0.939 [0.94,0.94]	0.858 [0.84,0.87]	0.923 [0.92,0.93]

Appendix

Table 14: Per-domain fine-tuned baselines (pruned, seed 42) across the complete metric family, with bootstrap 95% CIs in subscripts (10,000 resamples over test meetings; the macro CI resamples within each domain and averages the domain means). Each adapter is trained on its full training set: 0.66M (AMI), 0.58M (ICSI), 0.94M (ELITR), 19.7M (EPM) and 20.4M (MB) tokens per epoch. Column italics are test-set meeting counts. The balanced and natural blocks for comparison are in Table 22 .

Corpus	N	Mean	P95	P99	Max
AMI	20	376	517	582	599
ICSI	6	628	918	938	944
MB	862	101	230	313	959
ELITR (test)	55	539	1122	1240	1272
ELITR (test2)	10	907	1618	1808	1856
ELITR (test2023)	12	769	1330	1381	1394

Appendix

Table 15: Test-set reference-summary lengths in tokens (Mistral-7B tokenizer).

N

is the number of test references; ELITR is split by its three test sets, with all annotator references counted. The 2048-token generation budget covers 98.8% of references in full; the longer ones are undivided full-chapter EuroParlMin documents rather than per-session minutes.

Model	Dataset	$n$	Mean $\pm$ SD	95% CI
Llama-3.2-3B	AMI	20	457.7 $\pm$ 65.1	[427.2, 488.2]
	ICSI	6	420.5 $\pm$ 43.8	[374.5, 466.5]
	ELITR	38	567.1 $\pm$ 261.0	[481.3, 652.9]
	EPM	242	314.3 $\pm$ 157.4	[294.4, 334.3]
	MB	862	413.9 $\pm$ 284.4	[394.9, 433.0]
Llama-3.1-8B	AMI	20	443.2 $\pm$ 74.5	[408.4, 478.1]

Appendix

Table 16: Generated meeting-minute length in tokens (mean

\pm

standard deviation, with 95% confidence interval for the mean) per model and corpus. ELITR is collapsed to one prediction per meeting (

n{=}38

Corpus	Domain	Train	Dev	Test
AMI	product meetings	98	20	20
ICSI	academic meetings	43	9	6
MB	council meetings	5169	861	862
ELITR	project meetings	84	10	38 ∗
EPM	parliamentary sessions	2065	187	242
Total		7459	1087	1168

Appendix

Table 17: Final train/development/test partitions. Splits are canonical where available, seeded-and-released for ICSI (seed 42). ELITR figures are for the English portion; its Test column sums three test sets (

18{+}8{+}12

). ∗ ELITR test counts unique meetings (38), each carrying multiple reference minutes over which evaluation takes the maximum.

Hyperparameter	Value
Quantisation	4-bit NF4, double-quant, bf16 compute
LoRA rank $r$	32
LoRA $\alpha$	16
LoRA dropout	0.05
Adapted modules	q, k, v, o, gate, up, down
Optimiser	paged AdamW (8-bit)

Appendix

Table 18: QLoRA fine-tuning hyperparameters, applied identically across all schemes, budgets, and conditions. † Seeds 2 and 15 repeat both allocation schemes at 2M/8M/32M (Appendix E.1 ); seeds 1, 2, and 15 are used for the example-level baseline (Appendix E.2 ).

Figure 7: Per-epoch training loss (faint) and development loss (bold, with markers) for the balanced (top) and natural (bottom) schemes, pruned (left) and unpruned (right), on seed 42. The star marks the selected checkpoint, at the lowest development loss. Development loss bottoms out and then rises while training loss keeps falling; each run stops two epochs after its minimum under patience-2 early stopping.

Scheme	Cond.	2M	4M	8M	16M	32M
Balanced	pruned	0.906 (3)	0.826 (2)	0.801 (2)	0.760 (2)	0.728 (3)
Balanced	unpruned	0.959 (3)	0.845 (2)	0.792 (2)	0.777 (2)	0.723 (3)
Natural	pruned	0.833 (3)	0.749 (2)	0.691 (2)	0.636 (2)	0.583 (2)
Natural	unpruned	0.835 (3)	0.760 (2)	0.696 (2)	0.640 (2)	0.587 (2)

Appendix

Table 19: Checkpoint selection for all runs. Each cell gives the best development loss and, in parentheses, the epoch at which it occurred (the selected checkpoint). Development loss is used only for checkpoint selection, never as a quality measure (Section 3.6 ).

Figure 8: Inter-rater agreement on 741 judge-labelled facts from balanced-32M (Cohen’s

\kappa

, bootstrap 95% CIs, 4,000 resamples). Human–human agreement is the ceiling (dashed rule); shading marks the judge’s shortfall against it. The red dotted line is

\kappa=0.61

, the threshold for substantial agreement. The judge reaches the human band on completeness but falls below it on conciseness and faithfulness.

Cohen’s $\kappa$ , pairwise (bootstrap 95% CI)
	Compl.	Concis.	Faith.
$n$	257	267	213
H1–H2	0.791 [.71,.86]	0.835 [.77,.90]	0.799 [.71,.87]
Judge–H1	0.762 [.68,.84]	0.627 [.54,.71]	0.622 [.52,.72]
Judge–H2	0.706 [.61,.79]	0.634 [.54,.72]	0.571 [.47,.67]
Raw agreement, pairwise (%)

Appendix

Table 20: Agreement between the two annotators (H1, H2) and the LLM judge, per dimension. Top: chance-corrected agreement (Cohen’s

\kappa

) for each rater pair; H1–H2 is the human ceiling. Middle: the same pairs as raw percentage agreement. Bottom: among items where the two annotators agree (a consensus ground truth), the number of judge labels that are too generous (over-accept: judge positive, humans negative) versus too strict (over-reject), showing the judge errs by over-accepting in 87-94% of cases.

Condition	Metric	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
		20	6	38	242	862
Pruned	R-1	$0.339_{\pm.044}$	$0.310_{\pm.036}$	$0.237_{\pm.016}$	$0.277_{\pm.045}$	$0.532_{\pm.008}$	$0.339_{\pm.018}$	$0.465_{\pm.009}$
	R-2	$0.137_{\pm.016}$	$0.075_{\pm.010}$	$0.060_{\pm.007}$	$0.165_{\pm.036}$	$0.421_{\pm.007}$	$0.172_{\pm.011}$	$0.350_{\pm.007}$
	R-L	$0.206_{\pm.028}$	$0.170_{\pm.017}$	$0.137_{\pm.003}$	$0.223_{\pm.040}$	$0.498_{\pm.008}$	$0.247_{\pm.012}$	$0.423_{\pm.008}$
	R-Lsum	$0.328_{\pm.044}$	$0.292_{\pm.036}$	$0.229_{\pm.014}$	$0.271_{\pm.044}$	$0.505_{\pm.008}$	$0.325_{\pm.017}$	$0.443_{\pm.009}$
	BERT	$0.861_{\pm.004}$	$0.834_{\pm.003}$	$0.826_{\pm.006}$	$0.833_{\pm.013}$	$0.898_{\pm.000}$	$0.850_{\pm.004}$	$0.881_{\pm.003}$

Appendix

Table 21: Example-level results across all five metrics: mean

{}_{\pm\text{std}}

over seeds 1/2/15, per condition. Column italics are test-set meeting counts. Seed variation is small for aggregate measures (Macro/Micro std

\leq

0.018); minority domains with few test meetings (ICSI, AMI) show larger per-seed spread. R-1/2/L = ROUGE-1/2/L.

Balanced (equal-token) allocation
Metric	Budget	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
		20	6	38	242	862
ROUGE-1	2M	$0.389_{[0.30,0.47]}$	$0.272_{[0.24,0.30]}$	$0.159_{[0.12,0.20]}$	$0.227_{[0.20,0.25]}$	$0.585_{[0.57,0.60]}$	$0.326_{[0.31,0.35]}$	$0.492_{[0.48,0.51]}$
	4M	$0.359_{[0.27,0.45]}$	$0.224_{[0.12,0.33]}$	$0.166_{[0.13,0.21]}$	$0.326_{[0.29,0.36]}$	$0.587_{[0.57,0.61]}$	$0.333_{[0.30,0.36]}$	$0.514_{[0.50,0.53]}$
	8M	$0.465_{[0.40,0.52]}$	$0.443_{[0.40,0.49]}$	$0.251_{[0.20,0.30]}$	$0.443_{[0.41,0.47]}$	$0.626_{[0.61,0.64]}$	$0.446_{[0.43,0.46]}$	$0.572_{[0.56,0.59]}$
	16M	$0.522_{[0.50,0.55]}$	$0.440_{[0.41,0.47]}$	$0.383_{[0.34,0.42]}$	$0.392_{[0.36,0.42]}$	$0.649_{[0.63,0.67]}$	$0.477_{[0.46,0.49]}$	$0.584_{[0.57,0.60]}$

Appendix

Table 22: Full per-domain results ( pruned , seed 42) across the complete metric family (ROUGE-1/2/L/Lsum and BERTScore-F1), with bootstrap 95% CIs in subscripts. Rows are grouped by allocation scheme (Balanced vs. Natural) and, within each scheme, by metric; each metric block reports the five-budget ladder (2-32M). Column italics are test-set meeting counts. The per-domain crossover of RQ1 is visible in every metric: Balanced leads on the minority domains (AMI/ICSI/ELITR) while Natural leads on the majority domains (EPM/MB), with the gap widest at small budgets.

Balanced (equal-token) allocation
Metric	Budget	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
		20	6	38	242	862
ROUGE-1	2M	$0.281_{[0.20,0.37]}$	$0.113_{[0.08,0.14]}$	$0.143_{[0.11,0.18]}$	$0.332_{[0.31,0.35]}$	$0.559_{[0.54,0.58]}$	$0.285_{[0.27,0.31]}$	$0.491_{[0.48,0.51]}$
	4M	$0.318_{[0.23,0.41]}$	$0.260_{[0.17,0.34]}$	$0.174_{[0.13,0.22]}$	$0.189_{[0.17,0.21]}$	$0.615_{[0.60,0.63]}$	$0.311_{[0.28,0.34]}$	$0.505_{[0.49,0.52]}$
	8M	$0.393_{[0.31,0.47]}$	$0.367_{[0.27,0.44]}$	$0.241_{[0.20,0.28]}$	$0.406_{[0.37,0.44]}$	$0.637_{[0.62,0.65]}$	$0.409_{[0.38,0.43]}$	$0.571_{[0.56,0.58]}$
	16M	$0.508_{[0.47,0.54]}$	$0.449_{[0.40,0.50]}$	$0.306_{[0.26,0.35]}$	$0.354_{[0.32,0.38]}$	$0.664_{[0.65,0.68]}$	$0.456_{[0.44,0.47]}$	$0.585_{[0.57,0.60]}$

Appendix

Table 23: Full per-domain results ( unpruned , seed 42) across the complete metric family (ROUGE-1/2/L/Lsum and BERTScore-F1), with bootstrap 95% CIs in subscripts. Rows are grouped by allocation scheme (Balanced vs. Natural) and, within each scheme, by metric; each metric block reports the five-budget ladder (2–32M). Column italics are test-set meeting counts. This is the unpruned companion to the pruned grid; comparing the two at matched (scheme, budget, metric) cells isolates the effect of transcript pruning (RQ3).

Balanced (equal-token) allocation
Metric	Budget	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
		20	6	38	242	862
ROUGE-1	2M	0.105	0.090	0.096	0.219	0.530	0.208	0.442
	8M	0.324	0.185	0.128	0.256	0.587	0.296	0.497
	32M	0.497	0.424	0.238	0.224	0.613	0.399	0.518
ROUGE-2	2M	0.033	0.015	0.018	0.123	0.416	0.121	0.334

Appendix

Table 24: Llama-3.2-3B (RQ5): full per-domain results across the complete metric family, for both allocation schemes at the three shared budgets (2M/8M/32M; pruned, seed 42). Column italics are test-set meeting counts. The per-domain crossover of RQ1 reproduces here: balanced leads the minority domains (AMI/ICSI/ELITR) while natural leads the majority domains (EPM/MB).

System	Dim.	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
balanced 2M	Comple.	$0.370_{[0.29,0.46]}$	$0.147_{[0.07,0.22]}$	$0.077_{[0.05,0.11]}$	$0.310_{[0.28,0.34]}$	$0.478_{[0.45,0.50]}$	$0.276_{[0.25,0.30]}$	$0.427_{[0.41,0.45]}$
	Faithf.	$0.884_{[0.83,0.93]}$	$0.906_{[0.87,0.94]}$	$0.784_{[0.68,0.88]}$	$0.502_{[0.46,0.55]}$	$0.749_{[0.73,0.77]}$	$0.765_{[0.74,0.79]}$	$0.707_{[0.69,0.73]}$
	Concis.	$0.541_{[0.48,0.61]}$	$0.549_{[0.44,0.65]}$	$0.443_{[0.36,0.52]}$	$0.427_{[0.39,0.46]}$	$0.615_{[0.59,0.64]}$	$0.515_{[0.48,0.54]}$	$0.573_{[0.55,0.59]}$
balanced 32M	Comple.	$0.496_{[0.42,0.57]}$	$0.209_{[0.15,0.28]}$	$0.201_{[0.15,0.25]}$	$0.411_{[0.38,0.44]}$	$0.550_{[0.53,0.57]}$	$0.373_{[0.35,0.40]}$	$0.507_{[0.49,0.53]}$
	Faithf.	$0.892_{[0.86,0.92]}$	$0.916_{[0.87,0.96]}$	$0.791_{[0.74,0.83]}$	$0.418_{[0.38,0.45]}$	$0.691_{[0.67,0.71]}$	$0.742_{[0.73,0.76]}$	$0.642_{[0.62,0.66]}$
	Concis.	$0.481_{[0.42,0.54]}$	$0.457_{[0.39,0.51]}$	$0.313_{[0.26,0.36]}$	$0.469_{[0.44,0.50]}$	$0.639_{[0.62,0.66]}$	$0.472_{[0.45,0.49]}$	$0.589_{[0.57,0.61]}$

Appendix

Table 25: Per-domain fact-level LLM-judge scores (Qwen2.5-72B-Instruct-AWQ) for the balanced and natural schemes at the 2M and 32M budgets, with bootstrap 95% CIs in subscripts. Comple. (completeness) is the fraction of reference-minute facts recovered by the generated minute; Faithf. (faithfulness) is the fraction of generated-minute facts supported by the transcript; Concis. (conciseness) is the fraction of generated-minute facts that are salient with respect to the reference. Macro weights the five domains equally, micro weights meetings. This is the per-domain breakdown of the aggregate scores in Table 3 ; the RQ1 crossover on completeness (balanced leads the minority domains, natural the majority) is visible per domain, while faithfulness falls from 2M to 32M under both schemes, most sharply on EPM.

		Mistral-7B			Llama-3.2-3B
Dataset	Budget	Bal.	Nat.	$\Delta$	Bal.	Nat.	$\Delta$
ROUGE-Lsum
AMI	2M	0.376	0.331	$+0.045$	0.098	0.152	$-0.054$
	8M	0.449	0.230	$+0.219$	0.312	0.113	$+0.199$
	32M	0.495	0.393	$+0.102$	0.476	0.156	$+0.320$
ICSI	2M	0.258	0.129	$+0.129$	0.080	0.029	$+0.051$

Appendix

Table 26: RQ5 full numeric detail backing Table 5 . For each dataset, budget, and model, Bal. and Nat. are the raw balanced and natural scores and

\Delta=

balanced

-

natural (from the displayed rounded values); all runs are pruned, single seed 42. The sign of

\Delta

agrees across the two models for every dataset on both metrics—positive on AMI/ICSI/ELITR, negative on EPM/MB—the only per-cell exceptions being at 2M (AMI and ELITR on ROUGE-Lsum, EPM on BERTScore-F1).

Metric	Scheme	Budget	AMI	ICSI	ELITR	EPM	MB	Macro	Micro
			20	6	38	242	862
ROUGE-1	Balanced	2M	$0.380_{\pm 0.057}$	$0.271_{\pm 0.022}$	$0.161_{\pm 0.005}$	$0.253_{\pm 0.048}$	$0.576_{\pm 0.029}$	$0.328_{\pm 0.003}$	$0.491_{\pm 0.028}$
	Balanced	8M	$0.455_{\pm 0.022}$	$0.391_{\pm 0.046}$	$0.235_{\pm 0.024}$	$0.365_{\pm 0.087}$	$0.630_{\pm 0.005}$	$0.415_{\pm 0.031}$	$0.558_{\pm 0.016}$
	Balanced	32M	$0.527_{\pm 0.011}$	$0.452_{\pm 0.016}$	$0.370_{\pm 0.008}$	$0.524_{\pm 0.027}$	$0.675_{\pm 0.007}$	$0.510_{\pm 0.006}$	$0.630_{\pm 0.006}$
	Natural	2M	$0.316_{\pm 0.031}$	$0.176_{\pm 0.079}$	$0.176_{\pm 0.012}$	$0.355_{\pm 0.065}$	$0.611_{\pm 0.017}$	$0.327_{\pm 0.026}$	$0.536_{\pm 0.018}$
	Natural	8M	$0.241_{\pm 0.004}$	$0.131_{\pm 0.042}$	$0.105_{\pm 0.010}$	$0.481_{\pm 0.046}$	$0.669_{\pm 0.007}$	$0.325_{\pm 0.007}$	$0.602_{\pm 0.013}$

Appendix

Table 27: Multi-seed robustness (RQ1): per-domain scores as mean

\pm

std over seeds

\{2,15,42\}

for the balanced and natural schemes, pruned condition, at the 2M, 8M, and 32M budgets. Column italics are test-set meeting counts. The per-domain crossover of Section 5.1 reproduces across seeds: at 8M and 32M, balanced leads all three minority domains (AMI, ICSI, ELITR) and natural leads the majority domains (EPM, MB) on every metric, so macro favours balanced while micro favours natural. At 2M the minority advantage is smaller and ELITR falls within seed variation.

Explore similar work

DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
Jul 24, 2026He ZhangPretraining Pilot
Tsinghua University
TANDEM: Bi-Level Data Mixture Optimization with Twin Networks
Jun 3, 2026Jiaxing Wang, Deping Xiang, Jin Xu +9Mixture Models Reweighting
University of Oxford · Renmin University of China · University of Chinese Academy of Sciences
Scaling Laws for Mixture Pretraining Under Data Constraints
May 12, 2026Anastasiia Sedova, Skyler Seto, Natalie Schluter +1Pretraining Scaling Laws
Apple · ITU

Jul 24, 2026, cs.LGJ/K move · Enter open · S save

DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning

He Zhang

Tsinghua University

Pretraining Pilot

Open paper AskPDF arXiv

The training efficacy of large language models (LLMs) is fundamentally constrained by the quality and composition of training data. Existing dynamic data scheduling methods face critical limitations in industrial-scale pretraining and supervised fine-tuning (SFT): data selection incurs prohibitive O(N) costs on terabyte-scale corpora, mixture optimization schemes introduce severe I/O bottlenecks or require training auxiliary reference models, and sample-level reweighting strategies rely on loss signals that conflate noise, difficulty, and novelty. We present DomainPilot, a domain-level loss-guided two-stage data mixture optimization framework. DomainPilot introduces token-level domain loss monitoring to capture per-domain learning dynamics during training without halting the data pipeline. Building on these signals, we propose a Scaling Law guided coarse optimization stage that fits domain-specific convergence curves and derives a principled prior for mixture adjustment. A subsequent Mixing Law guided fine optimization stage refines the mixture by modeling cross-domain interaction effects through controlled sweep experiments. The entire mechanism is realized via a patch-based architecture that injects domain-aware loss computation into existing training frameworks (e.g., MindSpeed/Megatron-LM) with only ~30 lines of framework-specific adapter code. We validate DomainPilot on the Qwen3-1.7B model during SFT. Compared to the original data mixture, our optimized mixture achieves improvements of +2% on MMLU-Redux, +1.8% on AIME24, +3.8% on LiveCodeBench v5, and +3.6% on BFCL v3, without increasing total data volume or training cost. These results demonstrate that domain-level training signals provide an effective, lightweight alternative to expensive data selection or auxiliary model training for mixture optimization.

Abstract

Explore similar work

DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning

TANDEM: Bi-Level Data Mixture Optimization with Twin Networks

Scaling Laws for Mixture Pretraining Under Data Constraints