Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery

Hyojung Han

cs.CLSep 14, 2026

Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery

Authors: Hyojung Han

Organizations: ThakiCloud

Abstract

Korean dialect corpora are available but not redistributable: weights may be released, while reproducing training and evaluation from the underlying data cannot be. We ask how much of that supervision synthetic data recovers, and whether that recovery can be measured independently of the synthesis pipeline. We contribute KoDialectBench, 1,000 items across five regions on three axes, released as identifier hashes and scoring code so users reconstruct the items from their own licensed copy. Recovery is strongly axis-dependent: our best synthetic arm reaches 91.2% of the real-data gain on region identification but 63.7% on comprehension. On generation the answer depends on the metric: the deployed marker lexicon reports 92.3% on dialectness and 119.3% on region match, the latter exceeding the real-data reference, whereas reference-based generation reaches 72.4%. We find the marker metrics' scoring inventory is entirely contained in the inventory our transformation rules can emit. We test the effect of construction access directly with an exact-form construction-disjoint arm that withholds 20% of marker types from the rules. At exactly matched training size (8,600 examples) it reduces dialectness recovery from 91.8% to 8.1% and region-match recovery from 101.9% to 25.6% on the held-out marker inventory, while the three pipeline-independent measurements do not fall at all. A complementary evaluator sweep defines metric-construction coverage (MCC) and finds measured dialectness recovery increasing monotonically as overlap rises from MCC=0 to MCC=1. Shared construction and evaluation inventories can therefore substantially inflate estimates of synthetic-data recovery.

Figures & tables

Metric	base	SK5	SK6	SK4	SK11-D9	SK10-M	arm R
T1 chrF	50.69	69.61	71.44	69.45	69.47	69.80	80.14
T1 exact match (%)	2.2	11.7	11.8	11.6	10.7	11.0	24.7
T2 accuracy (%)	24.7	63.7	64.8	74.6	64.1	58.1	79.4
T3 dialectness †	0.0337	0.1145	0.1136	0.1329	0.0822	0.1136	0.1412
T3 region match †	0.1025	0.5889	0.5543	0.6879	0.5012	0.5847	0.5934
T3 ADI2 †	0.0118	0.0852	0.0800	0.1114	0.0619	0.0840	0.1065

Table 1: KoDialectBench v0, medians of three repeats, one serving configuration. † marker-lexicon family — see § 6.2 before interpreting. ‡ guardrail, not a gain axis (§ 6.5 ). SK6 repeats SK5 with a different seed.

Figure 1: Capability recovery of the best synthetic arm (SK4, 26,600 examples) under construction-independent evaluation: chrF for comprehension, accuracy for identification, and reference chrF for generation. Intermediate arms are in Table 1 ; the coupled generation metric is treated separately in Figure 2 .

Region	scoring markers	emittable forms
Gangwon	8,802	10,020
Gyeongsang	4,450	5,765
Jeolla	2,752	4,082
Chungcheong	1,991	3,263
Jeju	12,790	13,913

Table 2: Disjointness test. No scoring marker survives removal of the forms the transformation rules can emit.

Figure 2: Generation recovery under an independent metric (chrF against held-out human references) and the two coupled marker metrics as deployed (full scoring lexicon; Table 3 instead scores the held-out 20%), whose inventory the transformation rules can emit. The two marker metrics are shown separately rather than averaged: ADI2, the third marker metric, is their product. The ordering of arms is preserved; the level, and the crossing of the 100% line, is not.

Figure 3: The same training-side intervention, read by two metric families. Marker columns are scored on the held-out 20% inventory (not the deployed lexicon of Figure 2 ). Restricting the transformation rules to 80% of the marker inventory collapses the two marker-based measurements while the three pipeline-independent ones do not fall at all.

Arm	marker evaluator		pipeline-independent
	dialectness	region match	compr.	ident.	gen. ref
SK5 (coupled, 8,600)	91.8%	101.9%	64.2%	71.3%	67.9%
SK12-D9b (disjoint, 8,600)	8.1%	25.6%	67.6%	73.3%	70.2%
SK11-D9 (disjoint, 8,758)	8.4%	28.1%	63.8%	72.0%	68.1%
SK10-M (disjoint pool, 8,590 updates)	28.1%	47.5%	64.9%	61.1%	67.0%
SK4 (coupled, 26,600)	109.5%	122.3%	63.7%	91.2%	72.4%

Table 3: Construction-time intervention. SK12-D9b is the primary comparison : it is exactly size-matched to SK5 (8,600 examples) and matches it on task-by-region composition, source register and optimization recipe; 66.2% of its source sentences are shared with SK5. SK11-D9 is an independently sampled disjoint arm at 8,758, reported because the effect reproduces across two different synthetic samples. The treatment in both is a construction-disjoint synthesis pipeline: 20% of marker types are withheld from the transformation inventory, and residual evaluation-form leakage is filtered. The marker columns are scored by the same scorer code as Table 1 but on the held-out 20% inventory instead of the deployed lexicon, so the same arm carries a different marker figure in the two tables. Pipeline-independent columns report comprehension chrF, five-way region-identification accuracy, and generation chrF against held-out human references.

MCC	$\mathrm{MCC}_{w}$	arm R $-$ base		SK4 recovery
		[eval. pct.]	[item boot. 95% CI]	[eval. 2.5–97.5 pct.]
0.00	0.00	$+0.0146$ $[+0.0070,+0.0213]$	$+0.0146$ $[+0.0084,+0.0210]$	$-54.8$ %	$\,[-191.4,\ 9.5]$
0.25	0.37	$+0.0213$ $[+0.0123,+0.0301]$	$+0.0213$ $[+0.0146,+0.0281]$	$-0.9$ %	$\,[-70.4,\ 40.9]$
0.50	0.64	$+0.0281$ $[+0.0193,+0.0357]$	$+0.0281$ $[+0.0218,+0.0343]$	$32.8$ %	$\,[-7.9,\ 63.9]$
0.75	0.84	$+0.0356$ $[+0.0264,+0.0446]$	$+0.0357$ $[+0.0301,+0.0413]$	$58.6$ %	$\,[30.2,\ 79.6]$
1.00	1.00	$+0.0448$ $[+0.0392,+0.0508]$	$+0.0449$ $[+0.0395,+0.0500]$	$84.8$ %	$\,[73.7,\ 96.8]$

Table 4: MCC sweep on dialectness, 200 evaluator inventories per level stratified by region only. Two uncertainties are reported separately: “eval. pct.” is the 2.5–97.5 percentile across those evaluators, and “item boot.” fixes the median evaluator at that level and resamples benchmark items with replacement (2,000 draws, arms paired on item index). The real-data arm stays above base at every level under both. Endpoint recovery ranges do not overlap; adjacent levels do, so we do not claim a difference between neighbouring rows.

Figure 4: Metric–construction coverage sweep. Models and generations are fixed; only the evaluation inventory changes. Left : measured recovery of the synthetic arm (SK4) against the real-data arm, in percent. Right : “separation” is the evaluator’s own dialectness gap between the real-data arm and base; it shrinks as coverage falls, which is what makes the recovery ratio on the left progressively unstable — though it never reaches zero. Points are medians over 200 stratified evaluator samples; bars span the 2.5–97.5 percentiles.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	base	arm C	SK5	SK6	SK4	SK12-D9b	SK11-D9	SK10-M	SK7-D	SK9-D	arm R
T1 chrF	50.69	70.10	69.61	71.44	69.45	70.59	69.47	69.80	71.10	72.34	80.14
T1 exact match (%)	2.2	10.6	11.7	11.8	11.6	11.7	10.7	11.0	13.0	13.6	24.7
T2 accuracy (%)	24.7	63.1	63.7	64.8	74.6	64.8	64.1	58.1	71.0	66.8	79.4
T3 dialectness †	0.0337	0.1023	0.1145	0.1136	0.1329	0.0794	0.0822	0.1136	0.1030	0.1129	0.1412
T3 region match †	0.1025	0.5094	0.5889	0.5543	0.6879	0.4785	0.5012	0.5847	0.6828	0.6862	0.5934
T3 ADI2 †	0.0118	0.0653	0.0852	0.0800	0.1114	0.0584	0.0619	0.0840	0.0873	0.0972	0.1065

Appendix

Table 5: Every arm on every benchmark metric, medians of three runs. Marker-family values use the deployed full lexicon; the held-out intervention-evaluator results are reported separately in Table 3 .

Figure 5: Per-region transformation magnitude. “Edit ratio” is the character edit distance between each standard input and the generated output, normalised by input length, and uses no marker lexicon. All arms move far less than human references, and the synthetic arm’s between-region spread is narrower rather than wider.

	arm R (real)	SK4 (synthetic)
Region differentiation (5 distinct outputs)	77%	97%
chrF-exact round trip	93%	83%
Self-consistency of region label	40%	63%

Appendix

Table 6: Six self-written scenes. A qualitative probe, not a benchmark.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 239 papers

DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation
Jul 8, 2026Jordan Painter, Dipankar Srirag, Adarsh Kappiyath +3Language Modeling LLM Alignment
Institute for People-Centred AI, University of Surrey, Surrey, United Kingdom · University of New South Wales, Sydney, Australia
Dialect-Robust Speech Language Models with Synthetic Pseudo-Dialect Augmentation
Oct 7, 2026Shunsuke Mitsumori, Tomoya Mizumoto, Yusuke FujitaSpeech Language Models Language Model Robustness
SB Intuitions · Waseda University, Tokyo, Japan
Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models
Aug 30, 2026Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad +1Multilingual Language Model Evaluation LLM Evaluation
1Perle

Jul 8, 2026J/K move · Enter open · S save

DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia+2

Institute for People-Centred AI, University of Surrey, Surrey, United Kingdom · University of New South Wales, Sydney, Australia

Language Modeling LLM Alignment Text Generation Language Model Pretraining Language Model Post-Training Language Model Robustness+1

Open paper AskPDF arXiv

Large language models increasingly understand dialectal English, yet still produce only standard, US-leaning English, leaving dialectal generation, the harder half of the problem, largely unaddressed. We introduce DiaLLM, which continually pretrains three open-weight language model families on the International Corpus of English and applies implicit and explicit post-training paradigms, each combined with three model alignment strategies, giving the first controlled comparison of these components across Australian, Indian, and Northern British English. Our results reveal a robustness-generation gap: benchmarks are shaped by continual pretraining and SFT, while alignment visibly reshapes generation in ways benchmarks do not capture. Explicit variety-targeted adaptation produces output reliably recognised as dialectal and judged more dialectal than broad alignment, yet where human judgement was directly assessed, the method that most aggressively optimises the dialectal reward is not the one judged most dialectal. Independent linguistic analysis corroborates this reward-quality gap, most clearly on two of the three families. No single alignment method dominates, and closing the gap will require richer reward designs and continued investment in dialectal resources. We release all code, checkpoints, and preference datasets.

Abstract

Synthetic Data Evaluation

Explore similar work

DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

Dialect-Robust Speech Language Models with Synthetic Pseudo-Dialect Augmentation

Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models