Structured data exists in many forms (tables, knowledge graphs, charts, and time series), and converting it into text may involve different generation tasks. However, most prior work on data-to-text (D2T) generation has focused on specific tasks and datasets, relying either on task-specific training data or on the zero-shot capabilities of large language models. We study cross-domain D2T generation in a setting where neither in-domain training text nor test references are available, and where domains, generation goals, and input structures vary substantially. We compare data-driven knowledge distillation (DDKD) against zero-shot inference and fine-tuning on out-of-domain D2T data, and introduce structure-preserving augmentation via structural subsampling and perturbation. Experiments on five benchmarks show that, at constant model size (1.7B parameters), DDKD consistently outperforms both fine-tuning and zero-shot inference. Moreover, the resulting small models outperform a much larger finetuned model on two of the five domains, achieving comparable performance on the remaining three. We further construct QUINTD-5, a fivefold extension of QUINTD-1, and show that simply scaling real target-domain inputs yields only modest gains, whereas our augmentation strategy remains more effective and more cost-efficient for cross-domain distillation.
Figures & tables
Figure 1: Per-domain error rate ( lower is better ) across the five QUINTD-1 target domains, averaged over two independent LLM judges (GPT-5.1 and Gemini-2.5-Pro). Results compare three small models of identical scale: zero-shot inference, fine-tuning on WebNLG, and our DDKD-distilled model. Across both Qwen3 and Gemma3 backbones, DDKD consistently yields substantially lower error rates, indicating improved cross-domain transfer without target-domain text supervision.
Figure 2: Overview of our cross-domain Data-Driven Knowledge Distillation framework. Large teacher models are first trained on source-domain WebNLG using multiple structured input linearizations. Given limited target-domain structured inputs (QUINTD-1), the teacher generates synthetic texts for distillation. We compare two ways of improving target-domain supervision: collecting more real structured inputs as a naive baseline (QUINTD-5), and our proposed structural data augmentation , which increases diversity and yields progressively simpler training examples. A smaller student model is then distilled on the resulting data and evaluated with LLM judges and human annotation for faithfulness and content coverage, together with inter-judge agreement analysis.
Dataset
Format
Generation Goal
Structure Complexity
# Input Tokens †
Wikidata
MD
Structured entity description
#Props = 101, #Vals = 418
122 (84-190)
Ice Hockey
JSON
Event-based game summary
#Props = 20, #Vals = 975
335 (315-366)
OpenWeather
JSON
Time-series weather forecast reporting
#Props = 28, #Vals = 3,782
3,294 (3,163-3,487)
GSM Arena
JSON
Entity-centric specification description
#Props = 9, #Vals = 2,361
1,280 (886-3,164)
OWID
CSV
Chart Captioning
#Rows = 217, #Cols = 2
3,742 (178-8,076)
Table 1: QUINTD-1 Benchmark and Generation Tasks . #Props, #Vals: Number of distinct properties and Values present in each dataset. # Input Tokens: Average (Min-Max) number of input tokens from Qwen3 tokenizer.
Table 2: Example of deterministic WebNLG linearisation into the three target-domain input formats used in our experiments.
Domain
Base
Sub./Pert.
Mixed
Len. ( μ±σ )
Wikidata
100
32,628
65,256
173±52
Ice Hockey
100
25,500
51,000
421±28
OpenWeather
100
1,100
2,200
6,983±857
GSM Arena
100
1,381
2,762
1,601±643
OWID
100
1,000
2,000
5,067±3,355
Table 3: Statistics of the synthetic training data generated from the QUINTD-1 development sets under different augmentation strategies. Base: the number of QUINTD-1 instances. Sub./Pert.: the number of instances after subsampling or structural perturbation. Mixed: the number of instances for the mixed strategy. Len: mean and standard deviation of input-output sequence lengths (in Qwen3 tokenizer tokens).
System
Wikidata
Ice Hockey
OpenWeather
GSM Arena
OWID
NormAvg ↓
Zero-shot (ZS)
GPT-4.1-ZS
0.12
0.37
0.76
1.05
0.53
0.13
Qwen3-1.7B-ZS
0.74
1.38
6.10
1.22
2.11
0.64
Qwen3-8B-ZS
0.27
0.59
3.37
1.17
1.20
0.30
Qwen3-32B-ZS
0.45
0.63
2.53
0.49
0.99
0.25
WebNLG-SFT
Table 4: LLM-as-a-judge evaluation (all categories): average number of errors per output on the five QUINTD-1 target domains ( lower is better ). Results are based on GPT-5.1 as the judge. For DDKD, unless otherwise specified, results use 100 real seed instances (QUINTD-1 setting); rows marked with (500) use 500 real seed instances (QUINTD-5 setting). Bold indicates the best small model (1.7B) per domain and for NormAvg; underline indicates the best overall system. NormAvg denotes the mean of per-domain min-max normalised error counts, where normalisation is performed independently for each domain across all systems.
Appendix figures & tables66 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
WebNLG-SFT
Target-Domain SFT
Train. type
LoRA
LoRA
LoRA rank
8
8
LoRA α
32
32
Target mods
all-linear
all-linear
LR
1×10−4
5×10−5 / 1×10−4
Train BS
8
4 / 8
Appendix
Table 5: Training hyperparameters used for source-domain (WebNLG) and target-domain fine-tuning. For target-domain SFT, values are reported as Base / Augmented when they differ. For consistency, long maximum sequence lengths are used across training stages to match downstream inference conditions.
System
Peak Mem.
Train Time
Qwen3
Zero-shot (ZS)
Qwen3-1.7B-ZS
–
–
Qwen3-8B-ZS
–
–
Qwen3-32B-ZS
–
–
WebNLG-SFT
Appendix
Table 6: Training memory and time cost on a single NVIDIA A100 (80 GB). DDKD costs are reported on OWID to reflect the maximum input-length setting. Zero-shot models incur no training cost.
Domain
In-instance
Cross-instance
Wikidata
✓
Ice Hockey
✓
OpenWeather
✓
GSM Arena
✓
OWID
✓
Appendix
Table 7: Structural perturbation strategies adopted for each target domain.
Domain
QUINTD-1
QUINTD-5
#Props / Rows
#Vals / Cols
Input Len
#Props / Rows
#Vals / Cols
Input Len
Wikidata
95
416
122 (83–196)
176
1,762
123 (82–206)
Ice Hockey
20
971
333 (314–373)
20
3,047
332 (314–382)
OpenWeather
28
4,833
6,415 (6,214–6,749)
28
7,986
6,401 (6,159–6,886)
GSM Arena
9
2,432
1,307 (811–3,164)
9
8,140
1,273 (773–3,164)
OWID
236
2
4,004 (648–8,111)
273
2
4,465 (135–8,088)
Appendix
Table 8: Comparison of QUINTD-1 and QUINTD-5 development inputs. #Props / Rows denotes the number of distinct properties (JSON/Markdown) or rows (CSV), and #Vals denotes the number of distinct values. Input Len shows average input tokens (min–max, Qwen3 tokenizer). QUINTD-5 preserves similar input length distributions while substantially increasing knowledge diversity.
Figure 3: Kernel density estimates (KDE) of prompt token length distributions for QUINTD-1 and QUINTD-5 dev sets across five domains. Density normalization enables direct comparison despite differing sample sizes (100 vs. 500).
Level
Incorrect Fact
Not Checkable
Misleading
Other
All (Total Errors)
Token
0.618
0.490
0.169
0.580
0.582
Example
0.680
0.256
0.334
0.459
0.666
System
0.941
0.831
0.781
0.949
0.955
Appendix
Table 9: Cross-domain inter-judge agreement between GPT-5.1 and Gemini-2.5-Pro, macro-averaged across all target domains. All (Total Errors) is computed by correlating summed error counts across categories.
Domain
Level
Incorrect Fact
Not Checkable
Misleading
Other
All (Total Errors)
Wikidata
Token
0.563
0.663
0.147
0.255
0.523
Example
0.521
0.748
0.088
0.280
0.562
System
0.591
0.930
0.716
0.395
0.932
OpenWeather
Token
0.566
0.183
0.173
0.588
0.531
Example
0.557
0.172
0.291
0.361
0.550
System
0.915
0.444
0.075
0.826
0.913
Appendix
Table 10: Inter-judge Pearson correlation ( r ) between GPT-5.1 and Gemini-2.5-Pro at different aggregation levels, computed separately for each domain and error category. All (Total Errors) is computed by correlating the summed error counts across all error categories. System-level agreement is consistently higher than Token- and Example-level agreement, indicating robust consistency in system ranking across judges.
Target Domain
Selected DDKD Model
Teacher
Wikidata
Qwen3-1.7B-DDKD-Mixed
Qwen3-32B-SFT
Ice Hockey
Qwen3-1.7B-DDKD-Mixed
Qwen3-32B-SFT
OpenWeather
Qwen3-1.7B-DDKD-Sub
Qwen3-32B-SFT
GSM Arena
Qwen3-1.7B-DDKD-Mixed
Qwen3-32B-ZS
OWID
Qwen3-1.7B-DDKD-Mixed
Qwen3-32B-SFT
Appendix
Table 11: Final selection of distilled models per target domain. Model selection is primarily based on the average number of errors per output under GPT-5.1, and further informed by error rate and consistency with an independent judge (Gemini-2.5-Pro). Teacher models are selected on a per-domain basis according to their domain-level performance.
Granularity
Overall
Breakdown by Error Type ( α )
Agreement
Incorrect Fact
Not Checkable
Misleading
Other
Metric: Nominal α (Span-level detection)
Token-level
0.369
0.370
0.399
0.020
0.342
Metric: Interval α (Error counts)
Instance-level
0.578
0.549
0.530
0.033
0.710
System-level
0.902
0.810
0.498
-0.072
0.871
Appendix
Table 12: Inter-Annotator Agreement (Krippendorff’s α ) Analysis. We report the agreement scores for the overall dataset and a breakdown by specific error types. The Overall column represents the aggregate agreement calculated over all error types. The breakdown columns show the agreement when isolating each error type (binary distinction for token-level, counts for instance/system-level). Note that Misleading errors show near-zero or negative agreement, indicating high subjectivity, whereas Incorrect Fact and Other show high reliability at the system level.
Domain
Error Type
GPT-5.1
Gemini-2.5-Pro
Span #
Example #
Span #
Example #
OWID
Incorrect Fact
62
23
69
28
Not Checkable
13
8
2
2
Misleading
10
7
9
9
Other
12
11
28
25
GSM Arena
Incorrect Fact
23
16
31
20
Appendix
Table 13: Distribution of error annotations in the human evaluation set, aggregated over four systems. For each target domain and error category, we report the total number of error spans ( Span # ) and the number of examples containing at least one such error ( Example # ), as identified by GPT-5.1 and Gemini-2.5-Pro. All domains contain 48 annotated outputs (12 inputs × 4 systems).
Figure 4: Human Annotation Toolkit Interface - Screenshot 1
Figure 5: Human Annotation Toolkit Interface - Screenshot 2
Figure 6: Human Annotation Toolkit Interface - Screenshot 3
System
GSM-Arena
Ice Hockey
OWID
Weather
Wikidata
NormAvg ↓
Large Model (32B)
Teacher Model
1.06
0.42
1.25
0.78
0.44
0.15
Small Models (1.7B)
Zero-Shot
1.47
1.50
1.92
3.89
1.11
0.77
WebNLG-SFT
2.03
4.47
1.94
3.42
0.92
0.91
DDKD (Ours)
0.69
0.22
0.72
1.67
0.56
0.09
Appendix
Table 14: Human evaluation results (average error count per output; lower is better). For each domain, we report the mean number of annotated errors per output, averaged over the three human annotators on the 12-example diagnostic set. NormAvg is the mean of per-domain min–max normalized scores computed across the four evaluated systems (lower is better), to reduce scale differences between domains.
Domain
Overall
Breakdown by Error Type ( r )
All Errors
Fact
Unchk
Mis.
Oth.
GSM-Arena
0.941
0.997
0.502
-0.316
0.998
OWID
0.904
0.950
0.668
0.823
0.952
Weather
0.941
0.730
0.960
-0.422
0.884
Wikidata
0.829
-0.180
0.400
0.667
0.781
Ice Hockey
0.732
0.829
0.990
0.943
0.947
Appendix
Table 15: System-Level Human–LLM Agreement (Pearson r ). Correlation between human and GPT-5.1 evaluations across 4 systems. All Errors is computed on the total error count. Breakdown columns correspond to: Fact (Incorrect Fact), Unchk (Not Checkable), Mis. (Misleading), and Oth. (Other). High correlations ( >0.9 ) in All Errors indicate that GPT-5.1 largely reproduces the human system ranking.
Figure 9: System-level content coverage sanity check summarized by NormAvg (domain-wise min–max normalized coverage scores averaged across domains). Each row shows the mean NormAvg across two LLM judges (dot) and the judge-specific values (markers). DDKD-Best consistently outperforms same-size baselines and remains close to the selected best teacher, indicating gains are not achieved via content omission.
Domain
αord
αint
#Pairs
#Skip
OWID
0.525
0.504
396
4
GSM Arena
0.617
0.670
396
4
Weather
0.373
0.472
391
9
Ice Hockey
0.283
0.524
395
5
Wikidata
0.744
0.835
398
2
ALL
0.732
0.794
1,976
20
Appendix
Table 16: Instance-level inter-judge agreement between GPT-5.1 and Gemini-2.5-Pro . αord is computed on Coverage Score (1–5, ordinal) and αint on Estimated Coverage Ratio (0–1, interval). #Skip counts instances excluded due to parsing errors or missing values.
Domain
rs
ρs
rr
ρr
#Sys
OWID
0.924
1.000
0.916
1.000
4
GSM Arena
0.974
0.800
0.987
1.000
4
Weather
0.963
0.800
0.975
0.800
4
Ice Hockey
1.000
1.000
1.000
1.000
4
Wikidata
0.000
0.000
0.789
0.800
4
ALL
0.959
0.951
0.961
0.959
20
Appendix
Table 17: System-level inter-judge agreement between GPT-5.1 and Gemini-2.5-Pro . rs/ρs are correlations on per-system mean scores and rr/ρr on per-system mean ratios .
Dataset
#Inst.
Domain Scope
Reference
WebNLG
39,890
16 DBpedia categories
Human
E2E
46,733
Restaurant only
Human
KELM-Q1
18,723
Wikidata/Wikipedia
Automatic
Appendix
Table 18: Comparison of source-domain initialization datasets used in our ablation study. Statistics correspond to the actual training set used in our experiments.
Source Init.
Wikidata
IceHockey
OpenWeather
GSMArena
OWID
NE ↑
NC ↑
Err ↓
Cov ↑
Err ↓
Cov ↑
Err ↓
Cov ↑
Err ↓
Cov ↑
Err ↓
Cov ↑
WebNLG
0.09
4.61
0.02
3.99
1.86
2.76
0.65
3.23
0.43
3.41
0.954
1.000
E2E
0.09
4.38
0.00
2.56
3.84
2.07
0.39
2.43
0.90
2.18
0.828
0.066
KELM
0.31
4.40
0.39
2.93
5.06
2.08
1.86
2.21
2.39
2.02
0.000
0.072
Appendix
Table 19: Sensitivity to the source-domain initialization dataset. We fine-tune the same teacher model ( Qwen3-32B ) on three source datasets and directly evaluate transfer to the five QUINTD-1 target domains. Err denotes the average error number from our faithfulness evaluation (lower is better), and Cov denotes the content coverage score assigned by GPT-5.1 (higher is better). NE and NC denote domain-wise min–max normalized averages for Err and Cov, respectively; for Err, reversed min–max normalization is used so that higher is better. WebNLG provides the strongest overall initialization, achieving the most favorable trade-off between factual correctness and coverage across heterogeneous target domains.
System
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
NormAll ↓
Zero-shot (ZS)
GPT-4.1-ZS
0.14
0.47
0.07
0.00
0.68
0.16
Qwen3-1.7B-ZS
1.54
0.53
0.21
0.04
2.31
0.64
Qwen3-8B-ZS
0.75
0.38
0.16
0.03
1.32
0.30
Qwen3-32B-ZS
0.50
0.35
0.13
0.04
1.02
0.25
WebNLG-SFT
Appendix
Table 20: Qwen3 Error-Type Result Summary by GPT-5.1 . LLM-as-a-judge evaluation (error-type breakdown): macro-averaged number of errors per output across all target domains ( lower is better ). Results are based on GPT-5.1 as the judge. Bold marks the best 1.7B system per column (ties allowed); underline marks the best overall system per column (ties allowed). NormAll is the mean of per-domain min–max normalized All Categories error counts, where normalization is performed independently within each domain across all systems to ensure equal domain contribution ( lower is better ).
System
Wikidata
Ice Hockey
OpenWeather
GSM Arena
OWID
Zero-shot (ZS)
Qwen3-1.7B-ZS
0.98
1.71
7.19
1.34
2.57
Qwen3-8B-ZS
0.59
0.62
4.22
0.69
1.67
Qwen3-32B-ZS
0.53
0.49
3.69
0.10
1.60
WebNLG-SFT
Qwen3-1.7B-SFT
0.70
2.42
10.83
2.67
2.52
Appendix
Table 21: Qwen3 Result Summary by Gemini-2.5-Pro . LLM-as-a-judge evaluation ( all categories ): average number of errors per output on the five QUINTD-1 target domains ( lower is better ). Bold indicates the best small model (1.7B) per domain, while underlined values indicate the best overall system. All error categories (Incorrect Fact, Not Checkable, Misleading, Other) are counted and summed. Results are based on Gemini-2.5-Pro as the judge.
System
Wikidata
Ice Hockey
OpenWeather
GSM Arena
OWID
Zero-shot (ZS)
Gemma3-1B-IT-ZS
1.71
2.80
4.66
4.03
3.14
Gemma3-27B-IT-ZS
0.56
0.28
4.98
0.44
0.90
WebNLG-SFT
Gemma3-1B-IT-SFT
5.49
5.34
5.48
3.60
3.86
Gemma3-27B-IT-SFT
0.12
0.20
2.48
0.37
0.77
Appendix
Table 22: Gemma3 Result Summary by GPT-5.1 . LLM-as-a-judge evaluation ( all categories ): average number of errors per output on the five QUINTD-1 target domains ( lower is better ). Bold indicates the best small model per domain, while underlined values indicate the best overall system. All error categories (Incorrect Fact, Not Checkable, Misleading, Other) are counted and summed. Results are based on GPT-5.1 as the judge.
System
Wikidata
Ice Hockey
OpenWeather
GSM Arena
OWID
Zero-shot (ZS)
Qwen3-1.7B-ZS
33.17 ± 18.76
63.40 ± 12.40
125.86 ± 21.35
121.59 ± 19.23
52.61 ± 12.91
Qwen3-8B-ZS
35.57 ± 18.33
72.35 ± 8.22
120.54 ± 13.61
140.67 ± 15.46
52.32 ± 13.51
Qwen3-32B-ZS
34.79 ± 18.72
75.44 ± 11.90
127.22 ± 22.52
183.45 ± 34.34
71.73 ± 24.26
WebNLG-SFT
Qwen3-1.7B-SFT
29.64 ± 17.50
116.73 ± 304.27
767.44 ± 749.55
776.32 ± 916.36
189.57 ± 370.39
Appendix
Table 23: Qwen3 Outputs Length Analysis . Output length statistics ( tokens ): mean ± standard deviation over 100 outputs per system on each QUINTD-1 target domain. Tokenization uses the same tokenizer as the corresponding backbone (Qwen3) in our evaluation pipeline.
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
0.00
0.12
0.00
0.00
0.12
Qwen3-1.7B-ZS
–
0.18
0.50
0.02
0.04
0.74
Qwen3-8B-ZS
–
0.01
0.23
0.00
0.03
0.27
Qwen3-32B-ZS
–
0.04
0.39
0.01
0.01
0.45
WebNLG-SFT
Appendix
Table 24: Error counts on Wikidata evaluated by GPT-5.1 (Qwen3). We report the average number of errors per output across different error categories (lower is better). # Real denotes the number of real seed instances used for distillation. "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
0%
11%
0%
0%
11%
Qwen3-1.7B-ZS
–
16%
31%
2%
4%
46%
Qwen3-8B-ZS
–
1%
19%
0%
3%
22%
Qwen3-32B-ZS
–
4%
29%
1%
1%
33%
WebNLG-SFT
Appendix
Table 25: Error rates on Wikidata evaluated by GPT-5.1 (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. # Real denotes the number of real seed instances used for distillation. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
0.30
0.28
0.06
0.34
0.98
Qwen3-8B-ZS
0.05
0.20
0.08
0.26
0.59
Qwen3-32B-ZS
0.11
0.28
0.04
0.10
0.53
WebNLG-SFT
Qwen3-1.7B-SFT
0.19
0.08
0.20
0.23
0.70
Appendix
Table 26: Error counts on Wikidata evaluated by Gemini-2.5-Pro (Qwen3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen-1.7B-ZS
26%
20%
4%
29%
62%
Qwen3-8B-ZS
5%
18%
8%
20%
44%
Qwen3-32B-ZS
11%
24%
4%
9%
40%
WebNLG-SFT
Qwen3-1.7B-SFT
18%
8%
14%
20%
44%
Appendix
Table 27: Error rates on Wikidata evaluated by Gemini-2.5-Pro (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
0.17
1.28
0.06
0.20
1.71
Gemma3-27B-IT-ZS
0.04
0.48
0.00
0.04
0.56
WebNLG-SFT
Gemma3-1B-IT-SFT
2.65
1.69
0.16
0.98
5.49
Gemma3-27B-IT-SFT
0.04
0.06
0.02
0.00
0.12
Appendix
Table 28: Error counts on Wikidata evaluated by GPT-5.1 (Gemma3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
15%
67%
6%
18%
75%
Gemma3-27B-IT-ZS
4%
37%
0%
4%
39%
WebNLG-SFT
Gemma3-1B-IT-SFT
80%
76%
14%
66%
99%
Gemma3-27B-IT-SFT
4%
6%
2%
0%
11%
Appendix
Table 29: Error rates on Wikidata evaluated by GPT-5.1 (Gemma3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
0.38
0.96
0.16
0.53
2.03
Gemma3-27B-IT-ZS
0.05
0.38
0.04
0.70
1.17
WebNLG-SFT
Gemma3-1B-IT-SFT
5.64
0.35
0.37
1.08
7.44
Gemma3-27B-IT-SFT
0.08
0.05
0.06
0.06
0.25
Appendix
Table 30: Error counts on Wikidata evaluated by Gemini-2.5-Pro (Gemma3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
34%
58%
15%
50%
93%
Gemma3-27B-IT-ZS
5%
34%
4%
63%
77%
WebNLG-SFT
Gemma3-1B-IT-SFT
98%
14%
29%
84%
100%
Gemma3-27B-IT-SFT
7%
5%
6%
6%
22%
Appendix
Table 31: Error rates on Wikidata evaluated by Gemini-2.5-Pro (Gemma3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
0.01
0.36
0.00
0.00
0.37
Qwen3-1.7B-ZS
–
0.73
0.58
0.06
0.01
1.38
Qwen3-8B-ZS
–
0.30
0.27
0.02
0.00
0.59
Qwen3-32B-ZS
–
0.07
0.54
0.02
0.00
0.63
WebNLG-SFT
Appendix
Table 32: Error counts on Ice Hockey evaluated by GPT-5.1 (Qwen3). We report the average number of errors per output across different error categories (lower is better). # Real denotes the number of real seed instances used for distillation. "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
1%
25%
0%
0%
26%
Qwen3-1.7B-ZS
–
44%
47%
6%
1%
73%
Qwen3-8B-ZS
–
24%
20%
2%
0%
40%
Qwen3-32B-ZS
–
5%
45%
2%
0%
47%
WebNLG-SFT
Appendix
Table 33: Error rates on Ice Hockey evaluated by GPT-5.1 (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. # Real denotes the number of real seed instances used for distillation. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
0.88
0.27
0.19
0.37
1.71
Qwen3-8B-ZS
0.39
0.07
0.06
0.10
0.62
Qwen3-32B-ZS
0.10
0.19
0.05
0.15
0.49
WebNLG-SFT
Qwen3-1.7B-SFT
1.59
0.12
0.27
0.44
2.42
Appendix
Table 34: Error counts on Ice Hockey evaluated by Gemini-2.5-Pro (Qwen3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
53%
25%
17%
35%
91%
Qwen3-8B-ZS
33%
7%
4%
10%
46%
Qwen3-32B-ZS
8%
17%
5%
12%
33%
WebNLG-SFT
Qwen3-1.7B-SFT
79%
11%
26%
39%
97%
Appendix
Table 35: Error rates on Ice Hockey evaluated by Gemini-2.5-Pro (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
2.33
0.34
0.01
0.12
2.80
Gemma3-27B-IT-ZS
0.13
0.14
0.01
0.00
0.28
WebNLG-SFT
Gemma3-1B-IT-SFT
2.69
1.14
0.31
1.20
5.34
Gemma3-27B-IT-SFT
0.11
0.09
0.00
0.00
0.20
Appendix
Table 36: Error counts on Ice Hockey evaluated by GPT-5.1 (Gemma3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
92%
26%
1%
12%
94%
Gemma3-27B-IT-ZS
12%
13%
1%
0%
24%
WebNLG-SFT
Gemma3-1B-IT-SFT
88%
62%
24%
78%
100%
Gemma3-27B-IT-SFT
10%
9%
0%
0%
19%
Appendix
Table 37: Error rates on Ice Hockey evaluated by GPT-5.1 (Gemma3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
2.46
0.13
0.17
0.33
3.09
Gemma3-27B-IT-ZS
0.21
0.10
0.05
0.19
0.55
WebNLG-SFT
Gemma3-1B-IT-SFT
3.09
0.51
0.33
1.96
5.89
Gemma3-27B-IT-SFT
0.17
0.05
0.06
0.32
0.60
Appendix
Table 38: Error counts on Ice Hockey evaluated by Gemini-2.5-Pro (Gemma3).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
92%
10%
14%
31%
98%
Gemma3-27B-IT-ZS
17%
10%
5%
15%
40%
WebNLG-SFT
Gemma3-1B-IT-SFT
93%
32%
26%
96%
99%
Gemma3-27B-IT-SFT
16%
5%
6%
29%
50%
Appendix
Table 39: Error rates on Ice Hockey evaluated by Gemini-2.5-Pro (Gemma3).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
0.34
0.32
0.10
0.00
0.76
Qwen3-1.7B-ZS
–
5.06
0.50
0.52
0.02
6.10
Qwen3-8B-ZS
–
2.52
0.34
0.46
0.05
3.37
Qwen3-32B-ZS
–
1.70
0.35
0.31
0.17
2.53
WebNLG-SFT
Appendix
Table 40: Error counts on OpenWeather evaluated by GPT-5.1 (Qwen3). We report the average number of errors per output across different error categories (lower is better). # Real denotes the number of real seed instances used for distillation. "All Categories" represents the total error count. Bold indicates the best performing open-source model (lowest total errors).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
29%
23%
10%
5%
49%
Qwen3-1.7B-ZS
–
98%
23%
36%
2%
99%
Qwen3-8B-ZS
–
94%
26%
37%
5%
98%
Qwen3-32B-ZS
–
83%
20%
25%
17%
92%
WebNLG-SFT
Appendix
Table 41: Error rates on OpenWeather evaluated by GPT-5.1 (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. # Real denotes the number of real seed instances used for distillation. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
6.30
0.10
0.74
0.05
7.19
Qwen3-8B-ZS
3.61
0.06
0.52
0.03
4.22
Qwen3-32B-ZS
2.84
0.07
0.75
0.03
3.69
WebNLG-SFT
Qwen3-1.7B-SFT
9.54
0.90
0.61
0.59
10.83
Appendix
Table 42: Error counts on OpenWeather evaluated by Gemini-2.5-Pro (Qwen3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
100%
8%
42%
5%
100%
Qwen3-8B-ZS
100%
6%
42%
3%
100%
Qwen3-32B-ZS
96%
6%
47%
3%
97%
WebNLG-SFT
Qwen3-1.7B-SFT
95%
7%
31%
46%
99%
Appendix
Table 43: Error rates on OpenWeather evaluated by Gemini-2.5-Pro (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
3.02
1.05
0.57
0.02
4.66
Gemma3-27B-IT-ZS
3.57
1.00
0.34
0.07
4.98
WebNLG-SFT
Gemma3-1B-IT-SFT
2.65
1.69
0.16
0.98
5.48
Gemma3-27B-IT-SFT
2.26
0.06
0.12
0.04
2.48
Appendix
Table 44: Error counts on OpenWeather evaluated by GPT-5.1 (Gemma3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
94%
64%
45%
2%
100%
Gemma3-27B-IT-ZS
93%
27%
25%
7%
97%
WebNLG-SFT
Gemma3-1B-IT-SFT
80%
76%
14%
66%
99%
Gemma3-27B-IT-SFT
31%
6%
4%
4%
43%
Appendix
Table 45: Error rates on OpenWeather evaluated by GPT-5.1 (Gemma3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
4.06
0.65
0.57
0.09
5.37
Gemma3-27B-IT-ZS
5.72
0.10
0.69
0.10
6.52
WebNLG-SFT
Gemma3-1B-IT-SFT
5.64
0.35
0.37
1.08
7.44
Gemma3-27B-IT-SFT
2.86
0.05
1.61
0.33
4.85
Appendix
Table 46: Error counts on OpenWeather evaluated by Gemini-2.5-Pro (Gemma3).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
100%
41%
46%
9%
100%
Gemma3-27B-IT-ZS
100%
9%
46%
1%
100%
WebNLG-SFT
Gemma3-1B-IT-SFT
98%
14%
29%
84%
100%
Gemma3-27B-IT-SFT
52%
1%
45%
26%
89%
Appendix
Table 47: Error rates on OpenWeather evaluated by Gemini-2.5-Pro (Gemma3).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
0.01
1.04
0.00
0.00
1.05
Qwen3-1.7B-ZS
–
0.42
0.61
0.08
0.11
1.22
Qwen3-8B-ZS
–
0.21
0.79
0.10
0.70
1.17
Qwen3-32B-ZS
–
0.03
0.42
0.04
0.00
0.49
WebNLG-SFT
Appendix
Table 48: Error counts on GSM Arena evaluated by GPT-5.1 (Qwen3). We report the average number of errors per output across different error categories (lower is better). # Real denotes the number of real seed instances used for distillation. "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
1%
57%
0%
0%
57%
Qwen3-1.7B-ZS
–
33%
45%
8%
11%
71%
Qwen3-8B-ZS
–
19%
53%
9%
6%
69%
Qwen3-32B-ZS
–
3%
29%
4%
0%
34%
WebNLG-SFT
Appendix
Table 49: Error rates on GSM Arena evaluated by GPT-5.1 (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. # Real denotes the number of real seed instances used for distillation. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
0.65
0.25
0.29
0.15
1.34
Qwen3-8B-ZS
0.33
0.24
0.06
0.06
0.69
Qwen3-32B-ZS
0.04
0.03
0.03
0.00
0.10
WebNLG-SFT
Qwen3-1.7B-SFT
1.12
0.20
0.20
1.15
2.67
Appendix
Table 50: Error counts on GSM Arena evaluated by Gemini-2.5-Pro (Qwen3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
46%
20%
25%
13%
72%
Qwen3-8B-ZS
30%
21%
6%
6%
47%
Qwen3-32B-ZS
4%
3%
3%
0%
10%
WebNLG-SFT
Qwen3-1.7B-SFT
65%
12%
17%
76%
94%
Appendix
Table 51: Error rates on GSM Arena evaluated by Gemini-2.5-Pro (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
1.25
2.64
0.07
0.07
4.03
Gemma3-27B-IT-ZS
0.07
0.34
0.03
0.00
0.44
WebNLG-SFT
Gemma3-1B-IT-SFT
1.90
0.75
0.13
0.82
3.60
Gemma3-27B-IT-SFT
0.15
0.13
0.07
0.02
0.37
Appendix
Table 52: Error counts on GSM Arena evaluated by GPT-5.1 (Gemma3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
83%
89%
7%
7%
98%
Gemma3-27B-IT-ZS
6%
30%
3%
0%
44%
WebNLG-SFT
Gemma3-1B-IT-SFT
86%
47%
13%
59%
96%
Gemma3-27B-IT-SFT
12%
12%
7%
2%
29%
Appendix
Table 53: Error rates on GSM Arena evaluated by GPT-5.1 (Gemma3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
1.48
1.47
0.18
0.08
3.21
Gemma3-27B-IT-ZS
0.10
0.07
0.01
0.00
0.18
WebNLG-SFT
Gemma3-1B-IT-SFT
2.29
0.45
0.27
1.13
4.14
Gemma3-27B-IT-SFT
0.23
0.08
0.05
0.03
0.39
Appendix
Table 54: Error counts on GSM Arena evaluated by Gemini-2.5-Pro (Gemma3).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
84%
67%
14%
8%
99%
Gemma3-27B-IT-ZS
10%
7%
1%
0%
15%
WebNLG-SFT
Gemma3-1B-IT-SFT
90%
31%
23%
69%
96%
Gemma3-27B-IT-SFT
19%
8%
5%
2%
32%
Appendix
Table 55: Error rates on GSM Arena evaluated by Gemini-2.5-Pro (Gemma3).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
0.21
0.16
0.16
0.00
0.53
Qwen3-1.7B-ZS
–
1.29
0.44
0.36
0.02
2.11
Qwen3-8B-ZS
–
0.71
0.25
0.24
0.00
1.20
Qwen3-32B-ZS
–
0.68
0.05
0.25
0.01
0.99
WebNLG-SFT
Appendix
Table 56: Error counts on OWID evaluated by GPT-5.1 (Qwen3). We report the average number of errors per output across different error categories (lower is better). # Real denotes the number of real seed instances used for distillation. "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
# Real
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
GPT-4.1-ZS
–
21%
12%
15%
0%
38%
Qwen3-1.7B-ZS
–
68%
42%
35%
2%
87%
Qwen3-8B-ZS
–
51%
22%
22%
0%
73%
Qwen3-32B-ZS
–
48%
5%
24%
1%
61%
WebNLG-SFT
Appendix
Table 57: Error rates on OWID evaluated by GPT-5.1 (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. # Real denotes the number of real seed instances used for distillation. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
1.75
0.33
0.43
0.06
2.57
Qwen3-8B-ZS
1.12
0.13
0.42
0.00
1.67
Qwen3-32B-ZS
1.18
0.03
0.37
0.02
1.60
WebNLG-SFT
Qwen3-1.7B-SFT
2.00
0.08
0.13
0.31
2.52
Appendix
Table 58: Error counts on OWID evaluated by Gemini-2.5-Pro (Qwen3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Qwen3-1.7B-ZS
84%
32%
40%
6%
96%
Qwen3-8B-ZS
65%
13%
37%
0%
85%
Qwen3-32B-ZS
67%
3%
28%
2%
79%
WebNLG-SFT
Qwen3-1.7B-SFT
77%
6%
12%
30%
86%
Appendix
Table 59: Error rates on OWID evaluated by Gemini-2.5-Pro (Qwen3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
2.40
0.35
0.39
0.00
3.14
Gemma3-27B-IT-ZS
0.74
0.04
0.12
0.00
0.90
WebNLG-SFT
Gemma3-1B-IT-SFT
2.76
0.28
0.32
0.50
3.86
Gemma3-27B-IT-SFT
0.60
0.00
0.03
0.14
0.77
Appendix
Table 60: Error counts on OWID evaluated by GPT-5.1 (Gemma3). We report the average number of errors per output across different error categories (lower is better). "All Categories" represents the total error count. Bold indicates the best performing model (lowest total errors).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
93%
28%
30%
0%
97%
Gemma3-27B-IT-ZS
53%
3%
12%
0%
59%
WebNLG-SFT
Gemma3-1B-IT-SFT
50%
23%
32%
34%
92%
Gemma3-27B-IT-SFT
10%
0%
3%
14%
23%
Appendix
Table 61: Error rates on OWID evaluated by GPT-5.1 (Gemma3). Values represent the percentage of generated outputs containing at least one error of the specific type. The "All Categories" column reports the percentage of outputs containing any type of error. Lower percentages indicate better performance.
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
2.55
0.22
0.49
0.05
3.31
Gemma3-27B-IT-ZS
1.28
0.10
0.29
0.05
1.63
WebNLG-SFT
Gemma3-1B-IT-SFT
1.90
0.11
0.39
0.56
2.96
Gemma3-27B-IT-SFT
0.51
0.00
0.16
0.56
1.23
Appendix
Table 62: Error counts on OWID evaluated by Gemini-2.5-Pro (Gemma3).
Model
Incorrect Fact
Not Checkable
Misleading
Other
All Categories
Zero-Shot (ZS)
Gemma3-1B-IT-ZS
97%
20%
44%
5%
100%
Gemma3-27B-IT-ZS
72%
1%
26%
5%
83%
WebNLG-SFT
Gemma3-1B-IT-SFT
72%
10%
37%
48%
94%
Gemma3-27B-IT-SFT
28%
0%
16%
42%
68%
Appendix
Table 63: Error rates on OWID evaluated by Gemini-2.5-Pro (Gemma3).
System
Wikidata
IceHockey
GSM
Weather
OWID
NormAvg †
Score
Ratio
Score
Ratio
Score
Ratio
Score
Ratio
Score
Ratio
Qwen3-1.7B-ZS
4.61
0.912
3.82
0.736
3.20
0.601
2.78
0.463
2.93
0.501
0.5795
Qwen3-1.7B-SFT
4.56
0.899
3.42
0.628
3.00
0.551
2.20
0.294
2.52
0.385
0.0000
Qwen3-32B-SFT / ZS
4.61
0.916
3.99
0.776
3.84
0.761
2.76
0.456
3.41
0.609
0.8908
Qwen3-1.7B-DDKD-Best
4.66
0.926
3.97
0.766
3.85
0.754
2.57
0.404
2.94
0.493
0.8150
Appendix
Table 64: Content coverage sanity check using GPT-5.1 as the judge. Each cell reports mean Coverage Score (1–5) and mean Estimated Coverage Ratio (0–1). For Wikidata (entity description), the judge evaluates completeness over all input facts; for other domains, it evaluates coverage over task-relevant important content (salient entities/attributes and numeric patterns) without exhaustive row-level matching for long tables. Qwen3-32B-SFT/ZS denotes the teacher variant selected per domain (Table 11 ). † NormAvg is the average across domains of min–max normalized coverage score within each domain (normalization computed over the systems in this table; higher is better).
System
Wikidata
IceHockey
GSM
Weather
OWID
NormAvg †
Score
Ratio
Score
Ratio
Score
Ratio
Score
Ratio
Score
Ratio
Qwen3-1.7B-ZS
4.48
0.900
3.27
0.650
3.24
0.673
2.55
0.439
2.80
0.524
0.4833
Qwen3-1.7B-SFT
4.51
0.897
2.74
0.512
3.14
0.642
1.99
0.228
2.43
0.421
0.0857
Qwen3-32B-SFT / ZS
4.55
0.913
3.48
0.699
3.85
0.801
2.64
0.448
2.96
0.537
1.0000
Qwen3-1.7B-DDKD-Best
4.51
0.909
3.46
0.690
3.68
0.767
2.52
0.415
2.85
0.498
0.7540
Appendix
Table 65: Content coverage sanity check using Gemini-2.5-Pro as the judge. Each cell reports mean Coverage Score (1–5) and mean Estimated Coverage Ratio (0–1). For Wikidata (entity description), the judge evaluates completeness over all input facts; for other domains, it evaluates coverage over task-relevant important content (salient entities/attributes and numeric patterns) without exhaustive row-level matching for long tables. Qwen3-32B-SFT/ZS denotes the teacher variant selected per domain (Table 11 ). † NormAvg is the average across domains of min–max normalized coverage scores within each domain (normalization computed over the systems in this table; higher is better).
Adapting Large Language Models (LLMs) to specialized domains requires high-quality instruction tuning datasets, which are expensive to create through human annotation. Existing data synthesis methods focus on general-purpose tasks and fail to capture domain-specific terminology and reasoning patterns. To address this, we introduce DS2-Instruct, a zero-shot framework that generates domain-specific instruction datasets without human supervision. Our approach first generates task-informed keywords to ensure comprehensive domain coverage. It then creates diverse instructions by pairing these keywords with different cognitive levels from Bloom's Taxonomy. Finally, it uses self-consistency validation to ensure data quality. We apply this framework to generate datasets across seven challenging domains, such as mathematics, finance, and logical reasoning. Comprehensive evaluation demonstrates that models fine-tuned on our generated data achieve substantial improvements over existing data generation methods.
Adapting small instruction-tuned models to specialized domains often relies on supervised fine-tuning (SFT) on curated instruction-response examples, which is expensive to collect at scale. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross-passage or cross-document dependencies. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs, reassembles them using semantic structure inferred from embedding similarity, and then generates question-answer (QA) pairs via proximity, intra-cluster, and inter-cluster sampling with cluster-specialized system prompts. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three datasets of varied semantic heterogeneity, under fixed token budgets (5 and 20 million tokens). We use lexical overlap metrics, an LLM-as-a-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation. EmbGen improves Binary Accuracy on the most heterogeneous dataset by 12.5% at 5M and 88.9% at 20M tokens budget, relative to the strongest baseline, while remaining competitive across other datasets with lower heterogeneity.
Large-scale text corpora have become a quiet bottleneck in modern NLP, not just in storage, but in the accumulated cost of training, fine-tuning, and continual learning. We propose a text dataset distillation framework that reduces corpora to as little as 0.1% of their original size while preserving downstream task fidelity. We approach distillation through the lens of influence functions, which quantify each sample's contribution to the downstream objective, a natural and principled basis for selection. We introduce Trajectory-Aware Knowledge Estimation (TAKE), which convolves the knowledge-based influence along the training trajectory into a single per-sample knowledge score, capturing informative samples. These scores serve as sample weights within a discrete Optimal Transport objective, guiding prototype selection from a synthetically generated candidate pool. We evaluate TAKE on downstream accuracy across text classification and natural language inference tasks at extreme compression (0.1% or 20 samples/class), showing that data efficiency is achievable without sacrificing task fidelity. The approach is theoretically grounded, with broader implications for coreset construction and data-centric AI. We release our source code at https://github.com/votrinhan88/take.
Tri-Nhan Vo, Dang Nguyen, Sunil Gupta
Deakin Applied AI Initiative, Deakin University, Australia