A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer
Organizations: Flitto
Abstract
Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage's marginal value unknown. We design a production-ready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) are independently toggleable, yielding a 2x2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and fine-tune 9 models (5 LLM families, 2.4B-70B); we evaluate with four cross-provider LLM judges, a blind six-expert human evaluation, and three downstream MCQA benchmarks. Our central finding is that, under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains. 4-judge quality rises consistently, yet the cross-model mean MCQA gain is not significant; positive transfer concentrates on family-domain aligned pairs. All six human raters report higher full-pipeline quality, confirming the LLM-judge direction. An STT-engine swap to Whisper-medium confirms pipeline robustness. An LLM-difficulty audit shows the two frontier LLMs admit unknown on approximately 8% of QA on average; we release samples, prompts, code, and all SFT checkpoints.
Figures & tables
| Condition | Phase 0 | Phase 2 | Label |
|---|---|---|---|
| Exp 0 | OFF | OFF | no refinement |
| Exp 1-1 | ON | OFF | Phase 0 only |
| Exp 1-2 | OFF | ON | Phase 2 only |
| Exp 2 | ON | ON | full pipeline |
| Condition | Medical | Finance | Total |
|---|---|---|---|
| Exp 0 | 1,411 | 1,252 | 2,663 |
| Exp 1-1 | 1,468 | 1,297 | 2,765 |
| Exp 1-2 | 1,402 | 1,270 | 2,672 |
| Exp 2 | 1,376 | 1,222 | 2,598 |
| Source | Exp 0 | Exp 1-1 | Exp 1-2 | Exp 2 | ||||
| LLM (4-judge, Finance) | 3.76 | 3.84 | 3.76 | 3.93 | ||||
| LLM (4-judge, Medical) | 3.85 | 3.95 | 3.92 | 4.05 | ||||
| LLM (2 frontier, Finance) | 3.36 | 3.48 | 3.40 | 3.57 | ||||
| LLM (2 frontier, Medical) | 3.45 | 3.52 | 3.49 | 3.72 | ||||
| Human (Finance, 3 raters) | 3.00 | 3.20 | 3.13 | 3.29 | ||||
| Human (Medical, 3 raters) | 3.72 | 3.73 | 3.78 | 3.87 |
| Metric | ||||
|---|---|---|---|---|
| MMLU | ||||
| KMMLU | ||||
| KMMLU (med) | ||||
| KMMLU (fin) | ||||
| KMMLU-Pro (med) | ||||
| KMMLU-Pro (fin) |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment | Cells | Seeds |
|---|---|---|
| Main LoRA SFT | 72 | ( ) |
| MCQA eval (KMMLU/MMLU) | 72 | ( ) |
| KMMLU-Pro eval | 72 | ( ) |
| Full FT sanity | 12 | ( ) |
| W-Exp0 SFT | 4 | ( ) |
| W-Exp2 SFT | 4 | ( ) |
| # | sub-domain | ROW |
| Medical (19 sessions) | ||
| 1 | dental/implant | 659 |
| 2 | dental/AI implant | 789 |
| 3 | dental/PDRN | 671 |
| 4 | dental/implant | 460 |
| 5 | other/critical care | 419 |
| Stage | Exp 1-2 ( ) | Exp 2 ( ) |
|---|---|---|
| Phase 1 candidates (mean) | 60.0 | 63.1 |
| Judge pass rate | 87.8% | 82.6% |
| Refine recovery rate | 82.6% | 78.7% |
| Dedup duplicates / session | 1.0 | 1.7 |
| Mean exported / session | 56.8 | 58.8 |
| Net Phase 2 retention | 94.7% | 93.2% |
| Model | Metric | LoRA | FullFT | Sign |
|---|---|---|---|---|
| exaone3.5-2.4b | KMMLU-med | |||
| exaone3.5-2.4b | MMLU-med | |||
| exaone3.5-2.4b | KMMLU-fin | |||
| exaone3.5-2.4b | MMLU-fin | |||
| qwen3.5-4b | KMMLU-med | |||
| qwen3.5-4b | MMLU-med |
| Cond | factoid | explanatory | comparative | procedural |
|---|---|---|---|---|
| Exp 0 | 38% | 30% | 20% | 12% |
| Exp 1-1 | 46% | 30% | 18% | 6% |
| Exp 1-2 | 26% | 32% | 20% | 22% |
| Exp 2 | 30% | 40% | 20% | 10% |
| Type | Exp 0 ( ) | Exp 2 ( ) | Raw |
| factoid | 3.67 (19) | 3.75 (15) | |
| explanatory | 4.01 (15) | 4.08 (20) | |
| comparative | 3.74 (10) | 4.11 (10) | |
| procedural | 3.83 (6) | 4.10 (5) | |
| Pooled | 3.81 (50) | 3.99 (50) | |
| Type-controlled | – | 3.96 (50) |
| Model | Domain | Metric | Base | Exp 0 | Exp 1-1 | Exp 1-2 | Exp 2 | |
|---|---|---|---|---|---|---|---|---|
| EXAONE 3.5 2.4B | med | KMMLU | 43.37 | 40.80 | 40.86 | 41.33 | 41.17 | |
| EXAONE 3.5 2.4B | med | KMMLU-med | 40.28 | 36.38 | 35.80 | 36.77 | 36.47 | |
| EXAONE 3.5 2.4B | med | MMLU | 60.75 | 54.48 | 54.33 | 55.02 | 54.61 | |
| EXAONE 3.5 2.4B | fin | KMMLU | 43.37 | 40.26 | 40.22 | 41.19 | 41.00 | |
| EXAONE 3.5 2.4B | fin | KMMLU-fin | 44.83 | 38.65 | 39.39 | 40.45 | 40.33 | |
| EXAONE 3.5 2.4B | fin | MMLU | 60.75 | 55.54 | 55.12 | 55.74 | 55.74 |
| Model | Dom | Base | Exp 0 | Exp 1-1 | Exp 1-2 | Exp 2 | |
|---|---|---|---|---|---|---|---|
| EXAONE 3.5 2.4B | med | 38.84 | 34.36 | 33.40 | 35.15 | 34.44 | |
| EXAONE 3.5 2.4B | fin | 26.59 | 26.22 | 25.53 | 26.03 | 25.97 | |
| Gemma 3 4B | med | 38.26 | 35.27 | 36.10 | 36.14 | 37.39 | |
| Gemma 3 4B | fin | 28.09 | 24.09 | 24.78 | 24.72 | 25.34 | |
| Llama 3.2 3B | med | 32.86 | 33.28 | 33.20 | 34.23 | 32.95 | |
| Llama 3.2 3B | fin | 27.72 | 25.97 | 26.03 | 26.90 | 25.97 |
| Model | Dom | Base | Exp 0 | Exp 1-1 | Exp 1-2 | Exp 2 | |
|---|---|---|---|---|---|---|---|
| EXAONE 3.5 7.8B | med | 44.48 | 38.92 | 39.67 | 38.80 | 39.00 | |
| EXAONE 3.5 7.8B | fin | 30.34 | 27.59 | 27.84 | 27.90 | 28.90 | |
| Qwen 3.5 9B | med | 66.89 | 63.40 | 61.95 | 63.69 | 61.83 | |
| Qwen 3.5 9B | fin | 41.07 | 37.02 | 37.58 | 37.58 | 37.77 | |
| Gemma 3 27B | med | 62.32 | 58.84 | 59.29 | 58.88 | 58.96 | |
| Gemma 3 27B | fin | 36.33 | 35.58 | 36.58 | 35.64 | 35.71 |
| Sonnet 4.6 | Opus 4.7 | GPT-4o | GPT-5.4 | |
| Exp 0 | 4.092 | 3.464 | 4.312 | 3.352 |
| Exp 1-1 | 4.260 | 3.620 | 4.320 | 3.376 |
| Exp 1-2 | 4.116 | 3.544 | 4.352 | 3.344 |
| Exp 2 | 4.280 | 3.696 | 4.392 | 3.588 |
| Domain | sent. | WER (%) | CER (%) |
|---|---|---|---|
| medical (ko) | 100 | 21.42 | 8.26 |
| medical (ko+zh+en) | 100 | 36.54 | 15.45 |
| finance (ko+ar+fr+en) | 100 | 24.98 | 13.31 |
| weighted avg | 300 | 27.65 | 12.34 |
| Model | Cond | KMMLU ( ) | MMLU ( ) | ||
|---|---|---|---|---|---|
| EXAONE 3.5 7.8B | W-Exp0 / fin | 39.54 (0.08) | 62.35 (0.07) | ||
| EXAONE 3.5 7.8B | W-Exp0 / med | 38.69 (1.10) | 62.14 (0.30) | ||
| EXAONE 3.5 7.8B | W-Exp2 / fin | 39.50 (0.14) | 62.32 (0.29) | ||
| EXAONE 3.5 7.8B | W-Exp2 / med | 38.50 (0.01) | 61.47 (0.33) | ||
| Qwen 3.5 4B | W-Exp0 / fin | 49.34 (0.50) | 73.52 (0.03) | ||
| Qwen 3.5 4B | W-Exp0 / med | 45.63 (0.62) | 73.66 (0.06) |
| Bench | Endpoint | ||
|---|---|---|---|
| KMMLU | W-Exp0 | ||
| KMMLU | W-Exp2 | ||
| MMLU | W-Exp0 | ||
| MMLU | W-Exp2 |
| Cell | Sonnet 4.6 | Opus 4.7 | GPT-4o | GPT-5.4 | avg |
|---|---|---|---|---|---|
| Exp 0 med | |||||
| Exp 0 fin | |||||
| Exp 2 med | |||||
| Exp 2 fin |
| Cell | Seed | KMMLU-overall 95% CI | MMLU-overall 95% CI |
|---|---|---|---|
| EXAONE 3.5 7.8B / Exp 2 / fin | 1 | 39.39 [38.85, 39.89] | 62.04 [61.24, 62.83] |
| 2 | 39.89 [39.39, 40.43] | 62.05 [61.27, 62.83] | |
| EXAONE 3.5 7.8B / W-Exp0 / fin | 1 | 39.46 [38.95, 39.97] | 62.28 [61.52, 63.04] |
| 2 | 39.62 [39.11, 40.13] | 62.42 [61.66, 63.18] | |
| Llama 3.3 70B / Exp 2 / fin | 1 | 56.09 [55.58, 56.59] | 82.06 [81.46, 82.66] |
| 2 | 56.40 [55.90, 56.93] | 82.11 [81.49, 82.75] |
| Variance component | Share | |
|---|---|---|
| Between-cell (model domain) | 100.26 | |
| Residual (within-cell, seed-level) | 0 0.65 |
| Cond | Sonnet 4.6 | Opus 4.7 | GPT-4o | GPT-5.4 | 4-judge |
|---|---|---|---|---|---|
| Exp 0 | 3.75 | 3.78 | 4.53 | 3.81 | 3.97 |
| Exp 1-1 | 3.90 | 4.00 | 4.57 | 3.86 | 4.08 |
| Exp 1-2 | 3.86 | 3.94 | 4.49 | 3.92 | 4.06 |
| Exp 2 | 3.97 | 4.07 | 4.57 | 4.03 | 4.16 |
| GPT-4o | GPT-5.4 | Sonnet 4.6 | Opus 4.7 | avg | |
|---|---|---|---|---|---|
| Exp 0 | 54.0 | 12.0 | 0 7.0 | 0 5.0 | 19.5 |
| Exp 1-1 | 64.0 | 11.0 | 12.0 | 0 9.0 | 24.0 |
| Exp 1-2 | 66.0 | 0 8.0 | 0 4.0 | 0 1.0 | 19.8 |
| Exp 2 | 59.0 | 12.0 | 12.0 | 0 4.0 | 21.8 |
| strict | 60.8 | 10.8 | 0 8.8 | 0 4.8 | 21.3 |
| GPT-4o | GPT-5.4 | Sonnet 4.6 | Opus 4.7 | avg | |
| keyword | 12.0 | 24.0 | 18.5 | 19.0 | 18.4 |
| Cell | KoBEST avg | |
|---|---|---|
| EXAONE 3.5 7.8B / Exp 2 / fin | 71.56 | 0.29 |
| EXAONE 3.5 7.8B / W-Exp0 / fin | 71.34 | 1.19 |
| EXAONE 3.5 7.8B / W-Exp2 / fin | 71.60 | – |
| Llama 3.3 70B / Exp 2 / fin | 72.99 | 0.71 |
| Llama 3.3 70B / W-Exp0 / fin | 72.43 | – |
| Llama 3.3 70B / W-Exp2 / fin | 73.56 | – |
| Dimension | Exp 0 | Exp 1-1 | Exp 1-2 | Exp 2 | ||||
|---|---|---|---|---|---|---|---|---|
| faithfulness | 3.89 | 3.98 | 3.85 | 4.06 | ||||
| domain_accuracy | 3.88 | 3.98 | 3.83 | 4.05 | ||||
| question_quality | 3.42 | 3.53 | 3.60 | 3.60 | ||||
| answer_depth | 3.15 | 3.25 | 3.25 | 3.40 | ||||
| coherence | 4.70 | 4.72 | 4.67 | 4.84 |
| ID | Domain | Anonymized profile |
|---|---|---|
| M1 | Medical | Nurse, surgical specialty; 3 years clinical |
| M2 | Medical | Medical student (final year, M.D. candidate) |
| M3 | Medical | Korean medicine practitioner (private practice); 10 years clinical (Korean medicine & neuroscience focus) |
| F1 | Finance | Tax accountant (accounting & CS background); 2 years industry |
| F2 | Finance | Self-employed accountant; 30 years industry |
| F3 | Finance | Finance professional (business & MBA background); 4 years industry (financial services focus) |
| Rater | Exp 0 | Exp 1-1 | Exp 1-2 | Exp 2 | ||||
|---|---|---|---|---|---|---|---|---|
| F1 | ||||||||
| F2 | ||||||||
| F3 | ||||||||
| M1 | ||||||||
| M2 | ||||||||
| M3 |
| Pool | Exp 0 | Exp 1-1 | Exp 1-2 | Exp 2 | ||||
| All 6 raters ( ) | ||||||||
| z-normalized (all 6 pooled) | – | – | – | – | – |
| Domain | ICC(2,1) | ICC(2,3) | Krippendorff (interval) | (ordinal) |
|---|---|---|---|---|
| Medical | ||||
| Finance |
| Dimension | Human (6-rater pooled) |
|---|---|
| faithfulness | |
| domain accuracy | |
| question quality | |
| answer depth | |
| coherence | |
| overall |
Supervised Fine-Tuning
Supervised fine-tuning adapts pretrained models by learning from labeled examples, instructions, demonstrations, and task-specific trajectories.
See all 244 papers