Corpus Quality Improves, Retrieval Utility Does Not: An Ablation of Preprocessing and Refinement in a Speech-to-QA Pipeline

Wonsup Shin; Jingu Kim

cs.CLOct 20, 2026

Corpus Quality Improves, Retrieval Utility Does Not: An Ablation of Preprocessing and Refinement in a Speech-to-QA Pipeline

Authors: Wonsup Shin, Jingu Kim

Organizations: Flitto

Abstract

Practitioners building training corpora with a pipeline of language models keep a preprocessing or refinement (post-process) stage when switching it on raises the corpus quality. We ablate a production pipeline that turns conference speech into question-answer pairs. The preprocessing stage rewrites the transcript, and the refinement stage filters the generated pairs. Switching each on and off gives four corpora from the same 40 Korean medical and finance sessions. On the corpus itself, quality rises as the stages switch on. Four LLM judges and six human domain experts rank the four corpora the same way, and the fully refined corpus leads by +0.18 and +0.22 on a 1-5 scale. We then ask whether that gain carries downstream. We supply each corpus to six models at inference and measure retrieval accuracy. The ranking does not hold. Enabling the transcript rewrite lowers retrieval accuracy by 0.061, the only stage effect that survives correction for multiple comparisons. That stage also reads the whole session, so it almost doubles the cost of one. With both stages on, accuracy does not change. An equivalence test we declared in advance confirms this. The corpus itself is clearly valuable. It lifts retrieval accuracy from 0.231 to 0.748 in every model and domain we test. More than a third of its pairs carry knowledge that neither of two frontier models produces on its own. On our data, the null does not come from weak corpora. A rising score on a rubric of this kind did not predict what the corpus holds for a model.

Figures & tables

Figure 1: The same four corpora, measured two ways. Left: two scoring sources, each shifted to start at its value for Exp 0, the unrefined corpus, so they share an axis. Right: two downstream instruments, in percent change from Exp 0, with 95% intervals. Retrieval intervals come from the six models crossed with the two domains (§ 4.2 ). Novelty intervals come from the pairs. Note the different vertical scales.

corpus quality , 1–5
	Exp 0	Exp 1-1	Exp 1-2	Exp 2
4 LLM judges	3.81	3.90	3.84	3.99
6 human experts	3.36	3.46	3.46	3.58
downstream
Retrieval accuracy	0.767	0.706	0.785	0.748
Effectively novel, %	35.6	36.1	36.9	36.5

Table 1: The dissociation in one place. Best value in each row is bold. Novelty is the strict count, pairs that both frontier models miss (§ 4.3 ).

Figure 2: Retrieval accuracy with each corpus, on the 115 questions that pass the wording filter. Grey bars are the two controls, dark bars the four corpora. The shaded band marks the range the four corpora occupy.

anchoring
Contrast	Estimate	$p$
Exp 2 vs. no corpus	$+0.517$	$<0.0001^{*}$
Exp 2 vs. mismatched	$+0.648$	$<0.0001^{*}$
ablation
Exp 1-1 vs. Exp 0	$-0.061$	$0.0015^{*}$
Exp 1-2 vs. Exp 0	$+0.018$	$0.025$

Table 2: All nine planned contrasts. An asterisk marks the three that clear

0.05/9=0.0056

. Retrieval estimates are accuracy points on the cell unit; novelty estimates are percentage points over pairs (App. F ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: The pipeline and the four ablation conditions. Bold boxes are the two stages we switch on and off.

Fate of an input pair	0 $\to$ 1-2	1-1 $\to$ 2
Unchanged, character for character	67.8%	64.2%
Kept, answer reworded	0.7%	1.0%
Kept, answer expanded	0.2%	0.4%
Kept, a fact changed	0.0%	0.1%
Dropped or rewritten	31.3%	34.3%

Appendix

Table 3: What Phase 2 does to the pairs entering it, matched on exact question text. Columns name the two contrasts that toggle Phase 2 alone.

Dimension	What the rater judges
Faithfulness	The answer is factually right and invents nothing.
Domain accuracy	The medical or finance content is right.
Question quality	The question is worth learning from. Plain recall scores $1$ – $2$ , reasoning or synthesis $4$ – $5$ .
Answer depth	The answer lists facts ( $1$ – $2$ ) or reasons through cause, comparison, or synthesis ( $4$ – $5$ ).
Coherence	The answer addresses the question it was asked.

Appendix

Table 4: The quality rubric. Each dimension runs

1

5

, where

1

is very poor,

3

acceptable, and

5

excellent. Raters are told not to reward length, since a long recall answer is still shallow. A sixth dimension checks whether the difficulty and type labels match the content. It rates the labels rather than the pair, so it stays out of the average.

Missed by	Exp 0	Exp 1-1	Exp 1-2	Exp 2
sonnet-5	45.0	44.3	46.0	44.6
gpt-5.6	46.1	45.9	47.1	47.3
either model	55.6	54.2	56.1	55.3
both models	35.6	36.1	36.9	36.5

Appendix

Table 5: Percent of pairs whose core content the closed-book answer misses. Model names abbreviate claude-sonnet-5 and gpt-5.6-sol . Every row already requires the grounded and correct axes, and a pair clears those only when both models say yes. The last two rows are the lenient and strict counts of § 4.3 .

Threshold	Qs	1-1 $-$ 0	2 $-$ 0	Equiv.
$4$ pp	52	$-0.040^{*}$	$-0.029^{*}$	no
$6$ pp	68	$-0.058^{*}$	$-0.023^{*}$	yes
$8$ pp	95	$-0.070^{*}$	$-0.031^{*}$	no
$10.9$ pp	115	$\mathbf{-0.061^{*}}$	$\mathbf{-0.020}$	yes
$14$ pp	125	$-0.050^{*}$	$-0.014$	yes
$18$ pp	138	$-0.030$	$+0.005$	yes

Appendix

Table 6: Both ablation contrasts as the wording threshold varies. Columns are the threshold, the number of questions kept out of 163, the Exp 1-1 and Exp 2 contrasts against Exp 0, and whether the

90\%

interval for Exp 2 falls inside the

\pm 0.045

bound we registered in advance (App. F ). An asterisk marks a

95\%

interval that excludes zero. The bold row is the one we use.

Contrast	Cell means	Mixed effects
Exp 1-1 vs. Exp 0	$-0.061\ [-0.090,-0.032]$	$-0.058\ [-0.083,-0.034]$
Exp 1-2 vs. Exp 0	$+0.018\ [+0.004,+0.031]$	$+0.019\ [-0.005,+0.043]$
Exp 2 vs. Exp 0	$-0.020\ [-0.042,+0.003]$	$-0.018\ [-0.042,+0.006]$

Appendix

Table 7: The three ablation contrasts under both analyses, with

95\%

intervals. Cell means resample the 12 model-by-domain cells. The mixed-effects fit carries random effects for session, question, and model over the

2{,}760

scored answers.

Contrast	Shared	All pairs	Non-shared
Exp 0 $\to$ Exp 1-2	$69\%$	$+1.4$	$+4.2$
Exp 1-1 $\to$ Exp 2	$66\%$	$+0.4$	$+2.2$

Appendix

Table 8: Effective novelty under the strict count, in percentage points. Shared is the share of pairs whose question appears on both sides, which runs slightly above the share left unchanged character for character (Table 3 ). The last two columns score all pairs and then only the pairs one side does not share.

Source	Exp 0	Exp 1-1	Exp 1-2	Exp 2
4 LLM judges	3.81	3.90	3.84	3.99
6 human experts	3.36	3.46	3.46	3.58
2 frontier judges	3.41	3.50	3.45	3.65

Appendix

Table 9: The first two rows repeat Table 1 . The third is the frontier-tier subset of the LLM panel, which shows the ranking without the weaker two judges.

Training Data Curation

Training data curation improves the quality, diversity, organization, and reliability of datasets used to train machine-learning models.

See all 156 papers

Aug 24, 2026

Better Retrieval, Worse Robustness: How Multi-hop RAG Amplifies Upstream ASR Errors

Speech-based applications pass spoken queries through automatic speech recognition (ASR) before any retrieval module, so ASR errors enter the pipeline as a fixed upstream constraint. We empirically test whether two extensions to standard retrieval-augmented generation (RAG), entity-graph linking and iterative reformulation, absorb or amplify these errors. Using four English accents synthesized through neural TTS, we evaluate four RAG configurations on three multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA and MuSiQue) against a clean-text oracle. Although the structurally richer configurations generally retain higher absolute F1 under ASR input, both extensions amplify the error: the F1 gap from clean text to the highest-WER accent is 36-67% larger under their combination than under naive dense retrieval, on all three benchmarks. The dominant failure mode is corruption of one or more query entities, accounting for 87-96% of degradation cases on 2WikiMultiHopQA across all four methods. Two lightweight surface-form mitigations leave most of the gap intact, indicating that downstream retrieval structure amplifies remaining entity errors. We release code and data at https://github.com/Continuum-AI-Corp/spoken-multihop-rag .

Zhenghua Bao

Continuum AI

RAG Robustness Retrieval-Augmented Generation Graph Retrieval-Augmented Generation Multi-Hop QA

Ask PDF

Oct 6, 2026

Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training

Automatic assessment of communication quality in dyadic counseling conversations is bottlenecked by data: expert-rated corpora are small and expensive to grow. We study cross-domain transfer of expert overall-impression prediction across three German corpora of simulated counseling (two general-practice medical, one school-related parent-teacher;

n=195

expert-rated sessions, one corpus after scale equating). Training on the other domains beats training in-domain: leave-one-domain-out transfer reaches nested Spearman

ρ= 0.54

against

\le 0.48

within the target domain, a paired session-level gap of

+0.15

that holds at

+0.12

when the training-set sizes are matched, so it is not simply data volume. The decisive features are session-level construct scores from small open-weight LLMs reading the two-speaker transcript, with the constructs largely derived from the experts' rating instruments: the instrument-derived battery lifts a single judge from

0.32

0.41

over generic dialogue qualities, judges from three model families ensemble to

0.51

language-only, and a nonverbal-dyadic block adds

+0.03

more, not separable from noise at this sample size. We also price the recording setup: one corpus lost its per-speaker audio, 16% of its diarised segments carry the wrong speaker, and repair is worth

+0.07

there. At practically attainable corpus sizes, the expert's overall impression is carried by what is said, and by other communication programs' data more than by one's own.

Tobias Hallmen, Elisabeth André

Chair for Human-Centered Artificial Intelligence University of Augsburg

Mental Health Counseling Cross-Domain Transfer Learning Automated Evaluation

Ask PDF

Oct 6, 2026

Quality-Aware Self-Correcting Speech Translation on an Edge Device

We present a fully offline speech-to-speech translation pipeline that runs on a Jetson Nano (4 GB) and corrects its own weak translations without retraining. A Whisper-tiny ASR feeds an Opus-MT translator; multilingual BERT cosine similarity acts as a Quality Estimation (QE) gate, triggering a secondary-pass when confidence falls below a pre-defined threshold

τ

. We compare three correction methods: QE reranking (M1), Minimum Bayes-Risk decoding (M2), and constrained beam search (M3). On 1,012 FLORES-200 sentences (English-Spanish), M2 at

τ=0.90

produces statistically significant improvements over greedy decoding on BLEU (+0.67, p<0.001), ChrF (+0.51, p<0.001), and COMET (+0.0020 at N=3, p=0.002); M1 yields no significant gains, and M3 is significantly worse than baseline (p>0.99). Our central finding is that QE functions effectively as a gate but poorly as a ranker: removing the QE model from candidate selection (M1

\to

M2) does not hurt quality and frees 680 MB from the critical path. Using a gain-to-edit ratio adapted from the post-editing-effort literature, we further show that smaller candidate pools (N=3) yield more surgical corrections with better semantic adequacy, while larger pools (N=10) maximise lexical reward. We release the system and demonstrate live translation across six language pairs.

Zubair Ajmal Farooq, Diptesh Kanojia

School of CS and EE, University of Surrey, United Kingdom · Institute for People-Centred AI, University of Surrey, United Kingdom

Speech Translation Machine Translation

Ask PDF

Abstract

Training Data Curation

Explore similar work

Better Retrieval, Worse Robustness: How Multi-hop RAG Amplifies Upstream ASR Errors

Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training

Quality-Aware Self-Correcting Speech Translation on an Edge Device