Practitioners building training corpora with a pipeline of language models keep a preprocessing or refinement (post-process) stage when switching it on raises the corpus quality. We ablate a production pipeline that turns conference speech into question-answer pairs. The preprocessing stage rewrites the transcript, and the refinement stage filters the generated pairs. Switching each on and off gives four corpora from the same 40 Korean medical and finance sessions. On the corpus itself, quality rises as the stages switch on. Four LLM judges and six human domain experts rank the four corpora the same way, and the fully refined corpus leads by +0.18 and +0.22 on a 1-5 scale. We then ask whether that gain carries downstream. We supply each corpus to six models at inference and measure retrieval accuracy. The ranking does not hold. Enabling the transcript rewrite lowers retrieval accuracy by 0.061, the only stage effect that survives correction for multiple comparisons. That stage also reads the whole session, so it almost doubles the cost of one. With both stages on, accuracy does not change. An equivalence test we declared in advance confirms this. The corpus itself is clearly valuable. It lifts retrieval accuracy from 0.231 to 0.748 in every model and domain we test. More than a third of its pairs carry knowledge that neither of two frontier models produces on its own. On our data, the null does not come from weak corpora. A rising score on a rubric of this kind did not predict what the corpus holds for a model.
Figures & tables
Figure 1: The same four corpora, measured two ways. Left: two scoring sources, each shifted to start at its value for Exp 0, the unrefined corpus, so they share an axis. Right: two downstream instruments, in percent change from Exp 0, with 95% intervals. Retrieval intervals come from the six models crossed with the two domains (§ 4.2 ). Novelty intervals come from the pairs. Note the different vertical scales.
Exp 0
Exp 1-1
Exp 1-2
Exp 2
corpus quality , 1–5
4 LLM judges
3.81
3.90
3.84
3.99
6 human experts
3.36
3.46
3.46
3.58
downstream
Retrieval accuracy
0.767
0.706
0.785
0.748
Effectively novel, %
35.6
36.1
36.9
36.5
Table 1: The dissociation in one place. Best value in each row is bold. Novelty is the strict count, pairs that both frontier models miss (§ 4.3 ).
Figure 2: Retrieval accuracy with each corpus, on the 115 questions that pass the wording filter. Grey bars are the two controls, dark bars the four corpora. The shaded band marks the range the four corpora occupy.
Contrast
Estimate
p
anchoring
Exp 2 vs. no corpus
+0.517
<0.0001∗
Exp 2 vs. mismatched
+0.648
<0.0001∗
ablation
Exp 1-1 vs. Exp 0
−0.061
0.0015∗
Exp 1-2 vs. Exp 0
+0.018
0.025
Table 2: All nine planned contrasts. An asterisk marks the three that clear 0.05/9=0.0056 . Retrieval estimates are accuracy points on the cell unit; novelty estimates are percentage points over pairs (App. F ).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The pipeline and the four ablation conditions. Bold boxes are the two stages we switch on and off.
Fate of an input pair
0 → 1-2
1-1 → 2
Unchanged, character for character
67.8%
64.2%
Kept, answer reworded
0.7%
1.0%
Kept, answer expanded
0.2%
0.4%
Kept, a fact changed
0.0%
0.1%
Dropped or rewritten
31.3%
34.3%
Appendix
Table 3: What Phase 2 does to the pairs entering it, matched on exact question text. Columns name the two contrasts that toggle Phase 2 alone.
Dimension
What the rater judges
Faithfulness
The answer is factually right and invents nothing.
Domain accuracy
The medical or finance content is right.
Question quality
The question is worth learning from. Plain recall scores 1 – 2 , reasoning or synthesis 4 – 5 .
Answer depth
The answer lists facts ( 1 – 2 ) or reasons through cause, comparison, or synthesis ( 4 – 5 ).
Coherence
The answer addresses the question it was asked.
Appendix
Table 4: The quality rubric. Each dimension runs 1 to 5 , where 1 is very poor, 3 acceptable, and 5 excellent. Raters are told not to reward length, since a long recall answer is still shallow. A sixth dimension checks whether the difficulty and type labels match the content. It rates the labels rather than the pair, so it stays out of the average.
Missed by
Exp 0
Exp 1-1
Exp 1-2
Exp 2
sonnet-5
45.0
44.3
46.0
44.6
gpt-5.6
46.1
45.9
47.1
47.3
either model
55.6
54.2
56.1
55.3
both models
35.6
36.1
36.9
36.5
Appendix
Table 5: Percent of pairs whose core content the closed-book answer misses. Model names abbreviate claude-sonnet-5 and gpt-5.6-sol . Every row already requires the grounded and correct axes, and a pair clears those only when both models say yes. The last two rows are the lenient and strict counts of § 4.3 .
Threshold
Qs
1-1 − 0
2 − 0
Equiv.
4 pp
52
−0.040∗
−0.029∗
no
6 pp
68
−0.058∗
−0.023∗
yes
8 pp
95
−0.070∗
−0.031∗
no
10.9 pp
115
−0.061∗
−0.020
yes
14 pp
125
−0.050∗
−0.014
yes
18 pp
138
−0.030
+0.005
yes
Appendix
Table 6: Both ablation contrasts as the wording threshold varies. Columns are the threshold, the number of questions kept out of 163, the Exp 1-1 and Exp 2 contrasts against Exp 0, and whether the 90% interval for Exp 2 falls inside the ±0.045 bound we registered in advance (App. F ). An asterisk marks a 95% interval that excludes zero. The bold row is the one we use.
Contrast
Cell means
Mixed effects
Exp 1-1 vs. Exp 0
−0.061[−0.090,−0.032]
−0.058[−0.083,−0.034]
Exp 1-2 vs. Exp 0
+0.018[+0.004,+0.031]
+0.019[−0.005,+0.043]
Exp 2 vs. Exp 0
−0.020[−0.042,+0.003]
−0.018[−0.042,+0.006]
Appendix
Table 7: The three ablation contrasts under both analyses, with 95% intervals. Cell means resample the 12 model-by-domain cells. The mixed-effects fit carries random effects for session, question, and model over the 2,760 scored answers.
Contrast
Shared
All pairs
Non-shared
Exp 0 → Exp 1-2
69%
+1.4
+4.2
Exp 1-1 → Exp 2
66%
+0.4
+2.2
Appendix
Table 8: Effective novelty under the strict count, in percentage points. Shared is the share of pairs whose question appears on both sides, which runs slightly above the share left unchanged character for character (Table 3 ). The last two columns score all pairs and then only the pairs one side does not share.
Source
Exp 0
Exp 1-1
Exp 1-2
Exp 2
4 LLM judges
3.81
3.90
3.84
3.99
6 human experts
3.36
3.46
3.46
3.58
2 frontier judges
3.41
3.50
3.45
3.65
Appendix
Table 9: The first two rows repeat Table 1 . The third is the frontier-tier subset of the LLM panel, which shows the ranking without the weaker two judges.