Retrieval-augmented generation (RAG) lets large language models (LLMs) answer knowledge-intensive questions from retrieved evidence, but adapting a general-purpose RAG system to a specialized domain such as medicine is difficult when only an unlabeled corpus is available. Self-training approaches such as SimRAG address this by letting the LLM write its own question-answer pairs from domain passages and fine-tuning on the pairs that survive a round-trip consistency filter. We observe that these approaches treat every retained example identically: they never ask whether the student already knows the answer, already extracts it correctly from the retrieved context, or fails - and if so, whether the failure lies with the reader or with the retriever. To tackle this, we propose TriageRAG, a failure-aware self-training approach that probes the student on each self-generated question with and without retrieved context, triages it into one of four categories (known, solved, context-failure, retrieval-failure), and routes each category to the training signal that addresses it, including a parametric route that teaches the reader to answer when retrieval misses. Probing is cheap and repeated every round, so the training mixture follows the student, and the triage histogram doubles as a label-free diagnostic of where the domain gap lies. Experiments on 11 datasets spanning three domains with a Qwen3-4B backbone, and on CS-Bench with Qwen3-8B, demonstrate that TriageRAG outperforms a budget-matched SimRAG by 1.2-6.4 points and is the only method that recovers the closed-book level in every case where retrieval hurts the base model. A single ablation identifies the mechanism: round-trip filtering removes every training context that lacks the evidence, and a reader that never sees such contexts learns to over-trust noisy retrieval.
Figures & tables
Figure 1: Overview of TriageRAG. Each round, the student synthesizes (q,a,d) triples from the unlabeled corpus (span, multiple-choice and claim formats; only self-answerable pairs are kept), is probed closed-book ( pcb ) and open-book on the top- k retrieved passages ( pob , evidence hit h ), and each pair is triaged by the conditions shown and routed to a training signal; the routed mixture is used for LoRA fine-tuning and the updated student re-probes the pool in the next round. The retriever is frozen in our main configuration (Section 3.4 ).
Datasets
PubMedQA
BioASQ
MedQA
MedMCQA
MMLU-Med
MedXpertQA
Avg.
# questions
500
618
1,273
4,183
1,089
2,450
—
Backbone: Qwen3-4B-Instruct
Backbone (closed-book)
31.00
77.67
60.49
54.24
76.03
14.53
52.33
Backbone + RAG
68.20
92.07
61.35
56.94
76.49
14.82
61.65
Stage-I + RAG
82.00
92.56
61.12
59.22
75.02
13.84
63.96
SimRAG ‡
82.60
91.26
59.86
59.62
74.93
13.76
63.67
Table 1: Results in the medical domain (accuracy, %). All methods use RAG with the same bge-base retriever and top-5 passages at inference. Bold and underline mark the best and second-best result per column. ‡ : our own implementation with the same backbone, pool and budget. All Stage-II methods share the Stage-I model, the 40,540-tuple self-generated pool, a budget of 12,000 examples per round and 2,000 general replay examples. The notations are the same for the following tables.
Table 3
Method
Medical
CS
Science
PubMedQA
BioASQ
MedQA
MedMCQA
MMLU-Med
MedXpertQA
Avg.
CS Avg.
Sci. Avg.
TriageRAG
81.60
91.26
63.08
61.37
78.33
13.39
64.84
68.19
87.62
w/o parametric route
82.60
92.07
60.17
59.22
75.30
12.90
63.71
61.91
84.30
w/o triage (= random selection)
77.60
92.88
61.51
61.92
77.13
13.84
64.15
67.58
87.51
w/o triage, w/ round-trip filter (= SimRAG)
82.60
91.26
59.86
59.62
74.93
13.76
63.67
61.83
85.74
w/ retriever tuning
61.80
83.66
60.64
59.79
76.49
12.45
59.14
65.89
87.44
Table 4: Ablations of TriageRAG (Qwen3-4B, round 1). Medical results are per dataset; for CS and science we report the average. “w/o parametric route” puts the gold passage in the context of the retrieval-failure examples instead of training the reader to answer despite an unhelpful context (retriever frozen); “w/o triage” selects uniformly from the same pool at the same budget; “w/ retriever tuning” fine-tunes bge-base on the retrieval-failure pairs and re-indexes.
Figure 2: Top: triage of the self-generated pool by the Qwen3-4B student across rounds (shares in %). Bottom: average accuracy of each round-1 Stage-II mixture against the share of its training examples whose context does not contain the evidence; mixtures with at most 2% such examples (SimRAG, the ablation without the parametric route) stay at or below Stage-I, those with 10% or more reach within a point of the closed-book level or above.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Domain / backbone
Method
seed 0
seed 1
seed 2
mean ± sd
CS-Bench, Qwen3-4B
SimRAG ‡
61.83
62.25
63.01
62.36 ± 0.59
Random selection
67.58
65.47
67.58
66.88 ± 1.22
TriageRAG w/o parametric route
61.91
61.31
61.84
61.69 ± 0.33
TriageRAG
68.19
67.09
67.13
67.47 ± 0.62
Medical, Qwen3-4B
SimRAG ‡
63.67
62.51
62.92
63.03 ± 0.59
Random selection
64.15
65.04
63.78
64.32 ± 0.65
Appendix
Table 5: Average accuracy (%) of the round-1 Stage-II cells over three training seeds (seed 0 is the run reported in Tables 1 – 3 ). Seeds change the mixture sample and the LoRA initialisation; the Stage-I model, pool and round-1 probe are shared. On every seed TriageRAG beats SimRAG and the ablation without the parametric route; TriageRAG and random selection, and likewise the ablation and SimRAG, are within two points of each other.
Pool
Size
Known
Solved
Ctx.F.
Ret.F.
Hit@5
R@5
Medical
40,540
40.9
24.2
5.7
29.2
53.4
48.1
span
19,553
4.5
36.1
11.0
48.4
MC
11,775
75.9
8.8
0.1
15.2
claim
9,212
73.3
18.6
1.8
6.3
Science
32,686
56.6
30.3
6.0
7.0
84.2
77.9
span
18,622
31.5
47.7
9.9
11.0
Appendix
Table 6: Round-1 triage of the self-generated pools (Qwen3-4B Stage-I student, frozen retriever; shares in %). Hit@5: fraction of questions whose evidence is in the top-5 passages ( h=1 ); R@5: fraction whose source passage is among them.
Index at test time
PubMedQA
BioASQ
MedQA
Base bge-base
81.4
90.9
62.1
Tuned on round-1 failures
61.8
83.7
60.6
Recall@5 on the pool
0.48 (base) → 0.34 (tuned)
Appendix
Table 7: Controlled comparison holding one round-1 medical reader fixed (Qwen3-4B; the round-1 reader of the retriever-tuning run, trained on the same round-1 mixture as TriageRAG in a separate run, hence the small differences from Table 1 ): fine-tuning the retriever on retrieval-failure pairs degrades both recall of the source passage and downstream accuracy.
Figure 3: Per-dataset accuracy of the Stage-II variants in the medical domain (Qwen3-4B, round 1). Note the different y -ranges. The literature yes/no datasets (PubMedQA, BioASQ) move by at most 1.3 points under SimRAG, the ablation without the parametric route and TriageRAG; random selection ( −4.4 ) and RAFT-style ( −3.0 ) lower PubMedQA, and retriever tuning damages both. The exam-style datasets (MedQA, MedMCQA, MMLU-Med) carry the gains of TriageRAG over Stage-I.
known ( pcb=1.00 , pob=1.00 , source passage retrieved at rank 1) → 5% consolidation share
Question / Answer
Where does the lymphatic system filter the lymph? A: lymph nodes
Source (Textbooks)
Physiology, Levy. In addition to returning fluid and protein to the vascular bed, the lymphatic system filters the lymph at the lymph nodes and removes foreign particles such as bacteria.
What was the 10-year survival rate for relapsing polychondritis? A: 55%
Source (Textbooks)
Internal Medicine, Harrison. The course of relapsing polychondritis is highly variable. … In one study, the 5-year estimated survival rate was 74% and the 10-year survival rate was 55% .
Appendix
Table 8: Case studies: one self-generated medical question per triage category with its probe outcomes ( pcb , pob , evidence hit) and the route it receives. Green marks evidence present in the retrieved passages; red marks the answer-bearing sentence of the source passage when it was not retrieved.
Figure 4: Average accuracy per round (Qwen3-4B). Re-probing the updated student changes the mixture of TriageRAG every round; SimRAG’s selection does not depend on the student, so its later rounds re-train on the same distribution.
Domain / backbone
Method
Round 1
Round 2
Round 3
Medical / Qwen3-4B-Instruct
TriageRAG
64.84
64.74
64.50
TriageRAG w/ retriever tuning
59.14
64.41
58.75
SimRAG ‡ iterated
63.67
63.45
63.01
Science / Qwen3-4B-Instruct
TriageRAG
87.62
87.99
87.50
TriageRAG w/ retriever tuning
87.44
—
—
SimRAG ‡ iterated
85.74
—
—
Appendix
Table 9: Average accuracy per round for the iterated methods.