cs.AISep 28, 2026
SaveARCagent: An Adaptive Retrieval Calibration Agent for Clinical Question Answering
Abstract
In diseases where clinical guidelines are incomplete, contested, or mutually contradictory, knowledge completeness and dynamic conflict-aware synthesis are two safety-critical properties that standard Retrieval-Augmented Generation systems do not provide. Therefore, we present \sysname, an adaptive retrieval calibration clinical question-answering agent for ME/CFS, a disease where diagnostic frameworks coexist and major guidelines actively contradict each other on treatment. ARCagent contributes three components. First, a 1,706-chunk, 10-source knowledge base with a structured inter-guideline conflict registry spanning all active ME/CFS diagnostic frameworks. Second, a conflict-aware retrieval calibration pipeline that re-ranks retrieved evidence using query-specific focus and conflict signals. Third, a benchmark scored by LLM-as-Judge, avoiding systematic underestimation averaging 10.1 percentage points caused by keyword matching. ARCagent achieves 95.3%, outperforming all base LLMs. Code is available at https://github.com/Yukyin/ARCagent.
Figures & tables
Figure 1 . Overview of knowledge base and ARCagent construction. (a) Knowledge base construction: sources are collected and chunked into guideline, empirical, genomic, and observational knowledge, with a conflict registry annotating conflict zones. (b) ARCagent construction: a query is processed into a focus profile, resolved through conflict-aware retrieval, and answered via dual-mode response generation.
| Source | Organisation | Chunks | Words |
| IOM/NAM 2015 | US NAM | 929 | 109,145 |
| CDC ME/CFS | US CDC | 215 | 20,529 |
| NICE NG206 | UK NICE | 214 | 24,285 |
| Clinician Coalition | ME/CFS CC, USA | 102 | 35,700 |
| IACFS/ME Primer | Friedberg et al. | 82 | 24,464 |
| IQWiG N21-01 | IQWiG, Germany | 58 | 17,219 |
Table 1 . MECFS-KB source composition. IQWiG N21-01 chunks are annotated as conflicting with NICE NG206.
| Model | Auto | Diag | Navi | PEM | Scop | Conf | Spec | RevH | Over |
|---|---|---|---|---|---|---|---|---|---|
| Domain-specific medical LLMs | |||||||||
| BioMedLM | 76.3 | 77.8 | 76.7 | 72.6 | 42.6 | 60.8 | 73.5 | 68.3 | 68.6 |
| HuatuoGPT-II | 83.7 | 85.2 | 80.6 | 76.7 | 50.3 | 69.4 | 77.1 | 76.2 | 74.9 |
| Med-PaLM 2 | 85.1 | 83.5 | 82.7 | 81.2 | 54.6 | 73.1 | 80.4 | 81.7 | 77.8 |
| MedAlpaca | 70.8 | 73.2 | 71.3 | 69.4 | 39.5 | 54.6 | 71.2 | 63.3 | 64.2 |
| General-purpose LLMs | |||||||||
Table 2 . Full benchmark results. Auto denotes autonomic dysfunction and comorbidity queries. Diag denotes diagnostic criteria queries. Navi denotes care-navigation queries. PEM denotes post-exertional malaise management queries. Scop denotes out-of-scope refusal queries. Conf denotes inter-guideline conflict queries. Spec denotes guideline-specific detail queries. RevH denotes revision history queries. The lower block shows domain-specific medical LLMs. The ARCagent rows compare retrieval-first and adaptive retrieval calibration variants. Bold indicates per-category best.
| Model | LLM | Keyword | Human |
|---|---|---|---|
| Domain-specific medical LLMs | |||
| BioMedLM | 68.6 | 55.2 | 65.2 |
| HuatuoGPT-II | 74.9 | 60.1 | 75.3 |
| Med-PaLM 2 | 77.8 | 61.3 | 75.4 |
| MedAlpaca | 64.2 | 52.8 | 65.1 |
| General-purpose LLMs | |||
Table 3 . Comparison of three scoring methods. LLM refers to automated LLM-as-Judge scoring with llama3.1-8b. Keyword refers to keyword matching. Human refers to blinded ratings by three medical graduate student raters. The row shows per-column deviation from the Human-as-Judge average.
| Configuration | Chun | Auto | Diag | Navi | PEM | Scop | Conf | Spec | RevH | Overall |
|---|---|---|---|---|---|---|---|---|---|---|
| Knowledge type ablation | ||||||||||
| w/o guideline knowledge | 18 | 95.8 | 95.4 | 96.1 | 95.3 | 77.3 | 88.5 | 94.2 | 92.8 | 91.9 |
| w/o empirical knowledge | 1,688 | 98.3 | 97.8 | 97.7 | 98.1 | 80.1 | 90.5 | 95.7 | 94.6 | 94.1 |
| Source-level ablation | ||||||||||
| w/o diagnostic frameworks | 689 | 96.1 | 95.8 | 96.6 | 95.7 | 79.8 | 89.2 | 94.5 | 93.4 | 92.6 |
| w/o treatment guidelines | 1,492 | 97.2 | 96.5 | 97.3 | 97.4 | 77.6 | 91.5 | 95.1 | 95.8 | 93.6 |
Table 4 . Unified knowledge ablation using Grok-4.20 base model. The upper block removes one knowledge type and the lower block removes one guideline source category. Column labels match Table 2 . Full KB row shows the baseline. All values are LLM-as-Judge scores reported as percentages without the percent symbol.
| System | Response excerpt | Conflict surfaced |
|---|---|---|
| Query. “Is graded exercise therapy still recommended for ME/CFS?” (clinician mode) | ||
| Grok-4.20 | No, graded exercise therapy (GET) is not recommended for ME/CFS by major clinical guidelines. UK NICE Guidelines explicitly advise against GET, defining it as fixed incremental activity increases from a baseline, and state that such programs should not be offered due to reports of harm and lack of evidence for benefit, recommending pacing within energy limits to avoid post-exertional malaise (PEM) instead. US CDC similarly notes that standard exercise recommendations for healthy people can be harmful for ME/CFS patients and recommends pacing to manage energy and prevent PEM flare-ups, reflecting a broader expert shift away from the old deconditioning model toward recognising ME/CFS as a biological illness with PEM as a core feature. | No |
| ARCagent | Graded exercise therapy is no longer recommended for ME/CFS under current major guidelines. NICE NG206 [1] explicitly contraindicates GET on the grounds that the deconditioning model underpinning it has not been validated and that progressive exercise can provoke post-exertional malaise. Conflict detected. IQWiG N21-01 [2] reports low-certainty short-term benefit for structured activity in some subgroups, diverging from NICE NG206 based on a different appraisal of the PACE trial. Current North American and UK consensus supports individualised activity pacing within each patient’s energy envelope. [1][3] | Yes, NICE NG206 vs IQWiG N21-01 |
Table 5 . Qualitative comparison on the GET conflict query in clinician mode. Grok-4.20 without retrieval generates a response that does not surface the NICE NG206 prohibition or flag any inter-guideline disagreement. ARCagent detects the conflict intent, retrieves passages from both NICE NG206 and IQWiG N21-01, and presents both stances with inline source attribution.
Figure 2 . Case A1. GET conflict query in clinician mode. The response cites CCC 2003 ( Carruthers et al., 2003 ) and NICE NG206 ( National Institute for Health and Care Excellence, 2021 ) , flags the conflict with IQWiG N21-01 ( Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, 2023 ) , and recommends individualised activity pacing. The source panel shows the two retrieved guideline chunks.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3 . Case A2. Diagnostic criteria conflict in clinician mode. Four sources are retrieved and two inter-framework disagreements are surfaced. The source panel shows retrieved chunks from IOM 2015 ( Institute of Medicine, 2015 ) , ICC 2012 ( Carruthers et al., 2011 ) , and CCC 2003 ( Carruthers et al., 2003 ) .
Figure 4 . Case A3. CBT conflict query in clinician mode. The response presents the NICE NG206 ( National Institute for Health and Care Excellence, 2021 ) and IQWiG N21-01 ( Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, 2023 ) disagreement with source attribution. Three retrieved chunks are shown in the right panel.
Figure 5 . Case A4. Scope boundary enforcement on a numeric threshold query. The system correctly reports no validated cut-off and cites three sources. The source panel confirms that none of the retrieved chunks contains a numeric threshold.
Figure 6 . Case A5. Disease naming query in clinician mode. Six sources are retrieved. The response addresses historical, geographic, and diagnostic significance of naming differences using clinical framing.
Figure 7 . Case A6. Same disease naming query as Case A5 in patient mode. The same three sources are retrieved but the response uses plain language and omits guideline version specifics. Comparing A5 and A6 directly illustrates the dual-mode register shift.