When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models

Yubo Li; Ramayya Krishnan; Rema Padman

cs.IRMar 23, 2026

When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models

Authors: Yubo Li, Ramayya Krishnan, Rema Padman

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA

Abstract

Patient education materials for solid-organ transplantation vary substantially across U.S. centers, yet no systematic method exists to quantify this heterogeneity at scale. We introduce a framework that grounds the same patient questions in different centers' handbooks using retrieval-augmented language models and compares the resulting answers using a five-label consistency taxonomy. Applied to 102 handbooks from 23 centers and 1,115 benchmark questions, the framework quantifies heterogeneity across four dimensions: question, topic, organ, and center. We find that 20.8% of non-absent pairwise comparisons exhibit clinically meaningful divergence, concentrated in condition monitoring and lifestyle topics. Coverage gaps are even more prominent: 96.2% of question-handbook pairs miss relevant content, with reproductive health at 95.1% absence. Center-level divergence profiles are stable and interpretable, where heterogeneity reflects systematic institutional differences, likely due to patient diversity. These findings expose an information gap in transplant patient education materials, with document-grounded medical question answering highlighting opportunities for content improvement.

Figures & tables

Figure 1: Center distribution map for the handbook corpus. The 23 contributing U.S. transplant centers are geographically dispersed across the country.

	Heart	Kidney	Liver	Lung	Pancreas
Handbooks	26	22	17	26	11
Centers	17	14	11	15	8
Pre-transplant	10	8	5	10	4
Post-transplant	11	10	4	11	3
Combined	5	4	8	5	4

Table 1: Summary of the transplant handbook corpus by organ type. Centers indicates the number of distinct institutions contributing handbooks for each organ.

Panel of Figure 2 — Figure 2: Distribution of benchmark questions by organ type and topic category.

Figure 3: Overview of the experimental pipeline.

Label	Definition	Example
Absent	One or both answers indicate the handbook does not address the topic.	Center A provides dietary guidance; Center B’s handbook contains no relevant section.
Consistent	Both answers provide the same clinical recommendation.	Both centers advise avoiding grapefruit due to tacrolimus interactions.
Complementary	Clinically compatible but differing in detail or scope.	Center A lists side effects; Center B additionally describes management strategies.
Divergent	Substantive, clinically meaningful differences (e.g., different thresholds or timelines).	Center A recommends exercise at 6 weeks post-transplant; Center B at 8–12 weeks.
Contradictory	Directly opposing clinical guidance.	Center A allows ABO-incompatible live donors; Center B states they cannot proceed.

Table 2: Five-label taxonomy for pairwise comparison of center-specific answers.

Label	Count	% of non-absent
Absent	1,704,242	—
Complementary	44,870	66.0%
Divergent	14,132	20.8%
Consistent	8,874	13.0%
Contradictory	143	0.2%
Total non-absent	68,019	100%

Table 3: Global pairwise label distribution across all 1,115 questions. Percentages in the right column are computed over non-absent pairs only.

By organ type
Group	Questions	Pairs	Absence rate
General	311	31,722	0.905
Heart	137	3,562	0.818
Kidney	196	4,312	0.780
Liver	164	2,788	0.805
Lung	153	3,978	0.725

Table 4: Absence rates by organ type and selected topic categories. The absence rate is the fraction of question–handbook pairs in which the handbook did not address the question (Eq. 1 ).

Organ	$Q_{\text{total}}$	$Q_{\text{active}}$	$R_{\text{div}}$	$R_{\text{con}}$	%Div
General	311	122	0.188	0.285	28.6%
Heart	137	69	0.152	0.186	26.3%
Kidney	196	124	0.237	0.148	39.8%
Liver	164	91	0.209	0.142	30.5%
Lung	153	117	0.152	0.136	41.8%
Pancreas	154	44	0.195	0.223	11.0%

Table 5: Organ-level heterogeneity metrics.

Q_{\text{total}}

: total questions for the organ;

Q_{\text{active}}

: questions with

\geq

1 non-absent pair;

R_{\text{div}}

and

R_{\text{con}}

: mean question-level divergence and consistency rates (Eq. 4 ); %Div: percentage of questions with at least one divergent or contradictory pair.

Topic	$Q_{\text{total}}$	$Q_{\text{active}}$	$R_{\text{div}}$	$R_{\text{con}}$	%Div
Monitoring & Follow-up	108	59	0.277	0.073	38.9%
Lifestyle & Daily Living	225	134	0.235	0.128	40.4%
Pre-Transplant	176	105	0.221	0.179	29.5%
Medical Complications	319	168	0.179	0.177	26.6%
Reproductive Health	291	85	0.176	0.315	17.5%
Special Populations	73	39	0.172	0.242	24.7%

Table 6: Topic-level heterogeneity metrics (topics with

\geq

20 questions). Columns as in Table 5 .

Panel of Figure 4 — Figure 4: Selected comparison matrices illustrating heterogeneity patterns. Each cell represents a pairwise comparison between two center handbooks for a single question. Colors: grey = Absent , green = Consistent , yellow = Complementary , blue = Divergent , black = Contradictory . Handbook labels on axes are anonymized (organ-center_index-phase). Matrices are symmetric; diagonal entries are Consistent by convention.

Healthcare

Healthcare AI develops and evaluates models and systems for clinical care, medical research, public health, and healthcare operations.

See all 616 papers

Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG
May 27, 2026Yubo Li, Rema Padman, Ramayya KrishnanRetrieval-Augmented Generation Healthcare
Carnegie Mellon University
ThReadMed-QA: A Multi-Turn Medical Dialogue Benchmark from Real Patient Questions
Mar 11, 2026Monica Munnangi, Saiph SavageMedical QA Multi-Turn Dialogue Evaluation
Khoury College of Computer Sciences Northeastern University Boston
Rethinking Patient Education as Multi-turn Multi-modal Interaction
Apr 16, 2026Zonghai Yao, Zhipeng Tang, Chengtao Lin +5Multimodal Grounding Healthcare
1VA Bedford Health Care · 2UMass Amherst · 4Yale University +3

May 27, 2026J/K move · Enter open · S save

Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG

Yubo Li, Rema Padman, Ramayya Krishnan

Carnegie Mellon University

Retrieval-Augmented Generation Healthcare Clinical QA RAG Evaluation

Open paper AskPDF arXiv

A retrieval-augmented generation (RAG) system deployed over a multi-author institutional corpus can give a different answer to the same question depending on which source it retrieves -- a failure mode the dominant single-gold-answer paradigm cannot diagnose. We argue that source-dependence is a missing axis of NLP evaluation, and that auditing it means shifting the unit of evaluation from answer correctness to the inter-source relationship. We make this concrete in transplant patient education, where institutional sources demonstrably disagree, releasing three artefacts: TransplantQA, a benchmark of real patient questions, each answered by grounding generation in multiple institutional handbooks as candidate sources; HERO-QA, a hierarchical retrieval strategy that grounds and audits each answer; and a structured-output judge that scores inter-source relationships on a validated 5-label taxonomy. At scale, better retrieval reveals far more disagreement than prior estimates suggested -- understating its prevalence, not its intensity. The framework is domain-agnostic and transfers to legal and educational RAG: measuring source-dependence is a responsibility for deployed multi-source NLP generally.

Abstract

Healthcare

Explore similar work

Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG

ThReadMed-QA: A Multi-Turn Medical Dialogue Benchmark from Real Patient Questions

Rethinking Patient Education as Multi-turn Multi-modal Interaction