Every Edge Needs a Quote: Building a Quote-Grounded Knowledge Graph of Healthcare Software Adoption with Language-Model Agents

Juan Felipe Duran; Arthur Tobler; Martin Vallières

cs.CLOct 23, 2026

Every Edge Needs a Quote: Building a Quote-Grounded Knowledge Graph of Healthcare Software Adoption with Language-Model Agents

Authors: Juan Felipe Duran, Arthur Tobler, Martin Vallières

Organizations: Medical Physics Unit, Department of Oncology McGill University, Montreal, Canada · Toboggan Labs Montreal, Canada

Abstract

Hospitals regularly buy software that clinicians and patients depend on. Because these purchases are costly, hard to reverse and can affect patient care, buyers look for evidence of which peer institutions already use a product and which systems it connects to. This evidence is public but scattered across vendor websites, press releases and third-party pages. Language-model agents can now collect it at low cost, but the correctness of the extracted claims is difficult to establish, and a wrongly listed customer can mislead buyers and misrepresent an organisation. We present a pipeline that builds a knowledge graph of customer and integration relations from the open web under one rule: every edge must be supported by a verbatim quote from its source page. Pages come from a deterministic crawl of each vendor's site and a budgeted ReAct agent that searches the rest of the web. A permissive extractor proposes relations with quotes, and a layered grounding stage decides which to keep, using a referent step that reads the whole page when a quote names only one side, deterministic checks, and a panel of five model calls voting on nine criteria. On 61 vendors, the pipeline reaches 0.94 precision and 0.71 recall against a page-level key covering the 6,428 candidate relations proposed by its extractor, whose verdicts were drafted by one model and re-checked with a second. Against the golden reference, the marketplace's own curated list, it scores lower, but 89% of the relations it keeps that the reference lacks are stated on a page. A confidence score built from the panel's votes keeps more correct relations than the fixed rule at the same precision, and reaches 0.90 precision at 0.90 recall. We also discuss how narrow, bounded model calls, string-matched evidence and voting address long-context degradation and run-to-run variance.

Figures & tables

Term	Meaning in this paper
Customer relation	An organisation uses a product ( Adopts ).
Integration relation	Two named products are technically connected ( Integrates ).
Quote	The verbatim span of a page offered as evidence for a relation.
Candidate	One (page, relation) pair proposed by the extractor, with its quote.
Vendor page / third-party page	A page on the vendor’s own site / on any other site.
Referent	The step that reads the whole page when a quote names only one side of a relation.

Table 1: Terminology used throughout the paper.

Panel of Figure 1 — Figure 1: The pipeline. Discovery finds pages in two halves: a deterministic sweep of the vendor’s own site, and a ReAct agent that searches everything else under a tool budget. Customer names confirmed on the vendor’s site become search leads for the agent. Both halves feed the same extractor and the same grounding stage, which run in the order shown. Only relations that survive every step are written, each with its quote and provenance.

	Customers	Integrations	Total
Page-level key : relations proposed by the extractor	3,486	2,942	6,428
of which correct	2,952	1,861	4,813
Golden reference : relations listed	749	249	998

Table 2: The two benchmarks over the 61 vendors.

Benchmark	Scope	Precision	Recall	F1
Page-level key (page, relation)	All	0.94	0.71	0.81
	by type: customers	0.96	0.71	0.81
	by type: integrations	0.90	0.71	0.79
	by page: vendor’s site	0.94	0.71	0.81
	by page: third-party	0.92	0.65	0.76
	Each relation counted once (2,349)	0.92	0.74	0.82

Table 3: Accuracy on the 61-vendor corpus. On the page-level key, recall is over the candidates the extractor proposed; on the golden reference, over every organisation the reference lists. “Each relation counted once” counts a relation as kept if any page kept it and as correct if any page states it. The “by type” and “by page” rows are separate splits of the same candidates.

Outcome	Correct	Incorrect
Kept	3,415	217
Rejected by the semantic panel	1,106	1,212
page does not present them as users	546	305
all other criteria	560	907
Refused by the referent	276	108
Rejected by the shape checks	16	78

Table 4: Outcome of the 6,428 judged candidates under the final pipeline, by the step that rejected them. Correct and incorrect are according to the page-level key.

Configuration	Kept	P	R	F1	Golden F1
Without the referent	133	0.887	0.361	0.513	0.578
Final pipeline	291	0.932	0.808	0.866	0.585

Table 5: Customer relations on 266 vendor pages from 21 vendors, run end to end (one run each). P/R/F1 are against this set’s reviewed verdicts; golden F1 counts only customers that appear on these pages.

Panel of Figure 3 — Figure 3: Precision–recall curve of the confidence score, with every relation scored by a model that did not see it in training (five-fold cross-validation, all 22 features). Open circles mark score thresholds. The filled dot is the pipeline’s fixed decision on the same run (precision 0.94, recall 0.63). At that precision, the score reaches a recall of about 0.77.

Target precision	Recall	Relations kept
0.99	0.002	12
0.98	0.27	1,305
0.97	0.40	2,002
0.96	0.45	2,258
0.95	0.72	3,623
0.93	0.84	4,359

Table 6: Operating points of the confidence score on the page-level key, out-of-sample.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Rejects
incomplete_relationship	a missing object, object type or relation
unnamed_subject / unnamed_object	a generic endpoint (“hospitals”, “EHR systems”, “the platform”)
subject_is_individual	a person (title or clinical credential) given as the organisation
meta_commentary_quote	a quote that is the model’s own arrow notation, not page text
quote_not_in_source	a quote less than 75% of whose characters match the page
quote_not_contiguous	a quote whose longest contiguous match is under 95% (pre-empted by splice repair)

Appendix

Table 7: Deterministic shape checks. Two further codes (an endpoint absent from the evidence) exist but are disabled for both relation types.

Criterion	True when	Gates
is_named_organization	the subject is a specific, real, named organisation, not a person, a category, a placeholder (“your practice”), an anonymised description, or the vendor’s own brand	Adopts
relationship_actually_happened	the relation is stated as a completed fact, not a plan, a hope, or something cancelled	both
relationship_claim_supported	the quote states this type of relation between these entities, not a co-mention, an investment, an award or a network membership	both
is_third_party_vendor	the partner is independent of the vendor, not its own product, sub-brand or acquisition	Integrates
is_customer_facing_integration	the integration is one a customer would use, not swappable infrastructure (a cloud host, a generic model provider, a coding assistant)	Integrates
is_specific_named_entity	the partner is a named product, not a category (“EHRs”, “practice management system”)	Integrates

Appendix

Table 8: The semantic panel’s nine criteria, abridged from their definitions in the output schema. Every call answers all nine; each passes when at least half of the calls that returned say yes.

Step	Braintrust slug(s)
Extraction	customer-extraction-8f90 integration-extraction-4de1
Referent, third-party pages	referent-solution-customers-7f21 referent-solution-0cd9
Referent, vendor pages	referent-solution-customers-firstparty-1eb4 referent-solution-integrations-firstparty-71e0
Semantic panel	semantic-grounding-check-customers-fe3b semantic-grounding-check-integrations-a12e
Discovery agent	discovery-system-prompt-32a6 customer-discovery-f006 integration-discovery-c2a2

Appendix

Table 9: Prompt slugs. The extractor version hash covers the extraction, referent and semantic-panel prompts and their versions.

KG Construction

Knowledge graph construction turns text, tables, images, and databases into structured entities and relationships that machines can query and reason over.

See all 195 papers

Keep Your Receipts: Trustworthy Cleanup of Agent-Built Knowledge Graphs
Oct 13, 2026Vishisht ChoudharyKnowledge Graphs Data Provenance
Technical University of Munich
ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV
May 11, 2026Alex StinardRAG Evaluation Clinical QA
Department of Clinical Sciences, College of Medicine · University of Central Florida, Orlando, FL 32816
Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering
Jun 21, 2026Madhulatha Mandarapu, Sandeep KunkunuruFactual Knowledge in Language Models LLM Reasoning with Graphs
samyama

Oct 13, 2026J/K move · Enter open · S save

Keep Your Receipts: Trustworthy Cleanup of Agent-Built Knowledge Graphs

Vishisht Choudhary

Technical University of Munich

Knowledge Graphs Data Provenance KG Construction

Open paper AskPDF arXiv

Agents build knowledge graphs as memory, and building them has become easy. Maintaining them has not. When a source is retracted, overruled, or erased, everything built on top of it (entity merges, entailed edges, community summaries) must be re-examined, and the maintainer must choose between deleting too much and too little. Deployed pipelines delete only the directly extracted facts: on our corpus that reduces contaminated queries from 32.2% to 28.0% under a real retraction and changes nothing under predicate revocation. Rebuilding is no repair for a constructor that samples: two builds of the same 300-document corpus differ in 29,475 artifacts, five times the 6,248 that removing the 61 retracted documents changes. PolyGraph instead asks the constructor for receipts: for every artifact, alternative bundles of source evidence, each claimed sufficient on its own, emitted in the call that creates it. Cleanup is then set arithmetic over the receipts, with no model calls. Because receipts are the model's own self-report, a re-derivation audit checks them by deleting the cited evidence and testing whether the constructor still produces the artifact. The audit finds that a receipt is trustworthy when it is shaped like the claim it backs. Merge receipts, each backing one narrow claim, agree with re-derivation 68% and 85% of the time. Summary receipts, elicited per sentence but stored as one flat list for the whole text, are wrong about two thirds of the time, almost always by keeping dead content. One receipt per sentence instead is 100% and 88% faithful and lets cleanup keep 33 of 35 touched summaries. The result holds under a second oracle model, a claim-blind prompt, ten further deletions, a second domain and a second constructor. Distrusting the summary layer cuts contaminated queries to 0.4% and 0.8% at 99.8 to 100% of the recall a perfect maintainer would return.

Abstract

KG Construction

Explore similar work

Keep Your Receipts: Trustworthy Cleanup of Agent-Built Knowledge Graphs

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering