QureRadEmbed: Structuring Radiological Similarity through Attribute and Reasoning Supervision

Janhavi Prabhu; Sahil; Shivam Ashok Shukla; Manoj Tadepalli

cs.AISep 27, 2026

QureRadEmbed: Structuring Radiological Similarity through Attribute and Reasoning Supervision

Authors: Janhavi Prabhu, Sahil, Shivam Ashok Shukla, Manoj Tadepalli

Organizations: Research & Development, Qure.ai

Abstract

Radiological similarity depends on disease relationships and on fine details such as laterality, lobe, severity, size, and certainty. Broad biomedical similarity can overlook these qualifiers, particularly when several attributes vary together. We introduce QureRadEmbed, a 4B radiology-aware encoder trained with two complementary signals: RadSim supplies deterministic, attribute-decomposed ranking targets, while RadThought aligns reports with hierarchical evidence and reasoning descriptions. A three-stage curriculum combines these signals with report triplets, finding perturbations, and single- and cross-attribute contrasts. The final model achieves 0.996 mean ordering accuracy across ten controlled synthetic attributes and raises Spearman correlation with the designed joint-attribute targets from 0.501 to 0.976. On external findings-to-impression retrieval, Recall@1 reaches 10.5% on Open-I, 12.4% on testing XR, and 42.9% on testing CT, compared with 6.6%, 5.7%, and 31.4% for its backbone. Frozen embeddings support finding extraction with only 100 labeled testing-XR reports (macro-F1 0.481 versus 0.412 for the backbone). Whole-report comparison costs 8.8 seconds per 1,000 pairs in our benchmark, versus 2,755.1 seconds for the generative evaluator GREEN. Sentence-level comparison improves sensitivity to local discrepancies, although generative evaluation remains stronger on several expert-rated and subtle-error tasks. The results support reusable radiology-aware representations for search, structured report indexing, and efficient report comparison.

Figures & tables

Figure 1: RadSim supervision. InternS2 extracts clinical attributes before deterministic scoring. In the example, partial laterality and severity agreement yields

5.32/6.30\approx 0.84

; unstated attributes are excluded. The lower panels illustrate disease-family relatedness and continuous size agreement.

Figure 2: Supervision and training pipeline. InternS2 attribute extraction precedes RadSim scoring. Report ranking and perturbation, RadThought hierarchy, single-axis contrasts, and cross-axis pairs supply complementary objectives to one shared encoder.

Encoder / checkpoint	R@1	R@5	R@10	MRR@10	Abn. nDCG@10
Qwen3-Embedding-4B	.728	.857	.892	.784	.234
QureRadEmbed Stage 1	.852	.922	.940	.883	.342
QureRadEmbed Stage 2	.857	.929	.944	.889	.331
QureRadEmbed Stage 3	.860	.928	.946	.890	.327
zembed-1	.792	.894	.915	.836	.240
BGE-large	.773	.886	.914	.821	.210

Table 1: Chest findings-to-impression retrieval on the fixed 2,336-report gallery. All values are on a 0–1 scale; higher is better. Abn. nDCG@10 uses abnormality-label relevance rather than exact study identity. Bold marks the best value in each column across all displayed checkpoints and encoders.

Figure 3: Metrics across the base encoder and three training stages. Recall@1, Recall@10, and MRR@10 use the chest retrieval pool (2,336 reports); STS is Spearman correlation, and the ten-axis mean is average contrast-ordering accuracy. Colors and line styles distinguish the metrics.

Figure 4: Findings-to-impression Recall@1 on the chest retrieval pool (2,336 reports). All three training stages outperform the unadapted backbone and the public encoders evaluated on the same candidate gallery.

Pool	QureRadEmbed	Qwen3-4B	zembed-1	BGE-large
Open-I	10.5 / 11.2	6.6 / 7.8	7.7 / 5.3	5.8 / 8.5
Testing XR	12.4 / 12.9	5.7 / 8.7	7.0 / 6.6	3.5 / 7.3
Testing CT	42.9 / 45.6	31.4 / 39.4	28.9 / 33.4	9.8 / 24.5
Testing CT-B	39.9 / 44.1	32.0 / 36.7	34.4 / 33.8	25.0 / 35.2
Testing MR	54.9 / 60.4	47.5 / 52.8	44.2 / 47.8	31.0 / 44.9
Testing US	32.9 / 33.4	23.6 / 29.9	27.1 / 24.7	22.0 / 26.9

Table 2: External retrieval Recall@1 (%). Each cell lists two Recall@1 values separated by a slash: the first is findings-to-impression retrieval, and the second is impression-to-findings retrieval; the slash is a separator, not a ratio. Selected strong comparators are shown; QureRadEmbed leads all ten evaluated encoders on each task.

Figure 5: Controlled triplet-ordering accuracy across ten attributes. Stage 2 broadens discrimination beyond size and laterality, and Stage 3 retains high accuracy across the expanded suite. Stage 2 per-axis values are reported to two decimal places.

Figure 6: Graded attribute structure in the final model. The severity panel spans trace, mild, moderate, and severe; similarity decreases as the descriptions become further separated in severity. Count and density panels show the corresponding cosine structure for their displayed levels.

Figure 7: Mean cosine similarity versus cranio-caudal location separation for descriptions with the same finding and side. Separation is expressed in ordered zone steps, not physical distance. The final model shows a substantially stronger decrease with increasing separation than the comparison encoders. Shaded bands are descriptive and are not interpreted as confidence intervals.

Figure 8: Laterality response when another attribute varies, measured as the mean cosine of same-side pairs minus that of opposite-side pairs. Positive values favor the same-side group. The Stage 3 responses are positive for the size, severity, and count combinations that have negative responses at Stage 2.

Figure 9: t-SNE of findings embeddings from the chest retrieval pool (2,336 reports), shown for the backbone, final QureRadEmbed model, and five public encoders. Colors denote eight abnormalities. The highlighted model separates several clinical groups more clearly; projection distances are not a common metric across panels.

Figure 10: Finding-presence linear probes on frozen embeddings. Macro-F1 is measured on 4,000 testing-XR reports; the x-axis is the number of labeled training reports. “All” uses the 16,000-report training portion, with internal validation for regularization. Gains are largest in the low-label setting.

Metric	X-ray	CT
Test reports	189,312	28,645
Tags	131	256
Macro AUROC	0.988	0.975
Tags with AUROC $\geq 0.90$	130 / 131	243 / 256
Macro sensitivity	0.967	0.940
Macro specificity	0.959	0.938

Table 3: Discrimination across a broad tag vocabulary and exploratory operating-point results. AUROC uses continuous scores. Sensitivity, specificity, and F1 use tag-specific maximum-Youden thresholds selected on the evaluated test examples. Macro metrics weight tags equally.

Metric	Value
Mean own-attribute Spearman $\rho$	0.940
Mean AUC	0.994
Contradiction AUC	1.000
Modularity ratio	3.634
DCI disentanglement	0.073
DCI completeness	0.076

Table 4: Aggregate frozen-encoder attribute-head results. Modularity is the ratio of intended-attribute response to off-attribute response. The remaining measures characterize separation and concentration of attribute information; these do not establish orthogonal subspaces.

	Open-I		Testing XR		Testing CT
Method	P	C	P	C	P	C
QureRadEmbed cosine	.630	.721	.537	.436	.541	.358
QureRadEmbed directional	.864	.928	.618	.593	.717	.499
GREEN	.825	.538	.743	.361	.783	.433

Table 5: Discrepancy win rates on common GREEN subsets: 1,356 Open-I, 1,230 testing-XR, and 1,374 testing-CT pairs. P compares an erroneous report with an LLM paraphrase; C compares it with an LLM condensation. Each value is the fraction of comparisons in which the error receives a lower similarity score than the compatible rewrite (ties receive half credit). Higher is better.

	ReXVal	RadEval	RaTE-Eval
Method	Within study	Within study	Sentence	Paragraph
QureRadEmbed cosine	0.336	0.122	0.246	0.494
QureRadEmbed Align-F1	0.301	0.178	0.245	0.384
Qwen3-4B cosine	0.207	0.071	0.428	0.575
zembed-1 cosine	0.217	0.174	0.298	0.588
BGE-base cosine	0.378	0.100	0.351	0.489

Table 6: Agreement with human judgments on public benchmarks. Values are Kendall

\tau

(higher is better). ReXVal uses within-study agreement with clinically significant error counts (200 pairs); RadEval uses within-study agreement with reader error counts (624 pairs). RaTE-Eval evaluates sentence (441 pairs) and paragraph (370 pairs) ratings. The two RaTE columns are separate tasks, not retrieval directions. Selected embedding and report-metric comparators are shown with their scoring rule fixed across columns.

Method	Seconds per 1,000 pairs
BGE-base cosine	1.4
BERTScore	1.4
RadEvalModernBERT cosine	1.7
F1-RadBERT-CT	1.9
BGE-large cosine	3.7
QureRadEmbed cosine	8.8

Table 7: Measured report-comparison cost on one NVIDIA L40S. Values are seconds per 1,000 report pairs (lower is better). Cosine rows include encoding both reports, without cached vectors. Report-metric timings are normalized from the Open-I discrepancy runs; GREEN used a 200-report subset and generation batch size eight. These are implementation-specific measurements, not matched-throughput limits.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Configuration
Backbones	Qwen3-Embedding-4B, full fine-tuning, no LoRA.
Framework	sentence-transformers; plain DDP via torchrun .
Precision	bf16 + tf32.
Schedule	Cosine learning-rate schedule, 3% warmup, weight decay 0.01, seed 42.
Sequence length	128 in early development; 9,500 during the long-context portion of Stage 1 and throughout Stages 2–3.
Effective batch	256 during long-context Stage 1 training; 240 in Stages 2–3.

Appendix

Table 8: Documented configuration for Stage 1. Later refinements are described in Section 2.3 .

Dataset	Samples	Weight	Margin
Cross-axis pairs	50k	1.0	–
Findings / impression pairs	10k each	1.5	–
RadThought fused pairs	10k	0.6	–
RadThought per-level pairs	10k each	2.0	–
Perturbation pairs	10k	0.5	–
Size / count scored pairs	10k each	0.4	–

Appendix

Table 9: Stage 3 training mixture. Each batch comes from one dataset, sampled in proportion to its capped size.

Tag	Support	Threshold	Sens.	Spec.	F1	AUROC
aeration	414	0.039386	0.667	0.940	0.046	0.847
airspace_opacity	1,522	0.308242	0.929	0.919	0.155	0.974
aneurysm	33	0.038370	1.000	0.977	0.015	0.998
aortic_abnormality	1,101	0.153177	0.942	0.934	0.142	0.975
aortic_aneurysm	695	0.157237	0.944	0.948	0.117	0.990
aortic_ectasia	785	0.132587	0.995	0.989	0.421	0.999

Appendix

Table 10: Chest X-ray per-tag results (131 tags) at test-selected maximum-Youden thresholds. Support is reference-positive count; Sens. and Spec. denote sensitivity and specificity.

Tag	Support	Threshold	Sens.	Spec.	F1	AUROC
ABDOMINAL_WALL_HERNIA	72	0.128230	0.958	0.971	0.142	0.994
ABERRANT_RIGHT_SUBCLAVIAN_ARTERY	161	0.112896	0.944	0.967	0.241	0.990
ABSENT_GALLBLADDER	41	0.028526	0.780	0.879	0.018	0.909
ADRENAL_ADENOMA	429	0.389781	0.970	0.987	0.692	0.997
ADRENAL_CALCIFICATION	25	0.029913	0.880	0.953	0.032	0.964
ADRENAL_ENLARGEMENT	11	0.035492	1.000	0.971	0.026	0.994

Appendix

Table 11: CT per-tag results (256 tags) at test-selected maximum-Youden thresholds. Support is reference-positive count; Sens. and Spec. denote sensitivity and specificity.

Clinical NLP

Clinical natural language processing (NLP) develops methods to extract, interpret, and generate information from healthcare text and conversations.

See all 142 papers

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching
Sep 1, 2026Charles Corbière, Léo Machado, Aubin Charley +3LLM-as-a-Judge Medical Image Benchmarks
Raidium, Paris, France · Department of Radiology, Fondation Ophtalmologique Adolphe de Rothschild, Paris, France · Service de Radiologie Interventionnelle, Département d’Anesthésie, Chirurgie et Imagerie Interventionnelle (DACI), Gustave Roussy, Villejuif, France
RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports
Jun 3, 2026Zhenhong Yang, Zhuoyun Liu, Jintao Fei +4Factual Consistency Evaluation Automated Evaluation
JDH Algo, JD Health International Inc., China · Department of Big Data in Health Science, The First Affiliated Hospital of Wenzhou Medical University, China · Zhejiang Engineering Research Center for Hospital Emergency and Process Digitization, Wenzhou, China +1
Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context
Jul 7, 2026Suneeta Mall, Vladimir Nekrasov, Ashnil Kumar +5Radiology Report Generation Radiology VLMs

Sep 1, 2026J/K move · Enter open · S save

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

Charles Corbière, Léo Machado, Aubin Charley, Baptiste Callard+2

Raidium, Paris, France · Department of Radiology, Fondation Ophtalmologique Adolphe de Rothschild, Paris, France · Service de Radiologie Interventionnelle, Département d’Anesthésie, Chirurgie et Imagerie Interventionnelle (DACI), Gustave Roussy, Villejuif, France

LLM-as-a-Judge Medical Image Benchmarks Radiology Report Evaluation

Open paper AskPDF arXiv

As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language model (LLM)-based metrics are now the best-correlated with radiologist judgment, yet they output a single opaque score that neither a clinician nor a model builder can easily interpret or audit. We introduce RadMatch, a multi-stage, LLM-based metric that decomposes report comparison into a structured finding-level matching with significance-aware scoring and error characterization across seven clinical attribute dimensions (status, location, severity, morphology, certainty, longitudinal comparison, and measurement). The main score is the actionable-error count, both interpretable and auditable. Candidate findings are graded correct, partial, or incorrect, and unmatched findings are counted as missed or hallucinated. Triage and actionable safety recall/precision and per-subset views add complementary, deployment-oriented lenses. Across two expert benchmarks, RadMatch is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert. Relying only on few-shot prompting, it is designed to extend to other modalities and anatomies. We will release RadMatch as open-source code with an interactive dashboard for inspecting results.

Abstract

Clinical NLP

Explore similar work

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports

Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context