X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning
Authors: Yi Wang, Zhenlong Yuan, Yuchan Guo, Maolei Huang, Zijian Song, Dapeng Zhang, Rongbin Tan, Jiyuan Wang, +4 more
Organizations: Institute of Computing Technology · University of Chinese Academy of Sciences · Carnegie Mellon University · Alibaba Group · Sun Yat-sen University · National University of Singapore · Beijing Jiaotong University · City University of Hong Kong · Santa Clara University
Chest X-ray (CXR) interpretation often requires integrating visual findings with task-specific evidence rather than relying on direct visual recognition alone. While tool-augmented medical agents can supply such evidence via external modules, they often lack explicit mechanisms for learning when auxiliary evidence is needed and for preventing inapplicable quantitative evidence from entering diagnostic reasoning. To address this, we propose X-Ragent, a post-training framework for tool-augmented reinforcement learning that decouples learned auxiliary-evidence seeking from validity-gated quantitative measurement acquisition for evidence-grounded CXR reasoning. We construct Rad-CoT, a 7.7K-sample corpus combining causally ordered tool-augmented trajectories with report-style diagnostic reasoning, and use it to initialize Qwen3-VL-8B via agentic supervised fine-tuning (SFT). We then optimize the policy with Group Relative Policy Optimization (GRPO) using an accuracy-gated reward that applies process-level terms for measurement consistency and auxiliary-tool cost only to correct trajectories, preventing plausible process behavior from compensating for an incorrect diagnosis. Across three CXR benchmarks, X-Ragent achieves the strongest results, including an 8.6-point improvement on the fully held-out ChestAgentBench. By seeking evidence selectively and admitting only valid measurements, X-Ragent supports more faithful and evidence-grounded CXR reasoning.
Figures & tables
Figure 1: Evidence-grounded CXR reasoning with X-Ragent. (a) Single-shot MLLMs read the radiograph globally and miss localized evidence, leaving a visual–diagnostic gap. (b) Tool-augmented agents invoke tools but override the returned evidence with textual priors (top), or apply a measurement under incompatible acquisition conditions (bottom). (c) X-Ragent routes task-specific evidence on demand and integrates it with validity-gated quantitative measurements for evidence-grounded diagnosis.
Figure 2: Overview of X-Ragent. (i) Rad-CoT constructs structured CXR reasoning data through evidence triage and gating, evidence integration, and filtering. (ii) Agentic SFT cold-starts Qwen3-VL-8B on Rad-CoT, masking injected observations from the language-modeling loss. (iii) GRPO optimizes the policy with accuracy and format supervision, while tool-use cost and radiographic evidence consistency are gated by answer correctness. (iv) At inference, the policy selectively routes auxiliary tools, while an independent validity gate decides whether quantitative measurements are exposed; both are integrated in the second turn.
Figure 3: X-Ragent tool suite. The policy selectively invokes visual-refinement and evidence-lookup tools, while a separate validity-gated backend provides quantitative measurements.
Model
CheXbench
SLAKE
ChestAgentBench (OOD)
R-R
SLAKE
Reas.
Acc.
Det.
Cls.
Loc.
Comp.
Rel.
Diag.
Char.
All
General-purpose VLMs
GPT-4o
53.9
85.4
51.1
–
58.7
54.6
59.0
55.5
59.0
52.6
56.1
56.4
Llama-3.2-90B-Vision
62.6
74.0
49.2
–
58.1
56.5
59.9
57.5
59.3
55.9
58.0
57.9
Qwen3-VL-8B †
70.4
75.6
50.5
72.8
56.5
53.3
57.9
55.9
57.2
52.1
53.8
55.1
Medical VLMs
Table 1: Main results on three CXR benchmarks. Accuracy (%). CheXbench reports Rad-Restruct (R-R), SLAKE and OpenI reasoning (Reas.); ChestAgentBench is fully held out (OOD) and reported per capability. Following MedRAX, ChestAgentBench “All” is question-level accuracy over all 2,500 questions; since capability labels overlap, it is not the mean of the category columns. † Our base model, evaluated zero-shot with the direct-answer prompt (Appendix B.3 ). We denote best and second-best results, and highlight ours .
Table 5Table 6
View group
N
No gate
Gated
Δ
G-only / NG-only
All
2500
71.44
71.68
+0.24
17 / 11
Frontal
1887
72.18
72.18
+0.00
8 / 8
Non-frontal / atyp.
613
69.17
70.15
+0.98
9 / 3
Table 6: Controlled ablation of acquisition-validity gating on ChestAgentBench. The gated and no-gate variants use the same anatomy-measurement backend and differ only in whether view validity suppresses anatomy evidence. G-only / NG-only denote paired cases correct only with / without gating.
Figure 4: Counterfactual case study of validity-gated quantitative evidence. Without the validity gate, an inapplicable lung-field asymmetry measurement ( ln(AR/AL)=−1.01 ) is exposed to the policy and shifts the prediction from the correct option C to the incorrect option A. X-Ragent instead withholds invalid measurement bundle and returns correct answer based on radiographic evidence.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Hyperparameter
Value
SFT
Base model
Qwen3-VL-8B-Instruct
Trainable modules
LLM (vision encoder frozen)
Epochs / learning rate
1 / 1×10−5
Effective batch size
16
Max sequence length
8,192
LR schedule / warmup ratio
cosine / 0.1
Appendix
Table 7: Key hyperparameters of X-Ragent.
Figure 5: Composition of the training data. (a) The SFT corpus Rad-CoT. The outer ring shows the task type; the inner ring distinguishes single-turn report-style CoT from two-turn routed examples, in which the first-turn tool plan may be empty. (b) The GRPO corpus. Solid segments are reused from the routed SFT examples, while the hatched segment denotes multiple-choice questions unseen during SFT.
Subset
Format
Answer / reward
SFT
GRPO
Report-style diagnostic CoT
Single-turn
Free-text findings
4,000
–
Multiple-choice QA
Routed
Option-letter match
1,995
1,995
Multiple-choice QA (RL-only)
Routed
Option-letter match
–
1,000
Closed-ended VQA
Routed
Normalized exact match
720
720
Binary-choice VQA
Routed
Option-letter match
1,000
1,000
Total
7,715
4,715
Appendix
Table 8: Subset statistics of the SFT and GRPO corpora. “Routed” denotes the two-turn protocol in Sec. 2.1 , where the first-turn tool plan may be empty. GRPO uses only subsets with rule-verifiable answers; all GRPO rewards are binary.
Prompting Strategy
ChestAgent
CheXbench
SLAKE
Avg.
Direct answering (v4)
55.1
65.5
72.8
64.5
Reasoning v0
47.6
56.7
59.6
54.6
Reasoning v1
47.3
56.7
45.6
49.9
Reasoning v2
51.1
57.3
62.3
56.9
Reasoning v3
50.3
60.1
59.6
56.7
Appendix
Table 9: Effect of prompt-elicited reasoning on the Qwen3-VL-8B base model. We compare a direct-answer prompt with four variants that explicitly require intermediate reasoning. Accuracy (%) is reported using the same benchmark-level aggregation protocol as in the main experiments.
Evidence acquisition
ChestAgent
(a)
Full (learned routing)
71.7
(b)
w/o visual enh.
70.1
(c)
w/o aux. interp.
69.5
(d)
w/o obj. meas.
70.8
(e)
w/o all evidence modules
67.4
(f)
All auxiliary tools (static)
71.0
Appendix
Table 10: Ablation on evidence-acquisition modules at inference. A single trained checkpoint is evaluated with individual modules selectively disabled. The validity-gated measurement backend is retained unless objective measurement is explicitly removed; row (f) statically invokes all policy-callable auxiliary tools on every question.
Measurement pathway
ChestAgent
CheXbench
Independent validity gate (X-Ragent)
71.7
77.3
Policy routing + validity gate
71.0
76.1
Appendix
Table 11: Ablation on quantitative measurement routing. We compare the default X-Ragent design, where quantitative measurements are supplied by an independent validity-gated backend, with a variant that additionally requires the first-stage policy to select the measurement tool before validity checking. Accuracy (%) is reported on ChestAgentBench and CheXbench.
Figure 6: Selective auxiliary-evidence acquisition and integration. In the first turn, X-Ragent judges global inspection insufficient and routes localized visual refinement ( region_parse → roi_crop ) together with condition-specific evidence_lookup . Independently of this routing decision, the validity gate rejects the atypical input and withholds the measurement bundle ( M(I)=∅ ), preventing a physically implausible CTR (1.79 if ungated) from entering the reasoning. In the second turn, the policy integrates the magnified ROI with the retrieved criteria, identifying peripheral cavitated nodules without thoracic adenopathy, and selects the correct answer, localized nodular pulmonary amyloidosis.
Component
Execution
Backbone
Returned evidence
region_parse
Policy-routed
CXAS / BiomedParse
Region mask and bounding box
roi_crop
Policy-routed
Geometric crop
Local high-resolution view
evidence_lookup
Policy-routed
Qwen3.5-27B (local API)
Condition-specific radiographic knowledge
view_check
Backend
OpenCXR
Acquisition applicability
Measurement bundle
Backend
CXAS + deterministic rules
CTR, lung asymmetry, thoracic shift
Appendix
Table 12: Summary of the X-Ragent evidence-acquisition modules. Policy-routed tools are selected by the learned policy, whereas the validity check and quantitative measurements are executed automatically by the backend.
School of Computer Science, Hangzhou Dianzi University, Hangzhou, 310018, China · Department of Automation, Tsinghua University, Beijing 100084, China · Department of Colorectal Medical Oncology, Zhejiang Cancer Hospital, Hangzhou, 310022, China +2