DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents

Zikang Xu; Ruinan Jin; Xiaoxiao Li

cs.CVFeb 28, 2026

DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents

Authors: Zikang Xu, Ruinan Jin, Xiaoxiao Li

Organizations: Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Anhui, China · The University of British Columbia, Vancouver, BC V6Z 1Z4, Canada · Vector Institute, Toronto, ON M5G 1M1, Canada

Abstract

Fairness in medical agents is becoming critical as tool-using clinical AI systems orchestrate specialized vision and language modules for tasks such as chest X-ray question answering. While these medical AI agents can improve flexibility, their added pipeline complexity also creates new pathways for demographic bias beyond standalone models. We present DUCK, Decomposing Unfairness in Chest X-ray agents, a systematic audit of fairness in tool-using chest X-ray agents instantiated with MedRAX. To localize where disparities arise, we introduce a stage-wise fairness decomposition that separates end-to-end bias from three agent-specific sources: tool exposure bias, or utility gaps conditioned on tool presence; tool transition bias, or subgroup differences in tool-routing patterns; and model reasoning bias, or subgroup differences in synthesis behaviors. Extensive experiments on tool-using agentic frameworks across five driver backbones reveal that demographic gaps persist in end-to-end performance, with equalized odds up to 20.79% and the lowest fairness-utility tradeoff down to 28.65%. Intermediate behaviors, including tool usage, transition patterns, and reasoning traces, exhibit distinct subgroup disparities that are not predictable from end-to-end evaluation alone. For example, conditioned on segmentation-tool availability, the subgroup utility gap reaches as high as 50%. Our findings underscore the need for process-level fairness auditing and debiasing to ensure the equitable deployment of clinical agentic systems. Code: https://github.com/Nanboy-Ronan/DUCK.

Figures & tables

Figure 1: Overview of DUCX . (1) Dataset curation: we use CheXAgentBench and curate MIMIC-FairnessVQA into a standardized form. (2) MedRAX agent execution (ReAct): a driver LLM iteratively reasons, selects tools, and synthesizes a final answer, where we highlight multiple points where bias can be introduced (tool exposure, tool transitions, and final synthesis). (3) Fairness decomposition: We evaluate end-to-end fairness (ACC, $\Delta$ ACC, DP, EoD, FUT) and decompose it into tool exposure bias (gaps conditioned on tool presence), tool transition bias (gaps in tool routing), and LLM reasoning bias (gaps in synthesis behaviors).

Configuration		CheXAgentBench					MIMIC-FairnessVQA
Attr.	LLM	ACC $\uparrow$	$\Delta$ ACC $\downarrow$	DP $\downarrow$	EoD $\downarrow$	FUT $\uparrow$	ACC $\uparrow$	$\Delta$ ACC $\downarrow$	DP $\downarrow$	EoD $\downarrow$	FUT $\uparrow$
Gender	LLaMA3.1	$50.83_{[48.88,52.76]}$	${\color[rgb]{1,0,0}1.53_{[0.06,4.45]}}$	$5.04_{[2.32,7.93]}$	$15.10_{[5.63,26.98]}$	$50.29_{[48.20,52.30]}$	$32.89_{[30.85,34.95]}$	$2.00_{[0.10,5.86]}$	${\color[rgb]{1,0,0}2.99_{[1.12,6.28]}}$	$11.45_{[5.83,18.97]}$	$32.43_{[30.39,34.56]}$
	Ministral-3	$51.48_{[49.72,53.36]}$	${\color[rgb]{0,0,1}1.68_{[0.06,4.40]}}$	${\color[rgb]{1,0,0}3.66_{[1.48,6.73]}}$	$10.48_{[4.12,19.60]}$	$50.85_{[48.75,52.88]}$	$29.42_{[27.35,31.50]}$	$2.30_{[0.11,6.07]}$	$3.35_{[1.39,5.86]}$	${\color[rgb]{0,0,1}11.21_{[5.39,18.36]}}$	$28.95_{[26.86,31.07]}$
	Qwen3VL	${\color[rgb]{0,0,1}67.26_{[65.11,69.28]}}$	$3.21_{[0.21,7.11]}$	$4.47_{[1.42,8.12]}$	$11.28_{[4.87,22.64]}$	${\color[rgb]{0,0,1}65.98_{[63.41,68.45]}}$	${\color[rgb]{1,0,0}54.75_{[52.07,57.39]}}$	$2.17_{[0.11,6.29]}$	$4.22_{[1.74,7.56]}$	$11.47_{[4.93,20.61]}$	${\color[rgb]{1,0,0}53.92_{[51.02,56.77]}}$
	Qwen3	${\color[rgb]{1,0,0}69.10_{[67.24,70.92]}}$	$1.92_{[0.09,5.07]}$	${\color[rgb]{0,0,1}3.72_{[1.52,6.62]}}$	${\color[rgb]{1,0,0}9.16_{[3.45,19.62]}}$	${\color[rgb]{1,0,0}68.09_{[65.74,70.32]}}$	$49.89_{[47.55,52.10]}$	${\color[rgb]{1,0,0}1.90_{[0.09,5.30]}}$	${\color[rgb]{0,0,1}3.11_{[1.14,5.98]}}$	$11.32_{[5.06,19.56]}$	${\color[rgb]{0,0,1}49.23_{[46.77,51.37]}}$
	Gemini3	$50.60_{[48.64,52.64]}$	$3.62_{[0.36,7.47]}$	$4.99_{[2.61,7.99]}$	${\color[rgb]{0,0,1}9.56_{[4.83,16.61]}}$	$49.15_{[46.82,51.59]}$	$49.69_{[47.50,51.95]}$	${\color[rgb]{0,0,1}1.99_{[0.08,5.31]}}$	$3.13_{[1.36,5.38]}$	${\color[rgb]{1,0,0}11.01_{[5.31,19.10]}}$	$49.01_{[46.54,51.48]}$

Table 1: Overview of the end-to-end fairness performance.

\text{Mean}_{\text{CI}}

is reported. All values are in percentage. Best and Second Best in each group are highlighted.

Figure 2: Tool exposure bias across tools and subgroups. (a) Gender on CheXAgentBench, (b) Age on CheXAgentBench, (c) Gender on MIMIC-FairnessVQA, and (d) Age on MIMIC-FairnessVQA.

Figure 3: Tool transition biases represented as matrices. START: start indicator; CLS: classifier; RG: report generator; SEG: segmentator; VIS: visualizer; GRD: phrase grounding. Cells in red denote values larger than zero, while cells in blue denote values smaller than zero. $\Delta_{\text{TTB}}^{\text{gender}}=P^{\text{male}}-P^{\text{female}}$ , $\Delta_{\text{TTB}}^{\text{age}}=P^{\text{young}}-P^{\text{old}}$ . All subfigures present results averaged across different LLMs.

Configuration		CheXAgentBench			MIMIC-FairnessVQA
Attr.	LLM	$\Delta$ JudgeGap $\downarrow$	$\Delta$ Hedge $\downarrow$	$\Delta$ Demo. $\downarrow$	$\Delta$ JudgeGap $\downarrow$	$\Delta$ Hedge $\downarrow$	$\Delta$ Demo. $\downarrow$
Gender	LLaMA3.1	$6.19_{[0.21,19.00]}$	$4.97_{[0.25,11.66]}$	$0.09_{[0.02,1.66]}$	$0.10_{[0.30,21.70]}$	$3.10_{[0.20,12.40]}$	$0.10_{[0.00,0.30]}$
	Ministral-3	$2.86_{[0.15,8.92]}$	$12.20_{[0.63,31.73]}$	$0.56_{[0.06,3.74]}$	$3.10_{[0.40,23.40]}$	$4.90_{[0.60,44.40]}$	$0.20_{[0.00,1.20]}$
	Qwen3VL	$4.59_{[0.22,11.39]}$	$534.95_{[3.58,1438.18]}$	$1.57_{[0.08,4.21]}$	$21.67_{[2.66,41.90]}$	$831.10_{[44.75,3489.64]}$	$0.10_{[0.00,0.30]}$
	Qwen3	$5.58_{[0.47,11.69]}$	$12.58_{[0.65,51.20]}$	$3.63_{[0.57,6.88]}$	$1.90_{[0.30,24.80]}$	$4.40_{[1.00,69.20]}$	$0.00_{[0.00,0.00]}$
	Gemini3	$1.67_{[0.09,4.48]}$	$0.89_{[0.13,11.09]}$	$0.01_{[0.06,4.22]}$	$21.20_{[10.49,32.50]}$	$10.40_{[0.80,25.00]}$	$0.30_{[0.00,1.30]}$

Table 2: Subgroup gaps of reasoning-bias.

\text{Mean}_{\text{CI}}

is reported. All values are multiplied by 100 for better readability. Best and Second Best are highlighted.

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 419 papers

Feb 25, 2026

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning

AI agents with tool-use capabilities show promise for integrating the domain expertise of various tools. In the medical field, however, tools are usually AI models that are inherently error-prone and can produce contradictory responses. Existing research on medical agents lacks sufficient understanding of the tools' realistic reliability and thus cannot effectively resolve tool conflicts. To address this gap, this paper introduces a framework that enables an agent to interact with tools and empirically learn their practical trustworthiness across different types of multimodal queries via agentic learning. As a concrete instantiation, we focus on chest X-ray analysis and present a tool-expertise-aware chest X-ray agent (TEA-CXA). When tool outputs disagree, the agent experimentally accepts or rejects multimodal tool results, receives rewards, and learns which tool to trust for each query type. Importantly, TEA-CXA extends existing codebases for reinforcement learning with multi-turn tool-calling that focus on textual inputs, to support multimodal contexts effectively. In addition, we enhance the codebase for medical use scenarios by supporting multiple tool calls in one turn, parallel tool inference, and multi-image accommodation within a single user query. Our code framework is applicable to general medical research on multi-turn tool-calling reinforcement learning in multimodal settings. Experiments show that TEA-CXA outperforms the state-of-the-art methods and a comprehensive set of baselines. Code will be released.

Zheang Huai, Honglong Yang, Xiaomeng Li

The Hong Kong University of Science and Technology, Kowloon, Hong Kong

Medical AI Agents Tool-Using Vision-Language Agents Multimodal Tool Use RL for Tool Use

Ask PDF

Oct 5, 2026

Representation Disentanglement for Fair Chest X-Ray Diagnosis

Deep learning has advanced chest X-ray (CXR) diagnosis, yet demographic biases in learned representations may contribute to performance disparities across intersectional groups. We propose a single-encoder framework combining dual-level decorrelation with prototype-guided cross-group contrastive learning to reduce demographic dependence while accounting for within-class variation. We further propose Demographic Representation Alignment Reduction (DRAR), a new metric that quantifies the reduction in demographic structure within disease representations. The framework is evaluated on four classification tasks using 34,809 CheXpert test images across eight intersectional groups, defined by age, sex and ethnicity. Compared with empirical risk minimization (ERM), our method reduces the mean equalized-odds gap from 15.41% to 10.86% and the AUC gap from 5.95% to 5.01%. Our method achieves a DRAR of 59.04% relative to ERM, with only a slight decrease in mean AUC. These results demonstrate that representation disentanglement can reduce demographic bias and improve intersectional fairness. Code is available at https://github.com/06Yujie/Fair-Medical-Imaging.

Yujie Sun, Ruizhe Li, Xiaowu Sun

Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University, China · Nottingham Biomedical Research Centre, School of Medicine, University of Nottingham, UK

Chest X-Ray Classification Representation Disentanglement Algorithmic Bias Intersectional Fairness Fairness in Medical Image Classification

Ask PDF

Oct 12, 2026

X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning

Chest X-ray (CXR) interpretation often requires integrating visual findings with task-specific evidence rather than relying on direct visual recognition alone. While tool-augmented medical agents can supply such evidence via external modules, they often lack explicit mechanisms for learning when auxiliary evidence is needed and for preventing inapplicable quantitative evidence from entering diagnostic reasoning. To address this, we propose X-Ragent, a post-training framework for tool-augmented reinforcement learning that decouples learned auxiliary-evidence seeking from validity-gated quantitative measurement acquisition for evidence-grounded CXR reasoning. We construct Rad-CoT, a 7.7K-sample corpus combining causally ordered tool-augmented trajectories with report-style diagnostic reasoning, and use it to initialize Qwen3-VL-8B via agentic supervised fine-tuning (SFT). We then optimize the policy with Group Relative Policy Optimization (GRPO) using an accuracy-gated reward that applies process-level terms for measurement consistency and auxiliary-tool cost only to correct trajectories, preventing plausible process behavior from compensating for an incorrect diagnosis. Across three CXR benchmarks, X-Ragent achieves the strongest results, including an 8.6-point improvement on the fully held-out ChestAgentBench. By seeking evidence selectively and admitting only valid measurements, X-Ragent supports more faithful and evidence-grounded CXR reasoning.

Yi Wang, Zhenlong Yuan, Yuchan Guo +9

Institute of Computing Technology · University of Chinese Academy of Sciences · Carnegie Mellon University +6

Medical VLMs Medical Imaging RL for Tool Use Agentic RL

Ask PDF

Abstract

Algorithmic Fairness

Explore similar work

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning

Representation Disentanglement for Fair Chest X-Ray Diagnosis

X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning