DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents

Zikang Xu; Ruinan Jin; Xiaoxiao Li

cs.CVFeb 28, 2026

DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents

Authors: Zikang Xu, Ruinan Jin, Xiaoxiao Li

Organizations: Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Anhui, China · The University of British Columbia, Vancouver, BC V6Z 1Z4, Canada · Vector Institute, Toronto, ON M5G 1M1, Canada

Abstract

Fairness in medical agents is becoming critical as tool-using clinical AI systems orchestrate specialized vision and language modules for tasks such as chest X-ray question answering. While these medical AI agents can improve flexibility, their added pipeline complexity also creates new pathways for demographic bias beyond standalone models. We present DUCK, Decomposing Unfairness in Chest X-ray agents, a systematic audit of fairness in tool-using chest X-ray agents instantiated with MedRAX. To localize where disparities arise, we introduce a stage-wise fairness decomposition that separates end-to-end bias from three agent-specific sources: tool exposure bias, or utility gaps conditioned on tool presence; tool transition bias, or subgroup differences in tool-routing patterns; and model reasoning bias, or subgroup differences in synthesis behaviors. Extensive experiments on tool-using agentic frameworks across five driver backbones reveal that demographic gaps persist in end-to-end performance, with equalized odds up to 20.79% and the lowest fairness-utility tradeoff down to 28.65%. Intermediate behaviors, including tool usage, transition patterns, and reasoning traces, exhibit distinct subgroup disparities that are not predictable from end-to-end evaluation alone. For example, conditioned on segmentation-tool availability, the subgroup utility gap reaches as high as 50%. Our findings underscore the need for process-level fairness auditing and debiasing to ensure the equitable deployment of clinical agentic systems. Code: https://github.com/Nanboy-Ronan/DUCK.

Figures & tables

Figure 1: Overview of DUCX . (1) Dataset curation: we use CheXAgentBench and curate MIMIC-FairnessVQA into a standardized form. (2) MedRAX agent execution (ReAct): a driver LLM iteratively reasons, selects tools, and synthesizes a final answer, where we highlight multiple points where bias can be introduced (tool exposure, tool transitions, and final synthesis). (3) Fairness decomposition: We evaluate end-to-end fairness (ACC, $\Delta$ ACC, DP, EoD, FUT) and decompose it into tool exposure bias (gaps conditioned on tool presence), tool transition bias (gaps in tool routing), and LLM reasoning bias (gaps in synthesis behaviors).

Configuration		CheXAgentBench					MIMIC-FairnessVQA
Attr.	LLM	ACC $\uparrow$	$\Delta$ ACC $\downarrow$	DP $\downarrow$	EoD $\downarrow$	FUT $\uparrow$	ACC $\uparrow$	$\Delta$ ACC $\downarrow$	DP $\downarrow$	EoD $\downarrow$	FUT $\uparrow$
Gender	LLaMA3.1	$50.83_{[48.88,52.76]}$	${\color[rgb]{1,0,0}1.53_{[0.06,4.45]}}$	$5.04_{[2.32,7.93]}$	$15.10_{[5.63,26.98]}$	$50.29_{[48.20,52.30]}$	$32.89_{[30.85,34.95]}$	$2.00_{[0.10,5.86]}$	${\color[rgb]{1,0,0}2.99_{[1.12,6.28]}}$	$11.45_{[5.83,18.97]}$	$32.43_{[30.39,34.56]}$
	Ministral-3	$51.48_{[49.72,53.36]}$	${\color[rgb]{0,0,1}1.68_{[0.06,4.40]}}$	${\color[rgb]{1,0,0}3.66_{[1.48,6.73]}}$	$10.48_{[4.12,19.60]}$	$50.85_{[48.75,52.88]}$	$29.42_{[27.35,31.50]}$	$2.30_{[0.11,6.07]}$	$3.35_{[1.39,5.86]}$	${\color[rgb]{0,0,1}11.21_{[5.39,18.36]}}$	$28.95_{[26.86,31.07]}$
	Qwen3VL	${\color[rgb]{0,0,1}67.26_{[65.11,69.28]}}$	$3.21_{[0.21,7.11]}$	$4.47_{[1.42,8.12]}$	$11.28_{[4.87,22.64]}$	${\color[rgb]{0,0,1}65.98_{[63.41,68.45]}}$	${\color[rgb]{1,0,0}54.75_{[52.07,57.39]}}$	$2.17_{[0.11,6.29]}$	$4.22_{[1.74,7.56]}$	$11.47_{[4.93,20.61]}$	${\color[rgb]{1,0,0}53.92_{[51.02,56.77]}}$
	Qwen3	${\color[rgb]{1,0,0}69.10_{[67.24,70.92]}}$	$1.92_{[0.09,5.07]}$	${\color[rgb]{0,0,1}3.72_{[1.52,6.62]}}$	${\color[rgb]{1,0,0}9.16_{[3.45,19.62]}}$	${\color[rgb]{1,0,0}68.09_{[65.74,70.32]}}$	$49.89_{[47.55,52.10]}$	${\color[rgb]{1,0,0}1.90_{[0.09,5.30]}}$	${\color[rgb]{0,0,1}3.11_{[1.14,5.98]}}$	$11.32_{[5.06,19.56]}$	${\color[rgb]{0,0,1}49.23_{[46.77,51.37]}}$
	Gemini3	$50.60_{[48.64,52.64]}$	$3.62_{[0.36,7.47]}$	$4.99_{[2.61,7.99]}$	${\color[rgb]{0,0,1}9.56_{[4.83,16.61]}}$	$49.15_{[46.82,51.59]}$	$49.69_{[47.50,51.95]}$	${\color[rgb]{0,0,1}1.99_{[0.08,5.31]}}$	$3.13_{[1.36,5.38]}$	${\color[rgb]{1,0,0}11.01_{[5.31,19.10]}}$	$49.01_{[46.54,51.48]}$

Table 1: Overview of the end-to-end fairness performance.

\text{Mean}_{\text{CI}}

is reported. All values are in percentage. Best and Second Best in each group are highlighted.

Figure 2: Tool exposure bias across tools and subgroups. (a) Gender on CheXAgentBench, (b) Age on CheXAgentBench, (c) Gender on MIMIC-FairnessVQA, and (d) Age on MIMIC-FairnessVQA.

Figure 3: Tool transition biases represented as matrices. START: start indicator; CLS: classifier; RG: report generator; SEG: segmentator; VIS: visualizer; GRD: phrase grounding. Cells in red denote values larger than zero, while cells in blue denote values smaller than zero. $\Delta_{\text{TTB}}^{\text{gender}}=P^{\text{male}}-P^{\text{female}}$ , $\Delta_{\text{TTB}}^{\text{age}}=P^{\text{young}}-P^{\text{old}}$ . All subfigures present results averaged across different LLMs.

Configuration		CheXAgentBench			MIMIC-FairnessVQA
Attr.	LLM	$\Delta$ JudgeGap $\downarrow$	$\Delta$ Hedge $\downarrow$	$\Delta$ Demo. $\downarrow$	$\Delta$ JudgeGap $\downarrow$	$\Delta$ Hedge $\downarrow$	$\Delta$ Demo. $\downarrow$
Gender	LLaMA3.1	$6.19_{[0.21,19.00]}$	$4.97_{[0.25,11.66]}$	$0.09_{[0.02,1.66]}$	$0.10_{[0.30,21.70]}$	$3.10_{[0.20,12.40]}$	$0.10_{[0.00,0.30]}$
	Ministral-3	$2.86_{[0.15,8.92]}$	$12.20_{[0.63,31.73]}$	$0.56_{[0.06,3.74]}$	$3.10_{[0.40,23.40]}$	$4.90_{[0.60,44.40]}$	$0.20_{[0.00,1.20]}$
	Qwen3VL	$4.59_{[0.22,11.39]}$	$534.95_{[3.58,1438.18]}$	$1.57_{[0.08,4.21]}$	$21.67_{[2.66,41.90]}$	$831.10_{[44.75,3489.64]}$	$0.10_{[0.00,0.30]}$
	Qwen3	$5.58_{[0.47,11.69]}$	$12.58_{[0.65,51.20]}$	$3.63_{[0.57,6.88]}$	$1.90_{[0.30,24.80]}$	$4.40_{[1.00,69.20]}$	$0.00_{[0.00,0.00]}$
	Gemini3	$1.67_{[0.09,4.48]}$	$0.89_{[0.13,11.09]}$	$0.01_{[0.06,4.22]}$	$21.20_{[10.49,32.50]}$	$10.40_{[0.80,25.00]}$	$0.30_{[0.00,1.30]}$

Table 2: Subgroup gaps of reasoning-bias.

\text{Mean}_{\text{CI}}

is reported. All values are multiplied by 100 for better readability. Best and Second Best are highlighted.

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 419 papers

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning
Feb 25, 2026Zheang Huai, Honglong Yang, Xiaomeng LiMedical AI Agents Tool-Using Vision-Language Agents
The Hong Kong University of Science and Technology, Kowloon, Hong Kong
Representation Disentanglement for Fair Chest X-Ray Diagnosis
Oct 5, 2026Yujie Sun, Ruizhe Li, Xiaowu SunChest X-Ray Classification Representation Disentanglement
Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University, China · Nottingham Biomedical Research Centre, School of Medicine, University of Nottingham, UK
X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning
Oct 12, 2026Yi Wang, Zhenlong Yuan, Yuchan Guo +9Medical VLMs Medical Imaging
Institute of Computing Technology · University of Chinese Academy of Sciences · Carnegie Mellon University +6

Feb 25, 2026J/K move · Enter open · S save

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning

Zheang Huai, Honglong Yang, Xiaomeng Li

The Hong Kong University of Science and Technology, Kowloon, Hong Kong

Medical AI Agents Tool-Using Vision-Language Agents Multimodal Tool Use RL for Tool Use

Open paper AskPDF arXiv

AI agents with tool-use capabilities show promise for integrating the domain expertise of various tools. In the medical field, however, tools are usually AI models that are inherently error-prone and can produce contradictory responses. Existing research on medical agents lacks sufficient understanding of the tools' realistic reliability and thus cannot effectively resolve tool conflicts. To address this gap, this paper introduces a framework that enables an agent to interact with tools and empirically learn their practical trustworthiness across different types of multimodal queries via agentic learning. As a concrete instantiation, we focus on chest X-ray analysis and present a tool-expertise-aware chest X-ray agent (TEA-CXA). When tool outputs disagree, the agent experimentally accepts or rejects multimodal tool results, receives rewards, and learns which tool to trust for each query type. Importantly, TEA-CXA extends existing codebases for reinforcement learning with multi-turn tool-calling that focus on textual inputs, to support multimodal contexts effectively. In addition, we enhance the codebase for medical use scenarios by supporting multiple tool calls in one turn, parallel tool inference, and multi-image accommodation within a single user query. Our code framework is applicable to general medical research on multi-turn tool-calling reinforcement learning in multimodal settings. Experiments show that TEA-CXA outperforms the state-of-the-art methods and a comprehensive set of baselines. Code will be released.

Abstract

Algorithmic Fairness

Explore similar work

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning

Representation Disentanglement for Fair Chest X-Ray Diagnosis

X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning