Ablation Study of a Fairness Auditing Agentic System for Bias Mitigation in Early-Onset Colorectal Cancer Detection

Amalia Ionescu; Jose Guadalupe Hernandez; Jui-Hsuan Chang; Emily F. Wong; Paul Wang; Jason H. Moore; Tiffani J. Bright

cs.MAMar 17, 2026

Ablation Study of a Fairness Auditing Agentic System for Bias Mitigation in Early-Onset Colorectal Cancer Detection

Authors: Amalia Ionescu, Jose Guadalupe Hernandez, Jui-Hsuan Chang, Emily F. Wong, Paul Wang, Jason H. Moore, Tiffani J. Bright

Organizations: Cedars-Sinai Medical Center, Los Angeles, CA, USA

Abstract

Artificial intelligence (AI) is increasingly used in clinical settings, yet limited oversight and domain expertise can allow algorithmic bias and safety risks to persist. This study evaluates whether an agentic AI system can support auditing biomedical machine learning models for fairness in early-onset colorectal cancer (EO-CRC), a condition with documented demographic disparities. We implemented a two-agent architecture consisting of a Domain Expert Agent that synthesizes literature on EO-CRC disparities and a Fairness Consultant Agent that recommends sensitive attributes and fairness metrics for model evaluation. An ablation study compared three Ollama large language models (8B, 20B, and 120B parameters) across three configurations: pretrained LLM-only, Agent without Retrieval-Augmented Generation (RAG), and Agent with RAG. Across models, the Agent with RAG achieved the highest semantic similarity to expert-derived reference statements, particularly for disparity identification, suggesting agentic systems with retrieval may help scale fairness auditing in clinical AI.

Figures & tables

Ground Truth Agent 1

Ground Truth Agent 2

“Yes, there are well-documented disparities in early-onset colorectal cancer in both diagnosis and prognosis. Specifically, racial and ethnic minorities, particularly Black and Hispanic patients, and individuals from lower socioeconomic backgrounds are more likely to experience delayed diagnosis, present with more advanced disease, and have worse survival compared with non-Hispanic White patients. Men with early-onset colorectal cancer also have poorer survival outcomes than women. These disparities are driven by differences in access to care, delays in symptom evaluation and treatment, insurance and neighborhood socioeconomic factors, persistent inequities in stage at diagnosis and receipt of timely care, as well as emerging evidence of race- and ethnicity-associated genetic and epigenetic differences that may influence tumor behavior.”

“Given that the documented disparities lie in missed and delayed diagnoses, and worse survival among Black, Hispanic, lower-SES patients, and men, you should consider including equal opportunity, false negative rate parity, and equalized odds if feasible.”

Table 1: Expert-Driven Ground Truth statements used to evaluate performance for Agents 1 and 2.

Figure 1: Conceptual model of the agentic fairness auditing framework.

Model	Agent	Condition	$n$	$M$	$Mdn$	$IQR$
Llama 3.1 8B	Agent 1	LLM	100	0.775	0.780	0.081
		Agent (NR)	100	0.754	0.758	0.085
		Agent (R)	100	0.802	0.812	0.041
	Agent 2	LLM	100	0.605	0.600	0.042
		Agent (NR)	100	0.596	0.597	0.034
		Agent (R)	100	0.686	0.692	0.048

Table 2: Semantic similarity across model scales for Agent 1 and Agent 2 relative to the respective ground truth statements. Values represent mean (

M

), median (

Mdn

), and interquartile range (

IQR

Model	Agent	Overall test	Pairwise comparison ( $p$ Holm)
Llama 3.1 8B	Agent 1	$H(2)=39.56$ , $p<.001$	Agent (NR) $<$ LLM ( $p=.007$ )
			Agent (R) $>$ LLM ( $p<.001$ )
			Agent (R) $>$ Agent (NR) ( $p<.001$ )
	Agent 2	$H(2)=173.73$ , $p<.001$	Agent (NR) $<$ LLM ( $p=.014$ )
			Agent (R) $>$ LLM ( $p<.001$ )
			Agent (R) $>$ Agent (NR) ( $p<.001$ )

Table 3: Statistical tests for semantic similarity across model scales. Overall differences were evaluated using Kruskal–Wallis tests. Pairwise comparisons were conducted using Wilcoxon rank-sum tests with Holm correction.

Figure 2: Distribution of semantic similarity scores between output generated by Agents 1 and 2 across three conditions and the relevant ground truth statements. Results are shown for two agent roles: the Domain Expert Agent (Agent 1) and the Fairness Consultant Agent (Agent 2) across three model sizes. Violin plots display the distribution of similarity scores with embedded boxplots and individual observations. Comparisons are shown for the LLM and Agent without RAG (Agent NR) and Agent with RAG (Agent R). Statistical significance of pairwise comparisons is indicated by asterisks, with reported p-values where differences are not statistically significant.

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 419 papers

An Agentic Generative Large Language Model for Treatment Planning of Colorectal Cancer
Aug 10, 2026Mengxian Lyu, Cheng Peng, Tim Jang +18Clinical Decision-Making Domain Adaptation for LLMs
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA. · Division of Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA · University of Florida Health Cancer Institute, Gainesville, FL, USA +8
Does Splitting a Triage Decision Across Agents Hide Bias or Help Catch It? A Multi-Agent Simulation Study of LLM-Based Resource Allocation Under Audit Capacity Constraints
Aug 7, 2026Paul-Peter ArslanLLM Auditing Language Model Bias Evaluation
Institute for Future Technologies
DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents
Feb 28, 2026Zikang Xu, Ruinan Jin, Xiaoxiao LiAlgorithmic Fairness Algorithmic Bias
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Anhui, China · The University of British Columbia, Vancouver, BC V6Z 1Z4, Canada · Vector Institute, Toronto, ON M5G 1M1, Canada

Aug 10, 2026J/K move · Enter open · S save

An Agentic Generative Large Language Model for Treatment Planning of Colorectal Cancer

Mengxian Lyu, Cheng Peng, Tim Jang, Ang Li+17

Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA. · Division of Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA · University of Florida Health Cancer Institute, Gainesville, FL, USA+8

Clinical Decision-Making Domain Adaptation for LLMs Oncology Clinical Language Model Evaluation Medical LLMs

Open paper AskPDF arXiv

Treatment planning in precision oncology requires synthesizing heterogeneous patient information with rapidly evolving clinical guidelines to ensure guideline-concordant care. While large language models (LLMs) show promise in many diagnostic tasks, their adoption for high-stakes treatment planning is hindered by complex reasoning, adherence to timely clinical guidelines, and safety concerns. In this study, we present GatorOnco, an agentic LLM for colorectal cancer (CRC) treatment planning. GatorOnco is developed using a total of 282 billion tokens of biomedical text, including healthcare system-scale clinical text comprising 166 billion tokens from UF Health. We implemented a domain-adaptation method that integrates pre-training, model merging, a two-stage post-training approach, and agent-based reinforcement learning. An agentic retrieval-augmented generation (RAG) approach dynamically integrates time-sensitive clinical guidelines into the reasoning process. In a blind, randomized clinical evaluation conducted by five UF Health oncologists, GatorOnco significantly outperformed open-source LLMs (P < 0.01) and achieved expert-level performance comparable to UF Health oncologists. Compared with expert oncologists, GatorOnco received significantly higher ratings for readability (4.46 vs. 4.19, P < 0.01) and completeness (3.91 vs. 3.52, P < 0.01), while showing statistically comparable performance in correctness (4.09 vs. 4.11, P = 0.921), currency (4.04 vs. 3.98, P = 0.478), and safety (4.22 vs. 4.22, P = 0.999). These findings demonstrate that integrating agentic reasoning with large-scale domain adaptation can help bridge the gap for generative AI in high-stakes cancer treatment planning.

Abstract

Algorithmic Fairness

Explore similar work

An Agentic Generative Large Language Model for Treatment Planning of Colorectal Cancer

Does Splitting a Triage Decision Across Agents Hide Bias or Help Catch It? A Multi-Agent Simulation Study of LLM-Based Resource Allocation Under Audit Capacity Constraints

DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents