Anterior's Approach to Fairness Evaluation of Automated Prior Authorization System

Sai P. Selvaraj; Khadija Mahmoud; Anuj Iravane

cs.LGMar 15, 2026

Anterior's Approach to Fairness Evaluation of Automated Prior Authorization System

Authors: Sai P. Selvaraj, Khadija Mahmoud, Anuj Iravane

Organizations: Anterior, Inc.

Abstract

Increasing staffing constraints and turnaround-time pressures in Prior authorization (PA) have led to increasing automation of decision systems to support PA review. Evaluating fairness in such systems poses unique challenges because legitimate clinical guidelines and medical necessity criteria often differ across demographic groups, making parity in approval rates an inappropriate fairness metric. We propose a fairness evaluation framework for prior authorization models based on model error rates rather than approval outcomes. Using 7,166 human-reviewed cases spanning 27 medical necessity guidelines, we assessed consistency in sex, age, race/ethnicity, and socioeconomic status. Our evaluation combined error-rate comparisons, tolerance-band analysis with a predefined $\pm$ 5 percentage-point margin, statistical power evaluation, and protocol-controlled logistic regression. Across most demographics, model error rates were consistent, and confidence intervals fell within the predefined tolerance band, indicating no meaningful performance differences. For race/ethnicity, point estimates remain small, but subgroup sample sizes were limited, resulting in wide confidence intervals and underpowered tests, with inconclusive evidence within the dataset we explored. These findings illustrate a rigorous and regulator-aligned approach to fairness evaluation in administrative healthcare AI systems.

Figures & tables

Attr.	Groups	Error rates	$\boldsymbol{\Delta}$	NHST (z, p)	95% CI	CI $\boldsymbol{\in\delta}$	Power
Sex	F, M	5.8%, 6.3%	$-0.5$	$-0.91,\ 0.364$	$[-1.74,0.65]$	Yes	1.00
Age	A, O	5.8%, 6.3%	$-0.5$	$-0.79,\ 0.429$	$[-1.69,0.71]$	Yes	1.00
Race	NW, W	6.7%, 7.3%	$-0.6$	$-0.26,\ 0.798$	$[-4.82,3.68]$	Yes	0.51
SES	L, H	5.9%, 5.7%	$+0.1$	$+0.10,\ 0.920$	$[-2.55,2.83]$	Yes	0.88

Table 1:

|

Unadjusted evaluation. Subgroup error-rate comparisons and unadjusted fairness evaluation across protected attributes. The table reports group error rates, absolute error-rate difference (

\Delta

), z-statistics, and p-values for NHST, 95% confidence intervals (CI) for error rate difference, inclusion within a predefined tolerance band (

\delta

\pm

5pp), and statistical power for detecting

\delta

with

\alpha=0.05

. All observed disparities are small, non-significant, and the CI remains within the tolerance threshold. Attribute groups: Female (F), Male (M), Adult (A), Older (O), Non-white (NW), White (W), Low SES (L), High SES (H)

Attr.	Adj. Set	Adj. $\boldsymbol{\Delta}$	95% CI	CI $\boldsymbol{\in\delta}$	Evi. Ratio (Strength)
Sex	Protocol + Race	$+0.52$	$[-0.74,1.69]$	Yes	24% (Strong evidence)
	Protocol + Age	$+0.51$	$[-0.64,1.57]$	Yes	22% (Strong evidence)
	Protocol + SES	$+0.16$	$[-1.11,1.51]$	Yes	26% (Strong evidence)
Race	Protocol + Sex	$+0.28$	$[-4.72,5.50]$	No	102% (Weak evidence)
	Protocol + Age	$+0.11$	$[-5.01,5.69]$	No	107% (Weak evidence)
	Protocol + SES	$-0.14$	$[-5.79,4.67]$	No	105% (Weak evidence)

Table 2:

|

Adjusted evaluation. Adjusted error-rate disparities across protected attributes after controlling for protocol and additional demographic covariates. The table reports adjusted risk differences (percentage points), 95% bootstrap confidence intervals (CI), inclusion within the predefined operational tolerance band (

\delta=\pm

5pp), the Evidence Ratio (CI width relative to

\delta

), and qualitative evidence strength. Across sex, age, and SES, adjusted disparities are small and fully contained within the tolerance threshold with strong to moderate precision; race-adjusted comparisons exhibit wider intervals due to limited sample size and are therefore less conclusive

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Attr.	Category	Frequency (%)
Sex	Female	3,961 (55.3%)
	Male	2,544 (35.5%)
	Ignored	0 (0.0%)
	Missing	661 (9.2%)
Age group	22–50 (Adult)	2,241 (31.3%)
	51+ (Older)	4,545 (63.4%)

Appendix

Table 3:

|

Demographic characteristics. Distribution of protected attributes extracted from clinical in the reviewed prior authorization cases (

N=7{,}166

)

Protocol Characteristic	Value
Number of distinct protocols	27
Median cases per protocol (IQR)	99 (63–339)
Percentage of cases in top 5 protocols	62.8%

Appendix

Table 4:

|

Protocol distribution. Aggregated payer protocol representation in the reviewed prior authorization cases (

N=7{,}166

)

Attr.	Protocols with Overlap	Valid Protocols	Overlap Rate
Sex	27	27	100.0%
Age group	25	27	92.6%
Race / Ethnicity	21	24	87.5%
SES / Payer	23	26	88.5%

Appendix

Table 5:

|

Protocol overlap. The table reports the number of protocols containing at least one case from each comparison group for a given attribute after excluding missing values. High overlap rates indicate that most protocols include representation from both groups, supporting identifiability of within-protocol comparisons in adjusted analyses.

Review Outcome	Frequency (%)
Correct approval	2,901 (40.5%)
False approval	230 (3.2%)
Correct escalation	3,832 (53.5%)
Unnecessary escalation	203 (2.8%)

Appendix

Table 6:

|

Outcome distribution. Human review distribution in the reviewed prior authorization cases

Attr.	Adj. Set	Adj. $\boldsymbol{\Delta}$	95% CI	CI $\boldsymbol{\in\delta}$	Evi. Ratio (Strength)
Sex	Protocol	$+0.49$	$[-0.69,1.66]$	Yes	23% (Strong evidence)
	Race	$+0.40$	$[-2.93,3.67]$	Yes	66% (Moderate evidence)
	Age	$+0.61$	$[-0.63,1.88]$	Yes	25% (Strong evidence)
	SES	$+0.80$	$[-1.16,2.96]$	Yes	41% (Strong evidence)
Race	Protocol	$+0.18$	$[-5.37,4.80]$	No	102% (Weak evidence)
	Sex	$+1.04$	$[-3.22,4.72]$	Yes	79% (Moderate evidence)

Appendix

Table 7:

|

Sensitivity analysis. Single-Factor adjusted error-Rate disparities across protected attributes after controlling for demographic attributes or protocol. The table reports adjusted risk differences (percentage points), 95% bootstrap confidence intervals (CI), inclusion within the predefined operational tolerance band (

\delta=\pm

5pp), the Evidence Ratio (CI width relative to

\delta

), and qualitative evidence strength. Across race, sex, age, and SES, adjusted disparities are mostly small and contained within the tolerance threshold with strong to moderate precision; on a few occasions, we obtain inconclusive results

Attr.	Adj. Set	Adj. $\boldsymbol{\Delta}$	95% CI	CI $\boldsymbol{\in\delta}$	Evi. Ratio (Strength)
Sex	Protocol + Race	$+0.23$	$[-3.30,3.94]$	Yes	72% (Moderate evidence)
	Protocol + Age	$+0.58$	$[-0.60,1.90]$	Yes	25% (Strong evidence)
	Protocol + SES	$+0.58$	$[-1.71,3.00]$	Yes	47% (Strong evidence)
Race	Protocol + Sex	$+1.00$	$[-4.53,5.19]$	No	97% (Weak evidence)
	Protocol + Age	$+0.12$	$[-5.38,5.08]$	No	105% (Weak evidence)
	Protocol + SES	$-3.91$	$[-12.70,4.18]$	No	169% (Weak evidence)

Appendix

Table 8:

|

Adjusted evaluation with “Unknown”. Adjusted error-rate disparities across protected attributes after controlling for protocol and additional demographic covariates (Complete-case covariates; no “Unknown” levels). The table reports adjusted risk differences (percentage points), 95% bootstrap confidence intervals (CI), inclusion within the predefined operational tolerance band (

\delta=\pm

5pp), the Evidence Ratio (CI width relative to

\delta

), and qualitative evidence strength. Across sex, age, and SES, adjusted disparities are small and fully contained within the tolerance threshold with strong to moderate precision; race-adjusted comparisons (and cases where we use race as covariates) exhibit wider intervals due to limited sample size and are therefore less conclusive

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 419 papers

AI-Generated Prior Authorization Letters: Strong Clinical Content, Weak Administrative Scaffolding
Mar 31, 2026Moiz Sadiq Awan, Maryam RazaMedical LLMs Medical Report Generation
Independent Researcher
Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems
Sep 29, 2026Kelly McConvey, Angelina Zhai, Rebecca Li +1Algorithmic Auditing Algorithmic Fairness
University of Toronto, Toronto, Ontario, Canada · Georgia Institute of Technology, Atlanta, Georgia, USA
When Fairness Metrics Disagree: Evaluating the Reliability of Demographic Fairness Assessment in Machine Learning
Apr 16, 2026Khalid Adnan AlsayedAlgorithmic Fairness Group Fairness
School of Computing, Engineering & Digital Technologies, Teesside University, UK

Mar 31, 2026J/K move · Enter open · S save

AI-Generated Prior Authorization Letters: Strong Clinical Content, Weak Administrative Scaffolding

Moiz Sadiq Awan, Maryam Raza

Independent Researcher

Medical LLMs Medical Report Generation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Prior authorization remains one of the most burdensome administrative processes in U.S. healthcare, consuming billions of dollars and thousands of physician hours each year. While large language models have shown promise across clinical text tasks, their ability to produce submission-ready prior authorization letters has received only limited attention, with existing work confined to single-case demonstrations rather than structured multi-scenario evaluation. We assessed three commercially available LLMs (GPT-4o, Claude Sonnet 4.5, and Gemini 2.5 Pro) across 45 physician-validated synthetic scenarios spanning rheumatology, psychiatry, oncology, cardiology, and orthopedics. All three models generated letters with strong clinical content: accurate diagnoses, well-structured medical necessity arguments, and thorough step therapy documentation. However, a secondary analysis of real-world administrative requirements revealed consistent gaps that clinical scoring alone did not capture, including absent billing codes, missing authorization duration requests, and inadequate follow-up plans. These findings reframe the question: the challenge for clinical deployment is not whether LLMs can write clinically adequate letters, but whether the systems built around them can supply the administrative precision that payer workflows require.

Abstract

Algorithmic Fairness

Explore similar work

AI-Generated Prior Authorization Letters: Strong Clinical Content, Weak Administrative Scaffolding

Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems

When Fairness Metrics Disagree: Evaluating the Reliability of Demographic Fairness Assessment in Machine Learning