Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice

Syeda Anshrah Gillani; Mirza Samad Ahmed Baig

cs.CYAug 14, 2026

Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice

Authors: Syeda Anshrah Gillani, Mirza Samad Ahmed Baig

Organizations: Heidelberg University, Heidelberg, Germany · Fandaqah, Al Khobar, Saudi Arabia

Abstract

Patients increasingly ask large language model (LLM) assistants which doctor to see, making these systems AI infomediaries: algorithms that intermediate one person's choice among other people and thereby decide, silently and at scale, which physicians become visible. We report a prespecified randomized algorithm audit of what causally moves those recommendations. Seven models (six open-weight; gpt-4o-mini) each chose among five synthetic family-medicine physician cards whose attributes were independently randomized across 3,024 choice sets, three patient personas, nine prompt paraphrases and nine experimental arms, yielding 40,068 scored responses; gender and ethnicity were signaled through names following correspondence-audit methodology. Reputation signals dominate: raising a rating from 3.9 to 4.7 increases choice probability by 31.4 percentage points (pp), and raising the fee from $90 to$ 190 lowers it by 20.0 pp. Demographic parity is rejected, but not in the direction human audit studies predict: female-signaled names gain 2.5 pp, and Hispanic-, South-Asian- and Black-signaled names gain 1.3-2.9 pp over White-signaled names, tilts worth $7-$ 14 per visit in fee-equivalent terms, and a content-free first-listed position is worth $11. Yet models mentioned gender or ethnicity in at most 0.03% of their stated reasons and abstained in 0.39% of trials, so these effects are invisible in the models' own explanations, and transparency obligations relying on model self-report would not detect them. One reasoning model failed the prespecified auditability gate outright. The frozen design makes the audit repeatable: any new model can be assessed against identical stimuli, making recurring behavioural audit, rather than self-reported explanation, the monitoring technology fit for purpose.

Figures & tables

Panel of Figure 1 — Figure 1: Audit pipeline. A design frozen (and hashed) before any data collection generates 3,024 five-card choice sets whose attributes vary independently; each audited model answers every choice set under nine prespecified arms; average marginal component effects and prespecified equivalence tests are estimated from the pooled choices. Because randomization is at the attribute level, any systematic difference in choice probability identifies the causal weight the model places on that signal.

Panel of Figure 2 — Figure 2: One physician card as presented to the models (card layout shown; the grounded-arm snippet and reversed-layout variants are described in Methods). Black lines vary independently between cards; gray lines are identical on every card, so specialty, board certification, and distance cannot drive choices. Each choice set presents five such cards, and the model is asked to choose one for the persona and give a brief reason.

Model	letter	main	rank_top3	retest
gemma3:4b	0.0	0.8	0.0	0.0
gpt-4o-mini	0.0	0.0	0.0	0.0
llama3.1:8b	0.0	0.0	0.0	0.0
llama3.2:3b	1.7	0.0	0.7	0.0
mistral:7b-instruct-q4_K_M	0.0	0.0	14.1	32.0
phi3:mini	0.0	0.0	7.1	0.0

Table 1: Response parse-failure rates.

Model	Modal share	Entropy
gemma3:4b	0.993	0.015
gpt-4o-mini	0.967	0.064
llama3.1:8b	0.919	0.155
llama3.2:3b	0.856	0.295
mistral:7b-instruct-q4_K_M	0.732	0.560
phi3:mini	0.822	0.368

Table 2: Test–retest consistency of recommendations.

Figure 3: Average marginal component effects on choice probability (pooled and per model), with 95% choice-set–clustered confidence intervals. Baseline choice probability is 20% (five cards per set).

Attribute	AMCE (pp)	95% CI	$P$
Rating 4.3 (vs 3.9)	8.06	[7.34, 8.78]	¡.001
Rating 4.7 (vs 3.9)	31.38	[30.48, 32.29]	¡.001
85 reviews (vs 12)	4.79	[3.93, 5.64]	¡.001
400 reviews (vs 12)	8.44	[7.56, 9.31]	¡.001
Recent review (vs 11 mo.)	3.80	[3.09, 4.51]	¡.001
Responds to patient feedback	2.09	[1.37, 2.80]	¡.001

Table 3: Average marginal component effects on P(recommended), pooled across the model panel (percentage points).

Hypothesis	Wald	df	$P$ (raw)	$P$ (Holm)
H1_rating	4636.25	2	¡.001	¡.001***
H2_volume	363.61	2	¡.001	¡.001***
H3_recency	111.31	1	¡.001	¡.001***
H4_response	32.42	1	¡.001	¡.001***
H5_network	0.88	1	.349	.349
H6_price	1745.92	2	¡.001	¡.001***

Table 4: Confirmatory hypothesis tests (joint Wald tests, Holm-corrected).

Attribute	Range (pp)	Share
Patient rating	31.38	0.377
Visit fee	20.02	0.241
Review volume	8.44	0.101
List position	6.83	0.082
Recency	3.80	0.046
Ethnicity (name-signaled)	2.88	0.035

Table 5: Attribute importance: range of AMCEs and normalized share.

Figure 4: Choice-probability advantage of each gender

\times

ethnicity cell relative to White-male cards, with 95% confidence intervals and the prespecified

\pm

1.5 pp equivalence band.

Gender $\times$ ethnicity cell	AMCE (pp)	95% CI	$P$	TOST $P$
White male (ref.)	0.00	–	–	–
Black male	0.52	[-1.02, 2.05]	.510	.104
Hispanic male	1.74	[0.19, 3.30]	.028	.620
East Asian male	0.77	[-0.79, 2.32]	.333	.177
South Asian male	2.41	[0.81, 4.01]	.003	.868
White female	1.81	[0.26, 3.35]	.022	.651

Table 6: Intersectional gender

\times

ethnicity effects on P(recommended).

Model	$n$	Abstain (%)	Demographic-aware (%)
gemma3:4b	3024	0.0	0.0
gpt-4o-mini	3024	0.0	0.0
llama3.1:8b	3024	0.1	0.0
llama3.2:3b	3024	0.0	0.0
mistral:7b-instruct-q4_K_M	3024	0.6	0.0
phi3:mini	3024	1.2	0.1

Table 7: Abstention and demographic-aware response rates.

Figure 5: List-position effects on choice probability by model, holding card content constant. Slot 1 is the reference.

Term	Coef. (pp)	SE (pp)	$P$
Position 2 (vs 1)	4.42	3.50	.206
Position 3 (vs 1)	7.95	3.59	.027
Position 4 (vs 1)	3.99	3.44	.246
Position 5 (vs 1)	1.24	3.30	.707
Letter B (vs A)	0.82	3.68	.824
Letter C (vs A)	2.90	3.75	.438

Table 8: Letter-arm decomposition of list-position effects from letter-token preferences (linear probability model on first-token letter probabilities, trial-clustered SEs).

Attribute	Pooled	New in town	Chronic care	Budget
Rating 4.3 (vs 3.9)	40.3 [36.1, 44.6]	85.2 [72.5, 100.4]	93.0 [78.8, 110.0]	14.2 [10.3, 18.1]
Rating 4.7 (vs 3.9)	156.8 [147.4, 167.2]	366.4 [321.9, 425.1]	343.5 [301.3, 396.6]	46.5 [42.1, 51.1]
85 reviews (vs 12)	23.9 [19.4, 28.5]	66.6 [52.4, 83.1]	57.3 [43.3, 73.3]	4.3 [0.4, 8.2]
400 reviews (vs 12)	42.2 [37.2, 47.3]	106.5 [89.4, 127.1]	105.9 [89.0, 126.2]	7.6 [3.6, 11.7]
Recent review (vs 11 mo.)	19.0 [15.4, 22.7]	42.6 [31.3, 55.6]	41.8 [30.6, 54.2]	6.8 [3.5, 10.0]
Responds to patient feedback	10.4 [6.9, 14.1]	13.7 [3.0, 25.1]	22.3 [11.7, 33.0]	5.7 [2.3, 9.1]

Table 9: Fee-equivalent trade-offs ($/visit) with Krinsky–Robb 95% CIs.

Figure 6: Fee AMCEs by patient persona. The uninsured out-of-pocket persona shows the steepest price penalty.

Figure 7: Rating effects conditional on review volume: high volume amplifies the high-rating premium.

Figure 8: Stated versus revealed importance by attribute and model: share of stated reasons mentioning each attribute (stated) against its share of total AMCE range (revealed). Demographic attributes and position sit far below the diagonal.

Figure 9: Headline AMCEs across robustness arms (grounded snippets, reversed layout, field order, temperature 0, persona-free, plausibility-filtered): signs and ordering are stable throughout.

	Haiku 4.5	Sonnet 5	Opus 4.8	Fable 5
Rating 3.9	0.01	0.00	0.01	0.01
Rating 4.3	0.14	0.18	0.18	0.16
Rating 4.7	0.43	0.40	0.39	0.40
Fee $90	0.36	0.37	0.35	0.35
Fee $140	0.17	0.17	0.17	0.17
Fee $190	0.06	0.05	0.08	0.08

Table 10: Exploratory session pilot of four Claude-family models: share of cards chosen by attribute level (null

=0.20

). Each model saw the first 50 main-arm choice sets of the frozen design (250 cards; 0 parse failures). Responses were collected interactively inside an agentic coding harness with default decoding parameters, not via the Messages API under the audit protocol; estimates are descriptive and exploratory, are quarantined from the confirmatory pipeline, and enter no confirmatory test.

Large Language Model-Based Recommendation

Large language model-based recommendation uses language models to personalize ranking, retrieval, and conversational suggestions from text and behavior.

See all 210 papers

Sep 29, 2026

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.

Abinitha Gourabathina, Haoran Zhang, Yuexing Hao +2

Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science

Clinical Triage Clinical Decision-Making Language Model Bias Evaluation Clinical Language Model Evaluation Language Model Robustness

Ask PDF

Sep 16, 2026

Understanding AI Provider Recommendations in Local Service Markets

When someone asks an AI assistant which doctor to see or which firm to trust with their savings, the answer is a referral. We audit AI provider recommendations in four registry-backed service domains across the 100 largest U.S. metropolitan areas, matching every recommendation against the official registry for its domain (Medicare clinician and facility records, and SEC adviser disclosures), under three conditions: an open-weight model, a proprietary model without web search, and the same proprietary model with search. Without search, both models largely fabricate recommendations in the domains the web covers thinly. Only 4% of the open-weight model's recommended doctors and 11% of the proprietary model's match a clinician in the queried city, and the open-weight matches are name coincidences: its matched clinicians are no likelier to be primary-care doctors than names drawn at random from the registry. With search, 64-71% of recommendations in the same domains match a real provider. Search also changes who is recommended. Without it, recommended advisory firms carry SEC misconduct disclosures at 3.6 times the registry base rate, even after adjusting for firm size; with search, significantly below it. Restaurants, where quality and visibility are separately measurable, show a 3-5x review-count premium but a rating premium of at most a tenth of a star. Finally, search largely removes the metro-size penalty: without it, real recommendations concentrate in the largest metros; with it, match rates are similar across metro-size terciles. Whether an AI referral is trustworthy depends strongly on its retrieval configuration rather than on the underlying model alone, yet an answer produced without retrieval often carries no sign that its recommendations were never verified.

Hazem Ibrahim, Yasir Zaki

New York University Abu Dhabi Abu Dhabi, United Arab Emirates

Retrieval-Augmented Generation Large Language Model-Based Recommendation Trust in AI

Ask PDF

Sep 8, 2026

The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.

Siddharth Vohra, Manikandan Ravikiran

Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA · Indian Institute of Technology Mandi, India

Language Model Bias Evaluation Demographic Bias in Language Models Position Bias Black-Box Model Auditing Language Model Auditing

Ask PDF

Abstract

Large Language Model-Based Recommendation

Explore similar work

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

Understanding AI Provider Recommendations in Local Service Markets

The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits