Identifying and Mitigating Gender Cues in Academic Recommendation Letters: An Interpretability Case Study

Charlotte S. Alexander; Shane Storks; Souradip Pal; Sayak Chakrabarty; Arushi Sharma; Mlen-Too Wesley; Bailey Russo

cs.LGApr 14, 2026

Identifying and Mitigating Gender Cues in Academic Recommendation Letters: An Interpretability Case Study

Authors: Charlotte S. Alexander, Shane Storks, Souradip Pal, Sayak Chakrabarty, Arushi Sharma, Mlen-Too Wesley, Bailey Russo

Organizations: Scheller College of Business, Georgia Institute of Technology, Atlanta, GA 30308, USA · Weinberg Institute for Cognitive Science, University of Michigan - Ann Arbor, Ann Arbor, MI 48109, USA · School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA ·

Abstract

Letters of recommendation (LoRs) can carry patterns of implicitly gendered language that can inadvertently influence downstream decisions, e.g. in hiring and admissions. In this work, we investigate the extent to which Transformer-based encoder models as well as Large Language Models (LLMs) can infer the gender of applicants in academic LoRs submitted to an U.S. medical-residency program after explicit identifiers like names and pronouns are de-gendered. While using three models (DistilBERT, RoBERTa, and Llama 2) to classify the gender of anonymized and de-gendered LoRs, significant gender leakage was observed as evident from up to 68% classification accuracy. Text interpretation methods, like TF-IDF and SHAP, demonstrate that certain linguistic patterns are strong proxies for gender, e.g. "emotional'' and "humanitarian'' are commonly associated with LoRs from female applicants. As an experiment in creating truly gender-neutral LoRs, these implicit gender cues were remove resulting in a drop of up to 5.5% accuracy and 2.7% macro $F_1$ score on re-training the classifiers. However, applicant gender prediction still remains better than chance. In this case study, our findings highlight that 1) LoRs contain gender-identifying cues that are hard to remove and may activate bias in decision-making and 2) while our technical framework may be a concrete step toward fairer academic and professional evaluations, future work is needed to interrogate the role that gender plays in LoR review. Taken together, our findings motivate upstream auditing of evaluative text in real-world academic letters of recommendation as a necessary complement to model-level fairness interventions.

Figures & tables

Figure 1: Overview of our case-study pipeline: corpus construction, explicit de-gendering, model training, interpretability audit, and implicit de-gendering.

Panel of Figure 2 — Figure 2: Top 10 SHAP tokens with their corresponding values(+/-) for both genders (male/ female) grouped by part-of-speech category, identified by DistilBERT(last two encoder layers unfrozen), RoBERTa and Llama 2

Token	F $\rightarrow$ M Count	M $\rightarrow$ F Count	Absolute Difference
support	12	23.56	11.56
research	14	24.64	10.64
number	4	11.98	7.98
anesthesia	13	20.96	7.96
rotation	26	18.26	7.74
medicine	17	23.29	6.29

Table 1: Token-level gender prediction flips for TF-IDF derived tokens: comparison of top 10 tokens whose removal most frequently causes DistilBERT model prediction to flip from female to male or vice versa.

Token	F $\rightarrow$ M Count	M $\rightarrow$ F Count	Absolute Difference
liked	6	0.38	5.62
impressed	14	8.93	5.07
bedside	6	2.8	3.2
stellar	3	0.37	2.63
summary	1	3.16	2.16
thoughtful	4	1.97	2.03

Table 2: Token-level gender prediction flips for SHAP derived tokens: comparison of top tokens whose removal most frequently causes DistilBERT model prediction to flip from female to male or vice versa.

Model	Gender	Precision	Recall	$F_{1}$	Acc.	Macro Precision	Macro Recall	Macro $F_{1}$	Wtd. Precision	Wtd. Recall	Wtd. $F_{1}$
DistilBERT	Female	0.413	0.403	0.41	0.633	0.573	0.573	0.577	0.637	0.633	0.633
DistilBERT	Male	0.733	0.74	0.737	0.633	0.573	0.573	0.577	0.637	0.633	0.633
RoBERTa	Female	0.39	0.397	0.387	0.61	0.55	0.55	0.553	0.617	0.61	0.61
RoBERTa	Male	0.72	0.703	0.68	0.61	0.55	0.55	0.553	0.617	0.61	0.61
Llama 2	Female	0.395	0.444	0.418	0.612	0.560	0.563	0.561	0.623	0.612	0.615
Llama 2	Male	0.725	0.682	0.703	0.612	0.560	0.563	0.561	0.623	0.612	0.615

Table 3: Comparison of DistilBERT, RoBERTa, and Llama 2 performance on test EDG(Explicitly De-Gendered) dataset including individual class metrics (Precision, Recall and

F_{1}

) and aggregate metrics(Macro + Weighted Precision, Recall and

F_{1}

Dataset	DistilBERT				RoBERTa				Llama 2
Dataset	Acc.	Macro P	Macro R	Macro $F_{1}$	Acc.	Macro P	Macro R	Macro $F_{1}$	Acc.	Macro P	Macro R	Macro $F_{1}$
Original (non-EDG)	0.999	1.000	0.996	0.999	–	–	–	–	–	–	–	–
EDG (baseline)	0.633	0.573	0.573	0.577	0.61	0.55	0.55	0.553	0.612	0.56	0.563	0.561
EDG w/o SHAP Tokens	0.597	0.553	0.56	0.55 ( $\downarrow$ 2.7%)	0.585	0.56	0.57	0.55 $\downarrow$ 0.3%)	0.690	0.345	0.500	0.408 ( $\downarrow$ 15.3%)
EDG w/o TF-IDF Tokens	0.61	0.567	0.57	0.563 ( $\downarrow$ 1.4%)	0.555	0.54	0.54	0.53 ( $\downarrow$ 2.3%)	0.685	0.344	0.497	0.407 ( $\downarrow$ 15.4%)

Table 4: Comparison of DistilBERT, RoBERTa, and Llama 2 performance across three datasets: EDG (Explicitly De-Gendered), EDG with top SHAP-identified gender tokens removed, and EDG with top TF-IDF-identified gender tokens removed showing improvements(lowering) in macro precision, recall, and

F_{1}

scores

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Best hyper-parameters selected for DistilBERT and RoBERTa text classifiers..

Parameter	Value
Batch size (train / eval)	4 / 4
Epochs	5
Learning rate	$3.0\times 10^{-5}$
Weight decay	0.02
LoRA rank ( $r$ )	16
LoRA alpha	48

Appendix

Table 6: Best Llama 2 fine-tuning hyper-parameters

Model	Gender	Precision	Recall	$F_{1}$	Acc.	Macro Precision	Macro Recall	Macro $F_{1}$	Wtd. Precision	Wtd. Recall	Wtd. $F_{1}$
Topic Vectors Only	Female	0.370	0.480	0.420	0.580	0.550	0.560	0.550	0.620	0.580	0.600
Topic Vectors Only	Male	0.730	0.630	0.680	0.580	0.550	0.560	0.550	0.620	0.580	0.600
Topic + DistilBERT Embeddings	Female	0.410	0.440	0.430	0.630	0.580	0.580	0.580	0.640	0.630	0.640
Topic + DistilBERT Embeddings	Male	0.740	0.720	0.730	0.630	0.580	0.580	0.580	0.640	0.630	0.640

Appendix

Table 7: Classification performance of Random Forest models using (1) only topic vectors and (2) a combination of topic vectors and DistilBERT embeddings.

Figure 3: SHAP summary plot showing the most influential topics in predicting gender from recommendation letters using a Random Forest classifier trained on BERTopic vectors. Each row represents a topic, labeled with its BERTopic ID followed by the top five unigrams or bigrams (joined with underscores) extracted via TF-IDF. Each dot corresponds to a single letter; the horizontal position indicates the SHAP value, reflecting the topic’s contribution to the prediction (positive values push toward " male " negative toward " female "). Red dots represent the presence of a topic in a letter, while blue dots represent its absence.

Table 11

Table 12

Table 13

Gender Bias in Language Models

Gender bias in language models examines how systems encode and express gender stereotypes and how to evaluate and reduce their effects.

See all 66 papers

Gender bias across LLMs is common and highly heterogeneous
Sep 29, 2026Edoardo Bolzoni, Valerio CapraroGender Bias in Language Models LLM Auditing
University of Milan-Bicocca, Milan, Italy
Beyond the Name: Demographic Leakage in De-Identified Résumés and Evaluation Artifacts in LLM Bias Audits
Sep 15, 2026Qiangju Chen, Yang XiaoLLM-as-a-Judge Language Model Bias Evaluation
Macquarie University · The University of Melbourne
Quantifying Gender Bias in Large Language Models: When ChatGPT Becomes a Hiring Manager
Mar 10, 2026Nina Gerszberg, Janka Hamori, Andrew LoGender Bias in Language Models Algorithmic Bias
Massachusetts Institute of Technology Cambridge, Massachusetts, USA

Sep 29, 2026J/K move · Enter open · S save

Gender bias across LLMs is common and highly heterogeneous

Edoardo Bolzoni, Valerio Capraro

University of Milan-Bicocca, Milan, Italy

Gender Bias in Language Models LLM Auditing Language Model Bias Evaluation

Open paper AskPDF arXiv

Understanding gender biases in large language models (LLMs) is increasingly important as these systems become embedded in decision-support tools with real consequences. Prior research has focused only on a small set of models, leaving open the extent to which gender biases are common and heterogeneous across LLMs. We address this gap across ten models released between April 2025 and June 2026, spanning nine vendors, using two paradigms: gender attribution to stereotyped phrases (Study 1) and moral judgment of abuse or torture against a woman or a man to prevent a catastrophic outcome (Study 2). In Study 1, two of ten models attributed masculine-stereotyped phrases to female writers more often than the reverse, while three models showed the opposite pattern. In Study 2, several models converged on a male-disadvantaging asymmetry that was directionally consistent with a documented human tendency to protect female targets from harm, though the specific conditions under which this asymmetry emerged varied by model; three other models, by contrast, showed no variation across conditions. These results indicate that gender-related biases are common in LLMs. Their direction and magnitude, however, are highly heterogeneous, to the point that some models behave in diametrically opposite ways to others. Bias auditing should therefore be treated as an ongoing, multi-vendor process, rather than a one-time assessment.

Abstract

Gender Bias in Language Models

Explore similar work

Gender bias across LLMs is common and highly heterogeneous

Beyond the Name: Demographic Leakage in De-Identified Résumés and Evaluation Artifacts in LLM Bias Audits

Quantifying Gender Bias in Large Language Models: When ChatGPT Becomes a Hiring Manager