Identifying and Mitigating Gender Cues in Academic Recommendation Letters: An Interpretability Case Study
Authors: Charlotte S. Alexander, Shane Storks, Souradip Pal, Sayak Chakrabarty, Arushi Sharma, Mlen-Too Wesley, Bailey Russo
Organizations: Scheller College of Business, Georgia Institute of Technology, Atlanta, GA 30308, USA · Weinberg Institute for Cognitive Science, University of Michigan - Ann Arbor, Ann Arbor, MI 48109, USA · School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Iowa State University, Ames, IA 50011, USA · College of Computing, Georgia Institute of Technology, Atlanta, GA 30332, USA
Letters of recommendation (LoRs) can carry patterns of implicitly gendered language that can inadvertently influence downstream decisions, e.g. in hiring and admissions. In this work, we investigate the extent to which Transformer-based encoder models as well as Large Language Models (LLMs) can infer the gender of applicants in academic LoRs submitted to an U.S. medical-residency program after explicit identifiers like names and pronouns are de-gendered. While using three models (DistilBERT, RoBERTa, and Llama 2) to classify the gender of anonymized and de-gendered LoRs, significant gender leakage was observed as evident from up to 68% classification accuracy. Text interpretation methods, like TF-IDF and SHAP, demonstrate that certain linguistic patterns are strong proxies for gender, e.g. "emotional'' and "humanitarian'' are commonly associated with LoRs from female applicants. As an experiment in creating truly gender-neutral LoRs, these implicit gender cues were remove resulting in a drop of up to 5.5% accuracy and 2.7% macro F1 score on re-training the classifiers. However, applicant gender prediction still remains better than chance. In this case study, our findings highlight that 1) LoRs contain gender-identifying cues that are hard to remove and may activate bias in decision-making and 2) while our technical framework may be a concrete step toward fairer academic and professional evaluations, future work is needed to interrogate the role that gender plays in LoR review. Taken together, our findings motivate upstream auditing of evaluative text in real-world academic letters of recommendation as a necessary complement to model-level fairness interventions.
Figures & tables
Figure 1: Overview of our case-study pipeline: corpus construction, explicit de-gendering, model training, interpretability audit, and implicit de-gendering.
Figure 2: Top 10 SHAP tokens with their corresponding values(+/-) for both genders (male/ female) grouped by part-of-speech category, identified by DistilBERT(last two encoder layers unfrozen), RoBERTa and Llama 2
Token
F → M Count
M → F Count
Absolute Difference
support
12
23.56
11.56
research
14
24.64
10.64
number
4
11.98
7.98
anesthesia
13
20.96
7.96
rotation
26
18.26
7.74
medicine
17
23.29
6.29
Table 1: Token-level gender prediction flips for TF-IDF derived tokens: comparison of top 10 tokens whose removal most frequently causes DistilBERT model prediction to flip from female to male or vice versa.
Token
F → M Count
M → F Count
Absolute Difference
liked
6
0.38
5.62
impressed
14
8.93
5.07
bedside
6
2.8
3.2
stellar
3
0.37
2.63
summary
1
3.16
2.16
thoughtful
4
1.97
2.03
Table 2: Token-level gender prediction flips for SHAP derived tokens: comparison of top tokens whose removal most frequently causes DistilBERT model prediction to flip from female to male or vice versa.
Model
Gender
Precision
Recall
F1
Acc.
Macro Precision
Macro Recall
Macro F1
Wtd. Precision
Wtd. Recall
Wtd. F1
DistilBERT
Female
0.413
0.403
0.41
0.633
0.573
0.573
0.577
0.637
0.633
0.633
Male
0.733
0.74
0.737
RoBERTa
Female
0.39
0.397
0.387
0.61
0.55
0.55
0.553
0.617
0.61
0.61
Male
0.72
0.703
0.68
Llama 2
Female
0.395
0.444
0.418
0.612
0.560
0.563
0.561
0.623
0.612
0.615
Male
0.725
0.682
0.703
Table 3: Comparison of DistilBERT, RoBERTa, and Llama 2 performance on test EDG(Explicitly De-Gendered) dataset including individual class metrics (Precision, Recall and F1 ) and aggregate metrics(Macro + Weighted Precision, Recall and F1 ).
Dataset
DistilBERT
RoBERTa
Llama 2
Acc.
Macro P
Macro R
Macro F1
Acc.
Macro P
Macro R
Macro F1
Acc.
Macro P
Macro R
Macro F1
Original (non-EDG)
0.999
1.000
0.996
0.999
–
–
–
–
–
–
–
–
EDG (baseline)
0.633
0.573
0.573
0.577
0.61
0.55
0.55
0.553
0.612
0.56
0.563
0.561
EDG w/o SHAP Tokens
0.597
0.553
0.56
0.55 ( ↓ 2.7%)
0.585
0.56
0.57
0.55 ↓ 0.3%)
0.690
0.345
0.500
0.408 ( ↓ 15.3%)
EDG w/o TF-IDF Tokens
0.61
0.567
0.57
0.563 ( ↓ 1.4%)
0.555
0.54
0.54
0.53 ( ↓ 2.3%)
0.685
0.344
0.497
0.407 ( ↓ 15.4%)
Table 4: Comparison of DistilBERT, RoBERTa, and Llama 2 performance across three datasets: EDG (Explicitly De-Gendered), EDG with top SHAP-identified gender tokens removed, and EDG with top TF-IDF-identified gender tokens removed showing improvements(lowering) in macro precision, recall, and F1 scores
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 5: Best hyper-parameters selected for DistilBERT and RoBERTa text classifiers..
Parameter
Value
Batch size (train / eval)
4 / 4
Epochs
5
Learning rate
3.0×10−5
Weight decay
0.02
LoRA rank ( r )
16
LoRA alpha
48
Appendix
Table 6: Best Llama 2 fine-tuning hyper-parameters
Model
Gender
Precision
Recall
F1
Acc.
Macro Precision
Macro Recall
Macro F1
Wtd. Precision
Wtd. Recall
Wtd. F1
Topic Vectors Only
Female
0.370
0.480
0.420
0.580
0.550
0.560
0.550
0.620
0.580
0.600
Male
0.730
0.630
0.680
Topic + DistilBERT Embeddings
Female
0.410
0.440
0.430
0.630
0.580
0.580
0.580
0.640
0.630
0.640
Male
0.740
0.720
0.730
Appendix
Table 7: Classification performance of Random Forest models using (1) only topic vectors and (2) a combination of topic vectors and DistilBERT embeddings.
Figure 3: SHAP summary plot showing the most influential topics in predicting gender from recommendation letters using a Random Forest classifier trained on BERTopic vectors. Each row represents a topic, labeled with its BERTopic ID followed by the top five unigrams or bigrams (joined with underscores) extracted via TF-IDF. Each dot corresponds to a single letter; the horizontal position indicates the SHAP value, reflecting the topic’s contribution to the prediction (positive values push toward " male " negative toward " female "). Red dots represent the presence of a topic in a letter, while blue dots represent its absence.
Understanding gender biases in large language models (LLMs) is increasingly important as these systems become embedded in decision-support tools with real consequences. Prior research has focused only on a small set of models, leaving open the extent to which gender biases are common and heterogeneous across LLMs. We address this gap across ten models released between April 2025 and June 2026, spanning nine vendors, using two paradigms: gender attribution to stereotyped phrases (Study 1) and moral judgment of abuse or torture against a woman or a man to prevent a catastrophic outcome (Study 2). In Study 1, two of ten models attributed masculine-stereotyped phrases to female writers more often than the reverse, while three models showed the opposite pattern. In Study 2, several models converged on a male-disadvantaging asymmetry that was directionally consistent with a documented human tendency to protect female targets from harm, though the specific conditions under which this asymmetry emerged varied by model; three other models, by contrast, showed no variation across conditions. These results indicate that gender-related biases are common in LLMs. Their direction and magnitude, however, are highly heterogeneous, to the point that some models behave in diametrically opposite ways to others. Bias auditing should therefore be treated as an ongoing, multi-vendor process, rather than a one-time assessment.
De-identified résumé screening assumes that redacting explicit fields prevents ethnocultural inference; however, recent audits attribute residual leakage to declared languages. We investigate whether eliminating language fields resolves this leakage across nine open-weight models and 620 counterfactual résumés. By holding language attributes strictly identical, we isolate unstructured prose across five ethnocultural conditions and three cue-salience tiers. Target-group recovery averages 0.757 overall and saturates at 1.000 under high salience, demonstrating that non-language prose sustains demographic inference. Crucially, models diverge only under faint cues (0.086-0.690), establishing salience as an essential evaluation axis. Furthermore, pairwise LLM-as-a-judge outcomes are highly sensitive to evaluation design: forbidding ties yields an apparent selection-rate ratio of 0.39 alongside strong position and content effects, whereas permitting ties produces near-universal ties for most models (≥94%). Downstream scoring shows only very small between-condition differences, highlighting the need to distinguish demographic signals recoverable from résumé content from effects introduced by the evaluation protocol.
Qiangju Chen, Yang Xiao
Macquarie University · The University of Melbourne
The growing prominence of large language models (LLMs) in daily life has heightened concerns that LLMs exhibit many of the same gender-related biases as their creators. In the context of hiring decisions, we quantify the degree to which LLMs perpetuate societal biases and investigate prompt engineering as a bias mitigation technique. Our findings suggest that for a given resumé, an LLM is more likely to hire a female candidate and perceive them as more qualified, but still recommends lower pay relative to male candidates.
Nina Gerszberg, Janka Hamori, Andrew Lo
Massachusetts Institute of Technology Cambridge, Massachusetts, USA