cs.CLFeb 28, 2024
SaveFew-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification
Organizations: Indian Institute of Technology Delhi · Delhi Technological University · Indian Institute of Science Education and Research Kolkata, Mohanpur, India
Abstract
Large Language Models (LLMs) are increasingly being employed for downstream tasks such as tabular classification, where predictions can directly influence individuals in high-stakes domain such as hiring, education and criminal justice. As the adoption of LLMs for such decision making tasks continues to grow, ensuring fairness in their predictions becomes increasingly important. Existing approaches to ensure fairness in tabular data, typically process test instances independently, limiting the models awareness of the demographic composition and predictions of other instances, which is particularly important for reasoning about group-level fairness. To address this limitation, we propose FABLE (FAir Batch-based LLM framEwork) for tabular classification, a batch-based framework that jointly processes multiple test instances, enabling LLMs to reason over demographic information and prediction outcomes within a batch. To mitigate positional bias introduced by batch processing, the framework generates multiple shuffled versions of each batch and aggregates the resulting predictions through voting. We evaluate FABLE on three tabular classification datasets using three popular LLMs and four prompting configurations spanning zero-shot, few-shot, direct, and self-reflective prompting. Experimental results demonstrate that incorporating demographic information and explicit fairness constraints enable LLMs to achieve substantial fairness improvements across diverse datasets and fairness notions, highlighting their potential as fairness reasoners for tabular classification and promoting more equitable and accountable decision-making.
Figures & tables
Figure 1: Example shows a part of the conversation with GPT-5 about the Stop and Frisk Policy . When GPT-5 is queried about the proportion of Black individuals stopped by the police, it provides the requested statistics and further highlights the disproportionate representation of Black individuals among those stopped compared with White individuals. When prompted to determine a fair proportion of police stops, the model adopts the principle of Statistical Parity, also referred to as Proportional Representation Feldman et al. (2015) . Under this criterion, the model suggests that if Black individuals constitute 23% of the overall population, they should likewise account for 23% of police stops to satisfy the fairness criterion.
Figure 2: Example shows a part of the conversation with GPT-5 about the Fairness in Recruitment Process . When GPT-5 is queried about the fairness in recruitment process, it gives importance to causal independence and replies by saying that causal relationship between a candidate’s protected attributes and the recruitment outcome should be null.
| Notation | Explanation |
|---|---|
| Attributes describing an individual. | |
| The actual classification result. | |
| Predicted classification decision. | |
| Protected or sensitive attribute. | |
| True positives for sensitive group 1. | |
| True negatives for sensitive group 1. |
Table 1: Notations utilized in fairness definitions.
Figure 3: Comparison of two evaluation settings. (a) Independent evaluation: the LLM assesses each candidate individually in a separate prompt and returns an individual select or reject decision, without access to the demographic composition of the remaining test instances. Because every decision is made in isolation, this method cannot account for the overall group distribution, which limits its ability to enforce group fairness. (b) Batch evaluation: multiple test instances are presented to the LLM simultaneously in a single prompt, together with an instruction to satisfy a group fairness definition such as demographic parity. The LLM can therefore consider the demographic composition of the entire applicant pool (here, 30% female and 70% male) and incorporate group-level fairness into a collective decision.
Figure 4: Tabular data is converted into natural language representation, to make it suitable input for the LLM.
Figure 5: Overview of FABLE , a batch-based LLM framework designed to achieve fairer classification outcomes. In contrast to approaches that process test instances independently, our framework provides the LLM with a batch comprising multiple test instances simultaneously. (1) A batch of candidates is constructed. (2) To mitigate potential positional bias, variations of the batch are generated by shuffling the order of the test instances, so that each candidate appears at different positions (1st to last) in the prompt. (3) Each shuffled batch is independently provided to the LLM, resulting in predictions corresponding to one test instance. Owing to positional bias, the same candidate may receive different predictions across shuffles. (4) The predictions are aggregated using majority voting to determine the final prediction for each test instance. For example, candidate 1 is selected in two of the three shown shuffles and is therefore selected.
| Fairness Definition | Prompts |
| Demographic Parity | Ensure Demographic Parity on the basis of attribute . Demographic Parity implies that both g_1 and g_2 should have equal probability of being assigned to the positive_class . Probability of classifying g_1 and g_2 as positive_class should be equal. |
| Equal Opportunity | Ensure Equal Opportunity on the basis of attribute . True Positive Rate should be same across both g_1 and g_2 . Probability of a person with gold label as positive_class to be correctly classified as positive_class should be same for both g_1 and g_2 . |
| Equalized Odds | Ensure Equalized Odds on the basis of attribute . False Positive Rate should be same across both g_1 and g_2 . Probability of a person with gold label as negative_class to be incorrectly classified as positive_class should be same for both g_1 and g_2 . |
| Overall Accuracy Equality | Ensure Overall Accuracy Equality on the basis of attribute . Accuracy should be same across both g_1 and g_2 groups. Probability of a person with gold label as positive_class to be correctly classified as positive_class and a person with gold label as negative_class to be correctly assigned as negative_class should be same for both g_1 and g_2 group. |
| Causal Discrimination | Ensure Causal Discrimination on the basis of attribute . People having similar attributes should be assigned similar result. g_1 and g_2 who have same features should be assigned same classification result that is either both will be assigned negative_class or both will be assigned positive_class . |
| Fairness through Unawareness | Ensure Fairness Through Unawareness on the basis of attribute . Do not consider gender for decision making process. attribute should not be used while classification. Classification outcome should be same for any two people who have same attributes. |
Table 2: Prompts or Instructions provided for various fairness definitions in zero-shot and few-shot setup. attribute , g_1 and g_2 represents the sensitive attribute, demographic group 1 and demographic group 2 respectively.
Figure 6: Input prompts for the few-shot setting in Fewshot-Direct and Fewshot-Reflect. Both strategies share the same conversation: a system prompt containing the fairness definition, few-shot examples (user turn) and their labels (assistant turn), and the test batch of candidates (user turn). Fewshot-Direct calls the LLM once on this conversation, and its response gives the final decisions for the batch. In Fewshot-Reflect, the output of Fewshot-Direct is provided as input together with the same four turns, followed by an additional prompt asking the LLM to self-reflect on its answers and if required change them to satisfy the given fairness definition, a second LLM call then produces the revised decisions.
Figure 7: Method for selection of in-context examples for group fairness definitions, illustrated on a toy batch of six test instances (distances are illustrative). (1) Given a batch of test instances, RAG is used to retrieve, for each test instance , the nearest training example , subject to the constraint that both belong to the same demographic group; denotes the retrieval distance. This ensures that the retrieved examples preserve the demographic distribution of the test batch. (2) The positive-label ratio of the retrieved examples (number of positive examples in a group divided by the number of examples in a group) is computed for each demographic group. If the ratios are equal, the retrieved examples are selected as the final in-context examples. (3) Otherwise, the group with the lower positive-label ratio is identified, and its negative-labeled example with the greatest retrieval distance from its corresponding test instance is selected. Its label is flipped to positive, increasing the group’s positive-label count by one. The ratios are then recomputed, and this procedure is repeated until parity is achieved between the groups. In the example, the female ratio (1/3) is lower than the male ratio (2/3), so , the negative female example with the largest distance (0.34), is re-labeled positive, which brings both ratios to 2/3. (4) The balanced set is used as the in-context examples for the LLM; the flipped label is circled.
Figure 8: Effect of different fairness definitions relative to the No-Fairness (NF) baseline across the Adult, COMPAS, and Law School datasets. The rows represent changes in accuracy, F1-score, and fairness score, respectively, while the columns correspond to the three datasets. Incorporating fairness definitions generally reduces accuracy and F1-score while improving fairness. Group fairness improves in all 72 evaluated cases, whereas individual fairness improves in 34 out of 36 cases. These results highlight the fairness–accuracy trade-off, wherein improvements in fairness are accompanied by a decline in predictive performance.
Figure 9: Accuracy, F1-score, and fairness scores across the Adult, COMPAS, and Law School datasets under different fairness definitions. For group fairness, the scores generally range from 0.4 to 0.8, whereas individual fairness scores predominantly fall between 0.85 and 1.0. This suggests that models adhere more readily to individual fairness constraints than to group fairness.
Figure 10: Figure illustrating the impact of self-reflection on fairness outcomes. For group fairness, self-reflection improves fairness scores in 65 out of 72 cases, suggesting that prompting the LLM to reassess its initial predictions can enhance fairness performance.
More in this topic
Algorithmic Fairness
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
See all 424 papers