cs.CLFeb 28, 2024

Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification

Authors: Garima Chhikara, Bhaskara Hanuma Vedula, Anurag Sharma, Kripabandhu Ghosh, Ponnurangam Kumaraguru, Abhijnan Chakraborty

Organizations: Indian Institute of Technology Delhi · Delhi Technological University · Indian Institute of Science Education and Research Kolkata, Mohanpur, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are increasingly being employed for downstream tasks such as tabular classification, where predictions can directly influence individuals in high-stakes domain such as hiring, education and criminal justice. As the adoption of LLMs for such decision making tasks continues to grow, ensuring fairness in their predictions becomes increasingly important. Existing approaches to ensure fairness in tabular data, typically process test instances independently, limiting the models awareness of the demographic composition and predictions of other instances, which is particularly important for reasoning about group-level fairness. To address this limitation, we propose FABLE (FAir Batch-based LLM framEwork) for tabular classification, a batch-based framework that jointly processes multiple test instances, enabling LLMs to reason over demographic information and prediction outcomes within a batch. To mitigate positional bias introduced by batch processing, the framework generates multiple shuffled versions of each batch and aggregates the resulting predictions through voting. We evaluate FABLE on three tabular classification datasets using three popular LLMs and four prompting configurations spanning zero-shot, few-shot, direct, and self-reflective prompting. Experimental results demonstrate that incorporating demographic information and explicit fairness constraints enable LLMs to achieve substantial fairness improvements across diverse datasets and fairness notions, highlighting their potential as fairness reasoners for tabular classification and promoting more equitable and accountable decision-making.

Figures & tables

Figure 1: Example shows a part of the conversation with GPT-5 about the Stop and Frisk Policy . When GPT-5 is queried about the proportion of Black individuals stopped by the police, it provides the requested statistics and further highlights the disproportionate representation of Black individuals among those stopped compared with White individuals. When prompted to determine a fair proportion of police stops, the model adopts the principle of Statistical Parity, also referred to as Proportional Representation Feldman et al. (2015) . Under this criterion, the model suggests that if Black individuals constitute 23% of the overall population, they should likewise account for 23% of police stops to satisfy the fairness criterion.
Figure 2: Example shows a part of the conversation with GPT-5 about the Fairness in Recruitment Process . When GPT-5 is queried about the fairness in recruitment process, it gives importance to causal independence and replies by saying that causal relationship between a candidate’s protected attributes and the recruitment outcome should be null.
NotationExplanation
XXAttributes describing an individual.
YYThe actual classification result.
Y^\hat{Y}Predicted classification decision.
GGProtected or sensitive attribute.
TPg1TP_{g_{1}}True positives for sensitive group 1.
TNg1TN_{g_{1}}True negatives for sensitive group 1.
Table 1: Notations utilized in fairness definitions.
Figure 3: Comparison of two evaluation settings. (a) Independent evaluation: the LLM assesses each candidate individually in a separate prompt and returns an individual select or reject decision, without access to the demographic composition of the remaining test instances. Because every decision is made in isolation, this method cannot account for the overall group distribution, which limits its ability to enforce group fairness. (b) Batch evaluation: multiple test instances are presented to the LLM simultaneously in a single prompt, together with an instruction to satisfy a group fairness definition such as demographic parity. The LLM can therefore consider the demographic composition of the entire applicant pool (here, 30% female and 70% male) and incorporate group-level fairness into a collective decision.
Figure 4: Tabular data is converted into natural language representation, to make it suitable input for the LLM.
Figure 5: Overview of FABLE , a batch-based LLM framework designed to achieve fairer classification outcomes. In contrast to approaches that process test instances independently, our framework provides the LLM with a batch comprising multiple test instances simultaneously. (1) A batch of NN candidates is constructed. (2) To mitigate potential positional bias, MM variations of the batch are generated by shuffling the order of the test instances, so that each candidate appears at different positions (1st to last) in the prompt. (3) Each shuffled batch is independently provided to the LLM, resulting in MM predictions corresponding to one test instance. Owing to positional bias, the same candidate may receive different predictions across shuffles. (4) The MM predictions are aggregated using majority voting to determine the final prediction for each test instance. For example, candidate 1 is selected in two of the three shown shuffles and is therefore selected.
Fairness DefinitionPrompts
Demographic ParityEnsure Demographic Parity on the basis of << attribute >> . Demographic Parity implies that both << g_1 >> and << g_2 >> should have equal probability of being assigned to the << positive_class >> . Probability of classifying << g_1 >> and << g_2 >> as << positive_class >> should be equal.
Equal OpportunityEnsure Equal Opportunity on the basis of << attribute >> . True Positive Rate should be same across both << g_1 >> and << g_2 >> . Probability of a person with gold label as << positive_class >> to be correctly classified as << positive_class >> should be same for both << g_1 >> and << g_2 >> .
Equalized OddsEnsure Equalized Odds on the basis of << attribute >> . False Positive Rate should be same across both << g_1 >> and << g_2 >> . Probability of a person with gold label as << negative_class >> to be incorrectly classified as << positive_class >> should be same for both << g_1 >> and << g_2 >> .
Overall Accuracy EqualityEnsure Overall Accuracy Equality on the basis of << attribute >> . Accuracy should be same across both << g_1 >> and << g_2 >> groups. Probability of a person with gold label as << positive_class >> to be correctly classified as << positive_class >> and a person with gold label as << negative_class >> to be correctly assigned as << negative_class >> should be same for both << g_1 >> and << g_2 >> group.
Causal DiscriminationEnsure Causal Discrimination on the basis of << attribute >> . People having similar attributes should be assigned similar result. << g_1 >> and << g_2 >> who have same features should be assigned same classification result that is either both will be assigned << negative_class >> or both will be assigned << positive_class >> .
Fairness through UnawarenessEnsure Fairness Through Unawareness on the basis of << attribute >> . Do not consider gender for decision making process. << attribute >> should not be used while classification. Classification outcome should be same for any two people who have same attributes.
Table 2: Prompts or Instructions provided for various fairness definitions in zero-shot and few-shot setup. << attribute >> , << g_1 >> and << g_2 >> represents the sensitive attribute, demographic group 1 and demographic group 2 respectively.
Figure 6: Input prompts for the few-shot setting in Fewshot-Direct and Fewshot-Reflect. Both strategies share the same conversation: a system prompt containing the fairness definition, few-shot examples (user turn) and their labels (assistant turn), and the test batch of candidates (user turn). Fewshot-Direct calls the LLM once on this conversation, and its response gives the final decisions for the batch. In Fewshot-Reflect, the output of Fewshot-Direct is provided as input together with the same four turns, followed by an additional prompt asking the LLM to self-reflect on its answers and if required change them to satisfy the given fairness definition, a second LLM call then produces the revised decisions.
Figure 7: Method for selection of in-context examples for group fairness definitions, illustrated on a toy batch of six test instances (distances are illustrative). (1) Given a batch of NN test instances, RAG is used to retrieve, for each test instance TiT_{i} , the nearest training example EiE_{i} , subject to the constraint that both belong to the same demographic group; dd denotes the retrieval distance. This ensures that the retrieved examples preserve the demographic distribution of the test batch. (2) The positive-label ratio of the retrieved examples (number of positive examples in a group divided by the number of examples in a group) is computed for each demographic group. If the ratios are equal, the retrieved examples are selected as the final in-context examples. (3) Otherwise, the group with the lower positive-label ratio is identified, and its negative-labeled example with the greatest retrieval distance from its corresponding test instance is selected. Its label is flipped to positive, increasing the group’s positive-label count by one. The ratios are then recomputed, and this procedure is repeated until parity is achieved between the groups. In the example, the female ratio (1/3) is lower than the male ratio (2/3), so E6E_{6} , the negative female example with the largest distance (0.34), is re-labeled positive, which brings both ratios to 2/3. (4) The balanced set is used as the in-context examples for the LLM; the flipped label is circled.
Figure 8: Effect of different fairness definitions relative to the No-Fairness (NF) baseline across the Adult, COMPAS, and Law School datasets. The rows represent changes in accuracy, F1-score, and fairness score, respectively, while the columns correspond to the three datasets. Incorporating fairness definitions generally reduces accuracy and F1-score while improving fairness. Group fairness improves in all 72 evaluated cases, whereas individual fairness improves in 34 out of 36 cases. These results highlight the fairness–accuracy trade-off, wherein improvements in fairness are accompanied by a decline in predictive performance.
Figure 9: Accuracy, F1-score, and fairness scores across the Adult, COMPAS, and Law School datasets under different fairness definitions. For group fairness, the scores generally range from 0.4 to 0.8, whereas individual fairness scores predominantly fall between 0.85 and 1.0. This suggests that models adhere more readily to individual fairness constraints than to group fairness.
Figure 10: Figure illustrating the impact of self-reflection on fairness outcomes. For group fairness, self-reflection improves fairness scores in 65 out of 72 cases, suggesting that prompting the LLM to reassess its initial predictions can enhance fairness performance.
More in this topic

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 424 papers

Explore similar work

CardsList
Apr 21, 2026

Auditing LLMs for Algorithmic Fairness in Casenote-Augmented Tabular Prediction

LLMs are increasingly being considered for prediction tasks in high-stakes social service settings, but their algorithmic fairness properties in this context are poorly understood. In this short technical report, we audit the algorithmic fairness of LLM-based tabular classification on a real housing placement prediction task, augmented with street outreach casenotes from a nonprofit partner. We audit multi-class classification error disparities. We find that a fine-tuned model augmented with casenote summaries can improve accuracy while reducing algorithmic fairness disparities. We experiment with variable importance improvements to zero-shot tabular classification and find mixed results on resulting algorithmic fairness. Overall, given historical inequities in housing placement, it is crucial to audit LLM use. We find that leveraging LLMs to augment tabular classification with casenote summaries can safely leverage additional text information at low implementation burden. The outreach casenotes are fairly short and heavily redacted. Our assessment is that LLM zero-shot classification does not introduce additional textual biases beyond algorithmic biases in tabular classification. Combining fine-tuning and leveraging casenote summaries can improve accuracy and algorithmic fairness.

Xiao Qi Lee, Ezinne Nwankwo, Angela Zhou

Department of Computer Science San Jose State University · Department of Computer Science University of California, Berkeley · Department of Data Science and Operations University of Southern California

LLM AuditingAlgorithmic FairnessTabular Prediction
AskPDF
May 23, 2026

Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens

Every tabular classification task requires sufficient labeled data. When labels are scarce, practitioners can collect more examples, encode domain expertise into features, or accept lower performance. Large language models offer an alternative by leveraging knowledge acquired during pretraining. This raises three questions: (i) can zero-shot inference reduce the need for labeled data; (ii) can a richer prompt reduce the need for more examples; and (iii) can encoded domain knowledge compensate for missing data? To answer each under explicit, testable conditions, we introduce TabLens, a benchmark of 91 real-world and synthetic datasets across eight domains, plus eight industrial datasets with free-text columns. First, below a complexity threshold estimable from a dataset's intrinsic dimensionality before running an LLM, zero-shot inference matches the accuracy of up to about 16 labeled examples. This holds whether the model generates an answer or, more cheaply, scores candidate labels by log-probabilities, which loses almost no accuracy except for the smallest models. Above the threshold, the advantage disappears. On datasets with free-text columns, however, zero-shot LLMs outperform 16-shot TabPFN-3 Plus without manual feature engineering. Second, a richer prompt reduces the number of required demonstrations only when it includes both real column names and a plain-language task description; providing only one can be worse than neither, particularly at higher shot counts. Third, on LLM-synthetic datasets, explicit if--then expert rules help only when the prompt provides enough task context to interpret them, and even then, they substitute for at most about 16 real examples. Together, these results establish testable conditions under which LLMs can reduce data-collection and feature-engineering requirements for tabular classification.

Daria Grushina, Kseniia Kuvshinova, Alina Kostromina +3

Sb AI Lab, HSE University, NES, Russia · MBZUAI, United Arab Emirates · Sb AI Lab, HSE University, Russia +1

Zero-Shot LearningLLM EvaluationTabular ClassificationFew-Shot LearningTabular ICL
AskPDF
Aug 14, 2026

Training Fair Tabular Foundation Models

Tabular Foundation Models (TFMs) have emerged as leading methods for tabular predictive tasks, leveraging in-context learning to predict on new data without task-specific training. Despite the increased use of TFMs in high-stakes decision-making, their fairness properties remain largely unexplored. In this work, we incorporate fairness constraints directly into TFM training, enabling fair predictions in a single forward pass. Our approach addresses two key challenges: limited access to sensitive attributes in training data, and the incompatibility of existing fairness techniques with the in-context learning paradigm. We propose FairTFM, a scalable training strategy based on synthetic fairness tasks and a fairness-aware architecture using a gradient reversal layer, which encourages the model to learn representations invariant to sensitive attributes. Experiments on 132 fairness tasks show consistent improvements in fairness while maintaining competitive accuracy.

Patrik Kenfack, Jesse C. Cresswell, Anthony L. Caterini +2

ÉTS Montréal Mila - Quebec AI Institute · Layer 6 AI Toronto, Canada · University of Calgary Mila - Quebec AI Institute, CIFAR

Tabular Foundation ModelsAlgorithmic FairnessTabular ICL
AskPDF