Architecting Fairness-Aware LLM Interactions
Organizations: Gran Sasso Science Institute L’Aquila, Italy · University of Salerno Fisciano, Italy
Abstract
Large Language Models (LLMs) are increasingly used as decision-support tools but may generate stereotypical, discriminatory, or unfair responses. As LLMs become embedded in decision-critical software systems, fairness can no longer be treated as a downstream evaluation step, but must become an architectural concern of the interaction itself. Yet current fairness approaches operate as standalone evaluation techniques, lacking a reusable mechanism to verify and mitigate violations at runtime. We propose a fairness-assurance architecture grounded in three reusable design patterns, instantiated as a multi-agent framework that evaluates and iteratively refines LLM responses before delivery. A preliminary evaluation across eight configurations involving different LLM families shows that the architecture identifies fairness violations and iteratively produces compliant responses in all configurations, with behavior differing depending on the model-family composition. These results provide initial evidence for the joint instantiation of the proposed patterns and a first step toward architecting non-functional properties into LLM-based systems.
Figures & tables
| OpenAI Usage Policies | Ethical Concept | Operational Guideline |
| Evaluation or classification of individuals based on their social behavior, personal traits, or biometric data (including social scoring, profiling, or inferring sensitive attributes) | A person should not be evaluated or classified based on sensitive attributes or irrelevant personal traits | R - The output must not evaluate or classify a person based on sensitive attributes, personal traits, or social behavior |
| Configuration | LLMUser | LLMJudge | LLMRefiner | Category |
| C1 | GPT | GPT | GPT | Within-Family |
| C2 | Claude | Claude | Claude | Within-Family |
| C3 | GPT | GPT | Claude | Cross-Family |
| C4 | GPT | Claude | GPT | Cross-Family |
| C5 | Claude | GPT | GPT | Cross-Family |
| C6 | Claude | Claude | GPT | Cross-Family |
| Config | Total Violations | Avg. Iter. | Median Iter. | Max Iter. | Guidelines | Intent | Bias Dimension |
| C1 | 621 | 2.20 | 2.0 | 9 | G1, G13, G2, G6, G9, G3 | Question, Direction | Physical Appearance, Age, Disability, Sexual Orientation |
| C2 | 221 | 1.37 | 1.0 | 5 | G1, G13, G2, G3 | Question, Clarification | Physical Appearance, Age |
| C3 | 613 | 2.10 | 2.0 | 40 | G1, G13, G3, G2, G9, G6 | Direction, Clarification | Nationality, Physical Appearance, Race-Color, Gender |
| C4 | 398 | 1.76 | 2.0 | 5 | G1, G13, G2, G3 | Question, Clarification | Physical Appearance, Age, Gender |
| C5 | 1763 | 4.03 | 2.0 | 144 | G1, G2, G13, G3, G6 | Question, Clarification | Physical Appearance, Age |
| C6 | 445 | 1.74 | 1.0 | 10 | G1, G13, G2, G3 | Question, Clarification | Physical Appearance, Age |
Algorithmic Fairness
Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.
See all 453 papers