UGID: Unified Graph Isomorphism for Debiasing Large Language Models

Zikang Ding; Junchi Yao; Junhao Li; Yi Zhang; Wenbo Jiang; Hongbo Liu; Lijie Hu

cs.CLMar 19, 2026

UGID: Unified Graph Isomorphism for Debiasing Large Language Models

Authors: Zikang Ding, Junchi Yao, Junhao Li, Yi Zhang, Wenbo Jiang, Hongbo Liu, Lijie Hu

Organizations: University of Electronic Science and Technology of China, Chengdu, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · South China University of Technology, Guangzhou, China

Abstract

Large language models (LLMs) exhibit pronounced social biases. Output-level or data-optimization--based debiasing methods cannot fully resolve these biases, and many prior works have shown that biases are embedded in internal representations. We propose Unified Graph Isomorphism for Debiasing large language models (UGID), an internal-representation--level debiasing framework for large language models that models the Transformer as a structured computational graph, where attention mechanisms define the routing edges of the graph and hidden states define the graph nodes. Specifically, debiasing is formulated as enforcing invariance of the graph structure across counterfactual inputs, with differences allowed only on sensitive attributes. UGID jointly constrains attention routing and hidden representations in bias-sensitive regions, effectively preventing bias migration across architectural components. To achieve effective behavioral alignment without degrading general capabilities, we introduce a log-space constraint on sensitive logits and a selective anchor-based objective to preserve definitional semantics. Extensive experiments on large language models demonstrate that UGID effectively reduces bias under both in-distribution and out-of-distribution settings, significantly reduces internal structural discrepancies, and preserves model safety and utility.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of bias distribution shifts. The vertical layout allows for clear layer-wise comparison between GPT-2 (top) and LLaMA-3 (bottom).

Figure 2: Overview of the UGID framework.

Method	Debiasing Effectiveness ( $\downarrow$ 1.0)				Mechanism ( $\downarrow$ 0)		Safety	Utility
	ID		OOD		Edge	Node	Anchor	Anchor-PPL	IQ
	Mean	Max	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	( $\downarrow$ )	(Pass)
Original	7.14x	21.99x	9.00x	15.65x	0.211	5.198	100%	118.07	✓
CDA	1.16x	1.29x	1.16x	1.29x	0.110	3.813	100%	3.76	✓
KLAAD-LoRA	1.03x	1.13x	0.98x	1.00x	0.148	3.576	50%	10.66	✓
UGID (Ours)	0.94x	0.94x	1.06x	1.21x	0.007	0.058	100%	121.11	✓

Table 1: Detailed Performance on LLaMA-3-8B. Paleblue and underline indicate the best and second-best results within each block (excluding Original baseline).

Figure 3: Multi-dimensional Performance Analysis on LLaMA-3-8B.

Panel of Figure 4 — Figure 4: Attention routing differences under gender counterfactuals.

Variant Description	Effectiveness ( $\downarrow$ 1.0)		Mechanism ( $\downarrow$ 0)		Safety	Utility
	ID Bias		Edge	Node	Anchor	Anchor	IQ
	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	PPL ( $\downarrow$ )	(Pass)
Original	7.14x	21.99x	0.2111	5.1979	100%	118.07	✓
w/o (Edge + Node + Logit)	4.56x	12.22x	0.2173	5.2604	100%	117.41	✓
w/o (Edge + Node)	1.03x	1.07x	0.2065	5.2240	100%	114.91	✓
w/o Node	1.04x	1.06x	0.0114	0.6527	100%	120.96	✓

Table 2: Ablation Study on LLaMA-3-8B. Paleblue and underline indicate the best and second-best results (excluding Original baseline). The swap of Safety and Utility columns highlights the trade-off between structural alignment and linguistic proficiency.

Method	Robustness Diagnostics ( $\downarrow$ )				Safety ( $\uparrow$ )
	Template	Template	Dir.	Neutral	Unseen Anchor
	Mean	Var	Gap	Mass	Acc
Original	8.92	3.41	1.87	0.02	100%
CDA	1.21	0.0181	0.150	0.000013	100%
KLAAD	1.32	0.287	0.025	0.000004	50%
UGID (Ours)	0.98	0.0044	0.0625	0.0133	100%

Table 3: Robustness and Safety Diagnostics (LLaMA-3-8B). Paleblue and underline indicate the best and second-best results within each block (excluding Original baseline).

Figure 6: Layer-wise logit gap across models.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Debiasing Effectiveness ( $\downarrow$ 1.0)				Mechanism ( $\downarrow$ 0)		Safety	Utility
		ID Bias		OOD Bias		Edge	Node	Anchor	Anchor	IQ
		Mean	Max	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	PPL ( $\downarrow$ )	(Pass)
Llama-3-8B	Original	433.42x	1726.75x	214.79x	495.81x	0.2414	7.2188	100%	66.26	✓
Llama-3-8B	UGID (Ours)	0.97x	0.98x	0.99x	1.00x	0.0112	0.1029	50%	64.84	✓

Appendix

Table 4: Generalization Results on Regional Bias. Paleblue indicates the best results. The table demonstrates UGID’s robustness in mitigating regional stereotypes while maintaining structural integrity and linguistic utility on the Llama-3-8B architecture.

Method	BOLD		BBQ (Gender)		HolisticBias		RTP
	Bias	PPL-r	Ambig.	Acc	Bias	PPL-r	Bias	PPL-r
	( $\downarrow$ )	( $\approx 1$ )	( $\downarrow$ )	( $\uparrow$ )	( $\downarrow$ )	( $\approx 1$ )	( $\downarrow$ )	( $\approx 1$ )
Original	11.343	1.000	0.372	0.388	0.382	1.000	0.011	1.000
CDA	1.037	0.281	0.419	0.381	0.246	0.279	0.005	0.281
KLAAD	1.267	0.611	0.472	0.369	0.345	0.617	0.015	0.611
UGID (Ours)	1.998	1.000	0.342	0.398	0.241	1.006	0.003	1.000

Appendix

Table 5: Generalization Results on Four Benchmarks (LLaMA-3-8B). Paleblue and underline indicate the best and second-best results. UGID maintains superior linguistic utility (PPL-r

\approx 1

) while consistently achieving competitive debiasing performance.

Method	Training Time (s/it) $\downarrow$	Peak Memory (GB) $\downarrow$	Inference Latency
Original (Base)	0.2012	19.00	1.00 $\times$
UGID (Ours)	0.4159	6.92	1.00 $\times$

Appendix

Table 6: Efficiency Benchmarks on LLaMA-3-8B. The evaluation is conducted on a single NVIDIA A100 (80GB). UGID maintains identical inference latency to the base model while significantly reducing memory overhead via PEFT-based structural alignment.

Panel of Figure 8 — Figure 8: Performance Profiles across Diverse Architectures and Scales. Across all evaluated families, UGID consistently exhibits superior coverage (the largest blue area) compared to baselines, especially in preserving internal structural stability and safety.

Figure 9: Pareto Frontier of Debiasing Effectiveness vs. Structural Integrity.

Model Family	Method	Debiasing Effectiveness ( $\downarrow$ 1.0)				Mechanism ( $\downarrow$ 0)		Safety	Utility
		ID		OOD		Edge	Node	Anchor	Anchor-PPL	IQ
		Mean	Max	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	( $\downarrow$ )	(Pass)
Gemma-2-2B	Original	6.84x	37.45x	4.48x	8.38x	0.1502	73.00	100%	154.79	–
	CDA	1.13x	1.21x	1.18x	1.21x	0.0499	52.67	100%	5.94	–
	KLAAD	1.01x	1.06x	1.18x	1.29x	0.1572	54.13	50%	20.36	–
	UGID (Ours)	0.95x	1.00x	1.03x	1.13x	0.0067	0.67	100%	165.63	–

Appendix

Table 7: Results on Diverse Model Families. Paleblue and underline indicate the best and second-best results within each model family (excluding Original baseline).

Figure 10: Layer-wise Hidden-State Drift under Counterfactual Pronoun Swaps.

Figure 12: Activation patching / causal tracing results (He to She).

Figure 13: Activation patching / causal tracing results (She to He).

Figure 14: Spectral Signature Difference (layer=15). Token-wise

|s_{l}(x)-s_{l}(x^{\prime})|

for he/she counterfactuals (Original and CDA).

Figure 15: Spectral Signature Difference (layer=15). Token-wise

|s_{l}(x)-s_{l}(x^{\prime})|

for he/she counterfactuals (KLAAD and UGID).

Language Model Debiasing

See all 26 papers

Jan 29, 2026

Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement

Large language models (LLMs) exhibit social biases that reinforce harmful stereotypes, limiting their safe deployment. Most existing debiasing methods adopt a suppressive paradigm by modifying parameters, prompts, or neurons associated with biased behavior; however, such approaches are often brittle, weakly generalizable, data-inefficient, and prone to degrading general capability. We propose KnowBias, a lightweight and conceptually distinct framework that mitigates bias by strengthening, rather than suppressing, neurons encoding bias-knowledge. KnowBias identifies neurons encoding bias knowledge using a small set of bias-knowledge questions via attribution-based analysis, and selectively enhances them at inference time. This design enables strong debiasing while preserving general capabilities, generalizes across bias types and demographics, and is highly data efficient, requiring only a handful of simple yes/no questions and no retraining. Experiments across multiple benchmarks and LLMs demonstrate consistent state-of-the-art debiasing performance with minimal utility degradation. Data and code are available at https://github.com/JP-25/KnowBias.

Jinhao Pan, Chahat Raj, Anjishnu Mukherjee +4

Department of Computer Science, George Mason University, Fairfax, VA, USA · Lightridge High School, Aldie, VA, USA

Debiased ML Social Bias in Language Models Language Model Debiasing Demographic Bias in Language Models Activation Steering

Ask PDF

Sep 30, 2026

Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventions

Bias has long been studied in social psychology and cognitive science, where decades of research have produced a body of validated interventions that reduce stereotypical thinking and prejudiced responses in humans. We propose Debias It Yourself (DIY), a cognitively grounded framework that translates five such interventions into debiasing procedures for large language models and delivers them through three established paradigms: Show (in-context examples), Train (instruction tuning), and Revise (guided self-revision). Across three models, five bias benchmarks, eleven debiasing baselines, and three reasoning benchmarks, Train+Revise and Revise alone attain the top two average ranks, lead the bias-reasoning tradeoff (mean bias as low as 2% at 90% reasoning accuracy), and reduce bias on unseen dimensions by up to 14.8%. Our code and data are publicly available.

Chahat Raj, Sina Mansouri, Aylin Caliskan +2

George Mason University · University of Washington

Cognitive Bias in Language Models Language Model Debiasing Social Bias in Language Models Language Model Bias Evaluation

Ask PDF

Mar 23, 2026

CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMs

Large Language Models (LLMs) are deployed in high-stakes settings but can show demographic, gender, and geographic biases that undermine fairness and trust. Prior debiasing methods, including embedding-space projections, prompt-based steering, and causal interventions, often act at a single stage of the pipeline, resulting in incomplete mitigation and brittle utility trade-offs under distribution shifts. We propose CatRAG Debiasing, a dual-pronged framework that integrates functor with Retrieval-Augmented Generation (RAG) guided structural debiasing. The functor component leverages category-theoretic structure to induce a principled, structure-preserving projection that suppresses bias-associated directions in the embedding space while retaining task-relevant semantics. On the Bias Benchmark for Question Answering (BBQ) across three open-source LLMs (Meta Llama-3, OpenAI GPT-OSS, and Google Gemma-3), CatRAG achieves state-of-the-art results, improving accuracy by up to 40% over the corresponding base models and by more than 10% over prior debiasing methods, while reducing bias scores to near zero (from 60% for the base models) across gender, nationality, race, and intersectional subgroups.

Ravi Ranjan, Utkarsh Grover, Mayur Akewar +2

Florida International University Miami, USA · University of South Florida Tampa, USA

Language Model Debiasing Social Bias in Language Models Algorithmic Bias Demographic Bias in Language Models Language Model Bias Evaluation

Ask PDF

Abstract

Language Model Debiasing

Explore similar work

Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement

Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventions

CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMs