UGID: Unified Graph Isomorphism for Debiasing Large Language Models

Zikang Ding; Junchi Yao; Junhao Li; Yi Zhang; Wenbo Jiang; Hongbo Liu; Lijie Hu

cs.CLMar 19, 2026

UGID: Unified Graph Isomorphism for Debiasing Large Language Models

Authors: Zikang Ding, Junchi Yao, Junhao Li, Yi Zhang, Wenbo Jiang, Hongbo Liu, Lijie Hu

Organizations: University of Electronic Science and Technology of China, Chengdu, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · South China University of Technology, Guangzhou, China

Abstract

Large language models (LLMs) exhibit pronounced social biases. Output-level or data-optimization--based debiasing methods cannot fully resolve these biases, and many prior works have shown that biases are embedded in internal representations. We propose Unified Graph Isomorphism for Debiasing large language models (UGID), an internal-representation--level debiasing framework for large language models that models the Transformer as a structured computational graph, where attention mechanisms define the routing edges of the graph and hidden states define the graph nodes. Specifically, debiasing is formulated as enforcing invariance of the graph structure across counterfactual inputs, with differences allowed only on sensitive attributes. UGID jointly constrains attention routing and hidden representations in bias-sensitive regions, effectively preventing bias migration across architectural components. To achieve effective behavioral alignment without degrading general capabilities, we introduce a log-space constraint on sensitive logits and a selective anchor-based objective to preserve definitional semantics. Extensive experiments on large language models demonstrate that UGID effectively reduces bias under both in-distribution and out-of-distribution settings, significantly reduces internal structural discrepancies, and preserves model safety and utility.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of bias distribution shifts. The vertical layout allows for clear layer-wise comparison between GPT-2 (top) and LLaMA-3 (bottom).

Figure 2: Overview of the UGID framework.

Method	Debiasing Effectiveness ( $\downarrow$ 1.0)				Mechanism ( $\downarrow$ 0)		Safety	Utility
	ID		OOD		Edge	Node	Anchor	Anchor-PPL	IQ
	Mean	Max	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	( $\downarrow$ )	(Pass)
Original	7.14x	21.99x	9.00x	15.65x	0.211	5.198	100%	118.07	✓
CDA	1.16x	1.29x	1.16x	1.29x	0.110	3.813	100%	3.76	✓
KLAAD-LoRA	1.03x	1.13x	0.98x	1.00x	0.148	3.576	50%	10.66	✓
UGID (Ours)	0.94x	0.94x	1.06x	1.21x	0.007	0.058	100%	121.11	✓

Table 1: Detailed Performance on LLaMA-3-8B. Paleblue and underline indicate the best and second-best results within each block (excluding Original baseline).

Figure 3: Multi-dimensional Performance Analysis on LLaMA-3-8B.

Panel of Figure 4 — Figure 4: Attention routing differences under gender counterfactuals.

Variant Description	Effectiveness ( $\downarrow$ 1.0)		Mechanism ( $\downarrow$ 0)		Safety	Utility
	ID Bias		Edge	Node	Anchor	Anchor	IQ
	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	PPL ( $\downarrow$ )	(Pass)
Original	7.14x	21.99x	0.2111	5.1979	100%	118.07	✓
w/o (Edge + Node + Logit)	4.56x	12.22x	0.2173	5.2604	100%	117.41	✓
w/o (Edge + Node)	1.03x	1.07x	0.2065	5.2240	100%	114.91	✓
w/o Node	1.04x	1.06x	0.0114	0.6527	100%	120.96	✓

Table 2: Ablation Study on LLaMA-3-8B. Paleblue and underline indicate the best and second-best results (excluding Original baseline). The swap of Safety and Utility columns highlights the trade-off between structural alignment and linguistic proficiency.

Method	Robustness Diagnostics ( $\downarrow$ )				Safety ( $\uparrow$ )
	Template	Template	Dir.	Neutral	Unseen Anchor
	Mean	Var	Gap	Mass	Acc
Original	8.92	3.41	1.87	0.02	100%
CDA	1.21	0.0181	0.150	0.000013	100%
KLAAD	1.32	0.287	0.025	0.000004	50%
UGID (Ours)	0.98	0.0044	0.0625	0.0133	100%

Table 3: Robustness and Safety Diagnostics (LLaMA-3-8B). Paleblue and underline indicate the best and second-best results within each block (excluding Original baseline).

Figure 6: Layer-wise logit gap across models.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Debiasing Effectiveness ( $\downarrow$ 1.0)				Mechanism ( $\downarrow$ 0)		Safety	Utility
		ID Bias		OOD Bias		Edge	Node	Anchor	Anchor	IQ
		Mean	Max	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	PPL ( $\downarrow$ )	(Pass)
Llama-3-8B	Original	433.42x	1726.75x	214.79x	495.81x	0.2414	7.2188	100%	66.26	✓
Llama-3-8B	UGID (Ours)	0.97x	0.98x	0.99x	1.00x	0.0112	0.1029	50%	64.84	✓

Appendix

Table 4: Generalization Results on Regional Bias. Paleblue indicates the best results. The table demonstrates UGID’s robustness in mitigating regional stereotypes while maintaining structural integrity and linguistic utility on the Llama-3-8B architecture.

Method	BOLD		BBQ (Gender)		HolisticBias		RTP
	Bias	PPL-r	Ambig.	Acc	Bias	PPL-r	Bias	PPL-r
	( $\downarrow$ )	( $\approx 1$ )	( $\downarrow$ )	( $\uparrow$ )	( $\downarrow$ )	( $\approx 1$ )	( $\downarrow$ )	( $\approx 1$ )
Original	11.343	1.000	0.372	0.388	0.382	1.000	0.011	1.000
CDA	1.037	0.281	0.419	0.381	0.246	0.279	0.005	0.281
KLAAD	1.267	0.611	0.472	0.369	0.345	0.617	0.015	0.611
UGID (Ours)	1.998	1.000	0.342	0.398	0.241	1.006	0.003	1.000

Appendix

Table 5: Generalization Results on Four Benchmarks (LLaMA-3-8B). Paleblue and underline indicate the best and second-best results. UGID maintains superior linguistic utility (PPL-r

\approx 1

) while consistently achieving competitive debiasing performance.

Method	Training Time (s/it) $\downarrow$	Peak Memory (GB) $\downarrow$	Inference Latency
Original (Base)	0.2012	19.00	1.00 $\times$
UGID (Ours)	0.4159	6.92	1.00 $\times$

Appendix

Table 6: Efficiency Benchmarks on LLaMA-3-8B. The evaluation is conducted on a single NVIDIA A100 (80GB). UGID maintains identical inference latency to the base model while significantly reducing memory overhead via PEFT-based structural alignment.

Panel of Figure 8 — Figure 8: Performance Profiles across Diverse Architectures and Scales. Across all evaluated families, UGID consistently exhibits superior coverage (the largest blue area) compared to baselines, especially in preserving internal structural stability and safety.

Figure 9: Pareto Frontier of Debiasing Effectiveness vs. Structural Integrity.

Model Family	Method	Debiasing Effectiveness ( $\downarrow$ 1.0)				Mechanism ( $\downarrow$ 0)		Safety	Utility
		ID		OOD		Edge	Node	Anchor	Anchor-PPL	IQ
		Mean	Max	Mean	Max	$\Delta$ Spec	$\Delta$ Hidden	Acc ( $\uparrow$ )	( $\downarrow$ )	(Pass)
Gemma-2-2B	Original	6.84x	37.45x	4.48x	8.38x	0.1502	73.00	100%	154.79	–
	CDA	1.13x	1.21x	1.18x	1.21x	0.0499	52.67	100%	5.94	–
	KLAAD	1.01x	1.06x	1.18x	1.29x	0.1572	54.13	50%	20.36	–
	UGID (Ours)	0.95x	1.00x	1.03x	1.13x	0.0067	0.67	100%	165.63	–

Appendix

Table 7: Results on Diverse Model Families. Paleblue and underline indicate the best and second-best results within each model family (excluding Original baseline).

Figure 10: Layer-wise Hidden-State Drift under Counterfactual Pronoun Swaps.

Figure 12: Activation patching / causal tracing results (He to She).

Figure 13: Activation patching / causal tracing results (She to He).

Figure 14: Spectral Signature Difference (layer=15). Token-wise

|s_{l}(x)-s_{l}(x^{\prime})|

for he/she counterfactuals (Original and CDA).

Figure 15: Spectral Signature Difference (layer=15). Token-wise

|s_{l}(x)-s_{l}(x^{\prime})|

for he/she counterfactuals (KLAAD and UGID).

Language Model Debiasing

See all 26 papers

Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement
Jan 29, 2026Jinhao Pan, Chahat Raj, Anjishnu Mukherjee +4Debiased ML Social Bias in Language Models
Department of Computer Science, George Mason University, Fairfax, VA, USA · Lightridge High School, Aldie, VA, USA
Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventions
Sep 30, 2026Chahat Raj, Sina Mansouri, Aylin Caliskan +2Cognitive Bias in Language Models Language Model Debiasing
George Mason University · University of Washington
CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMs
Mar 23, 2026Ravi Ranjan, Utkarsh Grover, Mayur Akewar +2Language Model Debiasing Social Bias in Language Models
Florida International University Miami, USA · University of South Florida Tampa, USA

Jan 29, 2026J/K move · Enter open · S save

Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement

Jinhao Pan, Chahat Raj, Anjishnu Mukherjee, Sina Mansouri+3

Department of Computer Science, George Mason University, Fairfax, VA, USA · Lightridge High School, Aldie, VA, USA

Debiased ML Social Bias in Language Models Language Model Debiasing Demographic Bias in Language Models Activation Steering

Open paper AskPDF arXiv

Large language models (LLMs) exhibit social biases that reinforce harmful stereotypes, limiting their safe deployment. Most existing debiasing methods adopt a suppressive paradigm by modifying parameters, prompts, or neurons associated with biased behavior; however, such approaches are often brittle, weakly generalizable, data-inefficient, and prone to degrading general capability. We propose KnowBias, a lightweight and conceptually distinct framework that mitigates bias by strengthening, rather than suppressing, neurons encoding bias-knowledge. KnowBias identifies neurons encoding bias knowledge using a small set of bias-knowledge questions via attribution-based analysis, and selectively enhances them at inference time. This design enables strong debiasing while preserving general capabilities, generalizes across bias types and demographics, and is highly data efficient, requiring only a handful of simple yes/no questions and no retraining. Experiments across multiple benchmarks and LLMs demonstrate consistent state-of-the-art debiasing performance with minimal utility degradation. Data and code are available at https://github.com/JP-25/KnowBias.

Abstract

Language Model Debiasing

Explore similar work

Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement

Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventions

CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMs