Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO

Jinquan Zheng; Jia Yuan; Jiacheng Yao; Chenyang Gu; Pujun Zheng; Guoxiu He

cs.CLMar 22, 2026

Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO

Authors: Jinquan Zheng, Jia Yuan, Jiacheng Yao, Chenyang Gu, Pujun Zheng, Guoxiu He

Organizations: School of Economics and Management, East China Normal University

Abstract

Large language models (LLMs) used for multiple-choice and pairwise evaluation tasks often exhibit selection bias due to non-semantic factors like option positions and label symbols. Existing inference-time debiasing is costly and may harm reasoning, while pointwise training ignores that the same question should yield consistent answers across permutations. To address this issue, we propose Permutation-Aware Group Relative Policy Optimization (PA-GRPO), which mitigates selection bias by enforcing permutation-consistent semantic reasoning. PA-GRPO constructs a permutation group for each instance by generating multiple candidate permutations, and optimizes the model using two complementary mechanisms: (1) cross-permutation advantage, which computes advantages relative to the mean reward over all permutations of the same instance, and (2) consistency-aware reward, which encourages the model to produce consistent decisions across different permutations. Experimental results demonstrate that PA-GRPO outperforms strong baselines across seven benchmarks, substantially reducing selection bias while maintaining high overall performance. The code is available on github (https://github.com/ECNU-Text-Computing/PA-GRPO).

Figures & tables

Figure 1: Example of selection bias in discrete-choice settings: swapping the order of answers changes only non-semantic factors.

Figure 2: Comparison between Standard GRPO (left) and the proposed PA-GRPO (right). Standard GRPO treats permuted prompts as independent samples, suffering from permutation blindness where inconsistency goes unpunished. In contrast, PA-GRPO organizes samples into Permutation Groups. It introduces (1) a Cross-Permutation Advantage (using the permutation group mean as a baseline) and (2) a Consistency-Aware Reward to explicitly enforce semantic invariance across different permutations of the same instance.

		LLM-as-a-Judge												Multiple Choice Questions
Model	Method	MT-Bench			JudgeBench			PreferenceBench			RewardBench			GPQA			ARC-Challenge			TinyMMLU
Model	Method	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA
Llama-3.1-8B	Base	59.6	25.2	22.2	35.0	34.8	6.1	60.8	22.6	22.1	60.5	31.5	26.2	27.3	40.0	27.2	73.0	82.5	78.9	54.2	68.9	57.0
	GRPO	75.7	80.6	65.4	48.2	56.1	28.2	82.2	85.1	76.3	70.9	76.9	61.5	29.5	55.6	29.7	89.0	92.1	91.5	67.3	77.9	73.0
	UniBias	68.5	59.4	48.2	50.2	23.0	10.9	79.7	65.6	62.5	68.4	64.4	50.6	32.3	56.1	31.8	80.4	90.0	83.6	61.1	76.5	65.0
	PriDe	65.6	46.2	38.9	49.1	16.2	7.2	81.6	69.1	66.2	65.1	52.8	41.5	31.0	49.1	27.4	79.1	83.0	79.0	59.7	69.5	58.0

Table 1: Main results on Llama-3.1-8B-Instruct, Qwen3-8B, and Qwen3-32B across 7 benchmarks. All scores are rounded to one decimal place. PA-GRPO results show the absolute score (top) and the gain/drop compared to the best baseline (bottom). Bold indicates the best result in each column.

	LLM-as-a-Judge												Multiple Choice Questions
Method	MT-Bench			JudgeBench			PreferenceBench			RewardBench			GPQA			ARC-Challenge			TinyMMLU
Method	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA
Base	59.6	25.2	22.2	35.0	34.8	6.1	60.8	22.6	22.1	60.5	31.5	26.2	27.3	40.0	27.2	73.0	82.5	78.9	54.2	68.9	57.0
GRPO	75.7	80.6	65.4	48.2	56.1	28.2	82.2	85.1	76.3	70.9	76.9	61.5	29.5	55.6	29.7	89.0	92.1	91.5	67.3	77.9	73.0
+ $r_{\text{con}}$	75.2	83.1	66.5	48.0	56.5	29.2	82.6	85.9	76.9	70.6	79.0	61.9	29.9	56.4	31.0	87.8	92.3	92.0	67.1	78.8	73.0
+ $A_{\text{PA}}$	76.6	84.2	67.8	49.3	55.4	30.5	83.4	86.4	77.8	69.8	78.7	61.1	30.7	57.1	32.2	89.7	92.3	91.9	67.5	79.4	74.0

Table 2: Ablation study on Llama-3.1-8B-Instruct across all 7 benchmarks. We group datasets into LLM-as-a-Judge tasks and MCQ tasks. The best results are bolded . Adding both components (

A_{\text{PA}}

and

r_{\text{con}}

) consistently yields the best robustness.

	LLM-as-a-Judge												Multiple Choice Questions
Setting	MT-Bench			JudgeBench			PreferenceBench			RewardBench			GPQA			ARC-Challenge			TinyMMLU
Setting	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA
Base	59.6	25.2	22.2	35.0	34.8	6.1	60.8	22.6	22.1	60.5	31.5	26.2	27.3	40.0	27.2	73.0	82.5	78.9	54.2	68.9	57.0
$\lambda=0.5$	78.0	86.4	69.8	56.5	55.4	31.9	85.3	86.8	78.9	70.7	80.9	60.5	32.1	56.6	32.0	89.4	92.0	92.4	67.1	78.2	74.0
$\lambda=1.0$ (Default)	77.6	88.0	71.7	57.1	58.3	32.4	86.2	87.2	79.8	71.0	82.7	62.3	32.1	57.2	33.2	89.3	92.9	92.6	67.9	81.9	75.0
$\lambda=2.0$	76.5	88.6	70.5	56.8	55.9	32.0	85.0	87.5	79.1	70.3	82.5	62.1	31.4	58.0	33.6	88.7	91.2	91.1	66.4	80.1	73.0

Table 3: Impact of the consistency reward coefficient (

\lambda

) on Llama-3.1-8B-Instruct. We group datasets into LLM-as-a-Judge tasks and MCQ tasks.

\lambda=1.0

achieves the best balance. The best results are bolded .

Figure 3: Impact of Permutation Group Size (

P

	LLM-as-a-Judge												Multiple Choice Questions
Method	MT-Bench			JudgeBench			PreferenceBench			RewardBench			GPQA			ARC-Challenge			TinyMMLU
Method	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA
Base (Direct)	59.6	25.2	22.2	35.0	34.8	6.1	60.8	22.6	22.1	60.5	31.5	26.2	27.3	40.0	27.2	73.0	82.5	78.9	54.2	68.9	57.0
Base (CoT)	68.7	68.2	58.0	39.0	48.5	17.1	70.3	61.3	58.1	67.5	76.2	54.8	24.5	53.9	26.6	84.6	84.2	81.1	62.4	74.3	64.0
PA-GRPO (Direct)	75.1	85.5	69.3	58.0	54.5	31.8	84.8	82.3	75.9	69.5	80.1	59.8	31.0	54.1	30.8	86.5	89.9	88.7	66.5	77.9	72.0
PA-GRPO (CoT)	77.6	88.0	71.7	57.1	58.3	32.4	86.2	87.2	79.8	71.0	82.7	62.3	32.1	57.2	33.2	89.3	92.9	92.6	67.9	81.9	75.0

Table 4: Performance comparison of Direct vs. Chain-of-Thought (CoT) inference strategies. We report Accuracy, Consistency, and Consistent Accuracy for both the Base model and PA-GRPO across seven benchmarks.

	LLM-as-a-Judge												Multiple Choice Questions
Bias Setting	MT-Bench			JudgeBench			PreferenceBench			RewardBench			GPQA			ARC-Challenge			TinyMMLU
Bias Setting	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA	Acc	Con	CA
Label Bias (Label-only)	76.7	92.2	72.8	56.3	79.0	45.8	74.9	62.8	56.2	70.7	87.5	64.4	22.0	67.2	20.8	80.7	95.6	81.3	63.5	83.8	66.0
Position Bias (Order-only)	77.4	85.1	69.9	55.0	45.5	27.7	74.2	66.9	57.7	71.6	82.0	62.6	27.9	65.0	27.7	80.9	91.9	82.8	64.3	80.5	72.0
Selection Bias (Standard)	77.6	88.0	71.7	57.1	58.3	32.4	86.2	87.2	79.8	71.0	82.7	62.3	32.1	57.2	33.2	89.3	92.9	92.6	67.9	81.9	75.0

Table 5: Bias decomposition results under three controlled protocols: Label Bias, Position Bias, and Selection Bias. We report Accuracy, Consistency, and Consistent Accuracy for PA-GRPO across seven benchmarks.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	LLM-as-a-Judge								Multiple Choice Questions
Method / Setting	MT-Bench		JudgeBench		PreferenceBench		RewardBench		GPQA		ARC-Challenge		TinyMMLU
Method / Setting	RStd	CKLD	RStd	CKLD	RStd	CKLD	RStd	CKLD	RStd	CKLD	RStd	CKLD	RStd	CKLD
Base	0.364	0.378	0.216	0.627	0.386	0.453	0.331	0.291	0.056	1.570	0.025	0.005	0.033	0.028
GRPO	0.036	0.010	0.060	0.156	0.057	0.009	0.024	0.006	0.042	1.331	0.019	0.001	0.026	0.030
UniBias	0.047	0.005	0.050	0.019	0.066	0.008	0.184	0.380	0.044	1.129	0.015	0.003	0.027	0.019
PriDe	0.078	0.012	0.107	0.023	0.051	0.006	0.029	0.002	0.036	0.983	0.014	0.002	0.029	0.031

Appendix

Table 6: Bias evaluation results using RStd and CKLD (lower is better) on Llama-3.1-8B-Instruct across all seven benchmarks. All rows use the default evaluation prompt unless otherwise noted. The final row reports PA-GRPO under the rewritten prompt shown below. Bold indicates the best (lowest) result among all compared settings.

Explore similar work

Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventions
Sep 30, 2026Chahat Raj, Sina Mansouri, Aylin Caliskan +2Social Bias in Language Models Language Model Bias Evaluation
George Mason University · University of Washington
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
Jun 3, 2026Saket Reddy, Ke Yang, ChengXiang ZhaiReward Modeling RL for Language Models
University of Illinois - Urbana-Champaign
Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation
May 4, 2026Muneeb Ur Raheem KhanDebiased ML Language Model Decoding
Lahore University of Management Sciences (LUMS)

Sep 30, 2026J/K move · Enter open · S save

Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventions

Chahat Raj, Sina Mansouri, Aylin Caliskan, Antonios Anastasopoulos+1

George Mason University · University of Washington

Social Bias in Language Models Language Model Bias Evaluation

Open paper AskPDF arXiv

Bias has long been studied in social psychology and cognitive science, where decades of research have produced a body of validated interventions that reduce stereotypical thinking and prejudiced responses in humans. We propose Debias It Yourself (DIY), a cognitively grounded framework that translates five such interventions into debiasing procedures for large language models and delivers them through three established paradigms: Show (in-context examples), Train (instruction tuning), and Revise (guided self-revision). Across three models, five bias benchmarks, eleven debiasing baselines, and three reasoning benchmarks, Train+Revise and Revise alone attain the top two average ranks, lead the bias-reasoning tradeoff (mean bias as low as 2% at 90% reasoning accuracy), and reduce bias on unseen dimensions by up to 14.8%. Our code and data are publicly available.

Abstract

Explore similar work

Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventions

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation