Overton Pluralistic Reinforcement Learning for Large Language Models

Yu Fu; Seongho Son; Ilija Bogunovic

cs.CLFeb 24, 2026

Overton Pluralistic Reinforcement Learning for Large Language Models

Authors: Yu Fu, Seongho Son, Ilija Bogunovic

Organizations: University College London · University of Basel

Abstract

Existing alignment paradigms remain limited in capturing the pluralistic nature of human values. Overton Pluralism addresses this gap by generating responses with diverse perspectives from a single query. This paper introduces OP-GRPO (Overton Pluralistic Group Relative Policy Optimization), a reinforcement learning framework for implicit Overton Pluralism that enables a single large language model to produce pluralistic responses without explicit prompting or modular orchestration. Our workflow consists of two main steps. First, similarity estimator training fine-tunes a Sentence Transformer for Overton Pluralism tasks to provide more accurate coverage evaluation of generated responses. Second, OP-GRPO training incorporates this similarity estimator into a dual-reward system designed to ensure both broad coverage of genuine human perspectives and the uniqueness of each perspective, thereby promoting diversity. Empirical results demonstrate a "small models, big perspective coverage" effect. The trained Qwen2.5-3B-Instruct model surpasses a 20B GPT-OSS baseline with a 37.4 percent relative accuracy gain on a Natural Language Inference benchmark, and also outperforms a modular architecture baseline with a 19.1 percent relative improvement. Additional evaluations using GPT-4.1 as a large language model judge further confirm the robustness of the approach.

Figures & tables

Figure 1: An overview of different examples aimed at generating Overton Pluralism windows. On the top left corner are real multi-perspective responses from diverse human groups to a given query. The figure then illustrates: (1) responses generated by a base LLM using an implicit OP prompt; (2) the same base LLM prompted with an explicit OP instruction; (3) a modular pluralism approach that combines outputs from different community-specific LLMs, summarized by a final LLM to form the OP window ( Feng et al., 2024 ) ; and (4) our RL-trained pipeline, which achieves the highest correct coverage of human reference perspectives in OP responses.

Figure 2: Example illustrating the perspective matching strategy when the number of candidate perspectives equals the number of human reference perspectives, using the OP-SBERT model as the similarity estimator. Top: In the initial approach, each candidate sentence is matched to the reference sentence with the highest similarity score, which can result in multiple candidates being linked to the same reference. Bottom: Left: The initial matching method, which often leads to a many-to-one matching problem. Right: The improved matching strategy using the MBGM algorithm, where once a perspective pair is selected, it is fixed, and the corresponding reference is excluded from subsequent matching steps—thereby preventing repeated pairings.

Figure 3: Comparison of initial and optimized matching strategies using different SBERT models under OP-reward evaluation. Each method shown here adopts its optimal threshold value.

Method	Matrix 1: N-o-p(cand) $\leq$ N-o-p(ref)						Matrix 2: N-o-p(cand) $\geq$ N-o-p(ref)				Total Avg	Efficiency (sec/prompt)
Method	$cp_{1}$	$cp_{2}$	$cp_{3}$	$cp_{4}$	$cp_{5}$	Avg 1	$rp_{3}$	$rp_{4}$	$rp_{5}$	Avg 2	Total Avg	Efficiency (sec/prompt)
LLM Judge
Qwen-2.5-3B-Instruct	0.825	0.600	0.600	0.400	0.625	0.610	0.100	0.050	0.050	0.0667	0.406	1.36
Llama-3.1-8B-Instruct	1.000	0.900	0.825	0.950	0.925	0.920	0.175	0.050	0.100	0.1083	0.6156	2.72
Qwen-2.5-7B-Instruct	1.000	0.975	0.925	0.900	0.825	0.925	0.500	0.600	0.700	0.600	0.8031	1.31
Qwen-3-8B (no think)	1.000	0.925	0.975	0.925	0.900	0.945	0.600	0.600	0.600	0.600	0.816	1.98

Table 1: OP Rewarding Evaluation across different methods. (Matrix 1) The number of candidate perspectives (N-o-p) is smaller than the number of human reference perspectives, covering tasks

cp_{1}

cp_{5}

. (Matrix 2) The number of candidate perspectives is larger than the number of human references, covering tasks

rp_{3}

rp_{5}

. We compare various methods, including LLM-based and SBERT (ST)-based approaches, and further distinguish between the base ST model, ST combined with the MBGM algorithm, and ST with optimized hyperparameter settings. For completeness, we also report the average processing time per question on a single A100 GPU.

Method	OP-V2 Evaluation (Natural Language Inference)
Method	5 persp (Avg./[email protected])	6 persp (Avg./[email protected])	7 persp (Avg./[email protected])	8 persp (Avg./[email protected])	9 persp (Avg./[email protected])	10 persp (Avg./[email protected])	$>$ 10 persp (Avg./[email protected])	Avg.
Qwen2.5-1.5B-Instruct
Implicit OP Prompting	27.3/33.0	19.8/19.0	20.5/18.7	18.6/14.7	17.8/12.7	15.2/8.00	17.0/11.0	19.5/14.1
Explicit OP Prompting	31.5/44.7	24.3/25.6	25.5/29.0	23.6/24.0	21.6/18.3	20.6/14.7	21.3/15.0	24.1/24.5
Implicit OP–GRPO	52.4/85.0	45.8/78.7	45.4/80.6	42.4/72.7	40.1/68.3	39.3/62.7	41.6/66.5	43.9/73.5
Qwen2.5-3B-Instruct

Table 2: OP-V2 Evaluation (Natural Language Inference) across 7 subtasks from (

5p

) to (

>10p

). For each subtask, we report both the Average Score and [email protected] . In addition, the Modular Pluralism method employs Qwen3-14B as the summary model to generate the final OP window. The rows highlighted in blue indicate that our OP-GRPO-trained models achieve the best performance.

Method	5 Perspectives						10 Perspectives						Total Avg.
Method	Help.	Clar.	Fact.	Depth	Engag.	Avg.	Help.	Clar.	Fact.	Depth	Engag.	Avg.	Total Avg.
Llama3.2-3B-Instruct
Implicit OP–GRPO	4.723	4.917	4.830	4.773	4.740	4.797	4.787	4.910	4.937	4.757	3.877	4.653	4.725
Explicit Prompting	3.870	4.370	4.673	4.210	3.443	4.113	4.047	4.487	4.783	3.817	3.333	4.093	4.103
Qwen2.5-1.5B-Instruct
Implicit OP–GRPO	4.480	4.647	4.873	4.520	3.780	4.460	4.567	4.710	4.890	4.500	3.733	4.480	4.470

Table 3: LLM-as-Judge evaluation results (GPT-4.1). The blue tags indicate our methods, evaluated on two subtasks (5p and 10p), with separate aspect scores reported for each.

Figure 4: Average output length of each model (sorted from high to low). Our trained models generate relatively shorter responses without explicitly constraining the output length. The output length of OP–GRPO models represents the final tokens extracted from the summary block.

Figure 8

Setting	$5p$ (Avg./[email protected])	$10p$ (Avg./[email protected])
Finetune + MBGM	54.6 / 88.0	42.3 / 73.3
Finetune, no MBGM	39.4 / 56.3	29.8 / 45.2
Pre-trained + MBGM	42.0 / 69.0	36.0 / 42.1
Pre-trained, no MBGM	30.1 / 44.5	25.2 / 27.8

Figure 6: Qwen2.5-3B-Instruct (full OP–GRPO pipeline ablation). We compare four configurations by enabling or disabling SBERT ( paraphrase-mpnet-base-v2 ) fine-tuning and the MBGM matching strategy on the 5p and 10p subtasks of the OP-V2 test set.

Figure 7: NLI evaluation results and average token generation counts across different training checkpoints. Left: 5-perspective task; Right: 10-perspective task.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Traning Data Example in OP-V1 Dataset

Figure 9: Overview of the data filtering and augmentation process for redundant perspectives. This example shows a user-provided situation with responses from the OP-V1 dataset. Several perspectives, though framed differently (e.g., legal vs. moral), express the same core idea of a property rights violation. To improve dataset quality for downstream RL, redundancies are first identified and removed, then new perspectives are added as needed to ensure each situation is represented by a diverse and semantically distinct set of responses.

Model	LLaMA3 8B	Qwen 2.5-7B	Qwen3-8B	Qwen3-14B	ChatGPT-4.1
Accuracy (%)	78.4	64.1	89.4	90.5	93.0

Appendix

Table 5: Accuracy comparison of different LLMs on 100 manually labeled sentence pairs for semantic redundancy judgment.

Number of Perspectives	Row Count
1	48
2	199
3	794
4	2,490

Appendix

Table 6: Number of rows with fewer than five perspectives before augmentation.

Figure 10: Overall data processing flowchart. The procedure illustrates the detailed changes in the number of rows throughout the processing steps.

Figure 11: Distribution of Perspective Ratio Retained (Uniqueness Score), where $s$ denotes the perspective ratio range. A value of $s=1$ indicates the highest uniqueness score, corresponding to no redundant perspectives. Left: distribution of uniqueness scores in the OP-V1 dataset; Right: distribution of uniqueness scores in the OP-V2 dataset. After preprocessing and data augmentation, OP-V2 exhibits fewer redundancies. Its distribution is more concentrated around high uniqueness scores, whereas OP-V1 is more dispersed, with a larger portion of data failing to achieve a uniqueness score of $1$ .

Threshold	HPO-scale=10	HPO-scale=20	HPO-scale=30	HPO-scale=40	HPO-scale=50	HPO-scale=60	HPO-scale=70
0.65	0.9281	0.9281	0.9219	0.9063	0.8719	0.8281	0.7813
0.66	0.9094	0.9094	0.9344	0.9188	0.8750	0.8313	0.8125
0.67	0.9063	0.9063	0.9306	0.9219	0.8844	0.8594	0.8313
0.68	0.9031	0.9031	0.9219	0.9375	0.8844	0.8625	0.8375
0.69	0.9063	0.9063	0.9313	0.9375	0.8781	0.8750	0.8469
0.70	0.9094	0.9094	0.9094	0.9438	0.8875	0.8875	0.8688

Appendix

Table 7: Scale factors and thresholds for paraphrase-mpnet-base-v2 with MBGM. The results show that the optimal configuration of our OP perspective-matching system is achieved with the fine-tuned paraphrase-mpnet-base-v2 , using a scale factor of

40

and a threshold of

0.70

, which yields an accuracy of

0.944

. This clearly outperforms the base model without hyperparameter optimization, which achieves only

0.909

Model	Method	5 p (Avg./[email protected])	10 p (Avg./[email protected])
Qwen2.5-1.5B-Instruct	Explicit OP Prompting	23.2 / 10.1	14.4 / 2.7
Qwen2.5-1.5B-Instruct	Implicit OP-GRPO (ours)	41.3 / 31.1	24.0 / 8.9
Qwen2.5-3B-Instruct	Explicit OP Prompting	24.4 / 10.5	17.2 / 4.3
Qwen2.5-3B-Instruct	Implicit OP-GRPO (ours)	43.1 / 32.2	25.8 / 10.5
Llama3-2-3B-Instruct	Explicit OP Prompting	19.8 / 7.66	12.6 / 2.33
Llama3-2-3B-Instruct	Implicit OP-GRPO (ours)	41.8 / 32.3	27.1 / 10.7

Appendix

Table 8: Robustness results using NLI backbone roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli with mini-accuracy threshold increased to 0.5

Model	Method	5 p (Avg./[email protected])	10 p (Avg./[email protected])
Qwen2.5-1.5B-Instruct	Explicit OP Prompting	47.4 / 43.8	37.1 / 18.5
Qwen2.5-1.5B-Instruct	Implicit OP-GRPO (ours)	68.4 / 73.8	45.4 / 45.6
Qwen2.5-3B-Instruct	Explicit OP Prompting	50.0 / 45.7	44.3 / 29.4
Qwen2.5-3B-Instruct	Implicit OP-GRPO (ours)	71.6 / 76.4	48.9 / 53.4
Llama3-2-3B-Instruct	Explicit OP Prompting	40.5 / 33.3	32.4 / 16.0
Llama3-2-3B-Instruct	Implicit OP-GRPO (ours)	69.9 / 76.7	51.3 / 54.6

Appendix

Table 9: Robustness results using NLI backbone xlm-roberta-large-xnli with mini-accuracy threshold increased to 0.5

Method	5 p (Var of Avg. Acc.)	10 p (Var of Avg. Acc.)
OP-GRPO (Qwen2.5-1.5B)	0.0422	0.0398
OP-GRPO (Qwen2.5-3B)	0.0518	0.0433
OP-GRPO (Llama3-2-3B)	0.0411	0.0420

Appendix

Table 10: Prompt-style robustness measured as variance of average accuracy across three distinct input styles. Lower variance indicates stronger robustness.

Model	Method	5 p (Avg./[email protected])	10 p (Avg./[email protected])
Qwen2.5-3B	Explicit OP Prompting	36.8 / 38.5	30.1 / 33.0
Qwen2.5-3B	OP-GRPO (ours)	59.1 / 58.4	41.2 / 39.9
Llama3-2-3B	Explicit OP Prompting	33.2 / 30.1	26.5 / 19.0
Llama3-2-3B	OP-GRPO (ours)	57.7 / 54.5	39.5 / 33.9

Appendix

Table 11: Chinese evaluation using cross-lingual NLI backbone xlm-roberta-large-xnli .

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 403 papers

VISPA: Pluralistic Alignment via Automatic Value Selection and Activation
Jan 19, 2026Shenyan Zheng, Jiayou Zhong, Anudeex Shetty +3LLM Alignment Multi-Objective Language Model Alignment
University of Waterloo · University of Melbourne · Macquarie University +2
Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributions
Sep 29, 2026Meng-Chen Wu, Qipin Chen, Ansh Jain +3Human Preference Modeling Inference-Time Language Model Alignment
Amazon Science
Evaluating Pluralism in LLMs through Latent Perspectives
Jun 11, 2026Laura Majer, Jan Šnajder, Martin TutekLLM Evaluation LLM Alignment
TakeLab, University of Zagreb, Croatia.

Jan 19, 2026J/K move · Enter open · S save

VISPA: Pluralistic Alignment via Automatic Value Selection and Activation

Shenyan Zheng, Jiayou Zhong, Anudeex Shetty, Heng Ji+2

University of Waterloo · University of Melbourne · Macquarie University+2

LLM Alignment Multi-Objective Language Model Alignment Pluralistic Alignment Activation Steering

Open paper AskPDF arXiv

As large language models are increasingly used in high-stakes domains, it is essential that their outputs reflect not average} human preference, rather range of varying perspectives. Achieving such pluralism, however, remains challenging. Existing approaches consider limited values or rely on prompt-level interventions, lacking value control and representation. To address this, we introduce VISPA, a training-free pluralistic alignment framework, that enables direct control over value expression by dynamic selection and internal model activation steering. Across extensive empirical studies spanning multiple models and evaluation settings, we show VISPA is performant across all pluralistic alignment modes in healthcare and beyond. Further analysis reveals VISPA is adaptable with different steering initiations, model, and/or values. These results suggest that pluralistic alignment can be achieved through internal activation mechanisms, offering a scalable path toward language models that serves all.

Abstract

RL for Language Models

Explore similar work

VISPA: Pluralistic Alignment via Automatic Value Selection and Activation

Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributions

Evaluating Pluralism in LLMs through Latent Perspectives