Existing alignment paradigms remain limited in capturing the pluralistic nature of human values. Overton Pluralism addresses this gap by generating responses with diverse perspectives from a single query. This paper introduces OP-GRPO (Overton Pluralistic Group Relative Policy Optimization), a reinforcement learning framework for implicit Overton Pluralism that enables a single large language model to produce pluralistic responses without explicit prompting or modular orchestration. Our workflow consists of two main steps. First, similarity estimator training fine-tunes a Sentence Transformer for Overton Pluralism tasks to provide more accurate coverage evaluation of generated responses. Second, OP-GRPO training incorporates this similarity estimator into a dual-reward system designed to ensure both broad coverage of genuine human perspectives and the uniqueness of each perspective, thereby promoting diversity. Empirical results demonstrate a "small models, big perspective coverage" effect. The trained Qwen2.5-3B-Instruct model surpasses a 20B GPT-OSS baseline with a 37.4 percent relative accuracy gain on a Natural Language Inference benchmark, and also outperforms a modular architecture baseline with a 19.1 percent relative improvement. Additional evaluations using GPT-4.1 as a large language model judge further confirm the robustness of the approach.
Figures & tables
Figure 1: An overview of different examples aimed at generating Overton Pluralism windows. On the top left corner are real multi-perspective responses from diverse human groups to a given query. The figure then illustrates: (1) responses generated by a base LLM using an implicit OP prompt; (2) the same base LLM prompted with an explicit OP instruction; (3) a modular pluralism approach that combines outputs from different community-specific LLMs, summarized by a final LLM to form the OP window ( Feng et al., 2024 ) ; and (4) our RL-trained pipeline, which achieves the highest correct coverage of human reference perspectives in OP responses.
Figure 2: Example illustrating the perspective matching strategy when the number of candidate perspectives equals the number of human reference perspectives, using the OP-SBERT model as the similarity estimator. Top: In the initial approach, each candidate sentence is matched to the reference sentence with the highest similarity score, which can result in multiple candidates being linked to the same reference. Bottom: Left: The initial matching method, which often leads to a many-to-one matching problem. Right: The improved matching strategy using the MBGM algorithm, where once a perspective pair is selected, it is fixed, and the corresponding reference is excluded from subsequent matching steps—thereby preventing repeated pairings.
Figure 3: Comparison of initial and optimized matching strategies using different SBERT models under OP-reward evaluation. Each method shown here adopts its optimal threshold value.
Method
Matrix 1: N-o-p(cand) ≤ N-o-p(ref)
Matrix 2: N-o-p(cand) ≥ N-o-p(ref)
Total Avg
Efficiency (sec/prompt)
cp1
cp2
cp3
cp4
cp5
Avg 1
rp3
rp4
rp5
Avg 2
LLM Judge
Qwen-2.5-3B-Instruct
0.825
0.600
0.600
0.400
0.625
0.610
0.100
0.050
0.050
0.0667
0.406
1.36
Llama-3.1-8B-Instruct
1.000
0.900
0.825
0.950
0.925
0.920
0.175
0.050
0.100
0.1083
0.6156
2.72
Qwen-2.5-7B-Instruct
1.000
0.975
0.925
0.900
0.825
0.925
0.500
0.600
0.700
0.600
0.8031
1.31
Qwen-3-8B (no think)
1.000
0.925
0.975
0.925
0.900
0.945
0.600
0.600
0.600
0.600
0.816
1.98
Table 1: OP Rewarding Evaluation across different methods. (Matrix 1) The number of candidate perspectives (N-o-p) is smaller than the number of human reference perspectives, covering tasks cp1 to cp5 . (Matrix 2) The number of candidate perspectives is larger than the number of human references, covering tasks rp3 to rp5 . We compare various methods, including LLM-based and SBERT (ST)-based approaches, and further distinguish between the base ST model, ST combined with the MBGM algorithm, and ST with optimized hyperparameter settings. For completeness, we also report the average processing time per question on a single A100 GPU.
Table 2: OP-V2 Evaluation (Natural Language Inference) across 7 subtasks from ( 5p ) to ( >10p ). For each subtask, we report both the Average Score and [email protected] . In addition, the Modular Pluralism method employs Qwen3-14B as the summary model to generate the final OP window. The rows highlighted in blue indicate that our OP-GRPO-trained models achieve the best performance.
Method
5 Perspectives
10 Perspectives
Total Avg.
Help.
Clar.
Fact.
Depth
Engag.
Avg.
Help.
Clar.
Fact.
Depth
Engag.
Avg.
Llama3.2-3B-Instruct
Implicit OP–GRPO
4.723
4.917
4.830
4.773
4.740
4.797
4.787
4.910
4.937
4.757
3.877
4.653
4.725
Explicit Prompting
3.870
4.370
4.673
4.210
3.443
4.113
4.047
4.487
4.783
3.817
3.333
4.093
4.103
Qwen2.5-1.5B-Instruct
Implicit OP–GRPO
4.480
4.647
4.873
4.520
3.780
4.460
4.567
4.710
4.890
4.500
3.733
4.480
4.470
Table 3: LLM-as-Judge evaluation results (GPT-4.1). The blue tags indicate our methods, evaluated on two subtasks (5p and 10p), with separate aspect scores reported for each.
Figure 4: Average output length of each model (sorted from high to low). Our trained models generate relatively shorter responses without explicitly constraining the output length. The output length of OP–GRPO models represents the final tokens extracted from the summary block.
Figure 6: Qwen2.5-3B-Instruct (full OP–GRPO pipeline ablation). We compare four configurations by enabling or disabling SBERT ( paraphrase-mpnet-base-v2 ) fine-tuning and the MBGM matching strategy on the 5p and 10p subtasks of the OP-V2 test set.
Figure 7: NLI evaluation results and average token generation counts across different training checkpoints. Left: 5-perspective task; Right: 10-perspective task.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Traning Data Example in OP-V1 Dataset
Figure 9: Overview of the data filtering and augmentation process for redundant perspectives. This example shows a user-provided situation with responses from the OP-V1 dataset. Several perspectives, though framed differently (e.g., legal vs. moral), express the same core idea of a property rights violation. To improve dataset quality for downstream RL, redundancies are first identified and removed, then new perspectives are added as needed to ensure each situation is represented by a diverse and semantically distinct set of responses.
Model
LLaMA3 8B
Qwen 2.5-7B
Qwen3-8B
Qwen3-14B
ChatGPT-4.1
Accuracy (%)
78.4
64.1
89.4
90.5
93.0
Appendix
Table 5: Accuracy comparison of different LLMs on 100 manually labeled sentence pairs for semantic redundancy judgment.
Number of Perspectives
Row Count
1
48
2
199
3
794
4
2,490
Appendix
Table 6: Number of rows with fewer than five perspectives before augmentation.
Figure 10: Overall data processing flowchart. The procedure illustrates the detailed changes in the number of rows throughout the processing steps.
Figure 11: Distribution of Perspective Ratio Retained (Uniqueness Score), where s denotes the perspective ratio range. A value of s=1 indicates the highest uniqueness score, corresponding to no redundant perspectives. Left: distribution of uniqueness scores in the OP-V1 dataset; Right: distribution of uniqueness scores in the OP-V2 dataset. After preprocessing and data augmentation, OP-V2 exhibits fewer redundancies. Its distribution is more concentrated around high uniqueness scores, whereas OP-V1 is more dispersed, with a larger portion of data failing to achieve a uniqueness score of 1 .
Figure 12: Triplet Dataset Example
Threshold
HPO-scale=10
HPO-scale=20
HPO-scale=30
HPO-scale=40
HPO-scale=50
HPO-scale=60
HPO-scale=70
0.65
0.9281
0.9281
0.9219
0.9063
0.8719
0.8281
0.7813
0.66
0.9094
0.9094
0.9344
0.9188
0.8750
0.8313
0.8125
0.67
0.9063
0.9063
0.9306
0.9219
0.8844
0.8594
0.8313
0.68
0.9031
0.9031
0.9219
0.9375
0.8844
0.8625
0.8375
0.69
0.9063
0.9063
0.9313
0.9375
0.8781
0.8750
0.8469
0.70
0.9094
0.9094
0.9094
0.9438
0.8875
0.8875
0.8688
Appendix
Table 7: Scale factors and thresholds for paraphrase-mpnet-base-v2 with MBGM. The results show that the optimal configuration of our OP perspective-matching system is achieved with the fine-tuned paraphrase-mpnet-base-v2 , using a scale factor of 40 and a threshold of 0.70 , which yields an accuracy of 0.944 . This clearly outperforms the base model without hyperparameter optimization, which achieves only 0.909 .
Table 9: Robustness results using NLI backbone xlm-roberta-large-xnli with mini-accuracy threshold increased to 0.5
Method
5 p (Var of Avg. Acc.)
10 p (Var of Avg. Acc.)
OP-GRPO (Qwen2.5-1.5B)
0.0422
0.0398
OP-GRPO (Qwen2.5-3B)
0.0518
0.0433
OP-GRPO (Llama3-2-3B)
0.0411
0.0420
Appendix
Table 10: Prompt-style robustness measured as variance of average accuracy across three distinct input styles. Lower variance indicates stronger robustness.
As large language models are increasingly used in high-stakes domains, it is essential that their outputs reflect not average} human preference, rather range of varying perspectives. Achieving such pluralism, however, remains challenging. Existing approaches consider limited values or rely on prompt-level interventions, lacking value control and representation. To address this, we introduce VISPA, a training-free pluralistic alignment framework, that enables direct control over value expression by dynamic selection and internal model activation steering. Across extensive empirical studies spanning multiple models and evaluation settings, we show VISPA is performant across all pluralistic alignment modes in healthcare and beyond. Further analysis reveals VISPA is adaptable with different steering initiations, model, and/or values. These results suggest that pluralistic alignment can be achieved through internal activation mechanisms, offering a scalable path toward language models that serves all.
Shenyan Zheng, Jiayou Zhong, Anudeex Shetty +3
University of Waterloo · University of Melbourne · Macquarie University +2
Large language models (LLMs) are increasingly used in culturally sensitive settings, where alignment requires representing diverse preferences within populations. Yet existing methods model populations at coarse demographic or community levels and overlook within-group variation. We introduce Demographic Pluralism, an inference-time framework that estimates population-level opinion distributions without opinion-distribution training data or task-specific fine-tuning by generating multiple perspectives within demographically grounded groups. Across four backbones on GlobalOpinionQA and VITAL, it reduces Jensen-Shannon distance by 8.4%-26.4% over Modular Pluralism. Among weighted, equal-weighted, and inverse-weighted aggregation, equal weighting performs best overall; group-level error also increases with group weight, helping explain weighted aggregation's weaker performance.
The growing need to represent diverse perspectives has increased interest in pluralistic LLM generation. Although difficult to operationalize, identifying perspectives expressed in text would provide clear guidance on pluralistic alignment and more clearly articulate the pluralistic gap in LLM generation. While models have been shown to reduce the diversity of training data and generate homogeneously, this has been demonstrated primarily on multiple-choice questionnaires or using high-level characteristics of free-form text. In this paper, we introduce and implement a domain-agnostic multi-layered framework for unsupervised extraction of perspectives suitable for identifying the pluralistic gap in LLM-generated text. We evaluate our framework on book reviews, a highly opinionated dataset representing diverse perspectives, and compare various prompts and models. Our results show that while some models and prompting techniques come close to covering a broad spectrum of perspectives, rarer perspectives remain disproportionately underrepresented, resulting in distributions that diverge from human text.