Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals

Patrick Gerard; Svitlana Volkova

cs.AIMar 3, 2026

Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals

Authors: Patrick Gerard, Svitlana Volkova

Organizations: Information Sciences Institute, University of Southern California, Marina Del Rey, California, USA · Aptima Inc., Dayton, Ohio, USA

Abstract

Language models deployed in online communities must adapt to norms that vary across social, cultural, and domain-specific contexts. Prior alignment approaches rely on explicit preference supervision or predefined principles, which are effective for well-resourced settings but exclude most online communities -- particularly those without institutional backing, annotation infrastructure, or organized around sensitive topics -- where preference elicitation is costly, ethically fraught, or culturally misaligned. We observe that communities already express preferences implicitly through what content they accept, engage with, and allow to persist. We show that this acceptance behavior induces measurable geometric structure in representation space: accepted responses occupy coherent, high-density regions that reflect community-specific norms, while rejected content falls in sparser or misaligned areas. We operationalize this structure as an implicit preference signal for alignment and introduce density-guided response optimization (DGRO), a method that aligns language models to community norms without requiring explicit preference labels. Using labeled preference data, we demonstrate that local density recovers pairwise community judgments, indicating that geometric structure encodes meaningful preference signal. We then apply DGRO in annotation-scarce settings across diverse communities spanning platform, topic, and language. DGRO-aligned models consistently produce responses preferred by human annotators, domain experts, and model-based judges over supervised and prompt-based baselines. We position DGRO as a practical alignment alternative for communities where explicit preference supervision is unavailable or misaligned with situated practices, and discuss the implications and risks of learning from emergent acceptance behavior.

Figures & tables

Figure 1. Conceptual Representation of the Community Consensus Surface. The Z-axis represents a normative log-density, reflecting the implicit filtering of responses by community standards through moderation and collective feedback ( Lampe and Resnick, 2004 ; Chandrasekharan et al., 2017 ) . High-density regions correspond to a coherent, low-dimensional manifold of accepted responses in representation space ( Arora et al., 2018 ; Li et al., 2020 ) . The separation between preferred ( $r^{+}$ ) and non-preferred ( $r^{-}$ ) responses across this surface reflects an acceptance–preference correspondence , motivating preference learning and alignment without explicit annotation ( Christiano et al., 2017 ; Ouyang et al., 2022 ; Rafailov et al., 2023 ) .

Community	Platform	Scale	Acceptance Signal
Q&A	Reddit (SHP)	10K–50K pairs	Pairwise human preferences
Eating Disorder Support	Twitter	$\sim$ 43K posts	Replies, retweets
Eating Disorder Support	Reddit	$\sim$ 9.2M posts	Upvotes, comment depth
Eating Disorder Support	Forums	$\sim$ 1.6M posts	Replies, thread continuation
Conflict Documentation	VK	$\sim$ 8.34M posts	Likes, reposts

Table 1. Communities and data sources used in DGRO evaluation. Validation communities provide explicit preference supervision, while application communities lack pairwise labels and rely on behavioral acceptance signals.

Method	r/askhr	r/askbaking	r/askculinary	r/askhistorians	r/changemyview	r/asksocialscience	r/asksciencefiction
Random	${0.50}\pm{0.00}$	${0.50}\pm{0.00}$	${0.50}\pm{0.00}$	${0.50}\pm{0.00}$	${0.50}\pm{0.00}$	${0.50}\pm{0.00}$	${0.50}\pm{0.00}$
kNN	${0.55}\pm{0.03}$	${0.49}\pm{0.01}$	${0.50}\pm{0.02}$	${0.58}\pm{0.03}$	${0.49}\pm{0.03}$	${0.50}\pm{0.03}$	${0.52}\pm{0.04}$
Global Acceptance Density	${0.68}\pm{0.01}$	${0.53}\pm{0.03}$	${0.51}\pm{0.03}$	${0.60}\pm{0.09}$	${0.57}\pm{0.04}$	${0.59}\pm{0.03}$	${0.49}\pm{0.03}$
Local Acceptance Density	${0.71}\pm{0.03}$	${0.60}\pm{0.02}$	${0.57}\pm{0.04}$	${0.72}\pm{0.03}$	${0.61}\pm{0.03}$	${0.64}\pm{0.01}$	${0.65}\pm{0.02}$
Supervised Model (RM)	${0.75}\pm{0.03}$	${0.65}\pm{0.03}$	${0.72}\pm{0.01}$	${0.74}\pm{0.02}$	${0.68}\pm{0.02}$	${0.80}\pm{0.03}$	${0.72}\pm{0.02}$

Table 2. Pairwise accuracy across communities for unsupervised and supervised methods. Accuracy is reported as mean

\pm

bootstrap half-width,

\delta=\tfrac{1}{2}(\text{hi}-\text{lo})

, computed independently per subreddit. Supervised Model (RM) denotes the supervised reward model (stanfordnlp/SteamSHP-flan-t5-xl), trained with human preference annotations and included as a reference upper bound.

Panel of Figure 2 — Figure 2. Relative accuracy of DRGO-aligned models expressed as a percentage of baseline DPO performance, computed as $100\times(\text{DRGO}/\text{baseline})$ , where $100\%$ denotes parity with the baseline. Error bars denote $\pm 1$ standard error estimated via bootstrap resampling ( $n{=}500$ ), with uncertainty propagated using a first-order delta method.

Community	DGRO vs Base	DGRO vs ICL	DGRO vs SFT
ED-Reddit	$75.4\pm 2.9$ %	$65.8\pm 3.1$ %	$53.8\pm 3.1$ %
ED-Forum	$72.2\pm 3.2$ %	$64.1\pm 4.4$ %	$57.6\pm 3.3$ %
ED-Twitter	$76.1\pm 3.0$ %	$66.3\pm 4.1$ %	$56.9\pm 2.6$ %
VK State	$80.7\pm 3.1$ %	$59.9\pm 3.2$ %	$55.3\pm 2.0$ %

Table 3. LLM-as-judge head-to-head comparison of DGRO against baseline alignment approaches across annotation-scarce communities. Judges compare paired model outputs for the same prompt, using real community responses as contextual grounding for relevance and authenticity. Win rates indicate the percentage of comparisons in which DGRO is preferred (mean

\pm

95% CI).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Subreddit	Train	Validation	Test	Total
r/askhr	8,295	641	395	9,331
r/askbaking	44,007	2,096	1,544	47,647
r/askculinary	45,710	2,094	2,563	50,367
r/askhistorians	3,264	113	164	3,541
r/changemyview	38,173	1,637	1,836	41,646
r/asksocialscience	2,706	147	188	3,041

Appendix

Table 4. Dataset sizes for Reddit communities used in evaluation.

Embedding Model	r/askhr	r/askbaking	r/askculinary	r/askhistorians	r/changemyview	r/asksocialscience	r/asksciencefiction
MPNet (default)	${0.71}\pm{0.03}$	${0.60}\pm{0.02}$	${0.57}\pm{0.04}$	${0.72}\pm{0.03}$	${0.61}\pm{0.03}$	${0.64}\pm{0.01}$	${0.65}\pm{0.02}$
all-MiniLM-L6-v2	${0.70}\pm{0.03}$	${0.59}\pm{0.02}$	${0.56}\pm{0.04}$	${0.70}\pm{0.04}$	${0.60}\pm{0.03}$	${0.63}\pm{0.02}$	${0.64}\pm{0.02}$
E5-large-v2	${0.72}\pm{0.03}$	${0.61}\pm{0.02}$	${0.58}\pm{0.04}$	${0.73}\pm{0.03}$	${0.62}\pm{0.03}$	${0.65}\pm{0.02}$	${0.66}\pm{0.02}$

Appendix

Table 5. Effect of embedding model choice on local acceptance density performance. Accuracy is reported as mean

\pm

bootstrap half-width,

\delta=\tfrac{1}{2}(\text{hi}-\text{lo})

, computed independently per subreddit. Results are shown for the local density method using different sentence embedding models to construct the acceptance manifold.

Base Model	Preference Accuracy $\Delta$ (pp)
google/gemma-2b ( Team et al., 2025 )	$-0.4\pm 0.4$
google/gemma-7b ( Team et al., 2025 )	$+0.3\pm 0.5$
meta-llama/Llama-3.2-3B ( Dubey et al., 2024 )	$+0.1\pm 0.4$
meta-llama/Llama-3.1-8B ( Dubey et al., 2024 )	$+0.6\pm 0.6$

Appendix

Table 6. Deviation in length-normalized preference accuracy on held-out SHP human preference pairs relative to the Pythia-2.8B baseline. Reported values indicate mean difference (in percentage points)

\pm

bootstrap standard error, computed under identical prompts, objectives, and evaluation conditions. Deviations are small across base models, indicating that acceptance density-guided DPO induces consistent preference alignment behavior largely independent of model architecture, which is consistent with prior work ( Rafailov et al., 2023 ) .

Panel of Figure 3 — Figure 3. Local accuracy saturates quickly with neighborhood size. Shown is the worst-case absolute accuracy loss across communities relative to each community’s best-performing neighborhood size, demonstrating that performance remains within a few percentage points of optimal across a wide range of $k$ .

Panel of Figure 4 — Figure 4. Accuracy across communities. Bars show kNN, global density, and local density baselines, evaluated on the easy subset of examples. Vertical ticks denote supervised reward model (RM) accuracy. The dashed vertical line at $0.50$ marks random-chance performance.

Subreddit	AUSC	Pairs to 95% peak
r/askhr	0.971	50
r/askbaking	0.985	150
r/askculinary	0.981	250
r/askhistorians	0.920	1450
r/changemyview	0.978	250
r/asksocialscience	0.950	250

Appendix

Table 7. Data efficiency of the local method across communities. We report the normalized area under the saturation curve (AUSC) and the number of training pairs required to reach 95% of peak accuracy, both computed using accuracy expressed as a percentage of each method’s peak performance. Higher AUSC and lower pair counts indicate faster saturation under limited supervision.

Subreddit	$\rho_{s}$	$p$ -value
r/askhr	0.81	0.015*
r/askbaking	0.75	0.013*
r/askculinary	0.75	0.020*
r/askhistorians	0.45	0.197
r/changemyview	0.60	0.067
r/asksocialscience	0.26	0.500

Appendix

Table 8. Per-subreddit correlations between human agreement strength and local accuracy. For each subreddit, we bin comment pairs by agreement strength (median

\mathrm{score\_ratio}

per bin) and compute local pairwise accuracy within each bin. We then assess the monotonic relationship between bin-level agreement strength and bin-level accuracy using Spearman’s

\rho

. Five of seven subreddits show significant positive correlations (

p<0.05

), with particularly strong effects in r/asksciencefiction (

\rho_{s}=0.90

) and r/askhr (

\rho_{s}=0.81

). Asterisks denote significance levels: *

p<0.05

, ***

p<0.001

Panel of Figure 5 — Figure 5. Higher human agreement correlates with higher local accuracy. Each point is an agreement-strength bin from a subreddit. The moderately strong positive correlation ( $\rho_{s}=0.48$ , $p<10^{-4}$ ) suggests that judge accuracy improves in regions where community preferences are more clearly differentiated. The fitted line is shown for visualization only; significance is assessed with rank correlations.

Community	Expert IAA ( $\alpha$ )	Expert–LLM ( $\rho$ )	LLM vs Expert Majority (%)
ED-Reddit	0.46	0.50	77.9
ED-Forum	0.42	0.48	74.6
ED-Twitter	0.44	0.52	79.1
VK State	0.51	0.55	82.0
Aggregate	0.46	0.51	78.4

Appendix

Table 9. Reliability of human expert and LLM-based evaluation on a stratified subset of 200 examples (50 per domain). Inter-annotator agreement is measured using Krippendorff’s

\alpha

. Expert–LLM alignment is measured using Spearman rank correlation (

\rho

). LLM agreement with expert majority indicates the fraction of cases in which the aggregate LLM judgment matches the expert majority ranking.

Community	Context	Real	Base / ICL	SFT	DGRO
Eating Disorder Support – Reddit	“Please stop me. I am about to binge for the first time since the beginning of the year.”	“Take a deep breath. You’ve done a great job so far. One binge isn’t the end. Pause, breathe, and ask yourself what you need right now.”	“I’m sorry you’re feeling this way. Everything will be okay. Try to stay strong and remember you are not alone.”	“I’m so sorry. I can’t help you. I’m sorry you’re feeling this way too. I’m sorry you feel like that. You sound like a lot of people I know are suffering from a lot of things. I know I’m not alone, so I’m sorry you […]”	“I’m in recovery too, and moments like this are really hard. You’ve already made real progress. Maybe pause, breathe, and check in with what you actually need right now. I’m proud of you for reaching out.”
Conflict Documentation	News post about sanctions and “financial nuclear war”	“Will sanctions forbid treatment for our sick children? Medical equipment? Who is responsible for this madness?”	“Sanctions are a complex political tool. The situation is very complicated.”	“The people have suffered for many years and now face new challenges due to international pressure.”	“And why didn’t Putin, the oligarchs, or the media talk about this before? Now everyone pays the price. The country is run by fools, and ordinary people are left to suffer.”

Appendix

Table 10. Qualitative examples comparing model variants against real community responses.

Figure 6. UMAP visualization of response embeddings for the ED-Forum community. Real community responses are shown in gray, with model outputs overlaid in color using a shared embedding and projection. We display a random subset of 1,000 responses from the ED-Forum dataset for visualization purposes only. Note that these plots are just an illustrative example; they are not intended to support quantitative or comparative claims.

Context: Mike, I have talked to compounding pharmacies who assert that they can sell it because there is a shortage…and lots of people are getting their semaglutide this way from reputable compounding pharmacies. I am not saying it’s legal, but not necessarily unsafe.

Real (Community Response): You can’t guarantee the safety and efficacy of compounded semaglutide. We have no idea where the raw ingredients are being sourced. Overall it’s not a risk I’m willing to take as someone who took the Hippocratic Oath.

Pseudo-Candidate (Lower Ranked): I am a pharmacist and I have been told that there is a shortage of semaglutide from a reputable compounding pharmacy. Can you tell me if that’s true?

Pseudo-Candidate (Higher Ranked): Keto is a low-carb diet in which the body uses fat as fuel instead of carbohydrates, with the goal of improving health.

Appendix

Table 11. Representative failure case from an eating disorder support community. When both pseudo-candidates lie outside the local acceptance manifold, density-based ranking may prefer one misaligned response over another, even when a real community response is clearly preferable.

Preference Alignment

Preference alignment studies how AI systems are trained or steered to produce outputs that match human or user priorities.

See all 109 papers

Apr 6, 2026

Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment

Aligning language models with human preferences is essential for ensuring their safety and reliability. Although most existing approaches assume specific human preference models such as the Bradley-Terry model, this assumption may fail to accurately capture true human preferences, and consequently, these methods lack statistical consistency, i.e., the guarantee that language models converge to the true human preference as the number of samples increases. In contrast, direct density ratio optimization (DDRO) achieves statistical consistency without assuming any human preference models. DDRO models the density ratio between preferred and non-preferred data distributions using the language model, and then optimizes it via density ratio estimation. However, this density ratio is unstable and often diverges, leading to training instability of DDRO. In this paper, we propose a novel alignment method that is both stable and statistically consistent. Our approach is based on the relative density ratio between the preferred data distribution and a mixture of the preferred and non-preferred data distributions. Our approach is stable since this relative density ratio is bounded above and does not diverge. Moreover, it is statistically consistent and yields significantly tighter convergence guarantees than DDRO. We experimentally show its effectiveness with Qwen 2.5 and Llama 3.

Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai +4

NTT, Inc. · NTT DOCOMO, INC.

LLM Alignment Preference Alignment Density Ratio Estimation

Ask PDF

Jun 17, 2026

Steerable Cultural Preference Optimization of Reward Models

It is essential for large language model (LLM) technology to serve many different cultural sub-communities in a manner that is acceptable to each community. However, research on LLM alignment has so far predominantly focused on predicting a unified response preference of annotators from certain regions. This paper aims to advance the development of alignment models with a more global outlook, that are able to accurately represent the preferences of subcommunities and do not exhibit excessive bias towards any of them. We focus on the development of reward models for this purpose and present a novel reward model training algorithm (SCPO) that can incorporate diverse cultural preferences in a balanced manner. Our method results in performance increases of the minority reward model of up to 7 points over the baseline model across two datasets, PRISM and GlobalOpinionQA, and across 7 countries. SCPO is up to 280% more training data-efficient than full-data finetuning of reward models. In addition, we perform analysis of bias by separately evaluating on the preference of subcommunities and show that excessive bias is mitigated via our weighting method. Our code is available at https://github.com/minsik-ai/Steerable-Cultural-Preference

Minsik Oh, Advit Deepak, Sophie Wu +2

1Stanford University · University of Amsterdam

Reward Modeling Preference Modeling Cultural Bias in Language Models Multi-Objective Language Model Alignment

Ask PDF

Jun 4, 2026

RedditPersona: A Modular Framework for Community-Conditioned LLM Adaptation from Reddit

Community-conditioned language model adaptation requires choices about data collection, community definition, and evaluation that are currently made independently in each study, making it hard to compare assumptions or reuse artifacts. We present RedditPersona, a modular framework that standardizes these choices: it collects Reddit posts and comments, profiles active users, partitions them under five grouping strategies (subreddit-based, graph-structural, semantic, hybrid, and interaction-based), trains a parameter-efficient adapter per strategy via QLoRA, and evaluates them under a shared metric suite spanning fluency, fidelity, distributional alignment, and community identifiability. Applied to 112 subreddits in the urban well-being domain (301,429 user profiles, 16M+ comments), we find that adapters' behavioral identifiability tracks each strategy's intrinsic agreement with the subreddit baseline, and that a consistent trade-off between identifiability and distributional similarity to real text holds across all five strategies. The code and configuration files are available at: https://github.com/Ahghaffari/redditpersona.

Amirhossein Ghaffari, Ali Goodarzi, Huong Nguyen +3

Future Computing Group University of Oulu Oulu, Finland · Centre for Applied Computing University of Oulu Oulu, Finland

LLM Evaluation LLM Fine-Tuning Personality Modeling in Language Models Adapter Tuning

Ask PDF

Abstract

Preference Alignment

Explore similar work

Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment

Steerable Cultural Preference Optimization of Reward Models

RedditPersona: A Modular Framework for Community-Conditioned LLM Adaptation from Reddit