One Model Is Not a Crowd: Multi-LLM and Aspect-Conditioned Diverse Comment Generation

Nafis Irtiza Tripto; Delvin Ce Zhang; Mahjabin Nahar; Dongwon Lee

cs.CLSep 27, 2026

One Model Is Not a Crowd: Multi-LLM and Aspect-Conditioned Diverse Comment Generation

Authors: Nafis Irtiza Tripto, Delvin Ce Zhang, Mahjabin Nahar, Dongwon Lee

Organizations: College of Information Sciences and Technology, Pennsylvania State University, PA, USA · University of Sheffield, Sheffield, UK

Abstract

Human communication on the internet is shaped by diverse perspectives, most visibly expressed in online comment spaces. As large language model (LLM)based AI agents begin to inhabit these spaces, a key question arises: whether synthetic comment threads can capture the diversity inherent in human discourse. This concern is increasingly important, as the growing presence of homogenized AI-generated content risks reducing diversity over time, potentially leading to model collapse and degrading the richness of digital communication. Inspired by the plurality of human crowds and the aspect-driven nature of discourse, we hypothesize that comment diversity is better approximated by combining multiple LLMs with aspect-conditioned generation. We formalize and evaluate this approach using models from different providers and introduce a framework that characterizes diversity across semantic, linguistic, and socio-pragmatic features along three axes: dispersion, coverage, and alignment. Using this framework, we conduct a large-scale study on over 2 million YouTube comments across multiple domains. Our results reveal that multi-LLM and aspect-conditioned generation better align with human comment distributions and such data remains viable under pretraining style curation and is effective for downstream tasks. Yet, human diversity remains unmatched. Overall, our findings provide a practical foundation for generating more diverse and socially grounded discourse in AI-mediated environments.

Figures & tables

Figure 1 : We visualize comments from a YouTube video ( PUBG Season 4 Cinematic Trailer ) using 2D t-SNE ( Van der Maaten and Hinton, 2008 ) projections of text embeddings, comparing single-LLM generations ( S ) and multi-LLM aspect-conditioned generations ( $M_{A}$ ) against human ( $H$ ) comments (blue dots). While single-LLM outputs capture some common themes (e.g., trailer quality), they miss several semantic regions ( violet rectangle ) and exhibit repetition ( red rectangle ). In contrast, ( $M_{A}$ ) generation covers both dominant patterns (e.g., gameplay features, voice quality) and niche topics (e.g., “ winner winner chicken dinner ” PUBG reference), yielding a broader and more structured distribution.

Figure 2 : Overview of our framework. We collect YouTube videos across multiple domains, construct video context from metadata and transcripts, and derive aspect information from human comments via clustering. These are used to guide comment generation under different settings, followed by feature extraction and a comprehensive evaluation across dispersion , coverage , and alignment .

Domain	# Ch.	# Vid.	# Comm.	# Words	Asp./Vid.
News	14	3,857	518.7K	10.27M	$8.15\pm 4.19$
Pop	22	2,203	326.5K	4.56M	$8.21\pm 4.14$
Tech	10	1,514	269.0K	5.34M	$9.30\pm 4.38$

Table 1 : Human dataset summary.

Axis	Definition	Semantic	Lexical	Pragmatic
Dispersion	Intra-set variation within comments	Clipped pairwise cosine dissimilarity (*, $\uparrow$ ) ( Cann et al., 2023 ; Guo et al., 2025 ; Zhang et al., 2025a ) $\mathcal{D}_{\text{sem}}=\mathbb{E}_{i,j}[1-\widetilde{\cos}(c_{i},c_{j})]$	N-gram diversity ( $\uparrow$ ) ( Padmakumar and He, 2024 ) Self-repetition score ( $\downarrow$ ) ( Salkar et al., 2022 ) POS Compression ratio ( $\downarrow$ ) ( Shaib et al., 2025 ) Homogenization Score ( $\uparrow$ ) ( Lin, 2004 )	Simpson diversity index (*, $\uparrow$ ) ( Simpson, 1949 ) $\mathcal{D}_{\text{prag}}=1-\sum_{k}\bar{P}_{k}^{2}$ $\bar{P}_{k}=\frac{1}{N}\sum_{j}p_{j,k}$
Coverage	Fraction of human patterns recovered	Weighted manifold recall (*, $\uparrow$ ) ( Kynkäänniemi et al., 2019 ) $\mathcal{C}_{\text{sem}}=\frac{\sum_{k\in\mathcal{K}(H)}\mathbf{1}[\mathcal{K}_{k}\cap M\neq\emptyset]\cdot w_{k}}{\sum_{k\in\mathcal{K}(H)}w_{k}}$	Weighted n-gram & POS coverage (*, $\uparrow$ ) $\mathcal{C}_{\text{ling}}=\frac{\sum_{g\in\mathcal{P}(H)\cap\mathcal{P}(M)}f_{H}(g)}{\sum_{g\in\mathcal{P}(H)}f_{H}(g)}$	Categorical coverage ratio (*, $\uparrow$ ) $\mathcal{C}_{\text{prag}}=\frac{\|\mathcal{K}(H)\cap\mathcal{K}(M)\|}{\|\mathcal{K}(H)\|}$
Alignment	Distribution similarity between $H$ and $M$	Maximum Mean Discrepancy (RBF kernel) ( $\downarrow$ )	Avg. JSD score from ( $\downarrow$ ) $\mathrm{JSD}_{\text{POS}},\mathrm{JSD}_{\text{ngram}},\mathrm{JSD}_{\text{length}}$	Avg. category JSD ( $\downarrow$ ) $\mathcal{A}_{\text{prag}}=\frac{1}{\|\mathcal{K}\|}\sum_{k\in\mathcal{K}}\mathrm{JSD}_{k}$

Table 2 : Primary evaluation metrics across dispersion, coverage, and alignment for each feature set. * denotes metrics bounded in [0–1];

\uparrow

higher is better,

\downarrow

lower is better.

Panel of Figure 3 — Figure 3 : Dispersion across features. Left : semantic and middle : pragmatic dispersion distribution across videos respectively. Right : normalized linguistic dispersion aggregated across metrics.

Panel of Figure 4 — Figure 4 : Coverage, alignment, and perceived quality across settings. Left: mean coverage scores with std. Middle: alignment scores (lower is better). Right: Overall comment quality and realism.

Panel of Figure 5 — Figure 5 : Pre-training data curation and quality. Left: % of data retained after each step. Middle: distribution of educational value and right: % of document quality of retained samples respectively.

Figure 6 : Fine-tuning and instruction-tuning performance. Left: F1 score comparisons for different training portions. Middle: impact of augmenting human data with LLM-generated samples. Right: instruction-tuning performance across zero-shot, few-shot (human), and few-shot (

M_{A}

) settings.

Panel of Figure 7 — Figure 7 : Left: holistic comparison of aspect vs model swarms across different evaluations. Middle: impact of decoding variation ( $S_{D}$ ) relative to $S$ and other settings. Right: performance differences on unseen content compared to original data across settings (normalized avg. evaluation score).

Panel of Figure 8 — Figure 8 : Left: diversity trends (avg. of the metrics) as the number of comments increases. Middle: diversity trends with increasing number of LLMs per video ( $M$ , $M_{A}$ ). Right: cross-provider similarity across models, with inset showing sample pairwise distribution (Meta–Microsoft).

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Content and Comment Space
$v$	A content item (video).
$\mathcal{C}$	The space of all possible comments.
$C\in\mathcal{C}$	A comment, modelled as a random variable.
$\phi:\mathcal{C}\to\mathbb{R}^{d}$	Fixed embedding function mapping comments into a $d$ -dimensional semantic space.
$H=\{h_{1},\dots,h_{n}\}$	A set of $n$ human-generated comments for video $v$ .

Appendix

Table 3 : Summary of notation used throughout the paper.

Domain	#Channels	#Videos	#Comments	Channels (clickable, ordered by #videos)
News	14	3,857	518,718	Fox News (663, 134,427); ABC News (572, 78,937); CBS News (457, 47,384); MSNBC (379, 55,935); BBC News (379, 42,824); NBC News (275, 25,079); DW News (248, 19,516); CNBC (215, 29,415); Sky News (178, 15,488); CNN (169, 34,187); Al Jazeera (157, 15,315); Wall Street Journal (143, 18,314); AFP News (15, 1,124); Associated Press (7, 773)
Pop	22	2,203	326,542	Nintendo (469, 78,785); PlayStation (405, 63,392); Netflix (363, 58,753); GameSpot Trailers (253, 32,632); Xbox (189, 17,743); Sony Pictures (60, 8,754); Netflix India (53, 7,878); Prime Video (51, 7,470); Marvel (48, 8,164); Warner Bros. (46, 5,779); Universal (41, 5,910); GameTrailers (39, 4,967); 20th Century (31, 4,343); Hulu (31, 3,479); Paramount (28, 4,171); DC (24, 4,204); IGN (22, 3,915); Prime India (20, 3,201); Epic Games (13, 1,168); Prime UK (10, 1,129); Ubisoft (5, 540); Apple TV (2, 165)
Tech	10	1,514	268,954	Linus Tech Tips (272, 45,354); Unbox Therapy (207, 30,910); CNET (201, 25,949); Mrwhosetheboss (158, 36,402); Android Authority (135, 19,224); Austin Evans (131, 31,866); GSMArena (112, 17,750); Dave2D (109, 22,779); UrAvgConsumer (103, 28,077); MKBHD (86, 10,643)

Appendix

Table 4 : Dataset composition by domain. Channel names are clickable links to their respective YouTube pages with ( number of videos , number of comments considered in total ).

Panel of Figure 9 — Figure 9 : Distribution of video duration (minutes) in different domains in our dataset. We prefer shorter videos in specific channels to keep the context minimal.

Figure 10 : Distribution of selected videos per month in our time-frame.

Panel of Figure 11 — Figure 11 : Word cloud distribution from the titles of the videos in our dataset.

Panel of Figure 12 — Figure 12 : Distribution of the number of comments considered per video (after the filtering steps) in each domain.

Panel of Figure 13 — Figure 13 : Distribution of aspects in different videos.

Aspect	%	Pragmatic	Aspect Description
Enthusiastic Fan Reactions (Pragmatic)	65.07	informal joy non-sarcastic	Strong excitement around the film and its Bruce Springsteen connection, with nostalgic references to the soundtrack and repeated praise for the trailer, alongside expressive and hyperbolic fan reactions.
Music-driven Emotional Resonance (Topic)	13.97	–	Viewers share emotional connections to Springsteen’s music and highlight how the film’s coming-of-age narrative resonates with personal experiences and nostalgia.
Emotional & Relatable Storytelling (Topic)	13.10	–	Comments emphasize the film’s emotional depth, relatability, and themes of family, identity, and struggle, often describing cathartic and uplifting viewing experiences.
Springsteen Legacy Appreciation (Topic)	4.80	–	Discussion centers on admiration for Springsteen’s influence, with personal reflections on how his music shapes identity and enhances the film’s impact.
Cultural Identity & Representation (Topic)	2.18	–	Comments focus on South Asian and working-class identity, highlighting cultural pressures and the empowering role of music in shaping self-expression.
Playful Music Culture Observations (Pragmatic)	0.87	sentiment mixed light humor	Lighthearted remarks about music preferences and cultural trends, combining casual humor with appreciation for classic rock influences.

Appendix

Table 5: Example of extracted aspects from a sample video (BLINDED BY THE LIGHT - Official Trailer, ID = DmmHvnS0IKM ).

Provider	Models
OpenAI (7)	✓ gpt-oss-20b
	✓ gpt-oss-120b
	✗ gpt-5
	✗ gpt-5-mini
	✗ gpt-5-nano
	✗ gpt-4.1

Appendix

Table 6: Models grouped by provider. ✓ denotes open-source models and ✗ denotes proprietary models.

Panel of Figure 14 — Figure 14 : Comment length distribution in human comments and LLM-generated comments from different settings. No statistical significance difference was observed for each setting (wrt. human distribution) following a two-tailed t-test.

\mathcal{L}_{v}=\{(\ell_{b},\pi_{b})\}_{b=1}^{B},

Appendix

Algorithm 2 Comment Generation for Setting S: Single-LLM, No Aspects

\mathcal{L}_{v}=\{(\ell_{b},\pi_{b})\}_{b=1}^{B},

Appendix

Algorithm 3 Comment Generation for Setting M: Multi-LLM, No Aspects

\Lambda_{v}=\{(a_{j},\lambda_{j})\}_{j=1}^{A_{v}},

Appendix

Algorithm 4 Comment Generation for Setting

S_{A}

: Single-LLM, With Aspects

\Lambda_{v}=\{(a_{j},\lambda_{j})\}_{j=1}^{A_{v}},

Appendix

Algorithm 5 Comment Generation for Setting

M_{A}

: Multi-LLM, With Aspects

Panel of Figure 15 — Figure 15 : Distribution of text similarity in the proof-of-concept analysis. Left: Pairwise similarity of comments generated for the same video, comparing within-model and cross-model outputs. Right: Pairwise similarity of paragraphs generated on the same topic, comparing within-model and cross-model outputs.

Panel of Figure 16 — Figure 16 : Distribution of provider-level similarity in the proof-of-concept analysis. Left: Similarity between models grouped by provider, computed from comment generation on the same videos (derived from Figure 15 ). Right: Provider-level similarity (same vs. different) based on model similarity reported in [ 66 ] (Table 7-11).

Dataset	H	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	$0.65\pm 0.03$	$0.56\pm 0.04$	$0.57\pm 0.04$	$0.59\pm 0.03$	$0.59\pm 0.03$	$0.60\pm 0.03$	H $>$ $M_{A}$ $>$ $S_{A}$ $>$ M $>$ $S_{D}$ $>$ S
Pop	$0.64\pm 0.03$	$0.56\pm 0.04$	$0.56\pm 0.04$	$0.59\pm 0.03$	$0.62\pm 0.03$	$0.63\pm 0.03$	H $>$ $M_{A}$ $>$ $S_{A}$ $>$ M $>$ $S_{D}$ $>$ S
Tech	$0.64\pm 0.03$	$0.56\pm 0.04$	$0.56\pm 0.04$	$0.58\pm 0.03$	$0.60\pm 0.03$	$0.61\pm 0.03$	H $>$ $M_{A}$ $>$ $S_{A}$ $>$ M $>$ $S_{D}$ $>$ S
All	$0.65\pm 0.03$	$0.56\pm 0.04$	$0.57\pm 0.04$	$0.59\pm 0.03$	$0.61\pm 0.03$	$0.62\pm 0.03$	H $>$ $M_{A}$ $>$ $S_{A}$ $>$ M $>$ $S_{D}$ $>$ S

Appendix

Table 7 : Semantic dispersion results across datasets. Higher values indicate greater semantic diversity.

Dataset	Metric	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	Manifold recall $\uparrow$	$0.69\pm 0.33$	$0.53\pm 0.31$	$0.76\pm 0.54$	$0.90\pm 0.18$	$0.90\pm 0.17$	$M_{A}\sim S_{A}>M>S\sim S_{D}$
News	Semantic recall (MaxSim) $\uparrow$	$0.54\pm 0.04$	$0.54\pm 0.04$	$0.55\pm 0.04$	$0.59\pm 0.04$	$0.59\pm 0.04$	$S_{A}\sim M_{A}>M>S_{D}>S$
News	Semantic recall @ $\tau$ $\uparrow$	$0.26\pm 0.11$	$0.27\pm 0.11$	$0.30\pm 0.10$	$0.44\pm 0.14$	$0.44\pm 0.12$	$S_{A}\sim M_{A}>M>S_{D}>S$
Pop_culture	Manifold recall $\uparrow$	$0.72\pm 0.31$	$0.73\pm 0.30$	$0.75\pm 0.29$	$0.93\pm 0.14$	$0.93\pm 0.14$	$M_{A}\sim S_{A}>M\sim S_{D}\sim S$
Pop_culture	Semantic recall (MaxSim) $\uparrow$	$0.56\pm 0.04$	$0.57\pm 0.04$	$0.58\pm 0.04$	$0.62\pm 0.04$	$0.62\pm 0.04$	$S_{A}\sim M_{A}>M>S_{D}>S$
Pop_culture	Semantic recall @ $\tau$ $\uparrow$	$0.21\pm 0.10$	$0.24\pm 0.10$	$0.25\pm 0.09$	$0.39\pm 0.14$	$0.37\pm 0.12$	$S_{A}>M_{A}>M>S_{D}>S$

Appendix

Table 8 : Semantic coverage results across datasets. Higher values indicate better coverage of the human semantic space.

Dataset	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	$0.06\pm 0.02$	$0.06\pm 0.02$	$0.05\pm 0.01$	$0.02\pm 0.01$	$0.02\pm 0.01$	$M_{A}<S_{A}<M<S_{D}<S$
Pop_culture	$0.05\pm 0.02$	$0.05\pm 0.02$	$0.04\pm 0.02$	$0.02\pm 0.01$	$0.02\pm 0.01$	$M_{A}<S_{A}<M<S_{D}\sim S$
Tech	$0.05\pm 0.02$	$0.04\pm 0.02$	$0.04\pm 0.01$	$0.02\pm 0.01$	$0.02\pm 0.01$	$M_{A}<S_{A}<M<S_{D}\sim S$
All	$0.05\pm 0.02$	$0.05\pm 0.02$	$0.04\pm 0.01$	$0.02\pm 0.01$	$0.02\pm 0.01$	$M_{A}<S_{A}<M<S_{D}<S$

Appendix

Table 9 : Semantic alignment results across datasets. Lower values indicate better alignment with the human semantic distribution.

Dataset	Metric	H	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	n-gram diversity $\uparrow$	$3.35\pm 0.15$	$3.08\pm 0.22$	$2.89\pm 0.25$	$3.18\pm 0.13$	$3.26\pm 0.22$	$3.25\pm 0.18$	H $>$ $S_{A}\sim M_{A}$ $>$ M $>$ S $>$ $S_{D}$
News	TTR $\uparrow$	$0.47\pm 0.09$	$0.39\pm 0.08$	$0.34\pm 0.07$	$0.41\pm 0.07$	$0.46\pm 0.10$	$0.47\pm 0.09$	$M_{A}$ $>$ H $>$ $S_{A}$ $>$ M $>$ S $>$ $S_{D}$
News	Self-repetition $\downarrow$	$0.06\pm 0.07$	$0.52\pm 0.34$	$0.84\pm 0.41$	$0.28\pm 0.16$	$0.24\pm 0.23$	$0.26\pm 0.23$	H $<$ $S_{A}$ $<$ $M_{A}$ $<$ M $<$ S $<$ $S_{D}$
News	Compression ratio $\downarrow$	$2.12\pm 0.17$	$2.72\pm 0.30$	$2.96\pm 0.36$	$2.61\pm 0.18$	$2.40\pm 0.29$	$2.37\pm 0.28$	H $<$ $M_{A}$ $<$ $S_{A}$ $<$ M $<$ S $<$ $S_{D}$
News	POS comp. ratio $\downarrow$	$3.81\pm 0.37$	$4.31\pm 0.43$	$4.59\pm 0.48$	$4.16\pm 0.34$	$4.02\pm 0.42$	$4.03\pm 0.37$	H $<$ $S_{A}\sim M_{A}$ $<$ M $<$ S $<$ $S_{D}$
Pop_culture	n-gram diversity $\uparrow$	$3.34\pm 0.14$	$3.02\pm 0.21$	$2.81\pm 0.25$	$3.10\pm 0.11$	$3.23\pm 0.20$	$3.20\pm 0.17$	H $>$ $S_{A}$ $>$ $M_{A}$ $>$ M $>$ S $>$ $S_{D}$

Appendix

Table 10 : Linguistic dispersion results across datasets. Arrows indicate metric directionality: higher is better for

\uparrow

and lower is better for

\downarrow

Dataset	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	$0.91\pm 0.01$	$0.91\pm 0.01$	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.92\pm 0.01$	$M_{A}$ $>$ $S_{A}$ $>$ M $>$ $S_{D}$ $>$ S
Pop_culture	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.93\pm 0.01$	$0.93\pm 0.01$	$M_{A}$ $>$ $S_{A}$ $>$ M $>$ $S_{D}$ $>$ S
Tech	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.93\pm 0.01$	$0.93\pm 0.01$	$0.93\pm 0.01$	$M_{A}\sim$ M $>$ $S_{A}$ $>$ $S_{D}$ $>$ S
All	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.93\pm 0.01$	$M_{A}$ $>$ $S_{A}$ $>$ M $>$ $S_{D}$ $>$ S

Appendix

Table 11 : Linguistic coverage results across datasets. Higher values indicate better coverage of the human linguistic feature space.

Dataset	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	$0.04\pm 0.02$	$0.04\pm 0.02$	$0.03\pm 0.01$	$0.03\pm 0.02$	$0.02\pm 0.01$	$M_{A}$ $<$ $S_{A}$ $<$ M $<$ $S_{D}$ $<$ S
Pop_culture	$0.05\pm 0.02$	$0.04\pm 0.02$	$0.04\pm 0.02$	$0.03\pm 0.02$	$0.02\pm 0.02$	$M_{A}$ $<$ $S_{A}$ $<$ M $<$ $S_{D}$ $<$ S
Tech	$0.04\pm 0.02$	$0.03\pm 0.02$	$0.03\pm 0.01$	$0.02\pm 0.01$	$0.02\pm 0.01$	$M_{A}$ $<$ $S_{A}$ $<$ M $<$ $S_{D}\sim$ S
All	$0.04\pm 0.02$	$0.04\pm 0.02$	$0.03\pm 0.01$	$0.03\pm 0.02$	$0.02\pm 0.01$	$M_{A}$ $<$ $S_{A}$ $<$ M $<$ $S_{D}$ $<$ S

Appendix

Table 12 : Linguistic alignment results across datasets. Lower values indicate better alignment with the human linguistic distribution.

Dataset	H	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	$0.87\pm 0.00$	$0.85\pm 0.01$	$0.85\pm 0.01$	$0.87\pm 0.00$	$0.85\pm 0.01$	$0.86\pm 0.01$	M $>$ H $>$ $M_{A}$ $>$ $S_{A}$ $>$ S $>$ $S_{D}$
Pop_culture	$0.86\pm 0.01$	$0.85\pm 0.01$	$0.85\pm 0.01$	$0.86\pm 0.00$	$0.85\pm 0.01$	$0.85\pm 0.01$	M $>$ H $>$ $M_{A}$ $>$ $S_{A}$ $>$ S $>$ $S_{D}$
Tech	$0.86\pm 0.00$	$0.80\pm 0.03$	$0.85\pm 0.01$	$0.87\pm 0.00$	$0.85\pm 0.01$	$0.85\pm 0.01$	M $>$ H $>$ $M_{A}$ $>$ $S_{A}$ $>$ $S_{D}$ $>$ S
All	$0.86\pm 0.01$	$0.83\pm 0.03$	$0.85\pm 0.01$	$0.87\pm 0.00$	$0.85\pm 0.01$	$0.85\pm 0.01$	M $>$ H $>$ $M_{A}$ $>$ $S_{A}$ $>$ $S_{D}$ $>$ S

Appendix

Table 13 : Pragmatic dispersion results across datasets. Higher values indicate greater diversity in pragmatic attributes.

Dataset	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	$0.82\pm 0.10$	$0.83\pm 0.10$	$0.86\pm 0.07$	$0.83\pm 0.09$	$0.87\pm 0.07$	$M_{A}>M>S_{A}>S_{D}>S$
Pop_culture	$0.82\pm 0.11$	$0.82\pm 0.11$	$0.88\pm 0.06$	$0.85\pm 0.08$	$0.89\pm 0.06$	$M_{A}>M>S_{A}>S\sim S_{D}$
Tech	$0.84\pm 0.10$	$0.85\pm 0.09$	$0.90\pm 0.06$	$0.82\pm 0.09$	$0.90\pm 0.05$	$M_{A}\sim M>S_{D}>S>S_{A}$
All	$0.83\pm 0.10$	$0.83\pm 0.10$	$0.88\pm 0.07$	$0.83\pm 0.09$	$0.89\pm 0.06$	$M_{A}>M>S_{A}\sim S_{D}\sim S$

Appendix

Table 14 : Pragmatic coverage results across datasets. Higher values indicate broader coverage of the pragmatic characteristics observed in human discussions, including sentiment, stance, emotional tone, and discourse behavior.

Dataset	S	$S_{D}$	M	$S_{A}$	$M_{A}$	Comment
News	$0.09\pm 0.04$	$0.08\pm 0.03$	$0.07\pm 0.03$	$0.08\pm 0.03$	$0.06\pm 0.03$	$M_{A}<M<S_{A}\sim S_{D}<S$
Pop_culture	$0.07\pm 0.03$	$0.07\pm 0.03$	$0.05\pm 0.02$	$0.06\pm 0.02$	$0.05\pm 0.02$	$M_{A}\sim M<S_{A}<S_{D}\sim S$
Tech	$0.06\pm 0.02$	$0.05\pm 0.02$	$0.04\pm 0.01$	$0.05\pm 0.02$	$0.04\pm 0.01$	$M_{A}\sim M<S_{A}\sim S_{D}<S$
All	$0.07\pm 0.03$	$0.07\pm 0.03$	$0.05\pm 0.03$	$0.07\pm 0.03$	$0.05\pm 0.02$	$M_{A}\sim M<S_{A}\sim S_{D}\sim S$

Appendix

Table 15 : Pragmatic alignment results across datasets. Lower values indicate closer alignment with the pragmatic distribution of human discussions, including sentiment balance, discourse roles, and interaction styles.

Figure 17 : Semantic dispersion across domains and generation settings. The relative ordering of settings remains highly consistent across the news , pop_culture , and tech domains, with $M_{A}$ consistently producing the closest approximation to human semantic diversity. The smallest gap between human and synthetic comments is observed in the pop_culture domain.

Figure 18 : Domain-sensitivity heatmap across metrics and generation settings. Each cell represents the range (maximum - minimum) of a metric across domains for a given setting, where darker colors indicate greater sensitivity to domain variation. Pragmatic metrics exhibit the strongest domain sensitivity overall, particularly for the S and M settings, while aspect-conditioned settings ( $S_{A}$ and $M_{A}$ ) remain substantially more stable across domains.

Feature	Axis/Metric	Dataset	H	M	$\boldsymbol{S_{A}}$	$\boldsymbol{M_{A}}$
Semantic	Dispersion $\uparrow$	News-new	$0.65\pm 0.02$	$0.58\pm 0.03$	$0.61\pm 0.03$	$0.62\pm 0.02$
		Pop-culture-new	$0.64\pm 0.03$	$0.60\pm 0.03$	$0.63\pm 0.03$	$0.64\pm 0.02$
		Tech-new	$0.63\pm 0.04$	$0.57\pm 0.04$	$0.59\pm 0.04$	$0.60\pm 0.04$
Semantic	Manifold recall $\uparrow$	News-new	–	$0.77\pm 0.36$	$0.90\pm 0.24$	$0.94\pm 0.17$
		Pop-culture-new	–	$0.78\pm 0.33$	$0.84\pm 0.28$	$0.87\pm 0.25$
		Tech-new	–	$0.91\pm 0.20$	$0.92\pm 0.17$	$0.92\pm 0.16$

Appendix

Table 16 : Summary of the newer-video ablation results across domains. We report representative metrics from the semantic and linguistic evaluation axes. For dispersion metrics, higher values are better; for alignment metrics, lower values are better. Human results are included where applicable.

Metric	Krippendorff’s $\alpha$	Weighted Cohen’s $\kappa$	Agreement Level
Naturalness	0.601	0.591	Moderate
Authenticity	0.463	0.471	Moderate
Overall Quality	0.447	0.446	Moderate
Representativeness	0.413	0.360	Fair
Diversity	0.369	0.397	Fair
Appropriateness	0.298	0.312	Fair

Appendix

Table 17 : Inter-annotator agreement across evaluation metrics using Krippendorff’s

\alpha

and weighted Cohen’s

\kappa

. Agreement levels follow standard interpretation thresholds.

Metric	S	M	$M_{A}$	H	Comment
Diversity	$3.47\pm 0.52$	$3.58\pm 0.63$	$3.98\pm 0.70$	$4.12\pm 0.72$	H $\sim$ $M_{A}$ $>$ M $\sim$ S
Context	$3.90\pm 0.52$	$4.05\pm 0.51$	$4.03\pm 0.47$	$4.05\pm 0.56$	H $\sim$ $M_{A}$ $\sim$ M $\sim$ S
Naturalness	$2.74\pm 0.73$	$3.12\pm 0.57$	$3.60\pm 0.79$	$4.28\pm 0.65$	H $>$ $M_{A}$ $>$ M $>$ S
Authenticity	$3.07\pm 0.56$	$3.35\pm 0.53$	$3.60\pm 0.71$	$3.93\pm 0.58$	H $\sim$ $M_{A}$ $\sim$ M $\sim$ S
Representativeness	$3.17\pm 0.57$	$3.20\pm 0.41$	$3.57\pm 0.58$	$3.70\pm 0.64$	H $\sim$ $M_{A}$ $>$ M $\sim$ S
Appropriateness	$3.81\pm 0.62$	$3.97\pm 0.60$	$3.93\pm 0.64$	$3.77\pm 0.58$	H $\sim$ $M_{A}$ $\sim$ M $\sim$ S

Appendix

Table 18 : Human evaluation scores (mean

\pm

std) across different settings. S: single LLM, M: multi-LLM,

M_{A}

: aspect-guided multi-LLM, H: human comments.

\sim

means the difference between two settings is not statistically significant.

Metric	Human Judgment	LLM Judgment	Setting-level $\rho$	Sample-level $\rho$
Diversity	H $\sim$ $M_{A}$ $>$ M $\sim$ S	H $\sim$ $M_{A}$ $>$ M $\sim$ S	1.00	0.16
Context	H $\sim$ M $\sim$ $M_{A}$ $\sim$ S	S $\sim$ M $>$ $M_{A}$ $\sim$ H	-0.63	0.04
Naturalness	H $>$ $M_{A}$ $>$ M $>$ S	H $\sim$ $M_{A}$ $\sim$ M $\sim$ S	1.00	0.07
Authenticity	H $\sim$ $M_{A}$ $\sim$ M $\sim$ S	H $>$ $M_{A}$ $\sim$ M $>$ S	1.00	0.21
Representativeness	H $\sim$ $M_{A}$ $>$ M $\sim$ S	H $>$ $M_{A}$ $>$ M $\sim$ S	1.00	0.43
Appropriateness	M $\sim$ $M_{A}$ $\sim$ S $\sim$ H	S $\sim$ M $>$ $M_{A}$ $\sim$ H	0.63	0.05

Appendix

Table 19 : Comparison between human and LLM judgments across evaluation metrics. Spearman correlation (

\rho

) is reported at both the setting level and sample level.

Figure 19 : Overall comment quality results on whole dataset using LLM-as-judge.

Setting	Anger			Disgust			Fear			Joy			Neutral			Sadness			Surprise			Mac.F1
	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
H	0.55	0.56	0.55	0.57	0.53	0.55	0.73	0.34	0.46	0.77	0.80	0.79	0.84	0.86	0.85	0.57	0.56	0.56	0.72	0.71	0.71	0.64
S	0.44	0.56	0.49	0.45	0.60	0.52	0.68	0.46	0.55	0.79	0.73	0.76	0.82	0.83	0.82	0.62	0.57	0.59	0.68	0.64	0.66	0.63
M	0.42	0.64	0.51	0.51	0.43	0.47	0.73	0.39	0.51	0.78	0.74	0.76	0.82	0.84	0.83	0.61	0.54	0.58	0.71	0.67	0.69	0.62
$S_{A}$	0.53	0.55	0.54	0.54	0.52	0.53	0.75	0.41	0.54	0.74	0.81	0.77	0.84	0.83	0.84	0.63	0.53	0.57	0.68	0.71	0.70	0.64
$M_{A}$	0.57	0.56	0.57	0.56	0.50	0.53	0.81	0.43	0.56	0.75	0.77	0.76	0.82	0.85	0.83	0.62	0.49	0.54	0.68	0.71	0.69	0.64

Appendix

Table 20 : Full per-class classification report for 7-class emotion fine-tuning across all training settings. P = precision, R = recall, F1 = per-class F1. Mac.F1 = macro F1. All models evaluated on the held-out human test split.

Figure 20 : Heatmap of per-class F1 score while using different training portion in emotion classification task (to show utility as finetuning data).

Figure 21 : Confusion matrix in different settings using Mistral-7B-Instruct-v0.2 model. Providing few-shot examples improve the performance of separating neutral class from liberal class and we observe the similar effect for both using human comments and LLM comments as examples.

		Conservative			Liberal			Neutral			Overall
Model	Condition	P	R	F1	P	R	F1	P	R	F1	Acc	Mac.F1
Qwen-0.5B	Few-shot (H)	0.48	0.36	0.41	0.41	0.59	0.49	0.20	0.06	0.09	0.43	0.33
Qwen-0.5B	Few-shot ( $M_{A}$ )	0.51	0.19	0.28	0.42	0.72	0.53	0.22	0.23	0.22	0.42	0.34
Mistral-7B	Few-shot (H)	0.89	0.83	0.86	0.90	0.51	0.65	0.22	0.82	0.35	0.69	0.62
Mistral-7B	Few-shot ( $M_{A}$ )	0.91	0.84	0.87	0.93	0.47	0.63	0.20	0.82	0.32	0.68	0.61
Gemma-12B	Few-shot (H)	0.74	0.95	0.83	0.96	0.54	0.69	0.41	0.66	0.51	0.75	0.68

Appendix

Table 21 : Full per-class classification report for stance detection under few-shot prompting with human (H) and LLM (

M_{A}

) examples across three backbone models. Support counts: conservative 430, liberal 377, neutral 71 (total 878).

Explore similar work

Nov 5, 2024cs.CL

Growing a Tail: Increasing Output Diversity in Large Language Models

How diverse are the outputs of large language models when diversity is desired? We examine the diversity of responses of several language models to questions with multiple possible answers, comparing them with human responses. Our findings suggest that models' responses are highly concentrated, reflecting narrow, mainstream outputs, in comparison to humans, whose responses exhibit a much longer-tail. We examine three simple and practical ways to increase output diversity: 1) increasing generation randomness via temperature sampling; 2) prompting models to answer from diverse perspectives using a single prompt; 3) aggregating outputs from several models. We find that these interventions, especially when combined, can substantially increase output diversity, although single-model outputs generally remain less diverse than the human baseline. We discuss potential implications of these findings for future work in AI policy and governance that wishes to preserve cultural diversity, an essential building block of a democratic social fabric.

Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim +1

Law Faculty, Hebrew University of Jerusalem; Jerusalem, Israel. · Faculty of Computer Science, Technion – Israel Institute of Technology; Haifa, Israel.

Diversity Responses Eu Artificial Intelligence Act Democracy

Ask PDF

Jun 11, 2026cs.CL

Evaluating Pluralism in LLMs through Latent Perspectives

The growing need to represent diverse perspectives has increased interest in pluralistic LLM generation. Although difficult to operationalize, identifying perspectives expressed in text would provide clear guidance on pluralistic alignment and more clearly articulate the pluralistic gap in LLM generation. While models have been shown to reduce the diversity of training data and generate homogeneously, this has been demonstrated primarily on multiple-choice questionnaires or using high-level characteristics of free-form text. In this paper, we introduce and implement a domain-agnostic multi-layered framework for unsupervised extraction of perspectives suitable for identifying the pluralistic gap in LLM-generated text. We evaluate our framework on book reviews, a highly opinionated dataset representing diverse perspectives, and compare various prompts and models. Our results show that while some models and prompting techniques come close to covering a broad spectrum of perspectives, rarer perspectives remain disproportionately underrepresented, resulting in distributions that diverge from human text.

Laura Majer, Jan Šnajder, Martin Tutek

TakeLab, University of Zagreb, Croatia.

Internal Pluralism Large Language Model Bias Large Language Model Alignment Diversity Large Language Model Generation Perspective+1

Ask PDF

Oct 1, 2025cs.CL

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we identify a fundamental, pervasive data-level driver: typicality bias in preference data, whereby annotators systematically favor familiar text as a result of well-established findings in cognitive psychology. We formalize this bias theoretically, verify it on preference datasets empirically, and show that it plays a central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling, a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., "Generate 5 jokes about coffee and their corresponding probabilities"). Comprehensive experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), dialogue simulation, open-ended QA, and synthetic data generation, without sacrificing factual accuracy and safety. For instance, in creative writing, VS increases diversity by 1.6-2.1x over direct prompting. We further observe an emergent trend that more capable models benefit more from VS. In sum, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.

Jiayi Zhang, Simon Yu, Derek Chong +4

Northeastern University · Stanford University · West Virginia University

Diversity Large Language Model Alignment Single-Cell Foundation Models Verbalization Collapse

Ask PDF