cs.CLSep 27, 2026

One Model Is Not a Crowd: Multi-LLM and Aspect-Conditioned Diverse Comment Generation

Authors: Nafis Irtiza Tripto, Delvin Ce Zhang, Mahjabin Nahar, Dongwon Lee

Organizations: College of Information Sciences and Technology, Pennsylvania State University, PA, USA · University of Sheffield, Sheffield, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human communication on the internet is shaped by diverse perspectives, most visibly expressed in online comment spaces. As large language model (LLM)based AI agents begin to inhabit these spaces, a key question arises: whether synthetic comment threads can capture the diversity inherent in human discourse. This concern is increasingly important, as the growing presence of homogenized AI-generated content risks reducing diversity over time, potentially leading to model collapse and degrading the richness of digital communication. Inspired by the plurality of human crowds and the aspect-driven nature of discourse, we hypothesize that comment diversity is better approximated by combining multiple LLMs with aspect-conditioned generation. We formalize and evaluate this approach using models from different providers and introduce a framework that characterizes diversity across semantic, linguistic, and socio-pragmatic features along three axes: dispersion, coverage, and alignment. Using this framework, we conduct a large-scale study on over 2 million YouTube comments across multiple domains. Our results reveal that multi-LLM and aspect-conditioned generation better align with human comment distributions and such data remains viable under pretraining style curation and is effective for downstream tasks. Yet, human diversity remains unmatched. Overall, our findings provide a practical foundation for generating more diverse and socially grounded discourse in AI-mediated environments.

Figures & tables

Figure 1 : We visualize comments from a YouTube video ( PUBG Season 4 Cinematic Trailer ) using 2D t-SNE ( Van der Maaten and Hinton, 2008 ) projections of text embeddings, comparing single-LLM generations ( S ) and multi-LLM aspect-conditioned generations ( MAM_{A} ) against human ( HH ) comments (blue dots). While single-LLM outputs capture some common themes (e.g., trailer quality), they miss several semantic regions ( violet rectangle ) and exhibit repetition ( red rectangle ). In contrast, ( MAM_{A} ) generation covers both dominant patterns (e.g., gameplay features, voice quality) and niche topics (e.g., “ winner winner chicken dinner ” PUBG reference), yielding a broader and more structured distribution.
Figure 2 : Overview of our framework. We collect YouTube videos across multiple domains, construct video context from metadata and transcripts, and derive aspect information from human comments via clustering. These are used to guide comment generation under different settings, followed by feature extraction and a comprehensive evaluation across dispersion , coverage , and alignment .
Domain# Ch.# Vid.# Comm.# WordsAsp./Vid.
News143,857518.7K10.27M8.15±4.198.15\pm 4.19
Pop222,203326.5K4.56M8.21±4.148.21\pm 4.14
Tech101,514269.0K5.34M9.30±4.389.30\pm 4.38
Table 1 : Human dataset summary.
AxisDefinitionSemanticLexicalPragmatic
DispersionIntra-set variation within commentsClipped pairwise cosine dissimilarity (*, ↑\uparrow ) ( Cann et al., 2023 ; Guo et al., 2025 ; Zhang et al., 2025a ) Dsem=Ei,j[1−cos⁡~(ci,cj)]\mathcal{D}_{\text{sem}}=\mathbb{E}_{i,j}[1-\widetilde{\cos}(c_{i},c_{j})]N-gram diversity ( ↑\uparrow ) ( Padmakumar and He, 2024 ) Self-repetition score ( ↓\downarrow ) ( Salkar et al., 2022 ) POS Compression ratio ( ↓\downarrow ) ( Shaib et al., 2025 ) Homogenization Score ( ↑\uparrow ) ( Lin, 2004 )Simpson diversity index (*, ↑\uparrow ) ( Simpson, 1949 ) Dprag=1−∑kPˉk2\mathcal{D}_{\text{prag}}=1-\sum_{k}\bar{P}_{k}^{2} Pˉk=1N∑jpj,k\bar{P}_{k}=\frac{1}{N}\sum_{j}p_{j,k}
CoverageFraction of human patterns recoveredWeighted manifold recall (*, ↑\uparrow ) ( Kynkäänniemi et al., 2019 ) Csem=∑k∈K(H)1[Kk∩M≠∅]⋅wk∑k∈K(H)wk\mathcal{C}_{\text{sem}}=\frac{\sum_{k\in\mathcal{K}(H)}\mathbf{1}[\mathcal{K}_{k}\cap M\neq\emptyset]\cdot w_{k}}{\sum_{k\in\mathcal{K}(H)}w_{k}}Weighted n-gram & POS coverage (*, ↑\uparrow ) Cling=∑g∈P(H)∩P(M)fH(g)∑g∈P(H)fH(g)\mathcal{C}_{\text{ling}}=\frac{\sum_{g\in\mathcal{P}(H)\cap\mathcal{P}(M)}f_{H}(g)}{\sum_{g\in\mathcal{P}(H)}f_{H}(g)}Categorical coverage ratio (*, ↑\uparrow ) Cprag=∣K(H)∩K(M)∣∣K(H)∣\mathcal{C}_{\text{prag}}=\frac{|\mathcal{K}(H)\cap\mathcal{K}(M)|}{|\mathcal{K}(H)|}
AlignmentDistribution similarity between HH and MMMaximum Mean Discrepancy (RBF kernel) ( ↓\downarrow )Avg. JSD score from ( ↓\downarrow ) JSDPOS,JSDngram,JSDlength\mathrm{JSD}_{\text{POS}},\mathrm{JSD}_{\text{ngram}},\mathrm{JSD}_{\text{length}}Avg. category JSD ( ↓\downarrow ) Aprag=1∣K∣∑k∈KJSDk\mathcal{A}_{\text{prag}}=\frac{1}{|\mathcal{K}|}\sum_{k\in\mathcal{K}}\mathrm{JSD}_{k}
Table 2 : Primary evaluation metrics across dispersion, coverage, and alignment for each feature set. * denotes metrics bounded in [0–1]; ↑\uparrow higher is better, ↓\downarrow lower is better.
Figure 3 : Dispersion across features. Left : semantic and middle : pragmatic dispersion distribution across videos respectively. Right : normalized linguistic dispersion aggregated across metrics.
Figure 4 : Coverage, alignment, and perceived quality across settings. Left: mean coverage scores with std. Middle: alignment scores (lower is better). Right: Overall comment quality and realism.
Figure 5 : Pre-training data curation and quality. Left: % of data retained after each step. Middle: distribution of educational value and right: % of document quality of retained samples respectively.
Figure 6 : Fine-tuning and instruction-tuning performance. Left: F1 score comparisons for different training portions. Middle: impact of augmenting human data with LLM-generated samples. Right: instruction-tuning performance across zero-shot, few-shot (human), and few-shot ( MAM_{A} ) settings.
Figure 7 : Left: holistic comparison of aspect vs model swarms across different evaluations. Middle: impact of decoding variation ( SDS_{D} ) relative to SS and other settings. Right: performance differences on unseen content compared to original data across settings (normalized avg. evaluation score).
Figure 8 : Left: diversity trends (avg. of the metrics) as the number of comments increases. Middle: diversity trends with increasing number of LLMs per video ( MM , MAM_{A} ). Right: cross-provider similarity across models, with inset showing sample pairwise distribution (Meta–Microsoft).
Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

SymbolDescription
Content and Comment Space
vvA content item (video).
C\mathcal{C}The space of all possible comments.
C∈CC\in\mathcal{C}A comment, modelled as a random variable.
ϕ:C→Rd\phi:\mathcal{C}\to\mathbb{R}^{d}Fixed embedding function mapping comments into a dd -dimensional semantic space.
H={h1,…,hn}H=\{h_{1},\dots,h_{n}\}A set of nn human-generated comments for video vv .
Appendix
Table 3 : Summary of notation used throughout the paper.
Domain#Channels#Videos#CommentsChannels (clickable, ordered by #videos)
News143,857518,718Fox News (663, 134,427); ABC News (572, 78,937); CBS News (457, 47,384); MSNBC (379, 55,935); BBC News (379, 42,824); NBC News (275, 25,079); DW News (248, 19,516); CNBC (215, 29,415); Sky News (178, 15,488); CNN (169, 34,187); Al Jazeera (157, 15,315); Wall Street Journal (143, 18,314); AFP News (15, 1,124); Associated Press (7, 773)
Pop222,203326,542Nintendo (469, 78,785); PlayStation (405, 63,392); Netflix (363, 58,753); GameSpot Trailers (253, 32,632); Xbox (189, 17,743); Sony Pictures (60, 8,754); Netflix India (53, 7,878); Prime Video (51, 7,470); Marvel (48, 8,164); Warner Bros. (46, 5,779); Universal (41, 5,910); GameTrailers (39, 4,967); 20th Century (31, 4,343); Hulu (31, 3,479); Paramount (28, 4,171); DC (24, 4,204); IGN (22, 3,915); Prime India (20, 3,201); Epic Games (13, 1,168); Prime UK (10, 1,129); Ubisoft (5, 540); Apple TV (2, 165)
Tech101,514268,954Linus Tech Tips (272, 45,354); Unbox Therapy (207, 30,910); CNET (201, 25,949); Mrwhosetheboss (158, 36,402); Android Authority (135, 19,224); Austin Evans (131, 31,866); GSMArena (112, 17,750); Dave2D (109, 22,779); UrAvgConsumer (103, 28,077); MKBHD (86, 10,643)
Appendix
Table 4 : Dataset composition by domain. Channel names are clickable links to their respective YouTube pages with ( number of videos , number of comments considered in total ).
Figure 9 : Distribution of video duration (minutes) in different domains in our dataset. We prefer shorter videos in specific channels to keep the context minimal.
Figure 10 : Distribution of selected videos per month in our time-frame.
Figure 11 : Word cloud distribution from the titles of the videos in our dataset.
Figure 12 : Distribution of the number of comments considered per video (after the filtering steps) in each domain.
Figure 13 : Distribution of aspects in different videos.
Aspect%PragmaticAspect Description
Enthusiastic Fan Reactions (Pragmatic)65.07informal joy non-sarcasticStrong excitement around the film and its Bruce Springsteen connection, with nostalgic references to the soundtrack and repeated praise for the trailer, alongside expressive and hyperbolic fan reactions.
Music-driven Emotional Resonance (Topic)13.97–Viewers share emotional connections to Springsteen’s music and highlight how the film’s coming-of-age narrative resonates with personal experiences and nostalgia.
Emotional & Relatable Storytelling (Topic)13.10–Comments emphasize the film’s emotional depth, relatability, and themes of family, identity, and struggle, often describing cathartic and uplifting viewing experiences.
Springsteen Legacy Appreciation (Topic)4.80–Discussion centers on admiration for Springsteen’s influence, with personal reflections on how his music shapes identity and enhances the film’s impact.
Cultural Identity & Representation (Topic)2.18–Comments focus on South Asian and working-class identity, highlighting cultural pressures and the empowering role of music in shaping self-expression.
Playful Music Culture Observations (Pragmatic)0.87sentiment mixed light humorLighthearted remarks about music preferences and cultural trends, combining casual humor with appreciation for classic rock influences.
Appendix
Table 5: Example of extracted aspects from a sample video (BLINDED BY THE LIGHT - Official Trailer, ID = DmmHvnS0IKM ).
ProviderModels
OpenAI (7)✓ gpt-oss-20b
✓ gpt-oss-120b
✗ gpt-5
✗ gpt-5-mini
✗ gpt-5-nano
✗ gpt-4.1
Appendix
Table 6: Models grouped by provider. ✓ denotes open-source models and ✗ denotes proprietary models.
Figure 14 : Comment length distribution in human comments and LLM-generated comments from different settings. No statistical significance difference was observed for each setting (wrt. human distribution) following a two-tailed t-test.
 Lv={(ℓb,πb)}b=1B,\mathcal{L}_{v}=\{(\ell_{b},\pi_{b})\}_{b=1}^{B}, 
Appendix
Algorithm 2 Comment Generation for Setting S: Single-LLM, No Aspects
 Lv={(ℓb,πb)}b=1B,\mathcal{L}_{v}=\{(\ell_{b},\pi_{b})\}_{b=1}^{B}, 
Appendix
Algorithm 3 Comment Generation for Setting M: Multi-LLM, No Aspects
 Λv={(aj,λj)}j=1Av,\Lambda_{v}=\{(a_{j},\lambda_{j})\}_{j=1}^{A_{v}}, 
Appendix
Algorithm 4 Comment Generation for Setting SAS_{A} : Single-LLM, With Aspects
 Λv={(aj,λj)}j=1Av,\Lambda_{v}=\{(a_{j},\lambda_{j})\}_{j=1}^{A_{v}}, 
Appendix
Algorithm 5 Comment Generation for Setting MAM_{A} : Multi-LLM, With Aspects
Figure 15 : Distribution of text similarity in the proof-of-concept analysis. Left: Pairwise similarity of comments generated for the same video, comparing within-model and cross-model outputs. Right: Pairwise similarity of paragraphs generated on the same topic, comparing within-model and cross-model outputs.
Figure 16 : Distribution of provider-level similarity in the proof-of-concept analysis. Left: Similarity between models grouped by provider, computed from comment generation on the same videos (derived from Figure 15 ). Right: Provider-level similarity (same vs. different) based on model similarity reported in [ 66 ] (Table 7-11).
DatasetHSSDS_{D}MSAS_{A}MAM_{A}Comment
News0.65±0.030.65\pm 0.030.56±0.040.56\pm 0.040.57±0.040.57\pm 0.040.59±0.030.59\pm 0.030.59±0.030.59\pm 0.030.60±0.030.60\pm 0.03H >> MAM_{A} >> SAS_{A} >> M >> SDS_{D} >> S
Pop0.64±0.030.64\pm 0.030.56±0.040.56\pm 0.040.56±0.040.56\pm 0.040.59±0.030.59\pm 0.030.62±0.030.62\pm 0.030.63±0.030.63\pm 0.03H >> MAM_{A} >> SAS_{A} >> M >> SDS_{D} >> S
Tech0.64±0.030.64\pm 0.030.56±0.040.56\pm 0.040.56±0.040.56\pm 0.040.58±0.030.58\pm 0.030.60±0.030.60\pm 0.030.61±0.030.61\pm 0.03H >> MAM_{A} >> SAS_{A} >> M >> SDS_{D} >> S
All0.65±0.030.65\pm 0.030.56±0.040.56\pm 0.040.57±0.040.57\pm 0.040.59±0.030.59\pm 0.030.61±0.030.61\pm 0.030.62±0.030.62\pm 0.03H >> MAM_{A} >> SAS_{A} >> M >> SDS_{D} >> S
Appendix
Table 7 : Semantic dispersion results across datasets. Higher values indicate greater semantic diversity.
DatasetMetricSSDS_{D}MSAS_{A}MAM_{A}Comment
NewsManifold recall ↑\uparrow0.69±0.330.69\pm 0.330.53±0.310.53\pm 0.310.76±0.540.76\pm 0.540.90±0.180.90\pm 0.180.90±0.170.90\pm 0.17MA∼SA>M>S∼SDM_{A}\sim S_{A}>M>S\sim S_{D}
NewsSemantic recall (MaxSim) ↑\uparrow0.54±0.040.54\pm 0.040.54±0.040.54\pm 0.040.55±0.040.55\pm 0.040.59±0.040.59\pm 0.040.59±0.040.59\pm 0.04SA∼MA>M>SD>SS_{A}\sim M_{A}>M>S_{D}>S
NewsSemantic recall @ τ\tau ↑\uparrow0.26±0.110.26\pm 0.110.27±0.110.27\pm 0.110.30±0.100.30\pm 0.100.44±0.140.44\pm 0.140.44±0.120.44\pm 0.12SA∼MA>M>SD>SS_{A}\sim M_{A}>M>S_{D}>S
Pop_cultureManifold recall ↑\uparrow0.72±0.310.72\pm 0.310.73±0.300.73\pm 0.300.75±0.290.75\pm 0.290.93±0.140.93\pm 0.140.93±0.140.93\pm 0.14MA∼SA>M∼SD∼SM_{A}\sim S_{A}>M\sim S_{D}\sim S
Pop_cultureSemantic recall (MaxSim) ↑\uparrow0.56±0.040.56\pm 0.040.57±0.040.57\pm 0.040.58±0.040.58\pm 0.040.62±0.040.62\pm 0.040.62±0.040.62\pm 0.04SA∼MA>M>SD>SS_{A}\sim M_{A}>M>S_{D}>S
Pop_cultureSemantic recall @ τ\tau ↑\uparrow0.21±0.100.21\pm 0.100.24±0.100.24\pm 0.100.25±0.090.25\pm 0.090.39±0.140.39\pm 0.140.37±0.120.37\pm 0.12SA>MA>M>SD>SS_{A}>M_{A}>M>S_{D}>S
Appendix
Table 8 : Semantic coverage results across datasets. Higher values indicate better coverage of the human semantic space.
DatasetSSDS_{D}MSAS_{A}MAM_{A}Comment
News0.06±0.020.06\pm 0.020.06±0.020.06\pm 0.020.05±0.010.05\pm 0.010.02±0.010.02\pm 0.010.02±0.010.02\pm 0.01MA<SA<M<SD<SM_{A}<S_{A}<M<S_{D}<S
Pop_culture0.05±0.020.05\pm 0.020.05±0.020.05\pm 0.020.04±0.020.04\pm 0.020.02±0.010.02\pm 0.010.02±0.010.02\pm 0.01MA<SA<M<SD∼SM_{A}<S_{A}<M<S_{D}\sim S
Tech0.05±0.020.05\pm 0.020.04±0.020.04\pm 0.020.04±0.010.04\pm 0.010.02±0.010.02\pm 0.010.02±0.010.02\pm 0.01MA<SA<M<SD∼SM_{A}<S_{A}<M<S_{D}\sim S
All0.05±0.020.05\pm 0.020.05±0.020.05\pm 0.020.04±0.010.04\pm 0.010.02±0.010.02\pm 0.010.02±0.010.02\pm 0.01MA<SA<M<SD<SM_{A}<S_{A}<M<S_{D}<S
Appendix
Table 9 : Semantic alignment results across datasets. Lower values indicate better alignment with the human semantic distribution.
DatasetMetricHSSDS_{D}MSAS_{A}MAM_{A}Comment
Newsn-gram diversity ↑\uparrow3.35±0.153.35\pm 0.153.08±0.223.08\pm 0.222.89±0.252.89\pm 0.253.18±0.133.18\pm 0.133.26±0.223.26\pm 0.223.25±0.183.25\pm 0.18H >> SA∼MAS_{A}\sim M_{A} >> M >> S >> SDS_{D}
NewsTTR ↑\uparrow0.47±0.090.47\pm 0.090.39±0.080.39\pm 0.080.34±0.070.34\pm 0.070.41±0.070.41\pm 0.070.46±0.100.46\pm 0.100.47±0.090.47\pm 0.09MAM_{A} >> H >> SAS_{A} >> M >> S >> SDS_{D}
NewsSelf-repetition ↓\downarrow0.06±0.070.06\pm 0.070.52±0.340.52\pm 0.340.84±0.410.84\pm 0.410.28±0.160.28\pm 0.160.24±0.230.24\pm 0.230.26±0.230.26\pm 0.23H << SAS_{A} << MAM_{A} << M << S << SDS_{D}
NewsCompression ratio ↓\downarrow2.12±0.172.12\pm 0.172.72±0.302.72\pm 0.302.96±0.362.96\pm 0.362.61±0.182.61\pm 0.182.40±0.292.40\pm 0.292.37±0.282.37\pm 0.28H << MAM_{A} << SAS_{A} << M << S << SDS_{D}
NewsPOS comp. ratio ↓\downarrow3.81±0.373.81\pm 0.374.31±0.434.31\pm 0.434.59±0.484.59\pm 0.484.16±0.344.16\pm 0.344.02±0.424.02\pm 0.424.03±0.374.03\pm 0.37H << SA∼MAS_{A}\sim M_{A} << M << S << SDS_{D}
Pop_culturen-gram diversity ↑\uparrow3.34±0.143.34\pm 0.143.02±0.213.02\pm 0.212.81±0.252.81\pm 0.253.10±0.113.10\pm 0.113.23±0.203.23\pm 0.203.20±0.173.20\pm 0.17H >> SAS_{A} >> MAM_{A} >> M >> S >> SDS_{D}
Appendix
Table 10 : Linguistic dispersion results across datasets. Arrows indicate metric directionality: higher is better for ↑\uparrow and lower is better for ↓\downarrow .
DatasetSSDS_{D}MSAS_{A}MAM_{A}Comment
News0.91±0.010.91\pm 0.010.91±0.010.91\pm 0.010.92±0.010.92\pm 0.010.92±0.010.92\pm 0.010.92±0.010.92\pm 0.01MAM_{A} >> SAS_{A} >> M >> SDS_{D} >> S
Pop_culture0.92±0.010.92\pm 0.010.92±0.010.92\pm 0.010.92±0.010.92\pm 0.010.93±0.010.93\pm 0.010.93±0.010.93\pm 0.01MAM_{A} >> SAS_{A} >> M >> SDS_{D} >> S
Tech0.92±0.010.92\pm 0.010.92±0.010.92\pm 0.010.93±0.010.93\pm 0.010.93±0.010.93\pm 0.010.93±0.010.93\pm 0.01MA∼M_{A}\sim M >> SAS_{A} >> SDS_{D} >> S
All0.92±0.010.92\pm 0.010.92±0.010.92\pm 0.010.92±0.010.92\pm 0.010.92±0.010.92\pm 0.010.93±0.010.93\pm 0.01MAM_{A} >> SAS_{A} >> M >> SDS_{D} >> S
Appendix
Table 11 : Linguistic coverage results across datasets. Higher values indicate better coverage of the human linguistic feature space.
DatasetSSDS_{D}MSAS_{A}MAM_{A}Comment
News0.04±0.020.04\pm 0.020.04±0.020.04\pm 0.020.03±0.010.03\pm 0.010.03±0.020.03\pm 0.020.02±0.010.02\pm 0.01MAM_{A} << SAS_{A} << M << SDS_{D} << S
Pop_culture0.05±0.020.05\pm 0.020.04±0.020.04\pm 0.020.04±0.020.04\pm 0.020.03±0.020.03\pm 0.020.02±0.020.02\pm 0.02MAM_{A} << SAS_{A} << M << SDS_{D} << S
Tech0.04±0.020.04\pm 0.020.03±0.020.03\pm 0.020.03±0.010.03\pm 0.010.02±0.010.02\pm 0.010.02±0.010.02\pm 0.01MAM_{A} << SAS_{A} << M << SD∼S_{D}\sim S
All0.04±0.020.04\pm 0.020.04±0.020.04\pm 0.020.03±0.010.03\pm 0.010.03±0.020.03\pm 0.020.02±0.010.02\pm 0.01MAM_{A} << SAS_{A} << M << SDS_{D} << S
Appendix
Table 12 : Linguistic alignment results across datasets. Lower values indicate better alignment with the human linguistic distribution.
DatasetHSSDS_{D}MSAS_{A}MAM_{A}Comment
News0.87±0.000.87\pm 0.000.85±0.010.85\pm 0.010.85±0.010.85\pm 0.010.87±0.000.87\pm 0.000.85±0.010.85\pm 0.010.86±0.010.86\pm 0.01M >> H >> MAM_{A} >> SAS_{A} >> S >> SDS_{D}
Pop_culture0.86±0.010.86\pm 0.010.85±0.010.85\pm 0.010.85±0.010.85\pm 0.010.86±0.000.86\pm 0.000.85±0.010.85\pm 0.010.85±0.010.85\pm 0.01M >> H >> MAM_{A} >> SAS_{A} >> S >> SDS_{D}
Tech0.86±0.000.86\pm 0.000.80±0.030.80\pm 0.030.85±0.010.85\pm 0.010.87±0.000.87\pm 0.000.85±0.010.85\pm 0.010.85±0.010.85\pm 0.01M >> H >> MAM_{A} >> SAS_{A} >> SDS_{D} >> S
All0.86±0.010.86\pm 0.010.83±0.030.83\pm 0.030.85±0.010.85\pm 0.010.87±0.000.87\pm 0.000.85±0.010.85\pm 0.010.85±0.010.85\pm 0.01M >> H >> MAM_{A} >> SAS_{A} >> SDS_{D} >> S
Appendix
Table 13 : Pragmatic dispersion results across datasets. Higher values indicate greater diversity in pragmatic attributes.
DatasetSSDS_{D}MSAS_{A}MAM_{A}Comment
News0.82±0.100.82\pm 0.100.83±0.100.83\pm 0.100.86±0.070.86\pm 0.070.83±0.090.83\pm 0.090.87±0.070.87\pm 0.07MA>M>SA>SD>SM_{A}>M>S_{A}>S_{D}>S
Pop_culture0.82±0.110.82\pm 0.110.82±0.110.82\pm 0.110.88±0.060.88\pm 0.060.85±0.080.85\pm 0.080.89±0.060.89\pm 0.06MA>M>SA>S∼SDM_{A}>M>S_{A}>S\sim S_{D}
Tech0.84±0.100.84\pm 0.100.85±0.090.85\pm 0.090.90±0.060.90\pm 0.060.82±0.090.82\pm 0.090.90±0.050.90\pm 0.05MA∼M>SD>S>SAM_{A}\sim M>S_{D}>S>S_{A}
All0.83±0.100.83\pm 0.100.83±0.100.83\pm 0.100.88±0.070.88\pm 0.070.83±0.090.83\pm 0.090.89±0.060.89\pm 0.06MA>M>SA∼SD∼SM_{A}>M>S_{A}\sim S_{D}\sim S
Appendix
Table 14 : Pragmatic coverage results across datasets. Higher values indicate broader coverage of the pragmatic characteristics observed in human discussions, including sentiment, stance, emotional tone, and discourse behavior.
DatasetSSDS_{D}MSAS_{A}MAM_{A}Comment
News0.09±0.040.09\pm 0.040.08±0.030.08\pm 0.030.07±0.030.07\pm 0.030.08±0.030.08\pm 0.030.06±0.030.06\pm 0.03MA<M<SA∼SD<SM_{A}<M<S_{A}\sim S_{D}<S
Pop_culture0.07±0.030.07\pm 0.030.07±0.030.07\pm 0.030.05±0.020.05\pm 0.020.06±0.020.06\pm 0.020.05±0.020.05\pm 0.02MA∼M<SA<SD∼SM_{A}\sim M<S_{A}<S_{D}\sim S
Tech0.06±0.020.06\pm 0.020.05±0.020.05\pm 0.020.04±0.010.04\pm 0.010.05±0.020.05\pm 0.020.04±0.010.04\pm 0.01MA∼M<SA∼SD<SM_{A}\sim M<S_{A}\sim S_{D}<S
All0.07±0.030.07\pm 0.030.07±0.030.07\pm 0.030.05±0.030.05\pm 0.030.07±0.030.07\pm 0.030.05±0.020.05\pm 0.02MA∼M<SA∼SD∼SM_{A}\sim M<S_{A}\sim S_{D}\sim S
Appendix
Table 15 : Pragmatic alignment results across datasets. Lower values indicate closer alignment with the pragmatic distribution of human discussions, including sentiment balance, discourse roles, and interaction styles.
Figure 17 : Semantic dispersion across domains and generation settings. The relative ordering of settings remains highly consistent across the news , pop_culture , and tech domains, with MAM_{A} consistently producing the closest approximation to human semantic diversity. The smallest gap between human and synthetic comments is observed in the pop_culture domain.
Figure 18 : Domain-sensitivity heatmap across metrics and generation settings. Each cell represents the range (maximum - minimum) of a metric across domains for a given setting, where darker colors indicate greater sensitivity to domain variation. Pragmatic metrics exhibit the strongest domain sensitivity overall, particularly for the S and M settings, while aspect-conditioned settings ( SAS_{A} and MAM_{A} ) remain substantially more stable across domains.
FeatureAxis/MetricDatasetHMSA\boldsymbol{S_{A}}MA\boldsymbol{M_{A}}
SemanticDispersion ↑\uparrowNews-new0.65±0.020.65\pm 0.020.58±0.030.58\pm 0.030.61±0.030.61\pm 0.030.62±0.020.62\pm 0.02
Pop-culture-new0.64±0.030.64\pm 0.030.60±0.030.60\pm 0.030.63±0.030.63\pm 0.030.64±0.020.64\pm 0.02
Tech-new0.63±0.040.63\pm 0.040.57±0.040.57\pm 0.040.59±0.040.59\pm 0.040.60±0.040.60\pm 0.04
SemanticManifold recall ↑\uparrowNews-new–0.77±0.360.77\pm 0.360.90±0.240.90\pm 0.240.94±0.170.94\pm 0.17
Pop-culture-new–0.78±0.330.78\pm 0.330.84±0.280.84\pm 0.280.87±0.250.87\pm 0.25
Tech-new–0.91±0.200.91\pm 0.200.92±0.170.92\pm 0.170.92±0.160.92\pm 0.16
Appendix
Table 16 : Summary of the newer-video ablation results across domains. We report representative metrics from the semantic and linguistic evaluation axes. For dispersion metrics, higher values are better; for alignment metrics, lower values are better. Human results are included where applicable.
MetricKrippendorff’s α\alphaWeighted Cohen’s κ\kappaAgreement Level
Naturalness0.6010.591Moderate
Authenticity0.4630.471Moderate
Overall Quality0.4470.446Moderate
Representativeness0.4130.360Fair
Diversity0.3690.397Fair
Appropriateness0.2980.312Fair
Appendix
Table 17 : Inter-annotator agreement across evaluation metrics using Krippendorff’s α\alpha and weighted Cohen’s κ\kappa . Agreement levels follow standard interpretation thresholds.
MetricSMMAM_{A}HComment
Diversity3.47±0.523.47\pm 0.523.58±0.633.58\pm 0.633.98±0.703.98\pm 0.704.12±0.724.12\pm 0.72H ∼\sim MAM_{A} >> M ∼\sim S
Context3.90±0.523.90\pm 0.524.05±0.514.05\pm 0.514.03±0.474.03\pm 0.474.05±0.564.05\pm 0.56H ∼\sim MAM_{A} ∼\sim M ∼\sim S
Naturalness2.74±0.732.74\pm 0.733.12±0.573.12\pm 0.573.60±0.793.60\pm 0.794.28±0.654.28\pm 0.65H >> MAM_{A} >> M >> S
Authenticity3.07±0.563.07\pm 0.563.35±0.533.35\pm 0.533.60±0.713.60\pm 0.713.93±0.583.93\pm 0.58H ∼\sim MAM_{A} ∼\sim M ∼\sim S
Representativeness3.17±0.573.17\pm 0.573.20±0.413.20\pm 0.413.57±0.583.57\pm 0.583.70±0.643.70\pm 0.64H ∼\sim MAM_{A} >> M ∼\sim S
Appropriateness3.81±0.623.81\pm 0.623.97±0.603.97\pm 0.603.93±0.643.93\pm 0.643.77±0.583.77\pm 0.58H ∼\sim MAM_{A} ∼\sim M ∼\sim S
Appendix
Table 18 : Human evaluation scores (mean ±\pm std) across different settings. S: single LLM, M: multi-LLM, MAM_{A} : aspect-guided multi-LLM, H: human comments. ∼\sim means the difference between two settings is not statistically significant.
MetricHuman JudgmentLLM JudgmentSetting-level ρ\rhoSample-level ρ\rho
DiversityH ∼\sim MAM_{A} >> M ∼\sim SH ∼\sim MAM_{A} >> M ∼\sim S1.000.16
ContextH ∼\sim M ∼\sim MAM_{A} ∼\sim SS ∼\sim M >> MAM_{A} ∼\sim H-0.630.04
NaturalnessH >> MAM_{A} >> M >> SH ∼\sim MAM_{A} ∼\sim M ∼\sim S1.000.07
AuthenticityH ∼\sim MAM_{A} ∼\sim M ∼\sim SH >> MAM_{A} ∼\sim M >> S1.000.21
RepresentativenessH ∼\sim MAM_{A} >> M ∼\sim SH >> MAM_{A} >> M ∼\sim S1.000.43
AppropriatenessM ∼\sim MAM_{A} ∼\sim S ∼\sim HS ∼\sim M >> MAM_{A} ∼\sim H0.630.05
Appendix
Table 19 : Comparison between human and LLM judgments across evaluation metrics. Spearman correlation ( ρ\rho ) is reported at both the setting level and sample level.
Figure 19 : Overall comment quality results on whole dataset using LLM-as-judge.
SettingAngerDisgustFearJoyNeutralSadnessSurpriseMac.F1
 PRF1PRF1PRF1PRF1PRF1PRF1PRF1 
H0.550.560.550.570.530.550.730.340.460.770.800.790.840.860.850.570.560.560.720.710.710.64
S0.440.560.490.450.600.520.680.460.550.790.730.760.820.830.820.620.570.590.680.640.660.63
M0.420.640.510.510.430.470.730.390.510.780.740.760.820.840.830.610.540.580.710.670.690.62
SAS_{A}0.530.550.540.540.520.530.750.410.540.740.810.770.840.830.840.630.530.570.680.710.700.64
MAM_{A}0.570.560.570.560.500.530.810.430.560.750.770.760.820.850.830.620.490.540.680.710.690.64
Appendix
Table 20 : Full per-class classification report for 7-class emotion fine-tuning across all training settings. P = precision, R = recall, F1 = per-class F1. Mac.F1 = macro F1. All models evaluated on the held-out human test split.
Figure 20 : Heatmap of per-class F1 score while using different training portion in emotion classification task (to show utility as finetuning data).
Figure 21 : Confusion matrix in different settings using Mistral-7B-Instruct-v0.2 model. Providing few-shot examples improve the performance of separating neutral class from liberal class and we observe the similar effect for both using human comments and LLM comments as examples.
  ConservativeLiberalNeutralOverall
ModelConditionPRF1PRF1PRF1AccMac.F1
Qwen-0.5BFew-shot (H)0.480.360.410.410.590.490.200.060.090.430.33
Few-shot ( MAM_{A} )0.510.190.280.420.720.530.220.230.220.420.34
Mistral-7BFew-shot (H)0.890.830.860.900.510.650.220.820.350.690.62
Few-shot ( MAM_{A} )0.910.840.870.930.470.630.200.820.320.680.61
Gemma-12BFew-shot (H)0.740.950.830.960.540.690.410.660.510.750.68
Appendix
Table 21 : Full per-class classification report for stance detection under few-shot prompting with human (H) and LLM ( MAM_{A} ) examples across three backbone models. Support counts: conservative 430, liberal 377, neutral 71 (total 878).

Explore similar work

CardsList

  1. Growing a Tail: Increasing Output Diversity in Large Language Models

    Nov 5, 2024Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim +1DiversityResponses

    Law Faculty, Hebrew University of Jerusalem; Jerusalem, Israel. · Faculty of Computer Science, Technion – Israel Institute of Technology; Haifa, Israel.

  2. Evaluating Pluralism in LLMs through Latent Perspectives

    Jun 11, 2026Laura Majer, Jan Šnajder, Martin TutekInternal PluralismLarge Language Model Bias

    TakeLab, University of Zagreb, Croatia.

  3. Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

    Oct 1, 2025Jiayi Zhang, Simon Yu, Derek Chong +4DiversityLarge Language Model Alignment

    Northeastern University · Stanford University · West Virginia University