LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads?

Xinyi Liu; Rinat Khaziev; Dilek Hakkani-Tür; Tarek F. Abdelzaher

cs.CLOct 2, 2026

LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads?

Authors: Xinyi Liu, Rinat Khaziev, Dilek Hakkani-Tür, Tarek F. Abdelzaher

Organizations: University of Illinois Urbana-Champaign · Semantic Code Lab

Abstract

Long-context LLMs can now ingest entire online discussion threads, but understanding their social discourse requires more than reading a long document: models must track parent-reply relations, turning points, scoped subtrees, cross-branch contrasts, and participant trajectories. To test this structure-aware social reasoning, we introduce LongSocialBench, a benchmark of 1,462 verified human-authored multiple-choice items drawn from 94 complete Hacker News, Stack Exchange, and Reddit r/ChangeMyView episodes, with a median length of approximately 73K tokens. Each item pairs a complete serialized discussion and reply structure with a four-option question, requiring models to recover structured social evidence. Released items are verified for answerability, option uniqueness, and evidence grounding. Across 18 models and 29 evaluation settings, current long-context workflows remain far below human performance. The best individual result comes from Claude-Opus-4.7, which reaches 63.0% when prompted to eliminate incorrect options before answering, compared with 72.4% for independent human readers. Averaged across all 18 models, the full-context Baseline scores 43.9%. Supplying the gold evidence scope raises this to 55.0%, showing that substantial errors remain even after the relevant thread region is identified. LongSocialBench shows that the missing capability is not context access or prompting alone, but social understanding over structured reply trees.

Figures & tables

Figure 1: LongSocialBench construction and verification pipeline (§ 3.1 ). We sample long public discussions, author items guided by capability definitions, apply anti-shortcut controls, independently verify answers and evidence, and adjudicate residual disputes.

Area	Long	Thread	Social	Verif.
Long-context eval./agg.	✓	✗	✗	✓
Long-memory dialogue	✓	✗	$\blacktriangle$	✓
Social commonsense/ToM	✗	✗	✓	✓
Argument / stance	$\blacktriangle$	$\blacktriangle$	✓	$\blacktriangle$
Discussion summarization	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	✗
Ours	✓	✓	✓	✓

Table 1: Area-level comparison of related evaluation settings across long context, reply-tree structure, social interpretation, and verifiable evaluation.

Capability surface	Evidence focus
C1: Local social meaning	A comment’s social role or meaning in its local context.
C2: Reply-pair interaction	The relation between a reply and the parent comment it addresses.
C3: Collective structure	How positions or roles are distributed across a discussion region.
C4: Temporal dynamics	How an exchange changes over time, including shifts and turning points.
C5: Participant profile	A participant’s stance or interaction pattern across multiple comments.
C6: Cross-subdiscussion variation	How social patterns differ or remain consistent across branches or sub-discussions.

Table 2: Overview of the six capability surfaces. Each surface captures a recurring evidence pattern in reply-structured discussions.

(Closed-source frontier)
	Baseline (Full-context)							CoT (with $\Delta$ vs Baseline)
Model	C1	C2	C3	C4	C5	C6	Avg	C1	C2	C3	C4	C5	C6	Avg
Claude-Opus-4.7	74.5	65.2	49.1	67.0	54.1	50.0	60.0	79.4 $+4.8$	71.9 $+6.7$	53.1 $+4.1$	70.8 $+3.8$	65.6 $+11.5$	59.3 $+9.3$	66.7 $+6.7$
Gemini-2.5-Pro	65.5	58.4	52.1	49.7	45.9	36.4	51.3	71.5 $+6.1$	69.7 $+11.2$	55.9 $+3.8$	56.8 $+7.0$	56.9 $+11.0$	56.8 $+20.3$	61.2 $+9.9$
GPT-5	61.8	68.5	43.7	58.9	63.8	53.4	58.3	67.9 $+6.1$	80.9 $+12.4$	44.3 $+0.6$	49.7 $-9.2$	63.3 $-0.5$	58.5 $+5.1$	60.8 $+2.4$
Claude-Sonnet-4.5	73.3	59.6	46.6	49.7	50.5	46.6	54.4	71.5 $-1.8$	67.4 $+7.9$	46.1 $-0.4$	53.0 $+3.2$	56.0 $+5.5$	48.3 $+1.7$	57.1 $+2.7$

Table 3: Per-model capability accuracy under Baseline and CoT. Cells report accuracy; Avg is the C1–C6 macro mean. CoT superscripts show lift over Baseline, with green/red for positive/negative change. Bold marks the best model per column; C1–C6 are defined in Table 7 .

Figure 2: Method effect by capability. Mean

\Delta

accuracy relative to Baseline across

18

models for C1–C6. Reasoning methods are generally beneficial, while flat retrieval often reduces accuracy, especially on C3. Structure-aware and LLM-based retrieval recover part of these losses.

Model tier	Full	Gold-Scope	$\Delta$
Closed frontier (6)	$51.4$	$62.3$	$+10.9$
Strong open (4)	$43.4$	$53.9$	$+10.5$
Mid open (4)	$40.3$	$51.2$	$+10.9$
Small (4)	$37.0$	$48.9$	$+11.9$
All 18	$43.9$	$55.0$	$\mathbf{+11.1}$

Table 4: Gold evidence scope improves accuracy across all model tiers, but substantial error remains. Mean accuracy across models in each tier on all

1{,}462

items.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Primary topic family	Total	HN	Reddit	SE
Computing / AI / internet	25	20	0	5
Economy / labor / business	21	10	10	1
Science / engineering / health	17	3	5	9
Public policy / law / governance	11	6	4	1
Culture / media / social life	7	3	4	0
Education / academia	6	0	4	2

Appendix

Table 5: Coarse topic coverage of released episodes. Each episode is assigned one primary topic family based on its title and source domain.

Source	# eps	Tokens med.	Tokens max	Tokens total	Mean max-depth	Mean nodes	Mean root replies
Hacker News	43	73,397	118,981	3.29M	12.9	815.7	109.9
Stack Exchange	21	75,982	85,184	1.59M	2.0	601.0	385.2
Reddit r/CMV	30	56,322	79,597	1.75M	8.9	485.5	258.8
Overall	94	$\approx$ 73K	118,981	$\approx$ 6.63M	9.2	662.4	218.9

Appendix

Table 6: Episode-level scale and structure by source. Token counts use cl100k_base over the complete serialized discussion. Maximum depth is measured from the root, and root replies are comments at depth

1

Figure 3: Structural diversity of LongSocialBench episodes. (A) Episode lengths span $37$ to $119$ K tokens with platform-specific medians of $73$ K for Hacker News, $76$ K for Stack Exchange, and $56$ K for Reddit r/CMV. (B) Mean maximum depth differs substantially across sources: Stack Exchange $2.0$ , Reddit r/CMV $8.9$ , and Hacker News $12.9$ . (C) Depth-frequency curves show different threading regimes: Stack Exchange is concentrated near depths $1$ to $2$ , Reddit has more mass at depths $1$ to $6$ , and Hacker News retains substantial mass at depths $4$ to $10$ .

Capability surface	Decision tested	Abridged example item
C1: Local social meaning	Decide what one comment is doing in context.	A target comment replies to a parent claim. Decide its local role : support, pushback, reframing, or clarification.
C2: Reply-pair interaction	Compare a reply with the comment it answers.	A parent uses one basis of comparison; the reply switches basis. Decide the parent reply relation : refute, qualify, redirect, or concede.
C3: Collective structure	Identify the positions organizing a scoped discussion region.	In a scoped subtree, choose the position structure that best organizes the replies: framing, consequences, responsibility, or evidence.
C4: Temporal dynamics	Locate where an ordered exchange changes direction.	In a long reply chain, identify the first turning point where one argumentative framing gives way to another.
C5: Participant profile	Track one participant across comments.	Across one author’s comments, infer the participant trajectory : stable stance, narrowed claim, concession, escalation, or strategy shift.
C6: Cross-subdiscussion variation	Compare behavior across separate branches.	Across two branches, decide the cross-branch change : whether a role, stance, or strategy stays stable or shifts.

Appendix

Table 7: Six capability surfaces in LongSocialBench. Each surface is a recurring social-reading decision over a complete discussion episode.

Criterion	Requirement	Failure action
Answerability	The gold option is recoverable from the supplied episode alone, without external facts or author intent.	Rewrite or remove.
Option uniqueness	Exactly one option is supported under the stated scope and author-written criterion.	Revise options or remove.
Scope clarity	The stem specifies the relevant unit, such as a comment, direct-reply set, subtree, chain, participant trajectory, or cross-branch comparison.	Clarify stem and re-verify.
Evidence sufficiency	Evidence nodes support both the gold option and key premises such as branch boundary, counting unit, participant trajectory, or comparison set.	Add evidence, rewrite, or remove.
Observable social criterion	Interpretive questions are grounded in observable thread behavior rather than preference, speculation, or unstated norms.	Rewrite criterion or remove.
Option balance	Distractors are plausible wrong readings with comparable specificity, not fillers or wording artifacts.	Rewrite options and re-verify.

Appendix

Table 8: Release criteria for LongSocialBench items. Each candidate item must satisfy all criteria before release; failed items are clarified and re-verified, or removed if the issue cannot be resolved.

Control	Shortcut reduced	Authoring restriction
Minimal locators	The stem reveals the relevant node, chain, or subtree.	Use only the locator needed to define the scope; avoid exact quotes, usernames, redundant structural hints, or answer-implying descriptors.
Distributed evidence	A single salient comment solves a thread-level item.	Require evidence from the intended scope whenever the question asks for aggregation, comparison, trajectory, or cross-branch reasoning.
Controlled options	Option wording makes the answer obvious.	Use parallel option forms and close numeric or relational alternatives when appropriate; avoid answer-specific descriptive phrasing.
No hint vocabulary	Keyword search or hint words reveal the answer.	State the criterion without listing answer-bearing instances or repeating distinctive phrases from the gold evidence.
Diagnostic distractors	Distractors are implausible or too easy.	Tie distractors to plausible wrong scopes, wrong criteria, lexical matches, boundary errors, or aggregation mistakes.
Open-web contamination	The item is recoverable from open-web memory rather than the supplied episode.	Check distinctive stem/option phrases and named events; rewrite or exclude items whose answers can be recovered from open-web summaries or third-party transcripts.

Appendix

Table 9: Question-design controls used during authoring. The controls support shortcut-resistant multiple-choice evaluation over complete discussion episodes.

Domain	Pass	Corrected	Residual
Hacker News	44 / 46	2 / 46	0 / 46
Reddit r/CMV	32 / 32	0 / 32	0 / 32
Stack Exchange	22 / 22	0 / 22	0 / 22
All	98 / 100	2 / 100	0 / 100

Appendix

Table 10: Stratified release audit. Among

100

audited items,

98

require no correction, two require wording-level revisions without changing the gold option, and none contains a residual gold-answer error.

Model	Source	CW
Closed-source frontier systems
Claude-Opus-4.7	Anthropic	200K
GPT-5	OpenAI	400K
Claude-Sonnet-4.5	Anthropic	200K
Claude-4	Anthropic	200K
Gemini-2.5-Pro	Google	1M

Appendix

Table 11: Models used in the evaluation grid. CW = evaluation context cap in tokens.

Setting group	Variants	Input construction and prompt
Deployable workflows
Full-context prompting	Full	The model receives the complete serialized discussion, reply-structure information, question stem, and four options; the prompt asks it to locate relevant evidence, compare options, and output one answer letter.
Reasoning prompts	CoT, CoT-Ev, CoT-Tp, CoT-El, Reflect	Same input as Full; the prompt adds one reasoning strategy before the final answer: free-form reasoning, evidence enumeration, temporal tracking, option elimination, or self-reflection.
Sparse retrieval	BM25- $k20$ , BM25- $k50$	Comments are ranked by BM25 using the question stem as query; the top- $k$ comments are passed in original thread order and answered with the shared answer format.
Dense retrieval	E5- $k20$ , E5- $k50$	Comments and question stems are embedded with E5; the top- $k$ comments by cosine similarity are passed in original thread order and answered with the shared answer format.
RAG with anchor notes	BM25/E5 $\times$ $k\in\{20,50\}$ $\times$ anchors $\in\{1,2\}$	Retrieved comments are supplemented with brief model-generated anchor notes that identify what each retrieved region contributes before final answering.

Appendix

Table 12: Evaluation settings on LongSocialBench. Deployable workflows are used for the headline accuracy comparison. Diagnostic settings test shortcut artifacts, multiple-choice format effects, or evidence localization and are reported separately.

(Closed-source frontier)
Model	Full	CoT	LLM-r	Best †	C-mac
Claude-Opus-4.7	56.0	61.8	55.3	63.0	60.0
Gemini-2.5-Pro	51.5	58.8	50.4	59.3	51.3
GPT-5	52.9	53.8	50.3	56.4	58.3
Claude-Sonnet-4.5	51.4	52.8	50.8	53.4	54.4
Claude-4	52.0	52.5	50.5	53.5	57.0

Appendix

Table 13: Per-model accuracy on three representative workflows. Full = full-context prompting; CoT = chain-of-thought; LLM-r = LLM-as-retriever. Best † is the per-model maximum across all evaluated non-oracle workflows, reported as a diagnostic only. C-mac = macro mean over the six capability surfaces.

Figure 4: Method-family accuracy band on LongSocialBench. Each grey dot is one model-workflow cell from the

18\times 29

evaluation grid; markers and error bars show the family mean and

\pm 1

standard deviation across cells. All deployable workflows cluster below the human answerability reference of

72.4\%

; the Gold-Scope diagnostic reaches

55.0\%

Family	Cells	Mean	95% CI	$\Delta$ Full
Chain-of-thought	90	44.7	[43.0, 46.5]	$+0.7$
Full-context	18	43.9	[41.3, 46.9]	—
RAG $k{=}50$	72	42.7	[42.1, 43.2]	$-1.3$
RAG $k{=}20$	72	42.3	[41.8, 42.8]	$-1.6$
LLM-as-retriever	18	42.1	[39.1, 45.4]	$-1.8$
Structure-aware retrieval	54	41.1	[39.6, 42.6]	$-2.9$

Appendix

Table 14: Accuracy band by setting group, with cell-level bootstrap 95% CIs. Values are full-benchmark micro accuracy on the

1{,}462

released items, averaged over the per-cell accuracies inside each group.

\Delta

Full is the lift over Full-context (Baseline). Cells

=

number of (model, setting) cells in the group; the eight rows above the divider are deployable workflows, and the two rows below are scope diagnostics reported for reference.

Reader	Accuracy	# correct / 150
R1	74.7	112
R2	73.3	110
R3	72.7	109
R4	71.3	107
R5	70.0	105
Mean	72.4	108.6

Appendix

Table 15: Human answerability reference. Five readers answer the same stratified

150

-item sample without LongSocialBench-specific training.

Cohort	Mean	$95\%$ CI
Top single cell	$66.1$	$[61.1,71.8]$
Claude-Opus- $4.7$	$65.4$	$[59.7,71.8]$
Closed-frontier mean ( $n{=}6$ )	$59.3$	—
Strong-open mean ( $n{=}4$ )	$51.1$	—
$18$ -model fleet mean	$51.6$	—
Human readers ( $n{=}5$ )	$\mathbf{72.4}$	$\mathbf{[71.1,73.7]}$

Appendix

Table 16: Stratified

150

-item matched-aggregation subset. Bootstrap (

B{=}1000

) over draws of

150

items in the natural capability-platform proportions. Each row reports best-cell mean and the

95\%

CI across draws; human-reader numbers are for reference.

Figure 5: Per-model accuracy on each social capability under Baseline. Cells are mean accuracy on the $1{,}462$ -item benchmark restricted to that capability; the rightmost strip is the per-model mean.

Figure 6: Per-model accuracy by platform under Baseline. Hacker News, Reddit r/CMV, Stack Exchange, and per-model mean are shown for each model.

Figure 7: Domain

\times

method effect. Each panel is one platform; bars are mean

\Delta

accuracy vs. Baseline averaged across

18

models.

Model	CoT	CoT-Evid	CoT-Temp	CoT-Elim	Reflect	Mean
GLM-5	+11.8	+2.3	+9.6	+5.3	+8.8	+7.6
Gemini-2.5-Pro	+7.3	+4.4	+7.8	+7.0	+5.5	+6.4
Claude-Opus-4.7	+5.8	+4.4	+6.4	+7.0	+6.2	+6.0
Claude-Sonnet-4.6	+7.3	+3.8	+6.9	+6.1	+5.3	+5.9
Llama-4-Mav-17B	+2.5	-0.3	+4.1	+1.0	+1.6	+1.8
GPT-5	+0.9	+0.3	+1.2	+2.6	+3.4	+1.7

Appendix

Table 17: Reasoning workflow lift over Baseline, per model. Each cell is accuracy under the named workflow minus accuracy under Baseline (pp). Negative entries ( red ) indicate the workflow hurts the model. Models are sorted by mean lift descending. Reflect = self-reflection; CoT-Evid / CoT-Temp / CoT-Elim are chain-of-thought variants prompting evidence enumeration, temporal reasoning, or elimination of distractors, respectively.

Figure 8: Reasoning workflow lift over Baseline, per model. Cells are accuracy under the named workflow minus Baseline in percentage points; the rightmost strip is the per-model mean lift.

Retriever	Budget	Mean evidence recall
BM $25$	$k{=}20$	0.201
E $5$	$k{=}20$	0.203
BM $25$	$k{=}50$	0.308
E $5$	$k{=}50$	0.310

Appendix

Table 18: Evidence-node recall for standard comment-level retrievers. Values are mean recall against author-side evidence-node lists, averaged over the

1{,}462

released items. Only 8.8% of items reach

\geq 75\%

recall in the best of the four retrievers, and 3.5% attain full coverage at the largest budget.

Probe	Shortcut tested	Observed	Interpretation
Option-only	option prior	31.5%	option text alone is insufficient
Question-only	stem leakage	38.0%	question wording alone is insufficient
Title/root-only	topic familiarity	35.4%	topic-level information is insufficient without replies
Shuffled thread	bag-of-comments inference	40.4%	performance remains limited when local thread order is perturbed
Open-answer subset	multiple-choice artifact	$-7.6$ pp; $\rho{=}1.00$	absolute accuracy drops, but the model ranking is preserved

Appendix

Table 19: Shortcut robustness summary. Each probe removes or perturbs information that would be unnecessary if the corresponding shortcut were sufficient. Accuracy values are means across

18

models on the probe subsets; the full-benchmark Baseline is

43.9\%

for context. For open-answer, the table reports the same-item accuracy change and model-ranking correlation.

Tier	HN	Reddit	SE	$\Delta_{\text{HN}-\text{SE}}$
Closed frontier (6)	53.0	49.1	51.1	$+1.9$
Strong open (4)	42.4	44.1	44.5	$-2.1$
Mid open (4)	41.3	39.3	39.8	$+1.5$
Small (4)	38.8	35.4	35.3	$+3.5$
All 18	44.9	42.8	43.6	$\mathbf{+1.3}$

Appendix

Table 20: Pre- and post-cutoff contamination probe. Mean Baseline accuracy by tier on the two time-disjoint cohorts; Reddit is reported for context.

\Delta_{\text{HN}-\text{SE}}

is the post-cutoff minus pre-cutoff gap.

Figure 9: Episode length vs. Baseline accuracy. Each point is one of the

94

released episodes: mean Baseline accuracy across the

18

models against episode length in cl100k_base tokens, with the least-squares fit and the pooled Spearman

\rho

Condition	Accuracy
CoT baseline	$63.1$
Gold scope	$66.7$
Gold per-comment labels	$81.4$
Gold scope + gold labels	$98.6$
Structured enumerate-label-aggregate pipeline	$70.0$

Appendix

Table 21: Component oracle case study. Claude-Opus-

4.7

on the stratified

30

-episode /

510

-item subset.

Error type	Share (%)
Criterion use	$\mathbf{60}$
Thread structure	$16$
Counting rule	$14$
Surface cue	$10$

Appendix

Table 22: Manual audit of

50

Claude-Opus-

4.7

Gold-Scope errors. Each error is assigned the immediate source of the failure; percentages describe the audited sample.

Stage	Contributor task	Quality control
Authoring	Read the complete discussion, choose a clear scope, and write the question stem, four plausible options, gold option, plain-language criterion, and evidence nodes.	The item must require reasoning over the specified scope rather than topic familiarity or one salient comment. Distractors reflect plausible wrong readings, such as wrong scope, nearby count, boundary error, or near-synonym social label.
Artifact control	Remove answer-bearing wording from stems and options, including exact quotes, usernames, redundant structural hints, and distinctive phrases.	These controls reduce locator leakage, option-paraphrase matching, keyword shortcuts, and open-web contamination. Capability and item-form tags are retained only as metadata.
Independent verification	A different contributor re-answers the item from the same model-facing input used in evaluation: serialized thread, reply structure, question stem, and four answer options.	The verifier does not see the author’s gold option, criterion notes, evidence-node list, or rationale. Agreement checks answer stability and scope stability.
Adjudication	Phase-2 mismatches and author-flagged borderline cases receive a third-pass review.	The item is confirmed, clarified, rewritten and re-verified, or removed. A released item must have independent agreement or a stable adjudicated answer.

Appendix

Table 23: LongSocialBench annotation and verification protocol. Authoring records define the social criterion and supporting evidence; verification records test whether an independent reader can recover the gold option under the same input condition given to models.

Explore similar work

SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM Evaluation AI Agent Benchmarks
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3
Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
Nov 4, 2025Amanda Bertsch, Adithya Pratapa, Teruko Mitamura +2LLM Evaluation Long-Context Language Model Evaluation
Carnegie Mellon University, Language Technologies Institute
You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments
Sep 3, 2026Shiwei Hong, Junjie Ma, Emma Jiren Wang +5Pragmatic Reasoning in Language Models LLM Evaluation
George Mason University · Virginia Tech · University of Toronto +2

May 29, 2025J/K move · Enter open · S save

SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments

Zixiang Xu, Yanbo Wang, Yue Huang, Haomin Zhuang+12

Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame+3

LLM Evaluation AI Agent Benchmarks Social Reasoning LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed in socially grounded applications, where success requires interpreting context, inferring others' mental states, and reasoning about unreliable information. Yet existing benchmarks rarely evaluate these demands jointly in complex, evolving settings. We introduce SocialMaze, a benchmark that organizes six tasks across social deduction games, daily-life interactions, and digital community platforms along three descriptive design axes: deep reasoning, dynamic interaction, and information uncertainty. These axes characterize intended sources of task difficulty rather than latent, factor-analytic dimensions of model capability. Automated checks and human validation support data quality. Evaluations of twelve proprietary and open-weight LLMs show substantial variation in the use of evolving interaction histories; stronger chain-of-thought reasoners perform better on tasks requiring deeper inference, while uncertainty consistently degrades performance. Reasoning workflows help weaker short-chain-of-thought backbones but saturate on stronger reasoners. Finally, targeted fine-tuning on curated reasoning traces substantially improves structured social-reasoning tasks, whereas transfer to language-aggregation tasks remains statistically inconclusive. The project homepage is available at https://xzx34.github.io/socialmaze/.

Abstract

Explore similar work

SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments

Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments