Long-context LLMs can now ingest entire online discussion threads, but understanding their social discourse requires more than reading a long document: models must track parent-reply relations, turning points, scoped subtrees, cross-branch contrasts, and participant trajectories. To test this structure-aware social reasoning, we introduce LongSocialBench, a benchmark of 1,462 verified human-authored multiple-choice items drawn from 94 complete Hacker News, Stack Exchange, and Reddit r/ChangeMyView episodes, with a median length of approximately 73K tokens. Each item pairs a complete serialized discussion and reply structure with a four-option question, requiring models to recover structured social evidence. Released items are verified for answerability, option uniqueness, and evidence grounding. Across 18 models and 29 evaluation settings, current long-context workflows remain far below human performance. The best individual result comes from Claude-Opus-4.7, which reaches 63.0% when prompted to eliminate incorrect options before answering, compared with 72.4% for independent human readers. Averaged across all 18 models, the full-context Baseline scores 43.9%. Supplying the gold evidence scope raises this to 55.0%, showing that substantial errors remain even after the relevant thread region is identified. LongSocialBench shows that the missing capability is not context access or prompting alone, but social understanding over structured reply trees.
Figures & tables
Figure 1: LongSocialBench construction and verification pipeline (§ 3.1 ). We sample long public discussions, author items guided by capability definitions, apply anti-shortcut controls, independently verify answers and evidence, and adjudicate residual disputes.
Area
Long
Thread
Social
Verif.
Long-context eval./agg.
✓
✗
✗
✓
Long-memory dialogue
✓
✗
▲
✓
Social commonsense/ToM
✗
✗
✓
✓
Argument / stance
▲
▲
✓
▲
Discussion summarization
▲
▲
▲
✗
Ours
✓
✓
✓
✓
Table 1: Area-level comparison of related evaluation settings across long context, reply-tree structure, social interpretation, and verifiable evaluation.
Capability surface
Evidence focus
C1: Local social meaning
A comment’s social role or meaning in its local context.
C2: Reply-pair interaction
The relation between a reply and the parent comment it addresses.
C3: Collective structure
How positions or roles are distributed across a discussion region.
C4: Temporal dynamics
How an exchange changes over time, including shifts and turning points.
C5: Participant profile
A participant’s stance or interaction pattern across multiple comments.
C6: Cross-subdiscussion variation
How social patterns differ or remain consistent across branches or sub-discussions.
Table 2: Overview of the six capability surfaces. Each surface captures a recurring evidence pattern in reply-structured discussions.
Baseline (Full-context)
CoT (with Δ vs Baseline)
Model
C1
C2
C3
C4
C5
C6
Avg
C1
C2
C3
C4
C5
C6
Avg
(Closed-source frontier)
Claude-Opus-4.7
74.5
65.2
49.1
67.0
54.1
50.0
60.0
79.4 +4.8
71.9 +6.7
53.1 +4.1
70.8 +3.8
65.6 +11.5
59.3 +9.3
66.7 +6.7
Gemini-2.5-Pro
65.5
58.4
52.1
49.7
45.9
36.4
51.3
71.5 +6.1
69.7 +11.2
55.9 +3.8
56.8 +7.0
56.9 +11.0
56.8 +20.3
61.2 +9.9
GPT-5
61.8
68.5
43.7
58.9
63.8
53.4
58.3
67.9 +6.1
80.9 +12.4
44.3 +0.6
49.7 −9.2
63.3 −0.5
58.5 +5.1
60.8 +2.4
Claude-Sonnet-4.5
73.3
59.6
46.6
49.7
50.5
46.6
54.4
71.5 −1.8
67.4 +7.9
46.1 −0.4
53.0 +3.2
56.0 +5.5
48.3 +1.7
57.1 +2.7
Table 3: Per-model capability accuracy under Baseline and CoT. Cells report accuracy; Avg is the C1–C6 macro mean. CoT superscripts show lift over Baseline, with green/red for positive/negative change. Bold marks the best model per column; C1–C6 are defined in Table 7 .
Figure 2: Method effect by capability. Mean Δ accuracy relative to Baseline across 18 models for C1–C6. Reasoning methods are generally beneficial, while flat retrieval often reduces accuracy, especially on C3. Structure-aware and LLM-based retrieval recover part of these losses.
Model tier
Full
Gold-Scope
Δ
Closed frontier (6)
51.4
62.3
+10.9
Strong open (4)
43.4
53.9
+10.5
Mid open (4)
40.3
51.2
+10.9
Small (4)
37.0
48.9
+11.9
All 18
43.9
55.0
+11.1
Table 4: Gold evidence scope improves accuracy across all model tiers, but substantial error remains. Mean accuracy across models in each tier on all 1,462 items.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Primary topic family
Total
HN
Reddit
SE
Computing / AI / internet
25
20
0
5
Economy / labor / business
21
10
10
1
Science / engineering / health
17
3
5
9
Public policy / law / governance
11
6
4
1
Culture / media / social life
7
3
4
0
Education / academia
6
0
4
2
Appendix
Table 5: Coarse topic coverage of released episodes. Each episode is assigned one primary topic family based on its title and source domain.
Source
# eps
Tokens med.
Tokens max
Tokens total
Mean max-depth
Mean nodes
Mean root replies
Hacker News
43
73,397
118,981
3.29M
12.9
815.7
109.9
Stack Exchange
21
75,982
85,184
1.59M
2.0
601.0
385.2
Reddit r/CMV
30
56,322
79,597
1.75M
8.9
485.5
258.8
Overall
94
≈ 73K
118,981
≈ 6.63M
9.2
662.4
218.9
Appendix
Table 6: Episode-level scale and structure by source. Token counts use cl100k_base over the complete serialized discussion. Maximum depth is measured from the root, and root replies are comments at depth 1 .
Figure 3: Structural diversity of LongSocialBench episodes. (A) Episode lengths span 37 to 119 K tokens with platform-specific medians of 73 K for Hacker News, 76 K for Stack Exchange, and 56 K for Reddit r/CMV. (B) Mean maximum depth differs substantially across sources: Stack Exchange 2.0 , Reddit r/CMV 8.9 , and Hacker News 12.9 . (C) Depth-frequency curves show different threading regimes: Stack Exchange is concentrated near depths 1 to 2 , Reddit has more mass at depths 1 to 6 , and Hacker News retains substantial mass at depths 4 to 10 .
Capability surface
Decision tested
Abridged example item
C1: Local social meaning
Decide what one comment is doing in context.
A target comment replies to a parent claim. Decide its local role : support, pushback, reframing, or clarification.
C2: Reply-pair interaction
Compare a reply with the comment it answers.
A parent uses one basis of comparison; the reply switches basis. Decide the parent reply relation : refute, qualify, redirect, or concede.
C3: Collective structure
Identify the positions organizing a scoped discussion region.
In a scoped subtree, choose the position structure that best organizes the replies: framing, consequences, responsibility, or evidence.
C4: Temporal dynamics
Locate where an ordered exchange changes direction.
In a long reply chain, identify the first turning point where one argumentative framing gives way to another.
C5: Participant profile
Track one participant across comments.
Across one author’s comments, infer the participant trajectory : stable stance, narrowed claim, concession, escalation, or strategy shift.
C6: Cross-subdiscussion variation
Compare behavior across separate branches.
Across two branches, decide the cross-branch change : whether a role, stance, or strategy stays stable or shifts.
Appendix
Table 7: Six capability surfaces in LongSocialBench. Each surface is a recurring social-reading decision over a complete discussion episode.
Criterion
Requirement
Failure action
Answerability
The gold option is recoverable from the supplied episode alone, without external facts or author intent.
Rewrite or remove.
Option uniqueness
Exactly one option is supported under the stated scope and author-written criterion.
Revise options or remove.
Scope clarity
The stem specifies the relevant unit, such as a comment, direct-reply set, subtree, chain, participant trajectory, or cross-branch comparison.
Clarify stem and re-verify.
Evidence sufficiency
Evidence nodes support both the gold option and key premises such as branch boundary, counting unit, participant trajectory, or comparison set.
Add evidence, rewrite, or remove.
Observable social criterion
Interpretive questions are grounded in observable thread behavior rather than preference, speculation, or unstated norms.
Rewrite criterion or remove.
Option balance
Distractors are plausible wrong readings with comparable specificity, not fillers or wording artifacts.
Rewrite options and re-verify.
Appendix
Table 8: Release criteria for LongSocialBench items. Each candidate item must satisfy all criteria before release; failed items are clarified and re-verified, or removed if the issue cannot be resolved.
Control
Shortcut reduced
Authoring restriction
Minimal locators
The stem reveals the relevant node, chain, or subtree.
Use only the locator needed to define the scope; avoid exact quotes, usernames, redundant structural hints, or answer-implying descriptors.
Distributed evidence
A single salient comment solves a thread-level item.
Require evidence from the intended scope whenever the question asks for aggregation, comparison, trajectory, or cross-branch reasoning.
Controlled options
Option wording makes the answer obvious.
Use parallel option forms and close numeric or relational alternatives when appropriate; avoid answer-specific descriptive phrasing.
No hint vocabulary
Keyword search or hint words reveal the answer.
State the criterion without listing answer-bearing instances or repeating distinctive phrases from the gold evidence.
Diagnostic distractors
Distractors are implausible or too easy.
Tie distractors to plausible wrong scopes, wrong criteria, lexical matches, boundary errors, or aggregation mistakes.
Open-web contamination
The item is recoverable from open-web memory rather than the supplied episode.
Check distinctive stem/option phrases and named events; rewrite or exclude items whose answers can be recovered from open-web summaries or third-party transcripts.
Appendix
Table 9: Question-design controls used during authoring. The controls support shortcut-resistant multiple-choice evaluation over complete discussion episodes.
Domain
Pass
Corrected
Residual
Hacker News
44 / 46
2 / 46
0 / 46
Reddit r/CMV
32 / 32
0 / 32
0 / 32
Stack Exchange
22 / 22
0 / 22
0 / 22
All
98 / 100
2 / 100
0 / 100
Appendix
Table 10: Stratified release audit. Among 100 audited items, 98 require no correction, two require wording-level revisions without changing the gold option, and none contains a residual gold-answer error.
Model
Source
CW
Closed-source frontier systems
Claude-Opus-4.7
Anthropic
200K
GPT-5
OpenAI
400K
Claude-Sonnet-4.5
Anthropic
200K
Claude-4
Anthropic
200K
Gemini-2.5-Pro
Google
1M
Appendix
Table 11: Models used in the evaluation grid. CW = evaluation context cap in tokens.
Setting group
Variants
Input construction and prompt
Deployable workflows
Full-context prompting
Full
The model receives the complete serialized discussion, reply-structure information, question stem, and four options; the prompt asks it to locate relevant evidence, compare options, and output one answer letter.
Reasoning prompts
CoT, CoT-Ev, CoT-Tp, CoT-El, Reflect
Same input as Full; the prompt adds one reasoning strategy before the final answer: free-form reasoning, evidence enumeration, temporal tracking, option elimination, or self-reflection.
Sparse retrieval
BM25- k20 , BM25- k50
Comments are ranked by BM25 using the question stem as query; the top- k comments are passed in original thread order and answered with the shared answer format.
Dense retrieval
E5- k20 , E5- k50
Comments and question stems are embedded with E5; the top- k comments by cosine similarity are passed in original thread order and answered with the shared answer format.
RAG with anchor notes
BM25/E5 ×k∈{20,50}× anchors ∈{1,2}
Retrieved comments are supplemented with brief model-generated anchor notes that identify what each retrieved region contributes before final answering.
Appendix
Table 12: Evaluation settings on LongSocialBench. Deployable workflows are used for the headline accuracy comparison. Diagnostic settings test shortcut artifacts, multiple-choice format effects, or evidence localization and are reported separately.
Model
Full
CoT
LLM-r
Best †
C-mac
(Closed-source frontier)
Claude-Opus-4.7
56.0
61.8
55.3
63.0
60.0
Gemini-2.5-Pro
51.5
58.8
50.4
59.3
51.3
GPT-5
52.9
53.8
50.3
56.4
58.3
Claude-Sonnet-4.5
51.4
52.8
50.8
53.4
54.4
Claude-4
52.0
52.5
50.5
53.5
57.0
Appendix
Table 13: Per-model accuracy on three representative workflows. Full = full-context prompting; CoT = chain-of-thought; LLM-r = LLM-as-retriever. Best † is the per-model maximum across all evaluated non-oracle workflows, reported as a diagnostic only. C-mac = macro mean over the six capability surfaces.
Figure 4: Method-family accuracy band on LongSocialBench. Each grey dot is one model-workflow cell from the 18×29 evaluation grid; markers and error bars show the family mean and ±1 standard deviation across cells. All deployable workflows cluster below the human answerability reference of 72.4% ; the Gold-Scope diagnostic reaches 55.0% .
Family
Cells
Mean
95% CI
Δ Full
Chain-of-thought
90
44.7
[43.0, 46.5]
+0.7
Full-context
18
43.9
[41.3, 46.9]
—
RAG k=50
72
42.7
[42.1, 43.2]
−1.3
RAG k=20
72
42.3
[41.8, 42.8]
−1.6
LLM-as-retriever
18
42.1
[39.1, 45.4]
−1.8
Structure-aware retrieval
54
41.1
[39.6, 42.6]
−2.9
Appendix
Table 14: Accuracy band by setting group, with cell-level bootstrap 95% CIs. Values are full-benchmark micro accuracy on the 1,462 released items, averaged over the per-cell accuracies inside each group. Δ Full is the lift over Full-context (Baseline). Cells = number of (model, setting) cells in the group; the eight rows above the divider are deployable workflows, and the two rows below are scope diagnostics reported for reference.
Reader
Accuracy
# correct / 150
R1
74.7
112
R2
73.3
110
R3
72.7
109
R4
71.3
107
R5
70.0
105
Mean
72.4
108.6
Appendix
Table 15: Human answerability reference. Five readers answer the same stratified 150 -item sample without LongSocialBench-specific training.
Cohort
Mean
95% CI
Top single cell
66.1
[61.1,71.8]
Claude-Opus- 4.7
65.4
[59.7,71.8]
Closed-frontier mean ( n=6 )
59.3
—
Strong-open mean ( n=4 )
51.1
—
18 -model fleet mean
51.6
—
Human readers ( n=5 )
72.4
[71.1,73.7]
Appendix
Table 16: Stratified 150 -item matched-aggregation subset. Bootstrap ( B=1000 ) over draws of 150 items in the natural capability-platform proportions. Each row reports best-cell mean and the 95% CI across draws; human-reader numbers are for reference.
Figure 5: Per-model accuracy on each social capability under Baseline. Cells are mean accuracy on the 1,462 -item benchmark restricted to that capability; the rightmost strip is the per-model mean.
Figure 6: Per-model accuracy by platform under Baseline. Hacker News, Reddit r/CMV, Stack Exchange, and per-model mean are shown for each model.
Figure 7: Domain × method effect. Each panel is one platform; bars are mean Δ accuracy vs. Baseline averaged across 18 models.
Model
CoT
CoT-Evid
CoT-Temp
CoT-Elim
Reflect
Mean
GLM-5
+11.8
+2.3
+9.6
+5.3
+8.8
+7.6
Gemini-2.5-Pro
+7.3
+4.4
+7.8
+7.0
+5.5
+6.4
Claude-Opus-4.7
+5.8
+4.4
+6.4
+7.0
+6.2
+6.0
Claude-Sonnet-4.6
+7.3
+3.8
+6.9
+6.1
+5.3
+5.9
Llama-4-Mav-17B
+2.5
-0.3
+4.1
+1.0
+1.6
+1.8
GPT-5
+0.9
+0.3
+1.2
+2.6
+3.4
+1.7
Appendix
Table 17: Reasoning workflow lift over Baseline, per model. Each cell is accuracy under the named workflow minus accuracy under Baseline (pp). Negative entries ( red ) indicate the workflow hurts the model. Models are sorted by mean lift descending. Reflect = self-reflection; CoT-Evid / CoT-Temp / CoT-Elim are chain-of-thought variants prompting evidence enumeration, temporal reasoning, or elimination of distractors, respectively.
Figure 8: Reasoning workflow lift over Baseline, per model. Cells are accuracy under the named workflow minus Baseline in percentage points; the rightmost strip is the per-model mean lift.
Retriever
Budget
Mean evidence recall
BM 25
k=20
0.201
E 5
k=20
0.203
BM 25
k=50
0.308
E 5
k=50
0.310
Appendix
Table 18: Evidence-node recall for standard comment-level retrievers. Values are mean recall against author-side evidence-node lists, averaged over the 1,462 released items. Only 8.8% of items reach ≥75% recall in the best of the four retrievers, and 3.5% attain full coverage at the largest budget.
Probe
Shortcut tested
Observed
Interpretation
Option-only
option prior
31.5%
option text alone is insufficient
Question-only
stem leakage
38.0%
question wording alone is insufficient
Title/root-only
topic familiarity
35.4%
topic-level information is insufficient without replies
Shuffled thread
bag-of-comments inference
40.4%
performance remains limited when local thread order is perturbed
Open-answer subset
multiple-choice artifact
−7.6 pp; ρ=1.00
absolute accuracy drops, but the model ranking is preserved
Appendix
Table 19: Shortcut robustness summary. Each probe removes or perturbs information that would be unnecessary if the corresponding shortcut were sufficient. Accuracy values are means across 18 models on the probe subsets; the full-benchmark Baseline is 43.9% for context. For open-answer, the table reports the same-item accuracy change and model-ranking correlation.
Tier
HN
Reddit
SE
ΔHN−SE
Closed frontier (6)
53.0
49.1
51.1
+1.9
Strong open (4)
42.4
44.1
44.5
−2.1
Mid open (4)
41.3
39.3
39.8
+1.5
Small (4)
38.8
35.4
35.3
+3.5
All 18
44.9
42.8
43.6
+1.3
Appendix
Table 20: Pre- and post-cutoff contamination probe. Mean Baseline accuracy by tier on the two time-disjoint cohorts; Reddit is reported for context. ΔHN−SE is the post-cutoff minus pre-cutoff gap.
Figure 9: Episode length vs. Baseline accuracy. Each point is one of the 94 released episodes: mean Baseline accuracy across the 18 models against episode length in cl100k_base tokens, with the least-squares fit and the pooled Spearman ρ .
Condition
Accuracy
CoT baseline
63.1
Gold scope
66.7
Gold per-comment labels
81.4
Gold scope + gold labels
98.6
Structured enumerate-label-aggregate pipeline
70.0
Appendix
Table 21: Component oracle case study. Claude-Opus- 4.7 on the stratified 30 -episode / 510 -item subset.
Error type
Share (%)
Criterion use
60
Thread structure
16
Counting rule
14
Surface cue
10
Appendix
Table 22: Manual audit of 50 Claude-Opus- 4.7 Gold-Scope errors. Each error is assigned the immediate source of the failure; percentages describe the audited sample.
Stage
Contributor task
Quality control
Authoring
Read the complete discussion, choose a clear scope, and write the question stem, four plausible options, gold option, plain-language criterion, and evidence nodes.
The item must require reasoning over the specified scope rather than topic familiarity or one salient comment. Distractors reflect plausible wrong readings, such as wrong scope, nearby count, boundary error, or near-synonym social label.
Artifact control
Remove answer-bearing wording from stems and options, including exact quotes, usernames, redundant structural hints, and distinctive phrases.
These controls reduce locator leakage, option-paraphrase matching, keyword shortcuts, and open-web contamination. Capability and item-form tags are retained only as metadata.
Independent verification
A different contributor re-answers the item from the same model-facing input used in evaluation: serialized thread, reply structure, question stem, and four answer options.
The verifier does not see the author’s gold option, criterion notes, evidence-node list, or rationale. Agreement checks answer stability and scope stability.
Adjudication
Phase-2 mismatches and author-flagged borderline cases receive a third-pass review.
The item is confirmed, clarified, rewritten and re-verified, or removed. A released item must have independent agreement or a stable adjudicated answer.
Appendix
Table 23: LongSocialBench annotation and verification protocol. Authoring records define the social criterion and supporting evidence; verification records test whether an independent reader can recover the gold option under the same input condition given to models.
Large language models (LLMs) are increasingly deployed in socially grounded applications, where success requires interpreting context, inferring others' mental states, and reasoning about unreliable information. Yet existing benchmarks rarely evaluate these demands jointly in complex, evolving settings. We introduce SocialMaze, a benchmark that organizes six tasks across social deduction games, daily-life interactions, and digital community platforms along three descriptive design axes: deep reasoning, dynamic interaction, and information uncertainty. These axes characterize intended sources of task difficulty rather than latent, factor-analytic dimensions of model capability. Automated checks and human validation support data quality. Evaluations of twelve proprietary and open-weight LLMs show substantial variation in the use of evolving interaction histories; stronger chain-of-thought reasoners perform better on tasks requiring deeper inference, while uncertainty consistently degrades performance. Reasoning workflows help weaker short-chain-of-thought backbones but saturate on stronger reasoners. Finally, targeted fine-tuning on curated reasoning traces substantially improves structured social-reasoning tasks, whereas transfer to language-aggregation tasks remains statistically inconclusive. The project homepage is available at https://xzx34.github.io/socialmaze/.
Zixiang Xu, Yanbo Wang, Yue Huang +13
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3
As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evaluations have recently been released, these evaluations tend to rely on retrieval from one or more sections of the context, which allows nearly all of the context tokens to be disregarded as noise. This represents only one type of task that might be performed with long context. We introduce Oolong, a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions. Oolong is separated into two task sets: Oolong-synth, a set of naturalistic synthetic tasks, where we can easily ablate components of the reasoning problem; and Oolong-real, a downstream setting which requires reasoning over real-world conversational data. Oolong requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations. Even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text.
Chinese online comments often convey social meaning through indirect and playful language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models can distinguish plausible readings of what a naturally occurring comment is doing in a particular exchange. We introduce a benchmark for evaluating whether LLMs can recover such situated pragmatic meanings. From more than 200,000 public Chinese social media interaction records, we construct 4,735 human-validated diagnostic items, each pairing a target comment with reconstructed preceding context and plausible misreadings. We evaluate eight LLMs as both question writers and solvers in a cross-writer setting. The task is challenging: the strongest model achieves 81.42% leave-writer-out accuracy. Across all eight models, the mean leave-writer-out accuracy is 68.70% while human accuracy was 90.8%. Case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.
Shiwei Hong, Junjie Ma, Emma Jiren Wang +5
George Mason University · Virginia Tech · University of Toronto +2