We introduce GISTBench, a benchmark for evaluating Large Language Models' (LLMs) ability to understand users from their interaction histories in recommendation systems. Unlike traditional RecSys benchmarks that focus on item prediction accuracy, our benchmark evaluates how well LLMs can extract and verify user interests from engagement data. We propose two novel metric families: Interest Groundedness (IG), decomposed into precision and recall components to separately penalize hallucinated interest categories and reward coverage, and Interest Specificity (IS), which assesses the distinctiveness of verified LLM-predicted user profiles. We release a synthetic dataset constructed on real user interactions on a global short-form video platform. Our dataset contains both implicit and explicit engagement signals and rich textual descriptions. We validate our dataset fidelity against user surveys, and evaluate eight open-weight LLMs spanning 7B to 235B parameters, together with three proprietary frontier models (GPT-5, Claude 4.6, and Gemini 3.5 Flash). Our findings reveal performance bottlenecks in current LLMs, particularly their limited ability to accurately count and attribute engagement signals across heterogeneous interaction types.
Figures & tables
Figure 1: Overview of the GISTBench evaluation pipeline. User interaction histories are processed by an LLM to produce predicted interests. These are evaluated on two axes: Interest Groundedness (IG, decomposed into precision and recall) and Interest Specificity (IS, computed over verified categories only), then mapped to standardized taxonomy categories for normalization.
Dataset
Domain
Rich Text
Imp+
Imp–
Exp+
Exp–
Primary Signal
Density
KuaiRec Gao et al. (2022)
Short Video
Tags only
✓
✓
✓
✗
Watch ratio
100% ≥ 100 eng.
MIND Wu et al. (2020)
News
✓
✗
✓
✓
✗
Click/non-click
39% ≥ 100 eng.
Amazon Music Hou et al. (2024)
Music
✓
✓
✗
✓
✓
1–5 star ratings
0.12% ≥ 20 eng.
Goodreads Wan and McAuley (2018)
Books
✓
✓
✗
✓
✓
1–5 star ratings
Moderate
Ours (Synthetic)
Short Video
✓
✓
✓
✓
✗
Mixed behavioral
100% ≥ 100 eng.
Table 1: Comparison of datasets used in our benchmark. Rich Text : detailed natural language descriptions. Imp+/– : implicit positive/negative signals. Exp+/– : explicit positive/negative signals. Density : fraction of users with sufficient engagement history. Each dataset presents a distinct evaluation challenge based on its signal mix and sparsity.
Stand Up & Roasting, Sketches & Skits, Pranks, Memes, Reaction Comedy, …
Visual Arts & Design
Drawing & Painting, Photography, Graphic Design, Sculpture, Street & Public Art, …
Table 2: Examples of top-level interest domains and their corresponding interest categories. Each domain contains several fine-grained interest categories; only a representative subset is shown. The full list of categories is provided in Appendix A .
Synthetic
KuaiRec
Amazon Music
Goodreads
Model
IG F1
IS
IG F1
IS
IG F1
IS
IG F1
IS
GPT-OSS-120B
67.8
74.0
56.3
68.4
66.7
70.6
58.0
67.4
GPT-5
59.7
81.1
62.3
74.4
66.7
90.9
70.0
83.7
DeepSeek-R1
56.5
79.0
44.1
71.0
57.1
74.1
43.5
68.9
Qwen3-VL-235B
50.8
73.4
55.2
65.9
50.0
70.8
48.5
65.5
Claude 4.6
50.1
78.0
66.9
69.7
75.0
86.8
61.3
83.4
Table 3: Median IG F1 (%) and IS (%) across the four primary benchmark datasets. IG F1 is the harmonic mean of IG Precision and IG Recall; IS is computed over verified interest categories only (bounded [0,1] ). Best per-dataset results in bold .
Synthetic
KuaiRec
Model
IG P
IG R
IG P
IG R
DeepSeek-R1
82.4
43.9
89.8
29.5
GPT-OSS-120B
78.1
60.5
79.7
44.0
Qwen3-32B
73.6
34.3
82.4
31.8
GPT-5
67.8
53.5
73.5
54.2
Qwen2.5-72B
65.7
34.8
79.2
29.7
Table 4: IG Precision and IG Recall (%) on Synthetic and KuaiRec. IG P> IG R for every model. The primary bottleneck is coverage; hallucination is not. Best IG P and IG R per dataset in bold .
Survey-Based
Proposed Metrics
Model
S-Prec
S-Rec
IG P
IG R
IG F1
IS
DeepSeek-R1
62.3
60.0
57.3
33.7
42.9
83.3
GPT-OSS-120B
59.0
60.2
31.6
26.5
27.7
87.8
Qwen3-32B
59.5
52.8
50.0
26.1
33.9
80.6
Qwen3-VL-235B
57.1
66.1
25.0
26.9
25.7
84.8
DeepSeek-R1- Distill-70B
51.4
56.1
25.2
21.4
24.3
83.3
Table 5: Survey-Based metrics (S-Prec, S-Rec) versus proposed metrics (IG P , IG R , IG F1 , IS), all in %. S-Prec and S-Rec are computed against user self-reported interests; IG and IS are computed from engagement signals (Section 4 ). The full precision-recall decomposition reveals that IG P> IG R for seven of the eight models on the survey dataset (Qwen3-VL-235B is the recall-leaning exception), consistent with the benchmark results (Table 4 ): coverage, not hallucination, is the bottleneck. ρ : Spearman correlation between survey F1 and geometric mean of IG F1 and IS.
Model
Arena IF Elo
Median IG F1 (%)
DeepSeek-R1
1395
42.9
Qwen2.5-72B
1238
37.7
Qwen3-32B
1330
33.9
GPT-OSS-120B
1325
27.7
Qwen3-VL-235B
1325
25.7
DeepSeek-R1- Distill-70B
1395
24.3
Table 6: Instruction-following capability (Chatbot Arena IF Elo Chiang et al. (2024) ) and median IG F1 (%) on the survey dataset (real users). Among these models, Elo does not predict IG F1 rank.
Model
Ins. Imp+
Ins. Exp+
Exc. Neg
DeepSeek-R1
86.7
99.4
1.8
GPT-OSS-120B
97.3
99.2
1.3
Qwen3-32B
95.7
98.6
3.1
Qwen3-VL-235B
93.1
98.3
10.8
DeepSeek-R1- Distill-70B
90.5
94.8
9.6
Qwen2.5-72B
76.3
92.0
22.9
Table 7: Distribution of IG failure modes (%) among unverified interests, aggregated across all datasets. Categories are not mutually exclusive. Ins. Imp+ : insufficient implicit positive signals ( < 3); Ins. Exp+ : insufficient explicit positive signals ( < 2); Exc. Neg : excessive negative signals. Insufficient positive evidence dominates across all models. Excessive negative contamination is bimodal.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Comparison of synthetic and real UIH distributions. (a) Per-user engagement count densities show similar right-skewed patterns. (b) Box plots confirm that the relative ordering across action types is preserved.
Action Type
t-stat
p-value
Syn. Mean
Real Mean
Implicit Negative
15.65
4.80e-52
1569.39
796.30
Implicit Positive
21.44
6.02e-92
3308.83
1596.74
Explicit Positive
9.43
1.34e-20
105.66
43.79
Appendix
Table 10: Two-tailed t-tests comparing synthetic and real UIH engagement frequencies. Mean differences reflect intentional oversampling; relative proportions are preserved.
Figure 3: IG Precision vs. IG Recall per model on the survey dataset. Bubble size encodes the average number of predicted interest categories; color encodes IG F1 . Dotted curves are iso-F1 contours at 0.1–0.5. All models cluster in the low-recall region (IG R<0.4 ), confirming that coverage is the universal bottleneck. DeepSeek-R1 achieves the highest precision (57.3%) and IG F1 (42.9%). Qwen2.5-72B offers the best balance between precision and recall among capable models.
Figure 4: Per-user IG Precision vs. IG Recall on the benchmark datasets, colored by model. Diamond markers show per-model medians; small dots show individual users ( α=0.15 ). The diagonal reference line ( y=x ) highlights the universal asymmetry IG P> IG R . Within-model variance is substantial: even top models have many users with IG P<0.5 . This reflects user-level difficulty variation.
Figure 5: Forest plot showing 95% confidence intervals for mean IG F1 scores on the survey dataset. Overlapping intervals indicate that ranking differences between adjacent models may not be statistically significant.
Model
Unique Int.
Covered
Cov. %
GPT-OSS-120B
127,495
325
100.0
Qwen3-VL-235B
108,444
324
99.7
DeepSeek-R1-Distill-70B
37,340
323
99.4
Llama-3.1-8B-Instruct
40,690
323
99.4
DeepSeek-R1
78,534
320
98.5
Qwen2-7B
21,999
318
97.8
Appendix
Table 11: Per-model taxonomy coverage aggregated across all datasets. Unique Int. : number of distinct predicted interest strings mapped to the taxonomy. Covered : number of the 325 taxonomy categories with ≥ 1 mapped interest. All eight models cover ≥ 97% of categories.
Figure 6: Taxonomy coverage by model, aggregated across all datasets. The red dashed line marks the total 325 taxonomy categories. All models cover ≥ 97.5% of categories.
Figure 7: Cumulative distribution of predicted interests across taxonomy categories (sorted by size, largest first). The steep initial rise confirms heavy concentration: 10.1% of categories capture 50% of all predicted interests, and 29.4% capture 80%, reflecting the long-tailed nature of user engagement.
Figure 8: Taxonomy coverage across datasets for (a) GPT-OSS-120B alone and (b) the union of all evaluated models. The red dashed line marks the total 325 taxonomy categories. Ensembling substantially improves coverage on mid-range datasets (e.g., MIND: 53% → 75%; Goodreads: 59% → 76%) but has limited effect where a single model already saturates (Synthetic) or where data sparsity fundamentally constrains discovery (Amazon Music).
Figure 9: Box plot of per-user IG F1 scores by model on the survey dataset. The red dashed line shows the overall median. Even top models have users at IG F1=0 , showing substantial within-model variance.
Interest
Object
Content Description
Engagement
“NBA Basketball Highlights”
vid_12
#NBA #LeBron “LeBron’s game-winning dunk vs Celtics”
like ( exp+ )
vid_34
#Basketball “Top 10 plays of the week”
watch 90s ( imp+ )
vid_56
#NBAPlayoffs “Conference finals recap”
watch 120s ( imp+ )
vid_78
#NBA #GameDay “Buzzer beater compilation 2024”
share ( exp+ )
“Italian Cooking Recipes”
vid_91
#ItalianFood “How to make carbonara at home”
watch 45s ( imp+ )
vid_23
#Cooking “Knife skills tutorial for beginners”
skip ( imp − )
Appendix
Table 12: Evidence objects cited by the model for two predicted interests. Each object includes the content description (hashtags and title) and the user’s engagement type, classified as implicit positive ( imp+ : watch ≥ 30s), explicit positive ( exp+ : like, share), or implicit negative ( imp − : skip).
Interest
Imp+
Exp+
Imp −
Exp −
IG Verdict
IS
NBA Basketball
2
2
0
0
✓ Verified
3/4
Italian Cooking
2
0
2
0
✗ Failed
—
Appendix
Table 13: IG verification and IS retrieval results. “NBA Basketball Highlights” passes verification ( ≥ 2 explicit positives) and achieves IS = 3/4 (the judge correctly identifies 3 of 4 backing objects from a pool of 50). “Italian Cooking Recipes” fails: despite 2 implicit positives, it has 0 explicit positives and 2 implicit negatives (skips), indicating insufficient evidence of genuine interest. IS is not computed for unverified interests.
UIH Length
50
100
150
200
IG P (%)
74.4
80.9
84.9
88.9
IG R (%)
86.0
91.0
93.8
96.8
IG F1 (%)
79.7
85.6
89.2
92.5
IS (%)
70.7
72.4
73.9
74.6
# Interests
109
92
72
49
Appendix
Table 14: Effect of UIH length on GPT-OSS-120B performance (Synthetic dataset). IG R and IS both improve with more context, but at markedly different rates (+10.8pp vs. +3.9pp), confirming the two metrics respond to different input axes.
Models verifying
Count
%
1
37,672
41.4
2
19,929
21.9
3
13,325
14.6
4
9,680
10.6
5
6,011
6.6
6
3,276
3.6
Appendix
Table 15: Oracle category discovery distribution on the Synthetic dataset (7 models). Each row counts how many (user, interest category) pairs in the oracle are independently verified by exactly k models. 36.8% of oracle entries are verified by 3+ models (cross-model consensus), while 41.4% depend on a single model.
Model
Verified
Sole provider
% oracle
GPT-OSS-120B
60,971
16,065
17.6
Qwen3-VL-235B
48,260
8,844
9.7
Qwen2.5-72B
36,778
5,550
6.1
DeepSeek-R1
26,158
3,569
3.9
R1-Distill-70B
19,696
1,774
1.9
Qwen3-32B
13,900
1,308
1.4
Appendix
Table 16: Leave-one-out oracle impact on the Synthetic dataset. “Verified” = total (user, interest category) pairs this model verifies. “Sole provider” = pairs verified only by this model (lost if removed). GPT-OSS-120B (the broadest-coverage model) uniquely provides 17.6% of the oracle, while every model contributes at least some unique categories.
Evidence Relevance (IG)
A 1
A 2
A 3
A 4
A 1
—
0.40
0.70
0.47
A 2
—
0.44
0.43
A 3
—
0.65
A 4
—
Interest–Item Association (IS)
Appendix
Table 17: Pairwise Cohen’s κ between four annotators ( N=1,000 per task). IG average pairwise κ=0.52 (moderate); IS average pairwise κ=0.86 (almost perfect).
IG
IS
Agree%
κ
F 1
Agree%
κ
F 1
A 1
93.7
0.81
0.96
98.4
0.93
0.94
A 2
88.0
0.56
0.93
99.1
0.96
0.96
A 3
96.7
0.89
0.98
97.6
0.90
0.91
A 4
90.8
0.65
0.95
97.1
0.88
0.90
Appendix
Table 18: Agreement with majority vote ( ≥ 3/4). All annotators reach ≥ 88% agreement on IG and ≥ 97% on IS.
Agreement level
IG
IS
Unanimous (4/4 or 0/4)
77.0%
94.0%
3–1 split
15.2%
4.2%
2–2 split
7.8%
1.8%
Appendix
Table 19: Agreement-level distribution. 94% of IS items and 77% of IG items receive unanimous labels.
Setting
Exp+
Imp+
Hybrid
Imp − cap
Lenient ( −1 )
≥ 1
≥ 1
—
≤ 4
Base (paper)
≥ 2
≥ 3
≥ 1 & ≥ 2
≤ 3
Strict ( +1 )
≥ 3
≥ 4
≥ 2 & ≥ 3
≤ 2
Appendix
Table 20: Threshold configurations for the sensitivity sweep on Synthetic. Each component is shifted by one step; under Lenient the single-implicit-positive clause subsumes the hybrid clause.
Model
Lenient
Base
Strict
GPT-OSS-120B
76.1
67.8
43.1
DeepSeek-R1
60.2
56.5
39.6
Qwen3-VL-235B
75.1
50.8
34.6
Qwen3-32B
54.3
46.3
33.1
Qwen2.5-72B
56.9
45.4
36.3
DeepSeek-R1- Distill-70B
58.7
37.1
26.1
Appendix
Table 21: Median IG F1 (%) on Synthetic under the three threshold settings (Table 20 ). Absolute scores shift with the thresholds, but the model ordering is stable.
Metric
Base vs Lenient
Base vs Strict
IG P
+0.815
+0.893
IG R
+0.857
+1.000
IG F1
+0.821
+0.893
IS
+0.821
+1.000
Appendix
Table 22: Spearman rank correlation ( ρ , n=7 models) of the per-metric model ranking against the Base configuration. All ρ≥0.81 ; IG R and IS rankings are perfectly preserved under Strict.
Paper Name
HuggingFace Model ID
DeepSeek-R1
deepseek-ai/DeepSeek-R1
GPT-OSS-120B
openai/gpt-oss-120b
Qwen3-32B
Qwen/Qwen3-32B
Qwen3-VL-235B
Qwen/Qwen3-VL-235B-A22B-Instruct
R1-Distill-70B
deepseek-ai/DeepSeek-R1-Distill-Llama-70B
Qwen2.5-72B
Qwen/Qwen2.5-72B-Instruct
Appendix
Table 23: HuggingFace model identifiers for the eight evaluated models, followed by the judge model. † Judge model for both IG evidence filtering and IS retrieval evaluation.
Dataset
Prompts
Input tok.
Output tok.
Synthetic
46,961
175.3 M
198.3 M
KuaiRec
14,844
78.0 M
49.5 M
MIND
2,062
13.6 M
4.6 M
Amazon Music
126
0.6 M
0.1 M
Goodreads
2,219
34.7 M
8.1 M
Total
66,212
302.2 M
260.6 M
Appendix
Table 24: Stage A candidate-inference token cost for the reference candidate (Llama-3.1-8B-Instruct), by dataset. Other candidates’ token volume scales with the number of interests they generate (Table 27 ).
Judge pass
Prompts
Input tok.
Output tok.
IG evidence relevance (Sec. 4.2 )
2,497,314
3.27 B
66.5 M
IS distractor filtering (Sec. 4.3 )
38,977
2.70 B
12.2 M
IS retrieval (Sec. 4.3 )
2,399,632
4.65 B
34.2 M
Taxonomy mapping (Sec. 4.4 )
156,083
0.39 B
2.2 M
Total
5,092,006
11.07 B
117.7 M
Appendix
Table 25: Stage B benchmark-computation token cost (Llama-3.3-70B-Instruct judge), summed across all 8 evaluated candidates’ inputs. Per-pass subtotals are rounded sampled estimates; the Total is computed over all per-(model, dataset, judge) cells and may not equal the rounded subtotal sum.
Judge pass
H100-hours
IG evidence relevance filtering (Section 4.2 )
∼380
IS distractor-pool overlap filtering (Section 4.3 )
∼115
IS retrieval (Section 4.3 )
∼245
Interest-to-taxonomy mapping (Section 4.4 )
∼20
Total (across all 8 evaluated candidates)
∼760
Appendix
Table 26: Compute cost of the four Llama-3.3-70B-Instruct judge inference passes that constitute the benchmark computation, summed across the inputs from all 8 candidates evaluated in this paper, executed on 64 H100 GPUs. The per-candidate decomposition of this ∼760 H100-hour total is given in Table 27 .
Candidate
Generated interests
Benchmark computation H100-hr
DeepSeek-R1
35,550
∼56
Qwen3-32B
37,960
∼60
DeepSeek-R1-Distill-70B
48,040
∼76
Qwen2-7B
61,400
∼97
Qwen3-VL-235B
65,290
∼103
Llama-3.1-8B-Instruct
68,496
∼108
Appendix
Table 27: Benchmark computation cost per evaluated candidate, summing to the ∼760 H100-hour aggregate in Table 26 . Cost scales with each candidate’s generated interest count.
Figure 10: Ranking stability versus sample size on the Synthetic split. For each N we draw 50 random subsamples of N users, rank the models by median IG F1 within each, and compute the Spearman correlation against the full-set ( N=982 ) ranking; the line is the mean and the shaded band is ±1 standard deviation across replicates. The ranking is recovered to ρ=0.96 at N=10 and to 1.00 by N=200 .
Recently, the fast development of Large Language Models (LLMs) such as ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. However, the application of LLMs in the recommendation domain has not been thoroughly investigated. To bridge this gap, we propose LLMRec, a LLM-based recommender system designed for benchmarking LLMs on various recommendation tasks. Specifically, we benchmark several popular off-the-shelf LLMs, such as ChatGPT, LLaMA, ChatGLM, on five recommendation tasks, including rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. Furthermore, we investigate the effectiveness of supervised finetuning to improve LLMs' instruction compliance ability. The benchmark results indicate that LLMs displayed only moderate proficiency in accuracy-based tasks such as sequential and direct recommendation. However, they demonstrated comparable performance to state-of-the-art methods in explainability-based tasks. We also conduct qualitative evaluations to further evaluate the quality of contents generated by different models, and the results show that LLMs can truly understand the provided information and generate clearer and more reasonable results. We aspire that this benchmark will serve as an inspiration for researchers to delve deeper into the potential of LLMs in enhancing recommendation performance. Our codes, processed data and benchmark results are available at https://github.com/williamliujl/LLMRec.
Junling Liu, Chao Liu, Peilin Zhou +8
Alibaba Group · Ant Group · The Hong Kong University of Science and Technology (Guangzhou) +5
Large Language Models (LLMs) have reshaped user profiling, yet current evaluations mainly focus on static data snapshots. This paradigm overlooks the reality of personalized systems, where User-Generated Content (UGC) arrives continuously and fine-grained profile evolve rapidly. To bridge this gap, we introduce StreamProfileBench, a large-scale benchmark for fine-grained streaming user profiling. We formalize streaming user profiling as a continuous state maintenance task and curate a highly authentic dataset comprising over 120,000 UGC posts from 7,000+ real users across five diverse platforms. By leveraging the temporal correlation of user interests, we further propose a novel, annotation-free evaluation framework. Extensive experiments across 14 leading LLMs reveal that continuous profile updating remains an open challenge. Models exhibit a systemic conservative bias, over-retaining past interests while failing to recognize interest decay. Ablation experiments further validate the practical utility and necessity of the streaming paradigm.
Sizhe Wang, Feiyu Duan, Juelin Wang +2
School of Data Science, Fudan University · Shanghai University of Finance and Economics · Shanghai Innovation Institute +2
Traditional offline recommendation evaluation relies heavily on complex, manually maintained feature pipelines that are difficult to scale. While Large Language Models (LLMs) offer a promising alternative by predicting user engagement directly from raw text logs, empirical analysis in this study identifies a critical failure mode termed bidirectional rationalization. In a zero-shot setting, LLMs are found to convincingly argue for both positive and negative user engagement outcomes on the exact same item with identical evidence, highlighting the unreliability of off-the-shelf LLMs in predicting user engagement. To resolve this, we develop and apply a sequential behavioral alignment framework pairing fine-tuning with preference optimization over paired correct and counterfactual rationales. Evaluated on real-world homepage interaction logs, this aligned reasoning approach achieves a 32.19% lift in Macro-F1 score over the zero-shot baseline and matches the production feature-engineered baseline. The results demonstrate that behavioral alignment mitigates bidirectional rationalization while delivering human-interpretable reasoning traces without manual pipeline overhead.