GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

Iordanis Fostiropoulos; Muhammad Rafay Azhar; Abdalaziz Sawwan; Boyu Fang; Yuchen Liu; Jiayi Liu; Hanchao Yu; Qi Guo

cs.AIMar 31, 2026

GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

Authors: Iordanis Fostiropoulos, Muhammad Rafay Azhar, Abdalaziz Sawwan, Boyu Fang, Yuchen Liu, Jiayi Liu, Hanchao Yu, Qi Guo,

Organizations: Meta Recommendation Systems (MRS)

Abstract

We introduce GISTBench, a benchmark for evaluating Large Language Models' (LLMs) ability to understand users from their interaction histories in recommendation systems. Unlike traditional RecSys benchmarks that focus on item prediction accuracy, our benchmark evaluates how well LLMs can extract and verify user interests from engagement data. We propose two novel metric families: Interest Groundedness (IG), decomposed into precision and recall components to separately penalize hallucinated interest categories and reward coverage, and Interest Specificity (IS), which assesses the distinctiveness of verified LLM-predicted user profiles. We release a synthetic dataset constructed on real user interactions on a global short-form video platform. Our dataset contains both implicit and explicit engagement signals and rich textual descriptions. We validate our dataset fidelity against user surveys, and evaluate eight open-weight LLMs spanning 7B to 235B parameters, together with three proprietary frontier models (GPT-5, Claude 4.6, and Gemini 3.5 Flash). Our findings reveal performance bottlenecks in current LLMs, particularly their limited ability to accurately count and attribute engagement signals across heterogeneous interaction types.

Figures & tables

Figure 1: Overview of the GISTBench evaluation pipeline. User interaction histories are processed by an LLM to produce predicted interests. These are evaluated on two axes: Interest Groundedness (IG, decomposed into precision and recall) and Interest Specificity (IS, computed over verified categories only), then mapped to standardized taxonomy categories for normalization.

Dataset	Domain	Rich Text	Imp+	Imp–	Exp+	Exp–	Primary Signal	Density
KuaiRec Gao et al. (2022)	Short Video	Tags only	✓	✓	✓	✗	Watch ratio	100% $\geq$ 100 eng.
MIND Wu et al. (2020)	News	✓	✗	✓	✓	✗	Click/non-click	39% $\geq$ 100 eng.
Amazon Music Hou et al. (2024)	Music	✓	✓	✗	✓	✓	1–5 star ratings	0.12% $\geq$ 20 eng.
Goodreads Wan and McAuley (2018)	Books	✓	✓	✗	✓	✓	1–5 star ratings	Moderate
Ours (Synthetic)	Short Video	✓	✓	✓	✓	✗	Mixed behavioral	100% $\geq$ 100 eng.

Table 1: Comparison of datasets used in our benchmark. Rich Text : detailed natural language descriptions. Imp+/– : implicit positive/negative signals. Exp+/– : explicit positive/negative signals. Density : fraction of users with sufficient engagement history. Each dataset presents a distinct evaluation challenge based on its signal mix and sparsity.

Category Description	Example Interest Categories
Sports	Basketball, American Football, Soccer / Football, Combat Sports & Martial Arts, Golf, Winter Sports, …
Food & Cooking	Baked Goods & Dessert, Food Recipes & Cooking Tips, Local / Regional Cuisine, Mukbang, Restaurants & Eateries, …
Electronics & Technology	Artificial Intelligence, Software & Apps, Computers & Electronic Devices, Tech Builds & Repairs, …
Comedy & Humor	Stand Up & Roasting, Sketches & Skits, Pranks, Memes, Reaction Comedy, …
Visual Arts & Design	Drawing & Painting, Photography, Graphic Design, Sculpture, Street & Public Art, …

Table 2: Examples of top-level interest domains and their corresponding interest categories. Each domain contains several fine-grained interest categories; only a representative subset is shown. The full list of categories is provided in Appendix A .

	Synthetic		KuaiRec		Amazon Music		Goodreads
Model	IG F1	IS	IG F1	IS	IG F1	IS	IG F1	IS
GPT-OSS-120B	67.8	74.0	56.3	68.4	66.7	70.6	58.0	67.4
GPT-5	59.7	81.1	62.3	74.4	66.7	90.9	70.0	83.7
DeepSeek-R1	56.5	79.0	44.1	71.0	57.1	74.1	43.5	68.9
Qwen3-VL-235B	50.8	73.4	55.2	65.9	50.0	70.8	48.5	65.5
Claude 4.6	50.1	78.0	66.9	69.7	75.0	86.8	61.3	83.4

Table 3: Median IG F1 (%) and IS (%) across the four primary benchmark datasets. IG F1 is the harmonic mean of IG Precision and IG Recall; IS is computed over verified interest categories only (bounded

[0,1]

). Best per-dataset results in bold .

	Synthetic		KuaiRec
Model	IG P	IG R	IG P	IG R
DeepSeek-R1	82.4	43.9	89.8	29.5
GPT-OSS-120B	78.1	60.5	79.7	44.0
Qwen3-32B	73.6	34.3	82.4	31.8
GPT-5	67.8	53.5	73.5	54.2
Qwen2.5-72B	65.7	34.8	79.2	29.7

Table 4: IG Precision and IG Recall (%) on Synthetic and KuaiRec. IG

{}_{P}>

IG R for every model. The primary bottleneck is coverage; hallucination is not. Best IG P and IG R per dataset in bold .

	Survey-Based		Proposed Metrics
Model	S-Prec	S-Rec	IG P	IG R	IG F1	IS
DeepSeek-R1	62.3	60.0	57.3	33.7	42.9	83.3
GPT-OSS-120B	59.0	60.2	31.6	26.5	27.7	87.8
Qwen3-32B	59.5	52.8	50.0	26.1	33.9	80.6
Qwen3-VL-235B	57.1	66.1	25.0	26.9	25.7	84.8
DeepSeek-R1- Distill-70B	51.4	56.1	25.2	21.4	24.3	83.3

Table 5: Survey-Based metrics (S-Prec, S-Rec) versus proposed metrics (IG P , IG R , IG F1 , IS), all in %. S-Prec and S-Rec are computed against user self-reported interests; IG and IS are computed from engagement signals (Section 4 ). The full precision-recall decomposition reveals that IG

{}_{P}>

IG R for seven of the eight models on the survey dataset (Qwen3-VL-235B is the recall-leaning exception), consistent with the benchmark results (Table 4 ): coverage, not hallucination, is the bottleneck.

\rho

: Spearman correlation between survey F1 and geometric mean of IG F1 and IS.

Model	Arena IF Elo	Median IG F1 (%)
DeepSeek-R1	1395	42.9
Qwen2.5-72B	1238	37.7
Qwen3-32B	1330	33.9
GPT-OSS-120B	1325	27.7
Qwen3-VL-235B	1325	25.7
DeepSeek-R1- Distill-70B	1395	24.3

Table 6: Instruction-following capability (Chatbot Arena IF Elo Chiang et al. (2024) ) and median IG F1 (%) on the survey dataset (real users). Among these models, Elo does not predict IG F1 rank.

Model	Ins. Imp+	Ins. Exp+	Exc. Neg
DeepSeek-R1	86.7	99.4	1.8
GPT-OSS-120B	97.3	99.2	1.3
Qwen3-32B	95.7	98.6	3.1
Qwen3-VL-235B	93.1	98.3	10.8
DeepSeek-R1- Distill-70B	90.5	94.8	9.6
Qwen2.5-72B	76.3	92.0	22.9

Table 7: Distribution of IG failure modes (%) among unverified interests, aggregated across all datasets. Categories are not mutually exclusive. Ins. Imp+ : insufficient implicit positive signals (

<

3); Ins. Exp+ : insufficient explicit positive signals (

<

2); Exc. Neg : excessive negative signals. Insufficient positive evidence dominates across all models. Excessive negative contamination is bimodal.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Comparison of synthetic and real UIH distributions. (a) Per-user engagement count densities show similar right-skewed patterns. (b) Box plots confirm that the relative ordering across action types is preserved.

Action Type	t-stat	p-value	Syn. Mean	Real Mean
Implicit Negative	15.65	4.80e-52	1569.39	796.30
Implicit Positive	21.44	6.02e-92	3308.83	1596.74
Explicit Positive	9.43	1.34e-20	105.66	43.79

Appendix

Table 10: Two-tailed t-tests comparing synthetic and real UIH engagement frequencies. Mean differences reflect intentional oversampling; relative proportions are preserved.

Figure 3: IG Precision vs. IG Recall per model on the survey dataset. Bubble size encodes the average number of predicted interest categories; color encodes IG F1 . Dotted curves are iso-F1 contours at 0.1–0.5. All models cluster in the low-recall region (IG ${}_{R}<0.4$ ), confirming that coverage is the universal bottleneck. DeepSeek-R1 achieves the highest precision (57.3%) and IG F1 (42.9%). Qwen2.5-72B offers the best balance between precision and recall among capable models.

Figure 4: Per-user IG Precision vs. IG Recall on the benchmark datasets, colored by model. Diamond markers show per-model medians; small dots show individual users ( $\alpha=0.15$ ). The diagonal reference line ( $y=x$ ) highlights the universal asymmetry IG ${}_{P}>$ IG R . Within-model variance is substantial: even top models have many users with IG ${}_{P}<0.5$ . This reflects user-level difficulty variation.

Figure 5: Forest plot showing 95% confidence intervals for mean IG F1 scores on the survey dataset. Overlapping intervals indicate that ranking differences between adjacent models may not be statistically significant.

Model	Unique Int.	Covered	Cov. %
GPT-OSS-120B	127,495	325	100.0
Qwen3-VL-235B	108,444	324	99.7
DeepSeek-R1-Distill-70B	37,340	323	99.4
Llama-3.1-8B-Instruct	40,690	323	99.4
DeepSeek-R1	78,534	320	98.5
Qwen2-7B	21,999	318	97.8

Appendix

Table 11: Per-model taxonomy coverage aggregated across all datasets. Unique Int. : number of distinct predicted interest strings mapped to the taxonomy. Covered : number of the 325 taxonomy categories with

\geq

1 mapped interest. All eight models cover

\geq

97% of categories.

Figure 6: Taxonomy coverage by model, aggregated across all datasets. The red dashed line marks the total 325 taxonomy categories. All models cover $\geq$ 97.5% of categories.

Figure 7: Cumulative distribution of predicted interests across taxonomy categories (sorted by size, largest first). The steep initial rise confirms heavy concentration: 10.1% of categories capture 50% of all predicted interests, and 29.4% capture 80%, reflecting the long-tailed nature of user engagement.

Panel of Figure 8 — Figure 8: Taxonomy coverage across datasets for (a) GPT-OSS-120B alone and (b) the union of all evaluated models. The red dashed line marks the total 325 taxonomy categories. Ensembling substantially improves coverage on mid-range datasets (e.g., MIND: 53% $\to$ 75%; Goodreads: 59% $\to$ 76%) but has limited effect where a single model already saturates (Synthetic) or where data sparsity fundamentally constrains discovery (Amazon Music).

Figure 9: Box plot of per-user IG F1 scores by model on the survey dataset. The red dashed line shows the overall median. Even top models have users at IG ${}_{F1}=0$ , showing substantial within-model variance.

Interest	Object	Content Description	Engagement
“NBA Basketball Highlights”	vid_12	#NBA #LeBron “LeBron’s game-winning dunk vs Celtics”	like ( exp+ )
	vid_34	#Basketball “Top 10 plays of the week”	watch 90s ( imp+ )
	vid_56	#NBAPlayoffs “Conference finals recap”	watch 120s ( imp+ )
	vid_78	#NBA #GameDay “Buzzer beater compilation 2024”	share ( exp+ )
“Italian Cooking Recipes”	vid_91	#ItalianFood “How to make carbonara at home”	watch 45s ( imp+ )
	vid_23	#Cooking “Knife skills tutorial for beginners”	skip ( imp $-$ )

Appendix

Table 12: Evidence objects cited by the model for two predicted interests. Each object includes the content description (hashtags and title) and the user’s engagement type, classified as implicit positive ( imp+ : watch

\geq

30s), explicit positive ( exp+ : like, share), or implicit negative ( imp

-

: skip).

Interest	Imp+	Exp+	Imp $-$	Exp $-$	IG Verdict	IS
NBA Basketball	2	2	0	0	✓ Verified	3/4
Italian Cooking	2	0	2	0	✗ Failed	—

Appendix

Table 13: IG verification and IS retrieval results. “NBA Basketball Highlights” passes verification (

\geq

2 explicit positives) and achieves IS = 3/4 (the judge correctly identifies 3 of 4 backing objects from a pool of 50). “Italian Cooking Recipes” fails: despite 2 implicit positives, it has 0 explicit positives and 2 implicit negatives (skips), indicating insufficient evidence of genuine interest. IS is not computed for unverified interests.

UIH Length	50	100	150	200
IG P (%)	74.4	80.9	84.9	88.9
IG R (%)	86.0	91.0	93.8	96.8
IG F1 (%)	79.7	85.6	89.2	92.5
IS (%)	70.7	72.4	73.9	74.6
# Interests	109	92	72	49

Appendix

Table 14: Effect of UIH length on GPT-OSS-120B performance (Synthetic dataset). IG R and IS both improve with more context, but at markedly different rates (+10.8pp vs. +3.9pp), confirming the two metrics respond to different input axes.

Models verifying	Count	%
1	37,672	41.4
2	19,929	21.9
3	13,325	14.6
4	9,680	10.6
5	6,011	6.6
6	3,276	3.6

Appendix

Table 15: Oracle category discovery distribution on the Synthetic dataset (7 models). Each row counts how many (user, interest category) pairs in the oracle are independently verified by exactly

k

models. 36.8% of oracle entries are verified by 3+ models (cross-model consensus), while 41.4% depend on a single model.

Model	Verified	Sole provider	% oracle
GPT-OSS-120B	60,971	16,065	17.6
Qwen3-VL-235B	48,260	8,844	9.7
Qwen2.5-72B	36,778	5,550	6.1
DeepSeek-R1	26,158	3,569	3.9
R1-Distill-70B	19,696	1,774	1.9
Qwen3-32B	13,900	1,308	1.4

Appendix

Table 16: Leave-one-out oracle impact on the Synthetic dataset. “Verified” = total (user, interest category) pairs this model verifies. “Sole provider” = pairs verified only by this model (lost if removed). GPT-OSS-120B (the broadest-coverage model) uniquely provides 17.6% of the oracle, while every model contributes at least some unique categories.

Evidence Relevance (IG)
	A 1	A 2	A 3	A 4
A 1	—	0.40	0.70	0.47
A 2		—	0.44	0.43
A 3			—	0.65
A 4				—
Interest–Item Association (IS)

Appendix

Table 17: Pairwise Cohen’s

\kappa

between four annotators (

N=1{,}000

per task). IG average pairwise

\kappa=0.52

(moderate); IS average pairwise

\kappa=0.86

(almost perfect).

	IG			IS
	Agree%	$\kappa$	F 1	Agree%	$\kappa$	F 1
A 1	93.7	0.81	0.96	98.4	0.93	0.94
A 2	88.0	0.56	0.93	99.1	0.96	0.96
A 3	96.7	0.89	0.98	97.6	0.90	0.91
A 4	90.8	0.65	0.95	97.1	0.88	0.90

Appendix

Table 18: Agreement with majority vote (

\geq

3/4). All annotators reach

\geq

88% agreement on IG and

\geq

97% on IS.

Agreement level	IG	IS
Unanimous (4/4 or 0/4)	77.0%	94.0%
3–1 split	15.2%	4.2%
2–2 split	7.8%	1.8%

Appendix

Table 19: Agreement-level distribution. 94% of IS items and 77% of IG items receive unanimous labels.

Setting	Exp+	Imp+	Hybrid	Imp $-$ cap
Lenient ( $-1$ )	$\geq$ 1	$\geq$ 1	—	$\leq$ 4
Base (paper)	$\geq$ 2	$\geq$ 3	$\geq$ 1 & $\geq$ 2	$\leq$ 3
Strict ( $+1$ )	$\geq$ 3	$\geq$ 4	$\geq$ 2 & $\geq$ 3	$\leq$ 2

Appendix

Table 20: Threshold configurations for the sensitivity sweep on Synthetic. Each component is shifted by one step; under Lenient the single-implicit-positive clause subsumes the hybrid clause.

Model	Lenient	Base	Strict
GPT-OSS-120B	76.1	67.8	43.1
DeepSeek-R1	60.2	56.5	39.6
Qwen3-VL-235B	75.1	50.8	34.6
Qwen3-32B	54.3	46.3	33.1
Qwen2.5-72B	56.9	45.4	36.3
DeepSeek-R1- Distill-70B	58.7	37.1	26.1

Appendix

Table 21: Median IG F1 (%) on Synthetic under the three threshold settings (Table 20 ). Absolute scores shift with the thresholds, but the model ordering is stable.

Metric	Base vs Lenient	Base vs Strict
IG P	$+0.815$	$+0.893$
IG R	$+0.857$	$+1.000$
IG F1	$+0.821$	$+0.893$
IS	$+0.821$	$+1.000$

Appendix

Table 22: Spearman rank correlation (

\rho

n=7

models) of the per-metric model ranking against the Base configuration. All

\rho\geq 0.81

; IG R and IS rankings are perfectly preserved under Strict.

Paper Name	HuggingFace Model ID
DeepSeek-R1	deepseek-ai/DeepSeek-R1
GPT-OSS-120B	openai/gpt-oss-120b
Qwen3-32B	Qwen/Qwen3-32B
Qwen3-VL-235B	Qwen/Qwen3-VL-235B-A22B-Instruct
R1-Distill-70B	deepseek-ai/DeepSeek-R1-Distill-Llama-70B
Qwen2.5-72B	Qwen/Qwen2.5-72B-Instruct

Appendix

Table 23: HuggingFace model identifiers for the eight evaluated models, followed by the judge model. † Judge model for both IG evidence filtering and IS retrieval evaluation.

Dataset	Prompts	Input tok.	Output tok.
Synthetic	$46{,}961$	$175.3$ M	$198.3$ M
KuaiRec	$14{,}844$	$78.0$ M	$49.5$ M
MIND	$2{,}062$	$13.6$ M	$4.6$ M
Amazon Music	$126$	$0.6$ M	$0.1$ M
Goodreads	$2{,}219$	$34.7$ M	$8.1$ M
Total	$\mathbf{66{,}212}$	$\mathbf{302.2}$ M	$\mathbf{260.6}$ M

Appendix

Table 24: Stage A candidate-inference token cost for the reference candidate (Llama-3.1-8B-Instruct), by dataset. Other candidates’ token volume scales with the number of interests they generate (Table 27 ).

Judge pass	Prompts	Input tok.	Output tok.
IG evidence relevance (Sec. 4.2 )	$2{,}497{,}314$	$3.27$ B	$66.5$ M
IS distractor filtering (Sec. 4.3 )	$38{,}977$	$2.70$ B	$12.2$ M
IS retrieval (Sec. 4.3 )	$2{,}399{,}632$	$4.65$ B	$34.2$ M
Taxonomy mapping (Sec. 4.4 )	$156{,}083$	$0.39$ B	$2.2$ M
Total	$\mathbf{5{,}092{,}006}$	$\mathbf{11.07}$ B	$\mathbf{117.7}$ M

Appendix

Table 25: Stage B benchmark-computation token cost (Llama-3.3-70B-Instruct judge), summed across all

8

evaluated candidates’ inputs. Per-pass subtotals are rounded sampled estimates; the Total is computed over all per-(model, dataset, judge) cells and may not equal the rounded subtotal sum.

Judge pass	H100-hours
IG evidence relevance filtering (Section 4.2 )	$\sim 380$
IS distractor-pool overlap filtering (Section 4.3 )	$\sim 115$
IS retrieval (Section 4.3 )	$\sim 245$
Interest-to-taxonomy mapping (Section 4.4 )	$\sim 20$
Total (across all $8$ evaluated candidates)	$\mathbf{\sim 760}$

Appendix

Table 26: Compute cost of the four Llama-3.3-70B-Instruct judge inference passes that constitute the benchmark computation, summed across the inputs from all

8

candidates evaluated in this paper, executed on

64

H100 GPUs. The per-candidate decomposition of this

\sim 760

H100-hour total is given in Table 27 .

Candidate	Generated interests	Benchmark computation H100-hr
DeepSeek-R1	$35{,}550$	$\sim 56$
Qwen3-32B	$37{,}960$	$\sim 60$
DeepSeek-R1-Distill-70B	$48{,}040$	$\sim 76$
Qwen2-7B	$61{,}400$	$\sim 97$
Qwen3-VL-235B	$65{,}290$	$\sim 103$
Llama-3.1-8B-Instruct	$68{,}496$	$\sim 108$

Appendix

Table 27: Benchmark computation cost per evaluated candidate, summing to the

\sim 760

H100-hour aggregate in Table 26 . Cost scales with each candidate’s generated interest count.

Figure 10: Ranking stability versus sample size on the Synthetic split. For each $N$ we draw $50$ random subsamples of $N$ users, rank the models by median IG F1 within each, and compute the Spearman correlation against the full-set ( $N=982$ ) ranking; the line is the mean and the shaded band is $\pm 1$ standard deviation across replicates. The ranking is recovered to $\rho=0.96$ at $N=10$ and to $1.00$ by $N=200$ .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

LLMRec: Benchmarking Large Language Models on Recommendation Task
Aug 23, 2023Junling Liu, Chao Liu, Peilin Zhou +8Large Language Model-Based Recommendation Recommender Systems
Alibaba Group · Ant Group · The Hong Kong University of Science and Technology (Guangzhou) +5
StreamProfileBench: A Benchmark for Fine-Grained User Profile Inference in Real-World Streaming Scenarios
May 25, 2026Sizhe Wang, Feiyu Duan, Juelin Wang +2LLM Evaluation User Modeling
School of Data Science, Fudan University · Shanghai University of Finance and Economics · Shanghai Innovation Institute +2
From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation
Aug 11, 2026Alireza S. Ziabari, Kat Ellis, Colleen Chan +1LLM Alignment Recommender Systems
Netflix Los Angeles, USA · Netflix Los Gatos, USA

Aug 23, 2023J/K move · Enter open · S save

LLMRec: Benchmarking Large Language Models on Recommendation Task

Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye+7

Alibaba Group · Ant Group · The Hong Kong University of Science and Technology (Guangzhou)+5

Large Language Model-Based Recommendation Recommender Systems LLM Evaluation

Open paper AskPDF arXiv

Recently, the fast development of Large Language Models (LLMs) such as ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. However, the application of LLMs in the recommendation domain has not been thoroughly investigated. To bridge this gap, we propose LLMRec, a LLM-based recommender system designed for benchmarking LLMs on various recommendation tasks. Specifically, we benchmark several popular off-the-shelf LLMs, such as ChatGPT, LLaMA, ChatGLM, on five recommendation tasks, including rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. Furthermore, we investigate the effectiveness of supervised finetuning to improve LLMs' instruction compliance ability. The benchmark results indicate that LLMs displayed only moderate proficiency in accuracy-based tasks such as sequential and direct recommendation. However, they demonstrated comparable performance to state-of-the-art methods in explainability-based tasks. We also conduct qualitative evaluations to further evaluate the quality of contents generated by different models, and the results show that LLMs can truly understand the provided information and generate clearer and more reasonable results. We aspire that this benchmark will serve as an inspiration for researchers to delve deeper into the potential of LLMs in enhancing recommendation performance. Our codes, processed data and benchmark results are available at https://github.com/williamliujl/LLMRec.

Abstract

LLM Evaluation

Explore similar work

LLMRec: Benchmarking Large Language Models on Recommendation Task

StreamProfileBench: A Benchmark for Fine-Grained User Profile Inference in Real-World Streaming Scenarios

From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation