Data Processing for Offline Evaluation in Recommender Systems: a Survey
Authors: Alberto Carlo Maria Mancino, Angela Di Fazio, Danilo Danese, Matteo Attimonelli, Daniele Malitesta, Antonio Ferrara, Claudio Pomo, Tommaso Di Noia
Organizations: Politecnico di Bari · Sapienza Università di Roma · LUISS Guido Carli University
Offline evaluation is the dominant experimental paradigm in recommender systems research, enabling reproducible and cost-effective comparisons on historical interaction data. Yet, while considerable attention has been devoted to recommendation models and evaluation methodologies, the data processing decisions that precede model training have received less scrutiny. These decisions determine the information available to recommendation algorithms and can affect the comparability and reproducibility of experimental results. This survey provides a systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems. We examine the data-centric pipeline, from dataset selection and interaction representation to data preparation, multimodal feature extraction, and train-validation-test splitting. Our analysis spans recommendation paradigms, including collaborative, sequential, session-based, graph-based, knowledge-aware, context-aware, multimodal, federated, cross-domain, contrastive-learning, and LLM-based recommendation. Beyond reviewing existing practices, we introduce a unified framework and taxonomy for describing data transformations and feature-extraction strategies, distinguishing data preparation from the extraction of representations from multimodal side information. Our empirical analysis reveals a landscape dominated by a narrow set of dataset-level transformations, particularly support-driven filtering, while representation-dependent transformations remain less common. We further identify substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols, where similar labels may conceal different experimental conditions.
Figures & tables
Figure 1.1 : Distribution of papers across research topics and publication periods. Topics are ordered by temporal dominance, i.e., according to the period in which each topic becomes most prominent, from the earliest to the most recent interval.
Figure 1.2 : Distribution of papers across venue ranking categories.
Figure 3.1 : Taxonomy of recommendation datasets. Datasets can be characterized along three complementary dimensions: structural representations, information sources, and dataset organization.
Figure 3.2 : Most frequently used datasets in the analyzed literature. The figure shows the top 30 datasets ranked by the number of occurrences across publications, considering only datasets that appear in at least two studies. Different versions of the same dataset are treated as distinct entries, while variations due to preprocessing are not distinguished. The distribution reveals a strong concentration of empirical evaluation around a small number of widely reused benchmarks, which dominate the experimental landscape.
Figure 3.3 : Rank-frequency distribution of dataset usage in the analyzed literature. Datasets are ordered by decreasing frequency of occurrence across publications. The distribution exhibits a pronounced long-tail behavior: a small fraction of datasets is extensively reused (top 30), while a large portion appears only occasionally or in a single study. In particular, 46.1% of the datasets are used only once, highlighting a substantial degree of fragmentation in experimental practices.
Domain
Occurrences
Distinct
Avg. Usage
Movies
361
25
14.44
E-commerce / Marketplace
263
54
4.87
Fashion
138
14
9.86
POI / Travel / Tourism
125
31
4.03
Music
119
24
4.96
Multi Domain
110
7
15.71
Table 3.1 : Dataset usage across domains. “Occurrences” denotes the total number of dataset usages in the analyzed literature, while “Distinct” reports the number of unique datasets within each domain. “Avg. Usage” is defined as the ratio between occurrences and distinct datasets, capturing the average reuse of datasets within a domain. Domains are sorted by decreasing number of occurrences. Darker shades indicate higher values of Avg. Usage.
Figure 3.4 : Benchmark adoption across recommender systems research topics. Rows correspond to datasets (ordered by overall popularity) and columns to research topics. Cell colors represent the percentage of occurrences of each dataset within a topic, while numbers denote absolute occurrence counts. The visualization illustrates both the broad reuse of general-purpose benchmarks and the emergence of task-specific benchmark ecosystems.
Origin
Count
Percentage
Academic
156
59.5%
Industry
72
27.5%
Academic Challenge
23
8.8%
Industry Challenge
11
4.2%
Table 3.2 : Distribution of datasets by origin, distinguishing between academic, industrial, and challenge-based sources.
Group
Category
Count
Percentage
Dataset availability
Link available
186
70.2%
Not shared
30
11.3%
Proprietary
47
17.7%
Unknown
2
0.8%
Link quality
Working
154
82.8%
Broken link
23
12.4%
Table 3.3 : Dataset availability and link quality.
Dataset
Text
Image
Audio
Video
Reviews
KG
Alibaba iFashion
✓
✓
–
–
–
–
Allrecipes.com
✓
✓
–
–
–
–
Amadeus CME
⧫
–
–
–
–
✓
Amazon Reviews*
✓ ⧫
↗⧫
–
–
✓
★
Bing News
✓
–
–
–
–
✓
BookCrossing
✓
✓
–
–
–
★
Table 3.4 : Representative benchmark datasets and their auxiliary information. Symbols denote how each modality is released: ✓ raw data; ⧫ pre-computed representations; ↗ official external resources; ★ external enrichments introduced by subsequent studies. Multiple symbols indicate multiple release strategies. Datasets marked with * aggregate closely related versions.
Figure 4.1 : Taxonomy of data preparation transformations.
Family
Subfamily
Technique
Support-driven
Activity-based filtering
User filtering
Item filtering
Iterative k -core filtering
User and item subsampling
User subsampling
Item subsampling
Other
External-criterion filtering
Table 4.1 : Overview of dataset-level transformation techniques associated with the corresponding taxonomy categories. Each technique refers to its detailed description in the text.
Family
Subfamily
Technique
Sequential-based
Sequence transformations
Sequence truncation
Sequence-length filtering
Session-based
Session transformations
Session-length filtering
Graph-based
Connectivity-based transformations
Connectivity-based filtering
Context-based
Context transformations
Context selection
Context discretization
Table 4.2 : Overview of representation-dependent transformation techniques associated with the corresponding taxonomy categories. Each technique refers to its detailed description in the text.
Family
Encoder training cost
Per-study extraction cost
Experimenter control
Cross-study comparability
Reproducibility
Hand-crafted
None
Low
Full
Low
High
Frozen pre-trained
Amortized
Low
Medium
High
Medium
End-to-end trained
Per study
High
Full
Low
Low
Dataset-provided
By dataset
None
None
Exact
High
Table 4.3: Typical practical trade-offs across the four families of feature extractors. Individual implementations may deviate from these general characteristics.
Modality
Stage
Technique
Textual
Data preparation
Text-field selection and combination
Text normalization and cleaning
Tokenization, truncation, and padding
Feature extraction
Sparse lexical representations
Pre-trained language-model representations
Task-specific or dataset-provided representations
Table 4.4 : Overview of modality-specific side-information transformations. Each technique refers to its detailed description in the text.
Figure 4.2 : Distribution of the top 10 data transformation techniques applied to interaction logs. Occurrences are measured at the (paper, dataset) level to capture the specific preprocessing pipeline applied to each dataset within a study. Percentages indicate the relative prevalence of each technique across all annotated instances.
Figure 4.3 : Co-occurrence heatmap of the top 15 data transformation techniques. Each cell indicates the number of times two distinct operations were applied to the same (paper, dataset) instance. The diagonal is intentionally left blank as it represents single-technique occurrences. The high values at the intersection of user and item k-core (429 instances) and binarisation (over 100 instances each) highlight the prevalence of a standardised data preparation pipeline in the literature.
Figure 4.4 : Structural distribution of transformation types using a dual-scale visualisation. The top panel shows the global prevalence of Dataset-level transformations (0-100% scale), while the bottom panel provides a zoomed perspective (0-14% scale) to illustrate the distribution of Representation-dependent transformations. All percentages represent the global frequency of each sub-category across the entire surveyed corpus (Section 1.2 ).
Figure 4.5 : Distribution of the number of data preparation techniques applied per paper. The majority of studies employ a minimalist approach, with a mean of 2.22 operations, typically focusing on the standard k -core and binarisation pipeline.
Extractor category
Visual
Textual
Other (Audio/KG)
(a) Hand-crafted
0
3
0
(b) Frozen pre-trained, uni-modal
71
82
12
(b) Frozen pre-trained, cross-modal
2
0
0
(c) End-to-end trained
9
14
5
(d) Dataset-shipped pre-computed
28
11
2
Total declared
110
110
19
Table 4.5: Distribution of disclosed feature extractors by category. Counts refer to modality-specific paper–dataset observations. For visual and textual information, “N/A” denotes that the modality is absent. Video inputs are included under visual information when processed as frames, and under “Other” when their audio stream is processed. In the “Other” column, pairs without a reported audio or knowledge-graph extractor, combine cases in which the modality is absent and cases in which its extractor is not disclosed.
Figure 4.6 : Distribution of reported visual extractors assigned to family (b), over 73 paper–dataset observations. CLIP (darker bar) is the only cross-modal entry; all other extractors are unimodal. “Pretrained CNN” aggregates pairs in which the paper declares the use of a pre-trained CNN without specifying the architecture.
Figure 4.7 : Distribution of reported textual extractors assigned to family (b), over 82 paper–dataset observations. “Word2Vec/GloVe” aggregates pairs in which the paper declares a word-embedding pipeline (Skip-gram, gensim, or initialised GloVe) without further specification.
Splitting Family
Prevalence (%)
Random (Global & User-wise Hold-out)
35.8%
Random Leave-One-Out
21.2%
Temporal Leave-One-Out
14.5%
Temporal Hold-out (Global & User-wise)
12.3%
Fixed / Inherited Benchmark Splits
8.4%
Hybrid / Custom / Cross-Validation
4.1%
Table 5.1: Overall distribution of splitting families across the 1497 surveyed paper–dataset pairs.
Category
Rand. HO
Rand. LOO
Temp. LOO
Temp. HO
Fix./Inh. + Hyb./CV
ND
Collaborative
0.45
0.35
0.05
0.02
0.08
0.05
Sequential
0.05
0.05
0.75
0.10
0.03
0.02
Graph
0.28
0.08
0.02
0.25
0.30
0.07
Session
0.05
0.00
0.05
0.65
0.15
0.10
Cross-domain
0.55
0.20
0.00
0.15
0.08
0.02
Federated
0.40
0.35
0.00
0.05
0.15
0.05
Table 5.2: Row-wise distribution of splitting families across recommendation paradigms. The combined column Fix./Inh. + Hyb./CV groups fixed/inherited benchmark splits with hybrid/custom/cross-validation protocols; ND indicates protocols that are not declared or cannot be reliably reconstructed.
Figure 5.1 : Aggregate evolution of splitting families over time, measured over paper–dataset pairs. Random protocols dominate the earlier years of the corpus, whereas temporal protocols become more prevalent with the growth of sequential and session-based recommendation.
Offline evaluation of recommender systems is often affected by hidden, under-documented choices in data preparation. Seemingly minor decisions in filtering, handling repeats, cold-start treatment, and splitting strategy design can substantially reorder model rankings and undermine reproducibility and cross-paper comparability. In this paper, we introduce SplitLight, an open-source exploratory toolkit that enables researchers and practitioners designing preprocessing and splitting pipelines or reviewing external artifacts to make these decisions measurable, comparable, and reportable. Given an interaction log and derived split subsets, SplitLight analyzes core and temporal dataset statistics, characterizes repeat consumption patterns and timestamp anomalies, and diagnoses split validity, including temporal leakage, cold-user/item exposure, and distribution shifts. SplitLight further allows side-by-side comparison of alternative splitting strategies through comprehensive aggregated summaries and interactive visualizations. Delivered as both a Python toolkit and an interactive no-code interface, SplitLight produces audit summaries that justify evaluation protocols and support transparent, reliable, and comparable experimentation in recommender systems research and industry.
Anna Volodkevich, Dmitry Anikin, Danil Gusak +3
SB AI Lab, AI Center · SB AI Lab, Applied AI Institute · AXXX, AI Center +3
In recommender systems, collecting, storing, and processing large-scale interaction data is increasingly costly in terms of time, energy, and computation, yet it remains unclear when additional data stops providing meaningful gains. This paper investigates how offline recommendation performance evolves as the size of the training dataset increases and whether a saturation point can be observed. We implemented a reproducible Python evaluation workflow with two established toolkits, LensKit and RecBole, included 11 large public datasets with at least 7 million interactions, and evaluated 10 tool-algorithm combinations. Using absolute stratified user sampling, we trained models on nine sample sizes from 100,000 to 100,000,000 interactions and measured NDCG@10. Overall, raw NDCG usually increased with sample size, with no observable saturation point. To make result groups comparable, we applied min-max normalization within each group, revealing a clear positive trend in which around 75% of the points at the largest completed sample size also achieved the group's best observed performance. A late-stage slope analysis over the final 10-30% of each group further supported this upward trend: the interquartile range remained entirely non-negative with a median near 1.0. In summary, for traditional recommender systems on typical user-item interaction data, incorporating more training data remains primarily beneficial, while weaker scaling behavior is concentrated in atypical dataset cases and in the algorithmic outlier RecBole BPR under our setup.
Understanding the performance of large-scale recommender systems remains an underexplored challenge, especially for content creators and model developers. The raw engagement signals available to them, such as views and clicks, conflate content quality, model behavior, presentation bias, and audience reach, making it hard to attribute outcomes to the right cause. In this work, we present a general evaluation framework that enhances observability across multiple recommender systems at Netflix and demonstrate its effectiveness through several production deployments. The framework treats recommender-system observability as a counterfactual measurement problem: estimating what the recommender would have done, and what engagement would have followed, in the absence of a specific content item or model decision. We articulate three stakeholder-centered observability principles for content creators and model developers, and propose measurement methodologies covering bias reduction, relativity, and incrementality, applicable to both single-stage and cascading recommender systems and serving both audiences from a single measurement foundation.
Chaoran Guo, Ding Tong, Ting-Po Lee +1
Netflix Los Gatos, California, USA · San Jose, California, USA