cs.IRSep 18, 2026
SaveData Processing for Offline Evaluation in Recommender Systems: a Survey
Organizations: Politecnico di Bari · Sapienza Università di Roma · LUISS Guido Carli University
Abstract
Offline evaluation is the dominant experimental paradigm in recommender systems research, enabling reproducible and cost-effective comparisons on historical interaction data. Yet, while considerable attention has been devoted to recommendation models and evaluation methodologies, the data processing decisions that precede model training have received less scrutiny. These decisions determine the information available to recommendation algorithms and can affect the comparability and reproducibility of experimental results. This survey provides a systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems. We examine the data-centric pipeline, from dataset selection and interaction representation to data preparation, multimodal feature extraction, and train-validation-test splitting. Our analysis spans recommendation paradigms, including collaborative, sequential, session-based, graph-based, knowledge-aware, context-aware, multimodal, federated, cross-domain, contrastive-learning, and LLM-based recommendation. Beyond reviewing existing practices, we introduce a unified framework and taxonomy for describing data transformations and feature-extraction strategies, distinguishing data preparation from the extraction of representations from multimodal side information. Our empirical analysis reveals a landscape dominated by a narrow set of dataset-level transformations, particularly support-driven filtering, while representation-dependent transformations remain less common. We further identify substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols, where similar labels may conceal different experimental conditions.
Figures & tables
Figure 1.1 : Distribution of papers across research topics and publication periods. Topics are ordered by temporal dominance, i.e., according to the period in which each topic becomes most prominent, from the earliest to the most recent interval.
Figure 1.2 : Distribution of papers across venue ranking categories.
Figure 3.1 : Taxonomy of recommendation datasets. Datasets can be characterized along three complementary dimensions: structural representations, information sources, and dataset organization.
Figure 3.2 : Most frequently used datasets in the analyzed literature. The figure shows the top 30 datasets ranked by the number of occurrences across publications, considering only datasets that appear in at least two studies. Different versions of the same dataset are treated as distinct entries, while variations due to preprocessing are not distinguished. The distribution reveals a strong concentration of empirical evaluation around a small number of widely reused benchmarks, which dominate the experimental landscape.
Figure 3.3 : Rank-frequency distribution of dataset usage in the analyzed literature. Datasets are ordered by decreasing frequency of occurrence across publications. The distribution exhibits a pronounced long-tail behavior: a small fraction of datasets is extensively reused (top 30), while a large portion appears only occasionally or in a single study. In particular, 46.1% of the datasets are used only once, highlighting a substantial degree of fragmentation in experimental practices.
| Domain | Occurrences | Distinct | Avg. Usage |
| Movies | 361 | 25 | 14.44 |
| E-commerce / Marketplace | 263 | 54 | 4.87 |
| Fashion | 138 | 14 | 9.86 |
| POI / Travel / Tourism | 125 | 31 | 4.03 |
| Music | 119 | 24 | 4.96 |
| Multi Domain | 110 | 7 | 15.71 |
Table 3.1 : Dataset usage across domains. “Occurrences” denotes the total number of dataset usages in the analyzed literature, while “Distinct” reports the number of unique datasets within each domain. “Avg. Usage” is defined as the ratio between occurrences and distinct datasets, capturing the average reuse of datasets within a domain. Domains are sorted by decreasing number of occurrences. Darker shades indicate higher values of Avg. Usage.
Figure 3.4 : Benchmark adoption across recommender systems research topics. Rows correspond to datasets (ordered by overall popularity) and columns to research topics. Cell colors represent the percentage of occurrences of each dataset within a topic, while numbers denote absolute occurrence counts. The visualization illustrates both the broad reuse of general-purpose benchmarks and the emergence of task-specific benchmark ecosystems.
| Origin | Count | Percentage |
| Academic | 156 | 59.5% |
| Industry | 72 | 27.5% |
| Academic Challenge | 23 | 8.8% |
| Industry Challenge | 11 | 4.2% |
Table 3.2 : Distribution of datasets by origin, distinguishing between academic, industrial, and challenge-based sources.
| Group | Category | Count | Percentage |
| Dataset availability | Link available | 186 | 70.2% |
| Not shared | 30 | 11.3% | |
| Proprietary | 47 | 17.7% | |
| Unknown | 2 | 0.8% | |
| Link quality | Working | 154 | 82.8% |
| Broken link | 23 | 12.4% |
Table 3.3 : Dataset availability and link quality.
| Dataset | Text | Image | Audio | Video | Reviews | KG |
| Alibaba iFashion | ✓ | ✓ | – | – | – | – |
| Allrecipes.com | ✓ | ✓ | – | – | – | – |
| Amadeus CME | – | – | – | – | ✓ | |
| Amazon Reviews* | ✓ | – | – | ✓ | ||
| Bing News | ✓ | – | – | – | – | ✓ |
| BookCrossing | ✓ | ✓ | – | – | – |
Table 3.4 : Representative benchmark datasets and their auxiliary information. Symbols denote how each modality is released: ✓ raw data; pre-computed representations; official external resources; external enrichments introduced by subsequent studies. Multiple symbols indicate multiple release strategies. Datasets marked with * aggregate closely related versions.
Figure 4.1 : Taxonomy of data preparation transformations.
| Family | Subfamily | Technique |
| Support-driven | Activity-based filtering | User filtering |
| Item filtering | ||
| Iterative -core filtering | ||
| User and item subsampling | User subsampling | |
| Item subsampling | ||
| Other | External-criterion filtering |
Table 4.1 : Overview of dataset-level transformation techniques associated with the corresponding taxonomy categories. Each technique refers to its detailed description in the text.
| Family | Subfamily | Technique |
| Sequential-based | Sequence transformations | Sequence truncation |
| Sequence-length filtering | ||
| Session-based | Session transformations | Session-length filtering |
| Graph-based | Connectivity-based transformations | Connectivity-based filtering |
| Context-based | Context transformations | Context selection |
| Context discretization |
Table 4.2 : Overview of representation-dependent transformation techniques associated with the corresponding taxonomy categories. Each technique refers to its detailed description in the text.
| Family | Encoder training cost | Per-study extraction cost | Experimenter control | Cross-study comparability | Reproducibility |
| Hand-crafted | None | Low | Full | Low | High |
| Frozen pre-trained | Amortized | Low | Medium | High | Medium |
| End-to-end trained | Per study | High | Full | Low | Low |
| Dataset-provided | By dataset | None | None | Exact | High |
Table 4.3: Typical practical trade-offs across the four families of feature extractors. Individual implementations may deviate from these general characteristics.
| Modality | Stage | Technique |
| Textual | Data preparation | Text-field selection and combination |
| Text normalization and cleaning | ||
| Tokenization, truncation, and padding | ||
| Feature extraction | Sparse lexical representations | |
| Pre-trained language-model representations | ||
| Task-specific or dataset-provided representations |
Table 4.4 : Overview of modality-specific side-information transformations. Each technique refers to its detailed description in the text.
Figure 4.2 : Distribution of the top 10 data transformation techniques applied to interaction logs. Occurrences are measured at the (paper, dataset) level to capture the specific preprocessing pipeline applied to each dataset within a study. Percentages indicate the relative prevalence of each technique across all annotated instances.
Figure 4.3 : Co-occurrence heatmap of the top 15 data transformation techniques. Each cell indicates the number of times two distinct operations were applied to the same (paper, dataset) instance. The diagonal is intentionally left blank as it represents single-technique occurrences. The high values at the intersection of user and item k-core (429 instances) and binarisation (over 100 instances each) highlight the prevalence of a standardised data preparation pipeline in the literature.
Figure 4.4 : Structural distribution of transformation types using a dual-scale visualisation. The top panel shows the global prevalence of Dataset-level transformations (0-100% scale), while the bottom panel provides a zoomed perspective (0-14% scale) to illustrate the distribution of Representation-dependent transformations. All percentages represent the global frequency of each sub-category across the entire surveyed corpus (Section 1.2 ).
Figure 4.5 : Distribution of the number of data preparation techniques applied per paper. The majority of studies employ a minimalist approach, with a mean of 2.22 operations, typically focusing on the standard -core and binarisation pipeline.
| Extractor category | Visual | Textual | Other (Audio/KG) |
| (a) Hand-crafted | 0 | 3 | 0 |
| (b) Frozen pre-trained, uni-modal | 71 | 82 | 12 |
| (b) Frozen pre-trained, cross-modal | 2 | 0 | 0 |
| (c) End-to-end trained | 9 | 14 | 5 |
| (d) Dataset-shipped pre-computed | 28 | 11 | 2 |
| Total declared | 110 | 110 | 19 |
Table 4.5: Distribution of disclosed feature extractors by category. Counts refer to modality-specific paper–dataset observations. For visual and textual information, “N/A” denotes that the modality is absent. Video inputs are included under visual information when processed as frames, and under “Other” when their audio stream is processed. In the “Other” column, pairs without a reported audio or knowledge-graph extractor, combine cases in which the modality is absent and cases in which its extractor is not disclosed.
Figure 4.6 : Distribution of reported visual extractors assigned to family (b), over paper–dataset observations. CLIP (darker bar) is the only cross-modal entry; all other extractors are unimodal. “Pretrained CNN” aggregates pairs in which the paper declares the use of a pre-trained CNN without specifying the architecture.
Figure 4.7 : Distribution of reported textual extractors assigned to family (b), over paper–dataset observations. “Word2Vec/GloVe” aggregates pairs in which the paper declares a word-embedding pipeline (Skip-gram, gensim, or initialised GloVe) without further specification.
| Splitting Family | Prevalence (%) |
| Random (Global & User-wise Hold-out) | |
| Random Leave-One-Out | |
| Temporal Leave-One-Out | |
| Temporal Hold-out (Global & User-wise) | |
| Fixed / Inherited Benchmark Splits | |
| Hybrid / Custom / Cross-Validation |
Table 5.1: Overall distribution of splitting families across the surveyed paper–dataset pairs.
| Category | Rand. HO | Rand. LOO | Temp. LOO | Temp. HO | Fix./Inh. + Hyb./CV | ND |
| Collaborative | 0.45 | 0.35 | 0.05 | 0.02 | 0.08 | 0.05 |
| Sequential | 0.05 | 0.05 | 0.75 | 0.10 | 0.03 | 0.02 |
| Graph | 0.28 | 0.08 | 0.02 | 0.25 | 0.30 | 0.07 |
| Session | 0.05 | 0.00 | 0.05 | 0.65 | 0.15 | 0.10 |
| Cross-domain | 0.55 | 0.20 | 0.00 | 0.15 | 0.08 | 0.02 |
| Federated | 0.40 | 0.35 | 0.00 | 0.05 | 0.15 | 0.05 |
Table 5.2: Row-wise distribution of splitting families across recommendation paradigms. The combined column Fix./Inh. + Hyb./CV groups fixed/inherited benchmark splits with hybrid/custom/cross-validation protocols; ND indicates protocols that are not declared or cannot be reliably reconstructed.
Figure 5.1 : Aggregate evolution of splitting families over time, measured over paper–dataset pairs. Random protocols dominate the earlier years of the corpus, whereas temporal protocols become more prevalent with the growth of sequential and session-based recommendation.
More in this topic
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 514 papers