SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splits

Anna Volodkevich; Dmitry Anikin; Danil Gusak; Anton Klenitskiy; Evgeny Frolov; Alexey Vasilev

cs.IRFeb 22, 2026

SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splits

Authors: Anna Volodkevich, Dmitry Anikin, Danil Gusak, Anton Klenitskiy, Evgeny Frolov, Alexey Vasilev

Organizations: SB AI Lab, AI Center · SB AI Lab, Applied AI Institute · AXXX, AI Center · SB AI Lab ·

Abstract

Offline evaluation of recommender systems is often affected by hidden, under-documented choices in data preparation. Seemingly minor decisions in filtering, handling repeats, cold-start treatment, and splitting strategy design can substantially reorder model rankings and undermine reproducibility and cross-paper comparability. In this paper, we introduce SplitLight, an open-source exploratory toolkit that enables researchers and practitioners designing preprocessing and splitting pipelines or reviewing external artifacts to make these decisions measurable, comparable, and reportable. Given an interaction log and derived split subsets, SplitLight analyzes core and temporal dataset statistics, characterizes repeat consumption patterns and timestamp anomalies, and diagnoses split validity, including temporal leakage, cold-user/item exposure, and distribution shifts. SplitLight further allows side-by-side comparison of alternative splitting strategies through comprehensive aggregated summaries and interactive visualizations. Delivered as both a Python toolkit and an interactive no-code interface, SplitLight produces audit summaries that justify evaluation protocols and support transparent, reliable, and comparable experimentation in recommender systems research and industry.

Figures & tables

Figure 1. SplitLight in a data-preparation pipeline. From the raw dataset to split subsets, SplitLight audits data characteristics, flags problems, and enables side-by-side comparison of alternative splitting strategies to justify the selected evaluation protocol.

Figure 2. Global temporal split structure for (a) last-item and (b) all-items target definitions. Evaluation inputs include all within-user history before evaluation targets.

Figure 3. Analysis of cold items in test target subset for Diginetica dataset. Share of interactions with cold items grows in time from nearly 20% to 40%, resulting in 34.35% overall.

Figure 4. Temporal properties of the data in case of the leave-one-out split: dataset timeframe, train/evaluation durations and overlap, and per-user lifetime coverage.

Figure 5. Repeat consumption illustration: (a) repeated interactions (after first occurrence of an item) and (b) consecutive repeats (same item in immediate succession).

Figure 6. Temporal skew in ML-1M interactions. Test target subset spans nearly 76% of dataset timeframe under GTS $q_{0.9}$ .

Figure 7. Data leakage in Diginetica dataset (LOO): training and evaluation subsets are 100% overlapped, 89% targets demonstrate data leakage.

Visualization Type	Core & Temporal	Interactions Over Time	Repeat Consumption	Temporal Leakage	Cold Start	Time Deltas	Item Positions
Cards	✓	✓	✓	✓	✓	✓	✓
Tabular	✓		✓	✓	✓	✓	✓
Histogram	✓		✓			✓	✓
Timeline		✓		✓	✓

Table 1. Core SplitLight components and visualization types

Figure 8. Example of SplitLight summary dashboard showing an at-a-glance health check of the dataset and generated splits, with links to detailed diagnostics.

Dataset	#Users	#Items	#Interact.	Avg. Len.	Density (%)	Consec. Repeats (%)	Repeated Interact. (%)
Beauty ( McAuley et al., 2015 )	22 363	12 101	198 502	8.88	0.07	0.00	0.00
Diginetica ( Garg et al., 2024 )	310 324	122 993	1 235 380	3.98	0.003	5.85	13.62
Dressipi ( Landia et al., 2022 )	1 000 000	23 496	4 743 820	4.74	0.02	7.25	13.39
ML-1M ( Harper and Konstan, 2015 )	6 040	3 706	1 000 209	165.60	4.47	0.00	0.00
ML-20M ( Harper and Konstan, 2015 )	138 493	26 744	20 000 263	144.41	0.54	0.00	0.00
Zvuk ( Shevchenko et al., 2024 )	20 000	391 322	10 867 482	543.37	0.14	21.50	68.17

Table 2. Statistics of the raw datasets

Dataset	End Date	Length (years)	Mean $\Delta t$	Median $\Delta t$	Mean User Lt.	Median User Lt.	Ts. Collis. (%)
Beauty	2014-07-23	12.11	69.65 d	4.00 d	1.50 y	1.00 y	38.9931
Diginetica	2016-06-01	0.42	1.79 m	1.02 m	5.34 m	2.86 m	0.0003
Dressipi	2021-05-31	1.42	10.02 m	45.57 s	37.50 m	1.75 m	0.0139
ML-1M	2003-02-28	2.84	13.85 h	0.00 s	94.99 d	0.05 d	52.8935
ML-20M	2015-03-31	20.22	1.37 d	11.00 s	196.59 d	0.04 d	20.9617
Zvuk	2023-04-16	0.25	1.72 h	0.25 s	38.76 d	32.29 d	0.2195

Table 3. Temporal dataset statistics. Definitions:

\Delta t

– time between user interactions (d = days, h = hours, m = minutes, s = seconds), Lt. – lifetime, Ts. Collis. – timestamp collisions

	LOO	Shuffle Collis., LOO		GTS $q_{0.9}$	Keep Consec. Repeats, GTS $q_{0.9}$			Keep Cold, GTS $q_{0.9}$
Dataset	N@10	Collis. (%)	N@10 (shuffled)	N@10	Consec. Repeats (%)	N@10	N@10 (non- consec. targets)	Cold Targets (%)	N@10
Beauty	0.0566	38.99	0.0521	0.0348	—	—	—	22.32	0.0280
Diginetica	0.1663	$<1$	—	0.1443	5.85	0.2118	0.1364	1.50	0.1427
Dressipi	0.1318	$<1$	—	0.1343	7.25	0.2168	0.1225	32.59	0.0965
ML-1M	0.1861	52.89	0.1685	0.0905	—	—	—	0.50	0.0907
ML-20M	0.1601	20.96	0.1469	0.0836	—	—	—	28.40	0.0704

Table 4. Impact of different data curation protocols on RS evaluation results

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 514 papers

Data Processing for Offline Evaluation in Recommender Systems: a Survey
Sep 18, 2026Alberto Carlo Maria Mancino, Angela Di Fazio, Danilo Danese +5Recommender Systems
Politecnico di Bari · Sapienza Università di Roma · LUISS Guido Carli University
SPADE: Escaping the Popularity-Similarity Frontier to Measure Serendipitous Recommendations
Sep 25, 2026Tobias Vente, Maarten Peirsman, Noah Daniëls +2Recommender Systems
ADREM Data Lab, University of Antwerp, Middelheimlaan 1, 2020 Antwerpen, Belgium · Department of Computer Science, PO Box 68, FI-00014 University of Helsinki, Finland · Froomle, Post X, Borsbeeksebrug 34, 2600 Antwerpen, Belgium
From Top-1 to Top-K: A Reproducibility Study and Benchmarking of Counterfactual Explanations for Recommender Systems
Apr 21, 2026Quang-Huy Nguyen, Thanh-Hai Nguyen, Khac-Manh Thai +6Counterfactual Evaluation Counterfactual Explanations
VNU University of Engineering and Technology · Delft University of Technology

Sep 18, 2026J/K move · Enter open · S save

Data Processing for Offline Evaluation in Recommender Systems: a Survey

Alberto Carlo Maria Mancino, Angela Di Fazio, Danilo Danese, Matteo Attimonelli+4

Politecnico di Bari · Sapienza Università di Roma · LUISS Guido Carli University

Recommender Systems

Open paper AskPDF arXiv

Offline evaluation is the dominant experimental paradigm in recommender systems research, enabling reproducible and cost-effective comparisons on historical interaction data. Yet, while considerable attention has been devoted to recommendation models and evaluation methodologies, the data processing decisions that precede model training have received less scrutiny. These decisions determine the information available to recommendation algorithms and can affect the comparability and reproducibility of experimental results. This survey provides a systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems. We examine the data-centric pipeline, from dataset selection and interaction representation to data preparation, multimodal feature extraction, and train-validation-test splitting. Our analysis spans recommendation paradigms, including collaborative, sequential, session-based, graph-based, knowledge-aware, context-aware, multimodal, federated, cross-domain, contrastive-learning, and LLM-based recommendation. Beyond reviewing existing practices, we introduce a unified framework and taxonomy for describing data transformations and feature-extraction strategies, distinguishing data preparation from the extraction of representations from multimodal side information. Our empirical analysis reveals a landscape dominated by a narrow set of dataset-level transformations, particularly support-driven filtering, while representation-dependent transformations remain less common. We further identify substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols, where similar labels may conceal different experimental conditions.

Abstract

Recommender Systems

Explore similar work

Data Processing for Offline Evaluation in Recommender Systems: a Survey

SPADE: Escaping the Popularity-Similarity Frontier to Measure Serendipitous Recommendations

From Top-1 to Top-K: A Reproducibility Study and Benchmarking of Counterfactual Explanations for Recommender Systems