Improving recommendation algorithms for the Web requires coordinating interacting design choices and learning from experiments. We present RecRSI, an LLM-based research framework that uses mechanism hypotheses to organize algorithm discovery. Each hypothesis links a proposed mechanism change to a reference design, a rationale, and an expected observation. MechSpace derives implementable primitives and composition rules from the task source; Research Line uses this space and prior evidence to develop and test hypotheses; Mechanism Feedback carries outcomes and their design context into subsequent research. This organization makes each experiment inform which mechanisms to develop, revise, or combine next. Across four initial recommenders and four datasets, RecRSI yields average relative improvements of 8.48% in NDCG@5 and 6.88% in HR@5 over the initial algorithms, and 2.60% and 2.20%, respectively, over the strongest competing result for each setting and metric. Component and process studies link these gains to coordinated mechanism choices and evidence-guided follow-up. Our code is publicly available at https://github.com/DynamisG210/RecRSI.
Figures & tables
Figure 1. Research organization and algorithm discovery. (a) Candidate-level evolution organizes iteration around programs; RecRSI uses mechanism hypotheses to connect proposed changes, their rationale, and experimental evidence for the next design. Both workflows generate, evaluate, and refine algorithms. (b) Mean best-so-far validation NDCG@5 gain over the initial recommender on the same eight tasks as Figure 3 (a,b). Search repeats are averaged within each task, then tasks receive equal weight. The left panel contrasts candidate-level evolution with RecRSI. MechSpace supplies interventions for a mechanism hypothesis, which connects a change, rationale, and expected outcome. Evaluation produces hypothesis-linked evidence that guides the next hypothesis. The right panel compares eight-task algorithm discovery progress for RecRSI and five systems, using the mean relative gain in best-so-far validation NDCG at rank five. Repeats are averaged within tasks before equally weighting tasks.
Figure 2. Overview of RecRSI. MechSpace supplies the design space for mechanism hypotheses; Research Line develops them into candidates and experiments; Mechanism Feedback links outcomes to the tested hypotheses and designs. The shared research state carries this evidence into subsequent investigations. Task source and interfaces enter MechSpace. Four research roles develop mechanism hypotheses for selection, implementation, and evaluation. Feedback links observations to the tested hypotheses and reference designs. Research records, open questions, and the current best form the shared state used in subsequent rounds.
Seed Recommender
Method
CDs
Electronics
Office
MovieLens-1M
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
SASRec
Seed
0.3489
0.4572
0.3623
0.4860
0.2674
0.3980
0.5613
0.7176
RecHarness
0.3954
0.5126
0.3705
0.4947
0.2754
0.4080
0.5687
0.7241
DeepEvolve
0.3510
0.4605
0.3678
0.4932
0.2795
0.4099
0.5340
0.6917
OpenEvolve
0.3645
0.4841
0.3674
0.4920
0.2771
0.4139
0.5610
0.7188
ShinkaEvolve
0.3641
0.4841
0.3629
0.4871
0.2743
0.4100
0.5647
0.7212
Table 1. Final test performance across four seed recommenders and four datasets.
Figure 3. Discovery quality and search cost. (a,b) Eight-task final test NDCG@5 gain over Seed versus API reference cost (USD) and per GPU-hour. (c) Best validation gain at a common input-plus-output token ceiling on the same eight tasks; at 3 million tokens, RecRSI reaches 7.72% versus 5.09% for RecHarness. Repeats are averaged within tasks, then tasks equally. Search costs exclude final retraining. Two panels compare final test gain with search API reference cost in USD and GPU-hours. RecRSI lies on the quality–API-cost Pareto frontier, with the highest mean test gain of 7.36 percent versus 4.68 percent for RecHarness, at 2.38 dollars more API cost per search. Its gain per GPU-hour is 1.9603. Panel c plots best validation NDCG at 5 gain against cumulative input plus output tokens from zero to three million for the same eight tasks, using the same adopted search repeats. At three million tokens, RecRSI reaches 7.7228 percent and RecHarness 5.0857 percent.
Method
Mean frontier gain (%) ↑
Change vs. Full (pp)
Full (RecRSI)
5.6775
–
Prompt
5.4591
−0.2184
Unified
5.6387
−0.0388
Summary
5.1927
−0.4848
Prompt + Summary
5.5807
−0.0968
Table 2. Component contributions and space–feedback interaction on MoRec across four datasets.
Figure 4. Overlaid research progress on MoRec/CDs. Colors distinguish three experiment blocks, with solid Continue and dashed Restart trajectories. At 2k total evaluations, Restart takes the better of the two independent searches after k evaluations each. The divider marks evaluation 12; filled circles mark later bests, including the Block 3 scoring hypothesis. Table 3 reports final quality and API cost. A single plot overlays six step curves of best-so-far development NDCG at 5, sharing one total-evaluation axis. Colors denote Blocks 1–3 and solid/dashed lines denote continued research and independent restarts. After evaluation 12, Continue finds new bests at evaluations 13 and 15 in Block 1, 14 in Block 2, and 13 in Block 3. Block 3 combines the existing fusion and margin objective with cosine scoring.
Block
Continue (24)
Best restart ( 2×12 )
Gain after eval. 12
Continue − restart
1
0.4237
0.4195
+0.0057
+0.0043
2
0.4299
0.4299
+0.0043
≈0
3
0.4153
0.3884
+0.0352
+0.0269
Table 3. Final discovery quality and continuation gains at 24 total evaluations.
State
Raw
Summary
Mechanism
1
0.3647
0.3729
0.3767
2
0.3647
0.3729
0.3729
3
0.4123
0.4135
0.4150
4
0.3729
0.3729
0.3729
5
0.3681
0.3673
0.4037
6
0.3729
0.3729
0.3755
Table 4. Next-candidate quality in ten feedback comparisons at frozen MoRec/CDs decision points.
Space
NDCG@5
HR@5
A: MechSpace
0.4236±0.0036
0.5572±0.0062
H: Human-designed
0.3540±0.0051
0.4660±0.0053
Table 5. Final test performance of the validation-selected candidates from the automatically generated (A) and human-designed (H) mechanism spaces on SASRec/CDs. Results are reported as mean ± sample standard deviation.
Table 6. Mechanism combinations and development-set gains in all eight BERT4Rec/Electronics searches.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Seed Recommender
Method
CDs
Electronics
Office
MovieLens-1M
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
SASRec
RecHarness
0.0051
0.0114
0.0288
0.0335
0.0034
0.0047
0.0045
0.0039
DeepEvolve
0.0286
0.0356
0.0012
0.0019
0.0082
0.0154
0.0571
0.0538
OpenEvolve
0.0065
0.0074
0.0020
0.0035
0.0019
0.0011
0.0015
0.0034
ShinkaEvolve
0.0081
0.0142
0.0064
0.0057
0.0039
0.0084
0.0038
0.0052
Self-EvolveRec
0.0047
0.0105
0.0004
0.0010
0.0050
0.0087
0.0048
0.0048
Appendix
Table 7. Search-to-search standard deviation of final TEST NDCG@5 and HR@5, aligned with Table 1 .
NDCG@5
HR@5
Comparator
Δ
W/T/L
Δ
W/T/L
RecHarness
+5.52
13/0/3
+4.24
14/0/2
DeepEvolve
+5.70
15/0/1
+4.83
16/0/0
OpenEvolve
+4.38
15/0/1
+3.50
14/0/2
ShinkaEvolve
+4.31
15/0/1
+3.45
14/0/2
Self-EvolveRec
+7.23
15/0/1
+6.12
14/0/2
Appendix
Table 8. RecRSI versus each comparator on the 16 settings. Δ is mean relative gain (%); W/T/L counts wins, ties, and losses based on mean test scores.
Task
Split / evaluation history
Ranking protocol
SASRec / all four datasets
Per-user last-two holdout; test histories include training and validation interactions.
One positive plus 99 fixed negatives, evaluation seed 72026.
BERT4Rec/Electronics, Office
Amazon 2014 5-core; penultimate interaction for development, last for test.
Fixed uniform sampled pools.
BERT4Rec/MovieLens-1M
Per-user temporal split of the full MovieLens-1M data.
Fixed sampled pools; test negatives exclude train/dev and the test positive.
BERT4Rec/CDs
Common preprocessed data and user/item IDs across systems.
Fixed sampled candidate sets.
MoRec/CDs, Electronics
Common CDs preprocessing across systems; Electronics uses per-user last-two holdout. Test histories include training and validation interactions.
Fixed sampled candidate sets shared across systems.
MoRec/Office, MovieLens-1M
Per-user timestamp order (source-row tie-break); penultimate interaction for development, last for test. Test history: train + development.
One positive and 99 fixed uniform negatives; seed 72026. Negatives exclude the query history and target.
Appendix
Table 9. Task data splits and ranking protocols. Sampled pools contain one positive and 99 fixed negatives; all candidates within a task share the same evaluation protocol.
System
Task
R1
R2
R3
DeepEvolve
MoRec/CDs
2
6
0
MoRec/Office
0
1
–
MoRec/MovieLens-1M
11
–
–
BERT4Rec/Electronics
–
10
11
Self-EvolveRec
MoRec/CDs
2
–
10
MoRec/MovieLens-1M
–
–
1
Appendix
Table 10. Completed evaluations in searches stopped by the API safeguard. R1–R3 are search repeats; – marks a repeat not stopped by this safeguard.
School of Mathematics and Computing (Computational Science and Engineering), Yonsei University Seoul, Republic of Korea · School of Computer Science and Engineering, Kyungpook National University Daegu, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University Suwon, Republic of Korea +1