Improving recommendation algorithms for the Web requires coordinating interacting design choices and learning from experiments. We present RecRSI, an LLM-based research framework that uses mechanism hypotheses to organize algorithm discovery. Each hypothesis links a proposed mechanism change to a reference design, a rationale, and an expected observation. MechSpace derives implementable primitives and composition rules from the task source; Research Line uses this space and prior evidence to develop and test hypotheses; Mechanism Feedback carries outcomes and their design context into subsequent research. This organization makes each experiment inform which mechanisms to develop, revise, or combine next. Across four initial recommenders and four datasets, RecRSI yields average relative improvements of 8.48% in NDCG@5 and 6.88% in HR@5 over the initial algorithms, and 2.60% and 2.20%, respectively, over the strongest competing result for each setting and metric. Component and process studies link these gains to coordinated mechanism choices and evidence-guided follow-up. Our code is publicly available at https://github.com/DynamisG210/RecRSI.
Figures & tables
Figure 1. Research organization and algorithm discovery. (a) Candidate-level evolution organizes iteration around programs; RecRSI uses mechanism hypotheses to connect proposed changes, their rationale, and experimental evidence for the next design. Both workflows generate, evaluate, and refine algorithms. (b) Mean best-so-far validation NDCG@5 gain over the initial recommender on the same eight tasks as Figure 3 (a,b). Search repeats are averaged within each task, then tasks receive equal weight. The left panel contrasts candidate-level evolution with RecRSI. MechSpace supplies interventions for a mechanism hypothesis, which connects a change, rationale, and expected outcome. Evaluation produces hypothesis-linked evidence that guides the next hypothesis. The right panel compares eight-task algorithm discovery progress for RecRSI and five systems, using the mean relative gain in best-so-far validation NDCG at rank five. Repeats are averaged within tasks before equally weighting tasks.
Figure 2. Overview of RecRSI. MechSpace supplies the design space for mechanism hypotheses; Research Line develops them into candidates and experiments; Mechanism Feedback links outcomes to the tested hypotheses and designs. The shared research state carries this evidence into subsequent investigations. Task source and interfaces enter MechSpace. Four research roles develop mechanism hypotheses for selection, implementation, and evaluation. Feedback links observations to the tested hypotheses and reference designs. Research records, open questions, and the current best form the shared state used in subsequent rounds.
Seed Recommender
Method
CDs
Electronics
Office
MovieLens-1M
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
SASRec
Seed
0.3489
0.4572
0.3623
0.4860
0.2674
0.3980
0.5613
0.7176
RecHarness
0.3954
0.5126
0.3705
0.4947
0.2754
0.4080
0.5687
0.7241
DeepEvolve
0.3510
0.4605
0.3678
0.4932
0.2795
0.4099
0.5340
0.6917
OpenEvolve
0.3645
0.4841
0.3674
0.4920
0.2771
0.4139
0.5610
0.7188
ShinkaEvolve
0.3641
0.4841
0.3629
0.4871
0.2743
0.4100
0.5647
0.7212
Table 1. Final test performance across four seed recommenders and four datasets.
Figure 3. Discovery quality and search cost. (a,b) Eight-task final test NDCG@5 gain over Seed versus API reference cost (USD) and per GPU-hour. (c) Best validation gain at a common input-plus-output token ceiling on the same eight tasks; at 3 million tokens, RecRSI reaches 7.72% versus 5.09% for RecHarness. Repeats are averaged within tasks, then tasks equally. Search costs exclude final retraining. Two panels compare final test gain with search API reference cost in USD and GPU-hours. RecRSI lies on the quality–API-cost Pareto frontier, with the highest mean test gain of 7.36 percent versus 4.68 percent for RecHarness, at 2.38 dollars more API cost per search. Its gain per GPU-hour is 1.9603. Panel c plots best validation NDCG at 5 gain against cumulative input plus output tokens from zero to three million for the same eight tasks, using the same adopted search repeats. At three million tokens, RecRSI reaches 7.7228 percent and RecHarness 5.0857 percent.
Method
Mean frontier gain (%) ↑
Change vs. Full (pp)
Full (RecRSI)
5.6775
–
Prompt
5.4591
−0.2184
Unified
5.6387
−0.0388
Summary
5.1927
−0.4848
Prompt + Summary
5.5807
−0.0968
Table 2. Component contributions and space–feedback interaction on MoRec across four datasets.
Figure 4. Overlaid research progress on MoRec/CDs. Colors distinguish three experiment blocks, with solid Continue and dashed Restart trajectories. At 2k total evaluations, Restart takes the better of the two independent searches after k evaluations each. The divider marks evaluation 12; filled circles mark later bests, including the Block 3 scoring hypothesis. Table 3 reports final quality and API cost. A single plot overlays six step curves of best-so-far development NDCG at 5, sharing one total-evaluation axis. Colors denote Blocks 1–3 and solid/dashed lines denote continued research and independent restarts. After evaluation 12, Continue finds new bests at evaluations 13 and 15 in Block 1, 14 in Block 2, and 13 in Block 3. Block 3 combines the existing fusion and margin objective with cosine scoring.
Block
Continue (24)
Best restart ( 2×12 )
Gain after eval. 12
Continue − restart
1
0.4237
0.4195
+0.0057
+0.0043
2
0.4299
0.4299
+0.0043
≈0
3
0.4153
0.3884
+0.0352
+0.0269
Table 3. Final discovery quality and continuation gains at 24 total evaluations.
State
Raw
Summary
Mechanism
1
0.3647
0.3729
0.3767
2
0.3647
0.3729
0.3729
3
0.4123
0.4135
0.4150
4
0.3729
0.3729
0.3729
5
0.3681
0.3673
0.4037
6
0.3729
0.3729
0.3755
Table 4. Next-candidate quality in ten feedback comparisons at frozen MoRec/CDs decision points.
Space
NDCG@5
HR@5
A: MechSpace
0.4236±0.0036
0.5572±0.0062
H: Human-designed
0.3540±0.0051
0.4660±0.0053
Table 5. Final test performance of the validation-selected candidates from the automatically generated (A) and human-designed (H) mechanism spaces on SASRec/CDs. Results are reported as mean ± sample standard deviation.
Table 6. Mechanism combinations and development-set gains in all eight BERT4Rec/Electronics searches.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Seed Recommender
Method
CDs
Electronics
Office
MovieLens-1M
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
NDCG@5
HR@5
SASRec
RecHarness
0.0051
0.0114
0.0288
0.0335
0.0034
0.0047
0.0045
0.0039
DeepEvolve
0.0286
0.0356
0.0012
0.0019
0.0082
0.0154
0.0571
0.0538
OpenEvolve
0.0065
0.0074
0.0020
0.0035
0.0019
0.0011
0.0015
0.0034
ShinkaEvolve
0.0081
0.0142
0.0064
0.0057
0.0039
0.0084
0.0038
0.0052
Self-EvolveRec
0.0047
0.0105
0.0004
0.0010
0.0050
0.0087
0.0048
0.0048
Appendix
Table 7. Search-to-search standard deviation of final TEST NDCG@5 and HR@5, aligned with Table 1 .
NDCG@5
HR@5
Comparator
Δ
W/T/L
Δ
W/T/L
RecHarness
+5.52
13/0/3
+4.24
14/0/2
DeepEvolve
+5.70
15/0/1
+4.83
16/0/0
OpenEvolve
+4.38
15/0/1
+3.50
14/0/2
ShinkaEvolve
+4.31
15/0/1
+3.45
14/0/2
Self-EvolveRec
+7.23
15/0/1
+6.12
14/0/2
Appendix
Table 8. RecRSI versus each comparator on the 16 settings. Δ is mean relative gain (%); W/T/L counts wins, ties, and losses based on mean test scores.
Task
Split / evaluation history
Ranking protocol
SASRec / all four datasets
Per-user last-two holdout; test histories include training and validation interactions.
One positive plus 99 fixed negatives, evaluation seed 72026.
BERT4Rec/Electronics, Office
Amazon 2014 5-core; penultimate interaction for development, last for test.
Fixed uniform sampled pools.
BERT4Rec/MovieLens-1M
Per-user temporal split of the full MovieLens-1M data.
Fixed sampled pools; test negatives exclude train/dev and the test positive.
BERT4Rec/CDs
Common preprocessed data and user/item IDs across systems.
Fixed sampled candidate sets.
MoRec/CDs, Electronics
Common CDs preprocessing across systems; Electronics uses per-user last-two holdout. Test histories include training and validation interactions.
Fixed sampled candidate sets shared across systems.
MoRec/Office, MovieLens-1M
Per-user timestamp order (source-row tie-break); penultimate interaction for development, last for test. Test history: train + development.
One positive and 99 fixed uniform negatives; seed 72026. Negatives exclude the query history and target.
Appendix
Table 9. Task data splits and ranking protocols. Sampled pools contain one positive and 99 fixed negatives; all candidates within a task share the same evaluation protocol.
System
Task
R1
R2
R3
DeepEvolve
MoRec/CDs
2
6
0
MoRec/Office
0
1
–
MoRec/MovieLens-1M
11
–
–
BERT4Rec/Electronics
–
10
11
Self-EvolveRec
MoRec/CDs
2
–
10
MoRec/MovieLens-1M
–
–
1
Appendix
Table 10. Completed evaluations in searches stopped by the API safeguard. R1–R3 are search repeats; – marks a repeat not stopped by this safeguard.
Optimizing recommender models remains heavily reliant on manual iteration over architectures, objectives, and training strategies. While LLM-based agents can automate this trial-and-error process, relying on LLMs to both select modification directions and formulate concrete hypotheses can lead to unstable search under limited trial budgets. Inspired by the above challenge, we propose RecHarness, a Bandit-Routed Agentic Harness for automated recommender model optimization. RecHarness decouples the optimization process into two steps: a bandit router selects the next modification direction according to historical validation feedback, while the LLM generates a concrete optimization hypothesis and executable code edit within the selected direction. To sustain long-horizon exploration, RecHarness uses a jump-basin mechanism to activate a structural-jump arm when local edits stagnate. Across multiple recommendation tasks, datasets, and model backbones, RecHarness achieves more consistent performance improvements and uses limited trial budgets more effectively than LLM-directed search. During a 7-day online A/B test on a large-scale short-video advertising platform, the selected candidate improves ADVV by 2.084%, Revenue by 0.534%, and Exposure by 0.559%. Code is available at: https://github.com/6lyc/RecHarness.
Haoran Ling, Yuecheng Li, Zeyu Song +6
Georgia Institute of Technology · Kuaishou Technology, China
Large Language Models (LLMs) have emerged as a promising paradigm for next-generation recommender systems, offering strong semantic understanding and natural-language reasoning abilities. Despite recent progress, current LLM-based recommenders still face key challenges in constructing decision-relevant contexts from heterogeneous evidence. First, existing methods often rely on fixed context construction strategies: collaborative behavioral evidence and item-side metadata are typically incorporated through predefined prompts, static retrieval pipelines, or handcrafted injection mechanisms, making it difficult to determine what information is truly beneficial for each instance. Second, heterogeneous evidence introduces a severe context-efficiency bottleneck. Rich metadata and collaborative interaction records can quickly overwhelm the context window, while aggressive compression or heuristic filtering may discard fine-grained evidence critical for accurate recommendation. To address these challenges, we propose RRCM, a ranking-driven retrieval-and-reasoning framework over collaborative and metadata memories for LLM-based agentic recommendation. RRCM starts from a lightweight user-history context and learns whether to recommend directly, retrieve collaborative evidence, retrieve item metadata, or interleave both through reasoning. Both memories are represented in natural language and accessed through a unified retrieval interface, enabling flexible evidence acquisition without handcrafted CF injection or fixed retrieval rules. We optimize this memory-reading policy with an outcome-only ranking reward, instantiated using group relative policy optimization, so that retrieval decisions are directly driven by final top-k recommendation quality. Extensive experiments show that RRCM significantly outperforms traditional baselines and diverse LLM-based recommendation approaches.
Shijun Li, Pranav Belligundu, Tianxin Wei +3
The University of Texas at Austin · University of Illinois at Chicago · Capital One AI Foundations +1
Large language models (LLMs) can infer user preferences from interaction histories and reviews, yet the rationales they generate may not reflect the information actually used for recommendation. A preference claim may be weakly supported by its selected evidence, or may have little effect on the final ranking. We refer to these two failures as the grounding-influence gap. We introduce PROVE-REC, a general framework for verifiable preference reasoning in LLM-based recommendation. Pass A converts the complete pre-target history into a compact preference proof consisting of positive and avoidance claims linked to selected evidence entries. Pass B predicts the next item using only the proof and its selected evidence, preventing the recommender from bypassing the reasoning path. To verify evidence-to-proof grounding, we compare the effect of masking selected evidence with masking a comparable control entry. To verify proof-to-recommendation influence, we remove a preference claim and measure the resulting decrease in the target item's ranking margin. A ranking-preservation objective further retains useful information from the complete history. Comprehensive experiments on wide-ranging real-world datasets demonstrate that PROVE-REC consistently outperforms strong sequential, generative, and LLM-enhanced baselines, with improvements of up to 7.45%. Controlled ablations confirm the effectiveness of the two-pass architecture and verification objectives. Moreover, PROVE-REC produces claims that are more strongly grounded in historical evidence and more influential to recommendation while preserving ranking quality.
Yu Hou, Nathaniel Kang, Pengkai Wang +1
School of Mathematics and Computing (Computational Science and Engineering), Yonsei University Seoul, Republic of Korea · School of Computer Science and Engineering, Kyungpook National University Daegu, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University Suwon, Republic of Korea +1