LLM-based recommendation agents can search catalogs, inspect products, ask users, and revise rejected recommendations. However, these behaviors are often governed by hand-designed workflows, while existing training methods optimize isolated components such as query rewriting or list ranking rather than the interaction policy as a whole. We present SimRec, a reinforcement learning framework that trains the full interaction policy of a tool-integrated recommendation agent. SimRec introduces two components. A task-anchored user simulator provides feedback on every recommendation: conditioned on the current task need rather than on user profile, and it can be an external LLM or the recommender itself under self-play. A task-aligned reward optimizes the whole trajectory with GRPO, crediting natural retrieval quality and valid tool use alongside the final hit, with training-only target injection to handle sparse hits over large catalogs. On ESCI and Amazon C4, SimRec improves the NDCG@100 of a Qwen3-4B agent from 32.23% to 35.42% and from 22.94% to 24.72%}, outperforming training-free agents on far larger backbones and task-aligned baselines on NDCG. Analysis shows that the trained agent verifies candidates, uses preferences, and recovers from rejected recommendations far more often than its untrained counterpart. Our code is available at https://github.com/ZJU-REAL/SimRec .
Figures & tables
Figure 1: Compared to existing tool-integrated recommender agents (left), SimRec (right) learns to search, ask, and recommend from simulated interaction.
Figure 2: Case study of SimRec multi-turn recommendation refinement. SimRec uses user feedback to revise its recommendation.
Figure 3: Overview of SimRec. The recommendation agent interacts with a user simulator and external recommendation tools, while task-aligned GRPO optimizes the policy over complete multi-turn trajectories.
Method
ESCI
Amazon C4
NDCG@100
Avg.
NDCG@100
Avg.
Baby
Office
Sports
Game
Elec.
NDCG
Recall
Hit@1
Baby
Office
Sports
Game
NDCG
Recall
Hit@1
Standalone Retrieval Baselines
BM25
15.20
24.16
20.00
12.44
22.14
19.68
53.94
12.76
6.40
8.70
7.28
7.82
7.68
21.84
2.61
Qwen3 Dense
29.06
35.17
31.74
34.64
33.07
32.62
75.69
19.98
17.34
18.66
21.77
18.49
19.80
52.32
6.45
BLAIR-BASE
16.93
22.41
20.37
14.64
14.70
19.09
55.94
9.77
20.95
17.99
22.27
19.72
20.42
57.72
5.87
Table 1: Main results on ESCI and Amazon C4. Category values are NDCG@100; aggregate columns report NDCG@100, Recall@100, and Hit@1. Elec. denotes Electronics. LLM-backbone baselines share SimRec’s tool interface; SimRec uses Qwen3 Dense unless labeled BM25. Bold and underline denote the best and second-best distinct results, respectively.
Variant
NDCG@100
Hit@1
Full SimRec
24.72
9.76
w/o Preference
21.17
(−3.55)
7.65
(−2.11)
w/o Item Detail
23.42
(−1.30)
6.04
(−3.72)
w/o Feedback
22.14
(−2.58)
8.06
(−1.70)
Table 2: Component ablation on Amazon C4. Drops in parentheses are percentage points.
Feedback Mode
NDCG@10 (%) ↑
Hit@1 (%) ↑
Invalid (%) ↓
Fixed Feedback †
31.52
18.65
11.90
Qwen3-4B Base
31.10
19.53
30.70
SimRec-4B (Self-Play)
32.15
20.64
14.00
Table 4: Feedback-mode comparison on ESCI.
Figure 4: Recovery analysis on ESCI. (a) Hit@1 improvement from the first to the final recommendation for Qwen3-4B Base and SimRec-4B. (b) Category-wise and overall recovery rates among initial misses.
Figure 5: Training dynamics of natural retrieval and target injection. As training progresses, the Top-10 hit rate increases and the Top-100 miss rate decreases, indicating improved retrieval quality.
Model
Baby
Office
Sport
Game
Elec.
Avg.
Shuffled
20.91
26.05
18.39
14.49
13.56
19.73
Base
29.27
27.33
23.51
23.57
21.08
24.75
Self-Play
33.63
28.18
22.99
27.63
21.97
25.57
SimRec-4B
35.06
31.48
25.54
24.07
27.21
28.06
Table 5: Controlled ESCI reranking (NDCG@10, %). Base denotes Qwen3-4B; Self-Play denotes SimRec-4B with self-play. Game and Elec. denote Video Games and Electronics.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Baby
Office
Sports
Video Games
Electronics
AVG
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
Hit@1
Retrieval Baselines
BM25
15.20
45.63
24.16
63.04
20.00
53.92
12.44
41.75
22.14
58.25
19.68
53.94
12.76
Qwen3 Dense
29.06
70.87
35.17
78.80
31.74
75.25
34.64
78.64
33.07
73.79
32.62
75.69
19.98
BLAIR-BASE
16.93
54.37
22.41
63.59
20.37
58.09
14.64
41.75
14.70
49.51
19.09
55.94
9.77
BLAIR-LARGE
17.15
54.37
22.17
63.04
19.66
58.82
19.03
51.46
17.25
52.43
19.54
57.60
9.10
Appendix
Table 6: Detailed ESCI results. Category columns report NDCG@100 and Recall@100; AVG additionally reports Hit@1. SimRec uses Qwen3 Dense unless marked BM25. Bold and underline indicate best and second-best distinct values, respectively (including ties).
Method
Baby
Office
Sports
Video Games
AVG
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
Hit@1
Retrieval Baselines
BM25
6.40
21.05
8.70
24.26
7.28
20.45
7.82
20.67
7.68
21.84
2.61
Qwen3 Dense
17.34
51.82
18.66
48.53
21.77
55.61
18.49
51.33
19.80
52.32
6.45
BLAIR-BASE
20.95
61.94
17.99
52.17
22.27
62.03
19.72
50.67
20.42
57.72
5.87
BLAIR-LARGE
23.26
65.18
20.11
58.75
25.65
65.11
25.27
65.33
23.42
63.01
7.14
Appendix
Table 7: Detailed Amazon C4 results. Category columns report NDCG@100 and Recall@100; AVG additionally reports Hit@1. SimRec uses Qwen3 Dense unless marked BM25. Bold and underline indicate best and second-best values, respectively.
Setting
NDCG
Recall
Hit@1
Turns
Base + Dense
33.47
75.92
–
8.23
Base + BM25
19.63
53.50
–
7.49
SimRec + Dense
34.17
77.03
21.31
4.79
SimRec + BM25
29.91
71.81
18.20
3.84
Self-Play + Dense
34.42
76.25
21.53
4.00
Self-Play + BM25
25.76
65.48
17.20
4.02
Appendix
Table 8: ESCI results with Qwen3-1.7B. Dense refers to Qwen3 Dense; metrics are percentages and Turns is the average number of turns.
Benchmark
Base
RL
Δ
MMLU
55.83
57.93
+2.10
GSM8K
78.92
82.49
+3.57
IFEval
79.11
81.70
+2.59
MBPP pass@1
44.00
52.00
+8.00
HumanEval pass@1
71.95
71.34
−0.61
Appendix
Table 9: General-capability performance before and after recommendation training. Values are percentages; Δ denotes RL − Base in percentage points.
Ablation
Baby
Office
Sports
Video Games
ALL
w/o get_user_preference
20.72 / 55.06
19.09 / 51.65
23.30 / 56.82
19.34 / 46.00
21.17 / 53.89
w/o get_item_details
24.88 / 69.64
21.24 / 60.49
25.13 / 64.71
20.93 / 60.67
23.42 / 63.65
w/o multi-turn feedback
20.03 / 54.66
20.24 / 54.42
24.56 / 57.62
20.86 / 50.00
22.14 / 55.46
Appendix
Table 10: Amazon C4 component ablations across four categories. Values are percentages reported as NDCG@100 / Recall@100.
Ablation
Turns
Search
Details
Preference
Eff. pref.
Feedback
Len. pen.
w/o get_user_preference
4.99
99.88%
4.53%
0%
0%
0.26%
0.0213
w/o get_item_details
6.71
100%
0%
100%
19.11%
0.58%
0.0542
w/o multi-turn feedback
3.99
99.77%
99.25%
76.77%
2.33%
0%
0
Appendix
Table 11: Behavioral diagnostics for the Amazon C4 component ablations. These metrics characterize behavioral differences accompanying the main NDCG@100 and Recall@100 results.
Setting
NDCG@100
Hit@1
Train user, no reward sharing
21.75
9.38
Train user, reward sharing
20.97
7.03
No user-token training
20.57
7.81
Appendix
Table 12: Self-play optimization variants at step 500.
Metric
Qwen3.6-Plus
Kimi-K2.6
GLM-5.1
NDCG@100
35.42
34.41
36.93
Δ (pp)
–
−1.01
+1.51
Recall@100
77.91
78.69
79.02
Δ (pp)
–
+0.78
+1.11
Hit@1
24.08
22.42
26.42
Δ (pp)
–
−1.66
+2.34
Appendix
Table 13: Cross-simulator evaluation of SimRec. Changes ( Δ ) are percentage points relative to Qwen3.6-Plus.
Training steps
Episodes
Eligible
Top-10 hit
Rank 11–100
Top-100 miss
Injection
Inj./eligible
1–199
6,368
64.62%
61.81%
18.49%
19.70%
24.67%
38.18%
200–399
6,400
71.47%
66.19%
16.90%
16.92%
24.17%
33.82%
400–599
6,400
72.92%
65.31%
20.08%
14.61%
25.30%
34.69%
600–799
6,400
70.30%
67.67%
18.24%
14.09%
22.73%
32.34%
800–999
6,240
72.18%
68.52%
16.98%
14.50%
22.72%
31.48%
Appendix
Table 14: Training-stage dynamics of target injection. Eligible episodes satisfy the minimum user-turn condition. The injection rate additionally requires that the target be absent from the model-visible top k=20 .
Metric
SimRec
Base
Tool calls / session
5.19
2.43
Search calls / session
2.50
1.83
Item-detail calls / session
1.83
0.58
Preference calls / session
0.86
0.02
Turns / session
9.05
6.28
Appendix
Table 15: Tool-use statistics on the Amazon session test set. SimRec uses external tools more actively than the Qwen3-4B base policy, especially item-detail and preference calls; Base denotes Qwen3-4B Base.
Metric
0–2
3–4
≥5
Sessions
120
268
1,334
NDCG@100
0.9949
0.5274
0.1237
Recall@100
1.0000
0.9925
0.4745
Hit@1
0.9917
0.1381
0.0090
Avg. reward
2.2383
0.8719
0.3528
Appendix
Table 16: SimRec performance by tool-call count on the 1,722-session Amazon C4 test set. Quality metrics use a 0–1 scale; group differences are observational, not causal.
Although large language models (LLMs) have recently gained traction in recommender systems due to their strong reasoning capabilities and extensive world knowledge, previous LLM-based agents suffer from hallucination and context-length limitations, and thus are not suitable for full-ranking recommendation tasks. To circumvent these limitations through architectural design rather than modifying the LLM itself, we propose an agent-based recommendation framework, memory-based Personalized Recommendation Tool learning via autonomous language Agents (PRTA), in which an LLM acts as a central planner interacting with multiple recommendation models as tools. The LLM-based agent is responsible for high-level reasoning and personalized tool selection, while traditional recommendation models perform full-ranking scoring, leveraging their scalability in modeling behavioral patterns. To support personalized tool selection, we design reflection mechanisms that enable the agent to evaluate and compare tools for each user based on user profiles and candidate ranked lists. Extensive experiments across three public datasets demonstrate the superiority of \modelname over traditional recommendation and LLM-based baselines in improving full-ranking recommendation performance.
Mingdai Yang, Zhiwei Liu, Weizhi Zhang +3
Univ. of Illinois Chicago Chicago, IL, USA · Microsoft Mountain View, USA · Beihang University, & Hangzhou Innovation Institute of BUAA Beijing & Hangzhou, China
Recommender agents built on Large Language Models offer a promising paradigm for personalized recommendation. However, existing agents typically suffer from a misalignment between their tool-integrated reasoning trajectories and recommendation feedback, limiting their ability to distinguish fine-grained user preferences. To address these challenges, we propose AgenticRec, an agentic recommendation framework that formulates recommendation as a tool-integrated reasoning process over a recommendation-oriented tool suite. Built upon this framework, we further develop a dedicated two-stage training paradigm tailored for recommender agents. In the first stage, we introduce Recommendation-Oriented Trajectory Activation, optimize the agentic recommendation ability under implicit feedback. In the second stage, Progressive Preference Refinement further refines the agent through bidirectional preference reasoning over self-bootstrapped hard pairs, progressively sharpening preference boundaries. Theoretical analysis and extensive experiments demonstrate the effectiveness of AgenticRec. Our code is available at https://anonymous.4open.science/r/AgenticRec-FB16.
LLM recommender agents increasingly produce structured reports: sets of items accompanied by natural-language justifications. Yet existing benchmarks often focus on reranking small candidate sets or judging reports primarily by semantic plausibility, providing limited support for evaluating and training agents based on set-level utility. We introduce Recommender Atlas (Agentic Tasks for Learning and Assessment in Shopping), or RecoAtlas, a benchmark and reinforcement learning environment grounded in user behavior. RecoAtlas provides a shopping toolkit for product retrieval, complementarity, and diversity. Given a shopping query, an agent uses these tools to generate a structured recommendation report. Behavioral reference sets support exact-recovery evaluation and the training of a learned bundle-level reward that supplies feedback for policy optimization with RL. Its controlled tool environment exposes agents to semantic and behavior-aligned tools, enabling systematic analysis of how LLM reasoning, tool signals, and tool-use policies contribute to performance. Across controlled experiments, performance scales with model capacity and test-time compute, and improves with stronger tools, while semantic plausibility does not necessarily translate into behavior-grounded utility. RecoAtlas provides a foundation for training and evaluating shopping assistants to move beyond semantic plausibility toward coherent, behaviorally grounded recommendation sets.