In e-commerce, LLM agents show promise for shopping tasks such as recommendations, budget management, and bundle deals, where accurately capturing user preferences from long-horizon conversations is critical. However, progress is limited by two key challenges: (1) the absence of benchmarks for evaluating long-term preference-aware shopping tasks, and (2) the lack of fine-grained supervision for shopping agent training. To fill the benchmark gap, we introduce Shopping Companion Bench, a novel benchmark comprising two shopping tasks that require cross-session preference memory, grounded in a product pool of over 1.2 million real-world items. Our analysis further identifies two major sources of failure on this benchmark: cascading errors caused by preference hallucination, and insufficient verification of product attributes against user requirements. To address these failure modes, we design annotation-free, tool-wise rewards that provide process supervision for each tool call, alleviating reward sparsity in long-horizon tasks. Experimental results demonstrate that even state-of-the-art models such as GPT-5 achieve success rates below 70%, highlighting the difficulty of our benchmark. Notably, our fine-tuned lightweight 4B model consistently outperforms strong baselines in both preference capture and task performance, suggesting the effectiveness of our reward design.
Figures & tables
Figure 1: Overview of the synthesis pipeline for constructing Shopping Companion Bench . Starting from 1.2M real-world products and the LongMemEval corpus, we sample products and vouchers, generate single-product and add-on-deal instructions, synthesize preference dialogues with denoising, inject them as preference “needles” into long-term conversation histories, and verify that each instance is resolvable, searchable, and free of conflicting preferences.
Benchmark
Task
LTM
Intera.
WebShop ( Yao et al., 2022 )
✓
✗
✗
LongMemEval ( Wu et al., 2024 )
✗
✓
✗
ShoppingBench ( Wang et al., 2025a )
✓
✗
✗
ShopSimulator ( Wang et al., 2026 )
✓
✗
✓
Ours
✓
✓
✓
Table 1: Comparison of our benchmark with existing works. Task denotes downstream task evaluation. LTM signifies long-term memory evaluation. Intera. indicates whether the evaluation protocol supports multi-turn interaction, user corrections, and preference updates.
Figure 2: Overview of our agent framework, training workflow, and reward design. Top : The agent framework organizes shopping assistance into preference identification and task execution: Stage 1 retrieves and confirms user preferences from long-term memory; during training, the user simulator follows the No-Hint mode and directly passes the identified preferences to Stage 2, while Low-Hint and High-Hint modes are used for evaluation-time correction studies; Stage 2 iteratively searches products and verifies constraints. Bottom : The final reward R(τ)=Rz(τz)+Rtool(τ)+Rfmt(τ) , comprising outcome-level reward Rz , tool-wise reward Rtool (process-level scoring of each tool invocation for credit assignment), and format reward Rfmt (output protocol compliance).
Figure 3: Preliminary analysis of the GPT-5 zero-shot agent. Left : Tool-call frequency by turn over 200 trajectories. The agent exhibits a two-phase behavior: an initial memory-retrieval stage (Turns 0–2), followed by an iterative search–view stage (Turns 3+). Right : Distribution of failure modes among 69 failed trajectories. The two largest categories are Unverified Attributes and Same-Shop Alignment .
Category
Model
Single Product
Add-on Deals
Average
Acc.(%)
Succ.(%)
Acc.(%)
Succ.(%)
Acc.(%)
Succ.(%)
Closed
GPT-5
82.0
75.0
66.0
54.0
74.0
64.5
GPT-4.1
88.0
78.0
39.0
24.0
63.5
51.0
GPT-4o
79.0
72.0
41.0
26.0
60.0
49.0
Qwen3-Max
80.0
72.0
35.0
24.0
57.5
48.0
Open
DeepSeek-R1
64.0
57.0
40.0
24.0
52.0
40.5
Table 2: Main results on Shopping Companion Bench . Acc. measures preference grounding in Stage 1; Succ. measures final recommendation success in Stage 2. The best and second-best results are marked.
Strategy
Single
Add-on
Avg.
Oracle
85.0
73.0
79.0
One-Stage
73.0
32.0
52.5
Two-Stage (No Hint)
75.0
54.0
64.5
Two-Stage (Low Hint)
78.0
59.0
68.5
Two-Stage (High Hint)
80.0
60.0
70.0
Table 3: Success rates (%) for workflow variants and user simulator. The backbone model is GPT-5; No-Hint corresponds to proceeding without corrective feedback.
Figure 4: Tool-wise reward effects: (a) tool-wise score and (b) response length over training steps (solid=smoothed, translucent=raw).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Distribution of Product Category (Top 20)
Figure 8: Dialogue Generation Prompt
Figure 9: User Instruction Generation Prompt of Single Product Rec
Figure 10: User Instruction Generation Prompt of Add-On Deals
Figure 11: Single Product Rec Evaluation Prompt
Figure 12: Add-On Deals Evaluation Prompt
Figure 13: Stage 1 System Prompt
Figure 14: Stage 2 System Prompt
Figure 15: User Simulator Prompt for Low Hint
Figure 16: User Simulator Prompt for High Hint
Judge
GPT-5
GPT-4.1
GPT-4o
Qwen3-Max
GPT-5
64.5
51.0
49.0
48.0
Qwen3-Max
59.0
52.0
43.0
45.5
Kimi-K2
58.5
48.5
44.0
49.0
Gemini-3-Pro
66.5
54.5
45.0
48.5
Qwen3-Next-80B
63.0
49.5
43.0
46.0
Appendix
Table 5: Success rates (%) across different judge LLMs (columns are agent systems). All five judges consistently rank GPT-5 Agent as the top system.
Figure 17: Screenshot of the internal annotation console used for human meta-evaluation, showing the annotation instructions, an example instance, and the decision interface.
Metric
Single
Add-on
Avg.
iPAR
0.9200
0.8700
0.8950
PAR
0.9674
0.9310
0.9497
Appendix
Table 6: Human meta-evaluation results. iPAR measures human inter-annotator agreement, while PAR measures agreement between the GPT-5 judge and human consensus.
Error Type
Single
Add-on
Total
Unverified Attributes
10
17
27 (39.1%)
Same-shop Alignment
—
11
11 (15.9%)
Multi-SKU in Single Task
9
—
9 (13.0%)
Product ID / Sourcing
3
5
8 (11.6%)
Preference Hallucination
2
5
7 (10.1%)
Insufficient Tool Calls
—
7
7 (10.1%)
Appendix
Table 7: Distribution of error types across 69 failed GPT-5 zero-shot trajectories.
Figure 18: Case Study: Unverified Attributes. The agent asserts “ultrasonic” and “pet-safe” features that do not appear in the product_view response.
Figure 19: Case Study: Same-Shop Alignment Failure. The agent assembles products from three different shops but applies a same-shop voucher without verifying merchant consistency.
Figure 20: Case Study: Multi-SKU in Single Task. The agent outputs two product IDs for a task that requires exactly one.
Figure 21: Case Study: Preference Hallucination. The agent skips mem_view and treats mem_search snippets as confirmed preferences without verifying full dialogue context.
Figure 22: Case Study: Product ID / Sourcing Errors. The agent claims verified specifications (24V, 3kW, RS232, prices) that are absent from product_view responses.
Figure 23: Case Study: Insufficient Tool Calls. The agent completes Stage 2 with zero product_search or product_view calls, outputting fabricated product IDs.