In e-commerce, LLM agents show promise for shopping tasks such as recommendations, budget management, and bundle deals, where accurately capturing user preferences from long-horizon conversations is critical. However, progress is limited by two key challenges: (1) the absence of benchmarks for evaluating long-term preference-aware shopping tasks, and (2) the lack of fine-grained supervision for shopping agent training. To fill the benchmark gap, we introduce Shopping Companion Bench, a novel benchmark comprising two shopping tasks that require cross-session preference memory, grounded in a product pool of over 1.2 million real-world items. Our analysis further identifies two major sources of failure on this benchmark: cascading errors caused by preference hallucination, and insufficient verification of product attributes against user requirements. To address these failure modes, we design annotation-free, tool-wise rewards that provide process supervision for each tool call, alleviating reward sparsity in long-horizon tasks. Experimental results demonstrate that even state-of-the-art models such as GPT-5 achieve success rates below 70%, highlighting the difficulty of our benchmark. Notably, our fine-tuned lightweight 4B model consistently outperforms strong baselines in both preference capture and task performance, suggesting the effectiveness of our reward design.
Figures & tables
Figure 1: Overview of the synthesis pipeline for constructing Shopping Companion Bench . Starting from 1.2M real-world products and the LongMemEval corpus, we sample products and vouchers, generate single-product and add-on-deal instructions, synthesize preference dialogues with denoising, inject them as preference “needles” into long-term conversation histories, and verify that each instance is resolvable, searchable, and free of conflicting preferences.
Benchmark
Task
LTM
Intera.
WebShop ( Yao et al., 2022 )
✓
✗
✗
LongMemEval ( Wu et al., 2024 )
✗
✓
✗
ShoppingBench ( Wang et al., 2025a )
✓
✗
✗
ShopSimulator ( Wang et al., 2026 )
✓
✗
✓
Ours
✓
✓
✓
Table 1: Comparison of our benchmark with existing works. Task denotes downstream task evaluation. LTM signifies long-term memory evaluation. Intera. indicates whether the evaluation protocol supports multi-turn interaction, user corrections, and preference updates.
Figure 2: Overview of our agent framework, training workflow, and reward design. Top : The agent framework organizes shopping assistance into preference identification and task execution: Stage 1 retrieves and confirms user preferences from long-term memory; during training, the user simulator follows the No-Hint mode and directly passes the identified preferences to Stage 2, while Low-Hint and High-Hint modes are used for evaluation-time correction studies; Stage 2 iteratively searches products and verifies constraints. Bottom : The final reward R(τ)=Rz(τz)+Rtool(τ)+Rfmt(τ) , comprising outcome-level reward Rz , tool-wise reward Rtool (process-level scoring of each tool invocation for credit assignment), and format reward Rfmt (output protocol compliance).
Figure 3: Preliminary analysis of the GPT-5 zero-shot agent. Left : Tool-call frequency by turn over 200 trajectories. The agent exhibits a two-phase behavior: an initial memory-retrieval stage (Turns 0–2), followed by an iterative search–view stage (Turns 3+). Right : Distribution of failure modes among 69 failed trajectories. The two largest categories are Unverified Attributes and Same-Shop Alignment .
Category
Model
Single Product
Add-on Deals
Average
Acc.(%)
Succ.(%)
Acc.(%)
Succ.(%)
Acc.(%)
Succ.(%)
Closed
GPT-5
82.0
75.0
66.0
54.0
74.0
64.5
GPT-4.1
88.0
78.0
39.0
24.0
63.5
51.0
GPT-4o
79.0
72.0
41.0
26.0
60.0
49.0
Qwen3-Max
80.0
72.0
35.0
24.0
57.5
48.0
Open
DeepSeek-R1
64.0
57.0
40.0
24.0
52.0
40.5
Table 2: Main results on Shopping Companion Bench . Acc. measures preference grounding in Stage 1; Succ. measures final recommendation success in Stage 2. The best and second-best results are marked.
Strategy
Single
Add-on
Avg.
Oracle
85.0
73.0
79.0
One-Stage
73.0
32.0
52.5
Two-Stage (No Hint)
75.0
54.0
64.5
Two-Stage (Low Hint)
78.0
59.0
68.5
Two-Stage (High Hint)
80.0
60.0
70.0
Table 3: Success rates (%) for workflow variants and user simulator. The backbone model is GPT-5; No-Hint corresponds to proceeding without corrective feedback.
Figure 4: Tool-wise reward effects: (a) tool-wise score and (b) response length over training steps (solid=smoothed, translucent=raw).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Distribution of Product Category (Top 20)
Figure 8: Dialogue Generation Prompt
Figure 9: User Instruction Generation Prompt of Single Product Rec
Figure 10: User Instruction Generation Prompt of Add-On Deals
Figure 11: Single Product Rec Evaluation Prompt
Figure 12: Add-On Deals Evaluation Prompt
Figure 13: Stage 1 System Prompt
Figure 14: Stage 2 System Prompt
Figure 15: User Simulator Prompt for Low Hint
Figure 16: User Simulator Prompt for High Hint
Judge
GPT-5
GPT-4.1
GPT-4o
Qwen3-Max
GPT-5
64.5
51.0
49.0
48.0
Qwen3-Max
59.0
52.0
43.0
45.5
Kimi-K2
58.5
48.5
44.0
49.0
Gemini-3-Pro
66.5
54.5
45.0
48.5
Qwen3-Next-80B
63.0
49.5
43.0
46.0
Appendix
Table 5: Success rates (%) across different judge LLMs (columns are agent systems). All five judges consistently rank GPT-5 Agent as the top system.
Figure 17: Screenshot of the internal annotation console used for human meta-evaluation, showing the annotation instructions, an example instance, and the decision interface.
Metric
Single
Add-on
Avg.
iPAR
0.9200
0.8700
0.8950
PAR
0.9674
0.9310
0.9497
Appendix
Table 6: Human meta-evaluation results. iPAR measures human inter-annotator agreement, while PAR measures agreement between the GPT-5 judge and human consensus.
Error Type
Single
Add-on
Total
Unverified Attributes
10
17
27 (39.1%)
Same-shop Alignment
—
11
11 (15.9%)
Multi-SKU in Single Task
9
—
9 (13.0%)
Product ID / Sourcing
3
5
8 (11.6%)
Preference Hallucination
2
5
7 (10.1%)
Insufficient Tool Calls
—
7
7 (10.1%)
Appendix
Table 7: Distribution of error types across 69 failed GPT-5 zero-shot trajectories.
Figure 18: Case Study: Unverified Attributes. The agent asserts “ultrasonic” and “pet-safe” features that do not appear in the product_view response.
Figure 19: Case Study: Same-Shop Alignment Failure. The agent assembles products from three different shops but applies a same-shop voucher without verifying merchant consistency.
Figure 20: Case Study: Multi-SKU in Single Task. The agent outputs two product IDs for a task that requires exactly one.
Figure 21: Case Study: Preference Hallucination. The agent skips mem_view and treats mem_search snippets as confirmed preferences without verifying full dialogue context.
Figure 22: Case Study: Product ID / Sourcing Errors. The agent claims verified specifications (24V, 3kW, RS232, prices) that are absent from product_view responses.
Figure 23: Case Study: Insufficient Tool Calls. The agent completes Stage 2 with zero product_search or product_view calls, outputting fabricated product IDs.
As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.
Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3% of the mean final net assets achieved by human participants.
Qiming Shi, Yulong Tao, Linbo Jin +10
2State Key Lab of CAD&CG, Zhejiang University · 1Alibaba Group · School of Software and Microelectronics, Peking University +2
Understanding customer shopping trajectories is essential for enabling personalized shopping experiences. However, shopping records (i.e., customer's search, clicks, purchases, etc.) often span long time horizons over multiple years, resulting in extremely long trajectories that pose significant challenges for existing large language models (LLMs). Despite the importance of this problem, existing benchmarks are limited to short customer trajectories, while real-world trajectories from large e-commerce platforms are rarely accessible due to data privacy constraints. To address this gap, we introduce ShopTrajQA, a long-context evaluation benchmark constructed from real-world product information and simulated shopping trajectories. The dataset includes variants of up to 32k and 64k tokens, enabling systematic evaluation of model robustness under varying context lengths. Through comprehensive benchmarking of frontier LLMs, we identify critical performance gaps in reasoning over long shopping trajectory data. To address these challenges, we propose a Customer Agent Framework for ultra-long context management. Leveraging a Reinforcement Learning with Verifiable Rewards (RLVR) agentic training paradigm, our approach stores trajectories as external local files and trains the agent to autonomously retrieve and parse them through code-interpreter interactions (e.g., SQL queries), effectively bypassing the fixed in-context window constraints of LLMs. Experimental results demonstrate that our framework achieves strong performance for ShopTrajQA and shows generalization to other complex reasoning tasks.