Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.
Figures & tables
Figure 1: WebShop at a glance. Per model, the requested reasoning effort is varied from low to high. The graphs display success rate (left), mean stated confidence (middle) and mean reasoning tokens actually used (right) for each reasoning level. All 28 WebShop conditions lie between 74.0% and 88.1% success. We see that Qwen 3.8 Max goes from 394 to 2,033 reasoning tokens per task between low and high effort (488 to 3,926 on SQID), DeepSeek V4.1 Flash from 445 to 634, Gemini 3.1 Pro from 305 to 772 and Gemini 4 Argon from 0 to 774, whereas GPT-6 Astra and GPT-6.1 Sol stay below 100 reasoning tokens per task even at high effort. Similar graphs for SQID and WANDS can be found in Fig. S8 . Takeaway: Success rate does not rise significantly from low to high reasoning for most models, while the model output’s stated confidence does.
Figure 2: Paired within-task change on WebShop when requested effort is changed. The 95% confidence intervals for each case are added. The columns are low → medium, medium → high, and low → high. The top row shows change in success rate (percentage points), and the bottom row shows change in mean stated confidence (0–10 scale, ×10 so the units match). DeepSeek V4.1 Flash and Kimi K3 have no medium setting. Graphs for SQID and WANDS can be found in Figs. S4 and S6 . Takeaway: Most accuracy intervals cover zero (indicating no significant change in accuracy), and confidence intervals mostly sit above it (indicating an increase in confidence).
Success rate (pts)
Stated confidence (pts)
Model
WebShop
SQID
WANDS
WebShop
SQID
WANDS
GPT-6 Astra
+1.4
+0.8
+0.9
+1.9 †
+2.5 †
+1.7 †
GPT-6.1 Sol
+0.8
-0.2
+0.9
+1.5 †
+2.4 †
+1.9 †
GPT-6 Luna
+2.4
+2.0
+5.2 ∗
+1.6 †
+0.2
+1.3 ∗
Claude Fable 5.1
+1.2
+1.2
-1.8
+1.0 †
+0.5
+0.5
Claude Opus 5.5
+0.8
+0.2
-2.1
+0.4 ∗
+0.4
-0.3
Table 1: Paired low → high change in success rate (percentage points, teal) and mean stated confidence (0–10 scale ×10 , coral) on the three datasets. Shaded bold cells differ significantly from zero in a paired t -test at p<0.05 . A dagger ( † ) marks changes that remain significant ( p<0.05 ) after Holm correction [ 19 ] for the 26 contrasts per metric and dataset; an asterisk ( ∗ ) marks changes significant only before correction. Holm correction is strictly more conservative than the uncorrected test ( Sec. 3.3 ), so every † cell would also pass the uncorrected test. The contrasts from low → medium and medium → high per dataset can be found in Tabs. S4 , S7 and S11 . Takeaway: Increasing thinking level increases confidence but typically does not improve success rate in a statistically significant way.
ECE (pts) ↓
Brier ↓
AUROC ↑
Model
WebShop
SQID
WANDS
WebShop
SQID
WANDS
WebShop
SQID
WANDS
GPT-6 Astra
17.5
33.1
19.1
0.196
0.350
0.214
0.56
0.67
0.68
GPT-6.1 Sol
18.2
33.9
19.9
0.203
0.354
0.217
0.56
0.67
0.69
GPT-6 Luna
15.2
30.4
14.8
0.202
0.324
0.198
0.59
0.69
0.76
Claude Fable 5.1
3.4
13.2
5.0
0.135
0.225
0.155
0.69
0.74
0.80
Claude Opus 5.5
6.7
12.4
5.3
0.126
0.225
0.155
0.69
0.73
0.81
Table 2: Calibration by model on the three datasets (pooled over effort levels). ECE is measured in percentage points, Brier score is measured on a c/10 scale, and AUROC is measured on a 0−1 scale. Lower ECE and Brier and higher AUROC are better, and the best value per column is bold. Per-effort breakdowns are shown in Tabs. S5 , S8 and S12 . On ECE and Brier the Claude models lead by a wide margin everywhere (with the exception of WebShop Brier scores, where Gemini 4 Argon does better): ECE 3.4 and 6.7 points against 8.9–18.2 for the other eight on WebShop, 13.2 and 12.4 against 27.6–34.3 on SQID, 5.0 and 5.3 against 11.8–19.9 on WANDS. Takeaway: The relative calibration rankings are a model property rather than a dataset property: models are ordered nearly identically by ECE on all three datasets (Spearman correlation ρ=0.79 – 0.92 ) despite a large spread in accuracy.
Figure 3: WebShop: how many models are wrong on the same task . The graphs show at this and what independent errors would produce at each effort level. The dark bars are the observed number of tasks on which exactly k models chose wrong, and the light bars are the expected count if the models’ errors were independent, Binomial(n,pˉ) . The markers show the same binomial baseline. The medium panel covers the eight models with a medium setting. On WebShop at low effort, this distribution is bimodal: 319 of 493 tasks are solved by all ten models (independence predicts 50) and 47 are failed by all ten (independence predicts 0.0001). Similar graphs for SQID and WANDS can be found at Figs. S5 and S7 . Takeaway: At every effort level, the bimodal distribution of the errors indicates that models tend to fail together on the same tasks.
Dataset
Effort
n , pˉ
All wrong
All right
Var ratio
ρ
KL div.
obs / indep
obs / indep
WebShop
low
10, 0.201
47 / 0.0
319 / 51.8
7.4
0.71
2.65
medium
8, 0.183
43 / 0.0
340 / 97.0
5.9
0.70
1.90
high
10, 0.187
37 / 0.0
328 / 61.1
7.4
0.71
2.54
SQID
low
10, 0.528
185 / 0.8
171 / 0.3
8.1
0.79
4.12
medium
8, 0.519
194 / 2.6
185 / 1.4
6.6
0.80
3.21
Table 3: Shared errors on the three datasets, by effort level. For each effort level, there are n models, with mean error rate pˉ . We show the observed vs. independence-expected number of tasks on which models all-were-wrong or all-were-right, the variance ratio (observed variance of the number of wrong models over its variance under independence), the implied pairwise error correlation ρ , and the KL divergence of the observed distribution from Binomial(n,pˉ) . The pairwise error correlation implied by the variance is 0.70–0.71 on WebShop, 0.79–0.81 on SQID and 0.77–0.81 on WANDS. Takeaway: The variance of the number of wrong models is far more than in an independent-models baseline.
WebShop
SQID
WANDS
Tasks / valid decisions
496 / 13,875
500 / 13,990
330 / 9,234
Ground truth
simulator
human labels
human labels
Images shown
yes
yes
no
Success rate, all conditions (%)
74.0–88.1
43.7–50.6
65.8–73.7
Largest ∣ low → high ∣ success change (pts)
5.5
2.0
5.2
Success contrasts sig. (raw / Holm), of 26
8 / 2
1 / 0
2 / 1
Table S2 : Headline results on the three datasets. Rows are defined in the main paper; Claude values are Fable 5.1 / Opus 5.5.
Figure S1 : Per-model success, overconfidence and AUROC on the three datasets (pooled over effort; open circle WebShop, filled circle SQID, square WANDS). Success differs mainly by dataset; overconfidence and AUROC separate the models into the same groups on every dataset.
Model
Effort
Success (95% CI)
Conf.
Overconf.
10/10
ECE
Tokens
GPT-6 Astra
low
78.8 (75–82)
9.27
+13.9
71%
18.0
3
medium
78.4 (75–82)
9.33
+14.9
73%
17.4
8
high
80.2 (77–84)
9.46
+14.4
78%
17.2
44
GPT-6.1 Sol
low
77.6 (74–81)
9.34
+15.7
72%
18.5
3
medium
78.6 (75–82)
9.37
+15.1
74%
17.8
10
high
78.4 (75–82)
9.49
+16.5
78%
18.3
42
Table S3 : WebShop results per condition. Success in percent with Wilson 95% CI; mean stated confidence; overconfidence in points; share of answers rated 10/10; ECE in points; mean reasoning tokens.
Success rate (pts)
Stated confidence (0-10 scale)
Model
low → medium
medium → high
low → high
low → medium
medium → high
low → high
GPT-6 Astra
-0.4
+1.8 ∗
+1.4
+0.06 †
+0.13 †
+0.19 †
GPT-6.1 Sol
+1.0
-0.2
+0.8
+0.04 ∗
+0.11 †
+0.15 †
GPT-6 Luna
+3.2 ∗
-0.8
+2.4
+0.13 †
+0.03
+0.16 †
Claude Fable 5.1
+0.6
+0.6
+1.2
+0.03
+0.07 †
+0.10 †
Claude Opus 5.5
+0.4
+0.4
+0.8
+0.03
+0.01
+0.04 ∗
Table S4 : WebShop: paired change for every effort contrast. Success in percentage points; confidence on the 0–10 scale. Shaded bold cells are significant at p<0.05 in a paired t -test; † : still significant after Holm correction over the 26 contrasts per metric; ∗ : significant only before correction (see Tab. 1 ).
ECE (pts)
Brier
AUROC
Model
low
medium
high
low
medium
high
low
medium
high
GPT-6 Astra
18.0
17.4
17.2
0.203
0.200
0.187
0.55
0.56
0.56
GPT-6.1 Sol
18.5
17.8
18.3
0.208
0.202
0.198
0.56
0.54
0.57
GPT-6 Luna
15.0
13.9
16.7
0.208
0.193
0.205
0.61
0.59
0.58
Claude Fable 5.1
3.3
4.0
2.8
0.136
0.135
0.133
0.70
0.69
0.67
Claude Opus 5.5
6.5
6.6
6.9
0.129
0.126
0.122
0.69
0.69
0.70
Table S5 : WebShop calibration by effort level. ECE in points, Brier score, AUROC; metric blocks separated by heavy rules.
Figure 12
Model
Effort
Success (95% CI)
Conf.
Overconf.
10/10
ECE
Tokens
GPT-6 Astra
low
46.6 (42–51)
7.75
+30.9
41%
33.0
10
medium
46.6 (42–51)
7.84
+31.8
43%
32.5
22
high
47.4 (43–52)
8.00
+32.6
47%
33.8
77
GPT-6.1 Sol
low
47.2 (43–52)
7.86
+31.4
42%
33.4
13
medium
46.8 (42–51)
7.93
+32.5
43%
33.2
25
high
47.0 (43–51)
8.11
+34.1
47%
35.1
81
Table S6 : SQID results per condition. Columns as in Tab. S3 .
Figure S4 : SQID: paired within-task change with 95% intervals for success (top) and confidence (bottom) under each effort contrast. No success change is significant; confidence rises for five models.
Success rate (pts)
Stated confidence (0-10 scale)
Model
low → medium
medium → high
low → high
low → medium
medium → high
low → high
GPT-6 Astra
+0.0
+0.8
+0.8
+0.09 †
+0.16 †
+0.25 †
GPT-6.1 Sol
-0.4
+0.2
-0.2
+0.06 †
+0.18 †
+0.24 †
GPT-6 Luna
+2.6 ∗
-0.6
+2.0
-0.03
+0.06
+0.02
Claude Fable 5.1
-0.4
+1.6
+1.2
+0.06 ∗
-0.01
+0.05
Claude Opus 5.5
+0.8
-0.6
+0.2
+0.03
+0.01
+0.04
Table S7 : SQID: paired change for every effort contrast. Success in percentage points; confidence on the 0–10 scale. Shaded bold cells are significant at p<0.05 in a paired t -test; † : still significant after Holm correction over the 26 contrasts per metric; ∗ : significant only before correction (see Tab. 1 ).
ECE (pts)
Brier
AUROC
Model
low
medium
high
low
medium
high
low
medium
high
GPT-6 Astra
33.0
32.5
33.8
0.349
0.349
0.352
0.66
0.67
0.67
GPT-6.1 Sol
33.4
33.2
35.1
0.354
0.347
0.361
0.65
0.68
0.67
GPT-6 Luna
32.4
28.3
30.4
0.344
0.310
0.320
0.67
0.70
0.70
Claude Fable 5.1
12.8
14.3
12.6
0.223
0.226
0.227
0.74
0.74
0.73
Claude Opus 5.5
13.7
11.8
11.8
0.229
0.221
0.224
0.72
0.73
0.73
Table S8 : SQID calibration by effort level. ECE in points, Brier score, AUROC; metric blocks separated by heavy rules.
Figure S5 : SQID: number of models wrong per task at each effort level, observed versus the independence null. About 37% of tasks are failed by every model and 34-37% solved by every model; independence predicts under 1% for each.
Effort
n , pˉ
All wrong
All right
Var ratio
ρ
KL div.
obs / indep
obs / indep
low
10, 0.528
185 / 0.8
171 / 0.3
8.1
0.79
4.12
medium
8, 0.519
194 / 2.6
185 / 1.4
6.6
0.80
3.21
high
10, 0.525
189 / 0.8
173 / 0.3
8.3
0.81
4.24
Table S9 : SQID error concentration by effort level. Columns as in the main-paper concentration table.
Model
Effort
Success (95% CI)
Conf.
Overconf.
10/10
ECE
Tokens
GPT-6 Astra
low
71.5 (66–76)
8.98
+18.3
61%
18.7
6
medium
72.1 (67–77)
9.02
+18.1
63%
18.7
14
high
72.4 (67–77)
9.15
+19.1
68%
19.8
54
GPT-6.1 Sol
low
71.5 (66–76)
9.06
+19.1
63%
19.4
7
medium
71.2 (66–76)
9.13
+20.1
65%
20.1
15
high
72.4 (67–77)
9.25
+20.1
69%
20.1
49
Table S10 : WANDS results per condition. Columns as in Tab. S3 .
Figure S6 : WANDS: paired within-task change with 95% intervals under each effort contrast. GPT-6 Luna is the only model with a significant success gain, entirely in the low → medium step.
Success rate (pts)
Stated confidence (0-10 scale)
Model
low → medium
medium → high
low → high
low → medium
medium → high
low → high
GPT-6 Astra
+0.6
+0.3
+0.9
+0.04
+0.13 †
+0.17 †
GPT-6.1 Sol
-0.3
+1.2
+0.9
+0.08 †
+0.12 †
+0.19 †
GPT-6 Luna
+5.5 †
-0.3
+5.2 ∗
+0.08
+0.05
+0.13 ∗
Claude Fable 5.1
-0.6
-1.2
-1.8
+0.04
+0.01
+0.05
Claude Opus 5.5
-0.9
-1.2
-2.1
+0.00
-0.03
-0.03
Table S11 : WANDS: paired change for every effort contrast. Format as in Tab. S7 .
ECE (pts)
Brier
AUROC
Model
low
medium
high
low
medium
high
low
medium
high
GPT-6 Astra
18.7
18.7
19.8
0.213
0.214
0.215
0.68
0.68
0.67
GPT-6.1 Sol
19.4
20.1
20.1
0.215
0.221
0.214
0.69
0.68
0.69
GPT-6 Luna
18.8
12.8
12.7
0.224
0.186
0.184
0.75
0.76
0.77
Claude Fable 5.1
3.6
4.8
6.5
0.153
0.157
0.156
0.79
0.79
0.80
Claude Opus 5.5
3.1
7.2
5.7
0.155
0.156
0.154
0.79
0.81
0.82
Table S12 : WANDS calibration by effort level.
Figure S7 : WANDS: number of models wrong per task at each effort level, observed versus the independence null.
Effort
n , pˉ
All wrong
All right
Var ratio
ρ
KL div.
obs / indep
obs / indep
low
10, 0.285
55 / 0.0
184 / 11.4
7.9
0.77
3.48
medium
8, 0.282
62 / 0.0
205 / 23.3
6.7
0.81
2.88
high
10, 0.280
55 / 0.0
204 / 12.3
8.3
0.81
3.76
Table S13 : WANDS error concentration by effort level.
Figure S8 : Success, confidence and reasoning tokens by requested effort on all three datasets. Rows: WebShop, SQID, WANDS. All 28 WebShop conditions lie between 74.0% and 88.1% success, all SQID conditions between 43.7% and 50.6%, and all WANDS conditions between 65.8% and 73.7%. SQID has the hardest tasks (and therefore lowest accuracy) because its distractors are human-judged Substitutes, often the same product in another size, color or pack count.
Figure S9 : Reliability diagrams for SQID (left) and WANDS (right). On SQID every model lies below the diagonal; the Claude models are closest to it.
Figure S10 : Mean stated confidence on correct versus wrong answers for SQID (left) and WANDS (right), with 95% intervals. The drops are larger than on WebShop (1.3–2.1 against 0.3–0.8) but mean confidence on wrong answers still exceeds 6.5/10 for every non-Claude model. The Claude models drop on WANDS by 2.02 and 2.10 against 1.26–1.92, and on SQID by 1.84 and 1.72 against 1.37–1.85 (Qwen 3.8 Max is marginally first).
Figure S11 : Low-effort error concentration on the three datasets , observed versus independence.
Instruction
Correct product
Why distractors fail
“i need an oval soft rug that is easy to clean. also, it should be in eggplant purple color, and price lower than 270.00 dollars”
Unique Loom Solo Solid Shag area rug ($254.84), with shape: oval and color: eggplant purple selected
All nine are shag or area rugs under $270 from the same search, but none offers eggplant purple as a color option (one offers purple but not an oval shape).
“i want to get a super soft blue fleece throw that’s 50 inches by 63 inches, and price lower than 50.00 dollars”
HILHORA flannel fleece throw (16.99),withsize:50{}^{\prime\prime}\times$ 63 ′′ and color: blue selected
Six throws do not offer blue; two are not fleece (faux fur, woven); one is not described as super soft; none other than the target offers the 50 ′′× 63 ′′ size.
“i just love the matilde vicenzi brand macaroons and i want to try the amaretto d’italia macaroons flavor. the quality of the ingredients is my requirement … and price lower than 50.00 dollars”
Amaretto d’Italia Macaroons by Matilde Vicenzi ($14.99), with flavor: amaretto d’italia macaroons selected
Seven cookie products lack the quality ingredients attribute; two gift baskets ($100) exceed the price ceiling.
“i am looking for some alcohol free skin care, and price lower than 80.00 dollars”
Gaelle Organic Exfoliant Supérieure natural facial scrub ($48.00); no options
All nine are facial or body scrubs and serums under $80, but none is described as alcohol free .
Table S14 : Example WebShop tasks. Instruction text is shown verbatim; product titles are abridged. “Why distractors fail” summarizes the simulator’s verdict on the nine non-target candidates in that task. Construction rules are given in Sec. 3.1 .
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
Consumers increasingly delegate purchasing decisions to Large Language Models (LLMs) acting as surrogate consumers. Using "Tool-Lab," an adaptation of information-board process tracing that places product attributes behind costly tool calls, we examine how marketing pricing cues (i.e., just-below pricing and promotional framing) influence AI shopping agents. Across eight commercially deployed LLMs from three providers, we trace pre-choice information acquisition. Under zero cost, pricing cues rarely mislead. Imposing acquisition costs under a vague goal prompt leads LLMs to omit diagnostic attributes required to compute unit price and choose suboptimal choices resembling human heuristics. Relative to a specific goal prompt that mainly preserves diagnostic search and choice optimality, a vague goal prompt under constraints creates a search-mediated vulnerability. This research demonstrates that marketing heuristics in delegated AI shopping are governed by storefront information architecture, not necessarily immutable LLM flaws.
Davood Wadi, Yu Ma
Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada
As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.
Conversational shopping assistants now serve hundreds of millions of customers, yet no existing benchmark jointly evaluates the open-ended multi-turn reasoning, domain expertise, and criterion-level quality that real shopping conversations demand. Shopping reasoning is unique among language model applications. Unlike factual question answering or verifiable code generation, it requires balancing subjective preferences, budget constraints, and cross-product trade-offs across multi-turn dialogue, capabilities absent from previous e-commerce and general-purpose benchmarks. We introduce the Shopping Reasoning Bench, an expert-authored benchmark of 525 missions (232 single-turn, 293 multi-turn) with 10863 importance-weighted binary rubrics authored by retail domain experts. These criteria are organized under a taxonomy of five reasoning categories and fifteen subcategories covering diverse demands such as preference refinement, trade-off analysis, and compatibility assessment. An evaluation of nine models across three families (GPT, Claude, Gemini) shows that pass rates reach only 57--77% overall. On multi-turn missions, all models score 13--29 points lower on optional above-and-beyond criteria than on required ones, and performance degrades 4--18 points as conversations progress. These gaps show that current models handle basic shopping assistance but fall short of expert-level advice, making Shopping Reasoning Bench a challenging testbed for future shopping assistant development.