Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents
Organizations: University of Chicago
Abstract
Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.
Figures & tables
| Success rate (pts) | Stated confidence (pts) | |||||
|---|---|---|---|---|---|---|
| Model | WebShop | SQID | WANDS | WebShop | SQID | WANDS |
| GPT-6 Astra | +1.4 | +0.8 | +0.9 | +1.9 † | +2.5 † | +1.7 † |
| GPT-6.1 Sol | +0.8 | -0.2 | +0.9 | +1.5 † | +2.4 † | +1.9 † |
| GPT-6 Luna | +2.4 | +2.0 | +5.2 ∗ | +1.6 † | +0.2 | +1.3 ∗ |
| Claude Fable 5.1 | +1.2 | +1.2 | -1.8 | +1.0 † | +0.5 | +0.5 |
| Claude Opus 5.5 | +0.8 | +0.2 | -2.1 | +0.4 ∗ | +0.4 | -0.3 |
| ECE (pts) | Brier | AUROC | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | WebShop | SQID | WANDS | WebShop | SQID | WANDS | WebShop | SQID | WANDS |
| GPT-6 Astra | 17.5 | 33.1 | 19.1 | 0.196 | 0.350 | 0.214 | 0.56 | 0.67 | 0.68 |
| GPT-6.1 Sol | 18.2 | 33.9 | 19.9 | 0.203 | 0.354 | 0.217 | 0.56 | 0.67 | 0.69 |
| GPT-6 Luna | 15.2 | 30.4 | 14.8 | 0.202 | 0.324 | 0.198 | 0.59 | 0.69 | 0.76 |
| Claude Fable 5.1 | 3.4 | 13.2 | 5.0 | 0.135 | 0.225 | 0.155 | 0.69 | 0.74 | 0.80 |
| Claude Opus 5.5 | 6.7 | 12.4 | 5.3 | 0.126 | 0.225 | 0.155 | 0.69 | 0.73 | 0.81 |
| Dataset | Effort | , | All wrong | All right | Var ratio | KL div. | |
|---|---|---|---|---|---|---|---|
| obs / indep | obs / indep | ||||||
| WebShop | low | 10, 0.201 | 47 / 0.0 | 319 / 51.8 | 7.4 | 0.71 | 2.65 |
| medium | 8, 0.183 | 43 / 0.0 | 340 / 97.0 | 5.9 | 0.70 | 1.90 | |
| high | 10, 0.187 | 37 / 0.0 | 328 / 61.1 | 7.4 | 0.71 | 2.54 | |
| SQID | low | 10, 0.528 | 185 / 0.8 | 171 / 0.3 | 8.1 | 0.79 | 4.12 |
| medium | 8, 0.519 | 194 / 2.6 | 185 / 1.4 | 6.6 | 0.80 | 3.21 |
| WebShop | SQID | WANDS | |
| Tasks / valid decisions | 496 / 13,875 | 500 / 13,990 | 330 / 9,234 |
| Ground truth | simulator | human labels | human labels |
| Images shown | yes | yes | no |
| Success rate, all conditions (%) | 74.0–88.1 | 43.7–50.6 | 65.8–73.7 |
| Largest low high success change (pts) | 5.5 | 2.0 | 5.2 |
| Success contrasts sig. (raw / Holm), of 26 | 8 / 2 | 1 / 0 | 2 / 1 |
| Model | Effort | Success (95% CI) | Conf. | Overconf. | 10/10 | ECE | Tokens |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | low | 78.8 (75–82) | 9.27 | +13.9 | 71% | 18.0 | 3 |
| medium | 78.4 (75–82) | 9.33 | +14.9 | 73% | 17.4 | 8 | |
| high | 80.2 (77–84) | 9.46 | +14.4 | 78% | 17.2 | 44 | |
| GPT-6.1 Sol | low | 77.6 (74–81) | 9.34 | +15.7 | 72% | 18.5 | 3 |
| medium | 78.6 (75–82) | 9.37 | +15.1 | 74% | 17.8 | 10 | |
| high | 78.4 (75–82) | 9.49 | +16.5 | 78% | 18.3 | 42 |
| Success rate (pts) | Stated confidence (0-10 scale) | |||||
|---|---|---|---|---|---|---|
| Model | low medium | medium high | low high | low medium | medium high | low high |
| GPT-6 Astra | -0.4 | +1.8 ∗ | +1.4 | +0.06 † | +0.13 † | +0.19 † |
| GPT-6.1 Sol | +1.0 | -0.2 | +0.8 | +0.04 ∗ | +0.11 † | +0.15 † |
| GPT-6 Luna | +3.2 ∗ | -0.8 | +2.4 | +0.13 † | +0.03 | +0.16 † |
| Claude Fable 5.1 | +0.6 | +0.6 | +1.2 | +0.03 | +0.07 † | +0.10 † |
| Claude Opus 5.5 | +0.4 | +0.4 | +0.8 | +0.03 | +0.01 | +0.04 ∗ |
| ECE (pts) | Brier | AUROC | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | low | medium | high | low | medium | high | low | medium | high |
| GPT-6 Astra | 18.0 | 17.4 | 17.2 | 0.203 | 0.200 | 0.187 | 0.55 | 0.56 | 0.56 |
| GPT-6.1 Sol | 18.5 | 17.8 | 18.3 | 0.208 | 0.202 | 0.198 | 0.56 | 0.54 | 0.57 |
| GPT-6 Luna | 15.0 | 13.9 | 16.7 | 0.208 | 0.193 | 0.205 | 0.61 | 0.59 | 0.58 |
| Claude Fable 5.1 | 3.3 | 4.0 | 2.8 | 0.136 | 0.135 | 0.133 | 0.70 | 0.69 | 0.67 |
| Claude Opus 5.5 | 6.5 | 6.6 | 6.9 | 0.129 | 0.126 | 0.122 | 0.69 | 0.69 | 0.70 |
| Model | Effort | Success (95% CI) | Conf. | Overconf. | 10/10 | ECE | Tokens |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | low | 46.6 (42–51) | 7.75 | +30.9 | 41% | 33.0 | 10 |
| medium | 46.6 (42–51) | 7.84 | +31.8 | 43% | 32.5 | 22 | |
| high | 47.4 (43–52) | 8.00 | +32.6 | 47% | 33.8 | 77 | |
| GPT-6.1 Sol | low | 47.2 (43–52) | 7.86 | +31.4 | 42% | 33.4 | 13 |
| medium | 46.8 (42–51) | 7.93 | +32.5 | 43% | 33.2 | 25 | |
| high | 47.0 (43–51) | 8.11 | +34.1 | 47% | 35.1 | 81 |
| Success rate (pts) | Stated confidence (0-10 scale) | |||||
|---|---|---|---|---|---|---|
| Model | low medium | medium high | low high | low medium | medium high | low high |
| GPT-6 Astra | +0.0 | +0.8 | +0.8 | +0.09 † | +0.16 † | +0.25 † |
| GPT-6.1 Sol | -0.4 | +0.2 | -0.2 | +0.06 † | +0.18 † | +0.24 † |
| GPT-6 Luna | +2.6 ∗ | -0.6 | +2.0 | -0.03 | +0.06 | +0.02 |
| Claude Fable 5.1 | -0.4 | +1.6 | +1.2 | +0.06 ∗ | -0.01 | +0.05 |
| Claude Opus 5.5 | +0.8 | -0.6 | +0.2 | +0.03 | +0.01 | +0.04 |
| ECE (pts) | Brier | AUROC | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | low | medium | high | low | medium | high | low | medium | high |
| GPT-6 Astra | 33.0 | 32.5 | 33.8 | 0.349 | 0.349 | 0.352 | 0.66 | 0.67 | 0.67 |
| GPT-6.1 Sol | 33.4 | 33.2 | 35.1 | 0.354 | 0.347 | 0.361 | 0.65 | 0.68 | 0.67 |
| GPT-6 Luna | 32.4 | 28.3 | 30.4 | 0.344 | 0.310 | 0.320 | 0.67 | 0.70 | 0.70 |
| Claude Fable 5.1 | 12.8 | 14.3 | 12.6 | 0.223 | 0.226 | 0.227 | 0.74 | 0.74 | 0.73 |
| Claude Opus 5.5 | 13.7 | 11.8 | 11.8 | 0.229 | 0.221 | 0.224 | 0.72 | 0.73 | 0.73 |
| Effort | , | All wrong | All right | Var ratio | KL div. | |
|---|---|---|---|---|---|---|
| obs / indep | obs / indep | |||||
| low | 10, 0.528 | 185 / 0.8 | 171 / 0.3 | 8.1 | 0.79 | 4.12 |
| medium | 8, 0.519 | 194 / 2.6 | 185 / 1.4 | 6.6 | 0.80 | 3.21 |
| high | 10, 0.525 | 189 / 0.8 | 173 / 0.3 | 8.3 | 0.81 | 4.24 |
| Model | Effort | Success (95% CI) | Conf. | Overconf. | 10/10 | ECE | Tokens |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | low | 71.5 (66–76) | 8.98 | +18.3 | 61% | 18.7 | 6 |
| medium | 72.1 (67–77) | 9.02 | +18.1 | 63% | 18.7 | 14 | |
| high | 72.4 (67–77) | 9.15 | +19.1 | 68% | 19.8 | 54 | |
| GPT-6.1 Sol | low | 71.5 (66–76) | 9.06 | +19.1 | 63% | 19.4 | 7 |
| medium | 71.2 (66–76) | 9.13 | +20.1 | 65% | 20.1 | 15 | |
| high | 72.4 (67–77) | 9.25 | +20.1 | 69% | 20.1 | 49 |
| Success rate (pts) | Stated confidence (0-10 scale) | |||||
|---|---|---|---|---|---|---|
| Model | low medium | medium high | low high | low medium | medium high | low high |
| GPT-6 Astra | +0.6 | +0.3 | +0.9 | +0.04 | +0.13 † | +0.17 † |
| GPT-6.1 Sol | -0.3 | +1.2 | +0.9 | +0.08 † | +0.12 † | +0.19 † |
| GPT-6 Luna | +5.5 † | -0.3 | +5.2 ∗ | +0.08 | +0.05 | +0.13 ∗ |
| Claude Fable 5.1 | -0.6 | -1.2 | -1.8 | +0.04 | +0.01 | +0.05 |
| Claude Opus 5.5 | -0.9 | -1.2 | -2.1 | +0.00 | -0.03 | -0.03 |
| ECE (pts) | Brier | AUROC | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | low | medium | high | low | medium | high | low | medium | high |
| GPT-6 Astra | 18.7 | 18.7 | 19.8 | 0.213 | 0.214 | 0.215 | 0.68 | 0.68 | 0.67 |
| GPT-6.1 Sol | 19.4 | 20.1 | 20.1 | 0.215 | 0.221 | 0.214 | 0.69 | 0.68 | 0.69 |
| GPT-6 Luna | 18.8 | 12.8 | 12.7 | 0.224 | 0.186 | 0.184 | 0.75 | 0.76 | 0.77 |
| Claude Fable 5.1 | 3.6 | 4.8 | 6.5 | 0.153 | 0.157 | 0.156 | 0.79 | 0.79 | 0.80 |
| Claude Opus 5.5 | 3.1 | 7.2 | 5.7 | 0.155 | 0.156 | 0.154 | 0.79 | 0.81 | 0.82 |
| Effort | , | All wrong | All right | Var ratio | KL div. | |
|---|---|---|---|---|---|---|
| obs / indep | obs / indep | |||||
| low | 10, 0.285 | 55 / 0.0 | 184 / 11.4 | 7.9 | 0.77 | 3.48 |
| medium | 8, 0.282 | 62 / 0.0 | 205 / 23.3 | 6.7 | 0.81 | 2.88 |
| high | 10, 0.280 | 55 / 0.0 | 204 / 12.3 | 8.3 | 0.81 | 3.76 |
| Instruction | Correct product | Why distractors fail |
|---|---|---|
| “i need an oval soft rug that is easy to clean. also, it should be in eggplant purple color, and price lower than 270.00 dollars” | Unique Loom Solo Solid Shag area rug ($254.84), with shape: oval and color: eggplant purple selected | All nine are shag or area rugs under $270 from the same search, but none offers eggplant purple as a color option (one offers purple but not an oval shape). |
| “i want to get a super soft blue fleece throw that’s 50 inches by 63 inches, and price lower than 50.00 dollars” | HILHORA flannel fleece throw ({}^{\prime\prime}\times$ 63 ′′ and color: blue selected | Six throws do not offer blue; two are not fleece (faux fur, woven); one is not described as super soft; none other than the target offers the 50 63 ′′ size. |
| “i just love the matilde vicenzi brand macaroons and i want to try the amaretto d’italia macaroons flavor. the quality of the ingredients is my requirement … and price lower than 50.00 dollars” | Amaretto d’Italia Macaroons by Matilde Vicenzi ($14.99), with flavor: amaretto d’italia macaroons selected | Seven cookie products lack the quality ingredients attribute; two gift baskets ($100) exceed the price ceiling. |
| “i am looking for some alcohol free skin care, and price lower than 80.00 dollars” | Gaelle Organic Exfoliant Supérieure natural facial scrub ($48.00); no options | All nine are facial or body scrubs and serums under $80, but none is described as alcohol free . |
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 279 papers