Can LLMs Give Good Product Advice? A Framework for Diagnosing Failures
Organizations: NVIDIA Research · University of Groningen · Bar-Ilan University
Abstract
Evaluating product advice from a language model requires assessing whether the recommendation meets the user's needs and identifying where the model falls short when it does not. However, matching a product mentioned in an online reply cannot answer either question on its own: several products may be suitable, and the recorded answer may itself be poor advice. When shown a community-listed alternative alongside their own sampled car recommendation, six models prefer the alternative in 1.9 to 6.5 percent of eligible disagreements, consistently across both source-blind presentation orders. We propose a diagnostic framework that separates the user's stated needs, the model's added assumptions, candidate generation, filtering, and final choice, with 19 possible failure modes and the evidence needed to assess them. We apply this framework to product-advice data and to a separate product-selection task with externally assigned relevance labels. The experiments distinguish naming a community-listed product from selecting it and show how answer extraction and candidate-list construction affect the outcome.
Figures & tables
| Domain | Queries | Names | Replies | Score |
|---|---|---|---|---|
| Cars | 800 | 1,681 | 5 | 2 |
| Laptops | 800 | 737 | 4 | 1 |
| Headphones | 226 | 481 | 3 | 1 |
| SE software | 2,445 | 2,349 | 1 | 1 |
| SE hardware | 1,266 | 1,397 | 1 | 1 |
| Model | Supported | Conflicts (%) | Other |
|---|---|---|---|
| GPT-5 | 80 | 0 (0.0) | 367 |
| Claude Opus | 44 | 0 (0.0) | 403 |
| Llama-70B | 62 | 6 (1.3) | 379 |
| Qwen-80B | 57 | 3 (0.7) | 387 |
| Nemotron-49B | 59 | 3 (0.7) | 385 |
| Llama-8B | 36 | 5 (1.1) | 406 |
| Model | Count | Rate [95% CI] |
|---|---|---|
| GPT-5.4 | 26/402 | 6.5 [3.0, 10.3] |
| Claude Opus 4.8 | 16/392 | 4.1 [1.5, 7.3] |
| Llama-70B | 26/405 | 6.4 [2.9, 10.6] |
| Qwen-80B | 18/400 | 4.5 [2.2, 7.4] |
| Nemotron-49B | 13/317 | 4.1 [1.2, 7.7] |
| Llama-8B | 7/367 | 1.9 [0.5, 3.6] |
| Model | Cars | Laptops | Headphones |
|---|---|---|---|
| GPT-5 | 6.8 / 24.8 | 10.3 / 26.0 | 6.6 / 22.6 |
| Claude Opus | 6.5 / 24.0 | 9.6 / 24.8 | 7.2 / 24.6 |
| Llama-70B | 6.0 / 24.5 | 10.0 / 27.4 | 8.3 / 21.5 |
| Qwen-80B | 5.2 / 23.2 | 5.0 / 19.0 | 5.5 / 17.1 |
| Nemotron-49B | 6.0 / 22.0 | 8.6 / 23.3 | 4.9 / 17.7 |
| Llama-8B | 5.6 / 21.3 | 9.0 / 24.0 | 5.2 / 16.1 |
| Model | 1a | 1b | 1c | 2b | 4a |
|---|---|---|---|---|---|
| GPT-5 | 45 | 73 | 7 | 41 | 5 |
| Claude Opus | 51 | 85 | 3 | 0 | 2 |
| Llama-70B | 70 | 59 | 3 | 0 | 21 |
| Qwen-80B | 63 | 88 | 4 | 23 | 39 |
| Nemotron-49B | 67 | 96 | 2 | 19 | 25 |
| Llama-8B | 70 | 71 | 2 | 14 | 20 |
| Coverage (%) | Selection | |||
|---|---|---|---|---|
| Domain | % ( ) | |||
| Cars | 171 | 59.6 | 76.6 | 26.5 (131) |
| Laptops | 176 | 85.8 | 95.5 | 47.0 (168) |
| Headphones | 49 | 49.0 | 57.1 | 39.3 (28) |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Requests | Distinct ref. names | Replies (total) | Replies per query | Ref. names per query | Score per reply |
|---|---|---|---|---|---|---|
| Cars | 800 | 1,681 | 5,195 | 5 [3, 7] | 5 [4, 8] | 2 [1, 3] |
| Laptops | 800 | 737 | 3,092 | 4 [3, 5] | 4 [3, 5] | 1 [1, 2] |
| Headphones | 226 | 481 | 968 | 3 [2, 6] | 3 [3, 5] | 1 [1, 2] |
| SE software | 2,445 | 2,349 | 4,663 | 1 [1, 2] | 1 [1, 2] | 1 [1, 2] |
| SE hardware | 1,266 | 1,397 | 2,053 | 1 [1, 2] | 1 [1, 1] | 1 [1, 3] |
| Experiment | Domains | Models | Queries per cell |
|---|---|---|---|
| Reference-granularity reanalysis | Cars, laptops, headphones | All seven | 800 / 800 / 226 |
| Objective configuration audit | Cars | All seven | 149 shared; 3 saved generations each |
| Objective connection audit | Headphones | All seven | 44 shared; 3 saved generations each |
| Matched enumeration and choice | Cars, laptops, headphones | Llama-70B | 171 / 176 / 49 |
| Recovered paired coverage | Cars, laptops, headphones | Six; no Mistral | 160 / 164 / 49 |
| Generated-pool choice | Cars; laptops; headphones | GPT-5.4 | 171 / 176 / 49; plus 171 archived car queries |
| Model | Reviewed support | Rule support | Conflict minimum (%; 95% CI) | Other/unknown |
|---|---|---|---|---|
| GPT-5 | 9 | 71 | 0.00 [–] | 367 |
| Claude Opus | 5 | 39 | 0.00 [–] | 403 |
| Llama-70B | 8 | 54 | 1.34 [0.00, 3.36] | 379 |
| Qwen-80B | 3 | 54 | 0.67 [0.00, 2.01] | 387 |
| Nemotron-49B | 5 | 54 | 0.67 [0.00, 1.79] | 385 |
| Llama-8B | 2 | 34 | 1.12 [0.00, 2.91] | 406 |
| Model | Supported | Conflicts | Unknown | Supported % [95% CI] |
|---|---|---|---|---|
| GPT-5 | 125 | 0 | 7 | 94.7 [88.6, 99.2] |
| Claude Opus | 128 | 0 | 4 | 97.0 [93.9, 99.2] |
| Llama-70B | 110 | 0 | 22 | 83.3 [72.0, 93.2] |
| Qwen-80B | 127 | 0 | 5 | 96.2 [91.7, 100.0] |
| Nemotron-49B | 99 | 0 | 33 | 75.0 [65.2, 84.1] |
| Llama-8B | 106 | 0 | 26 | 80.3 [71.2, 88.6] |
| Model | Invalid | Abstain | Match | Excl. | Eligible | Bad comp. | Ref. / presentations |
|---|---|---|---|---|---|---|---|
| GPT-5.4 | 8 | 1 | 39 | 0 | 402 | 0 | 77/804 (9.6%) |
| Claude Opus 4.8 | 14 | 3 | 36 | 5 | 392 | 1 | 53/784 (6.8%) |
| Llama-70B | 13 | 0 | 25 | 7 | 405 | 6 | 70/810 (8.6%) |
| Qwen-80B | 5 | 3 | 36 | 6 | 400 | 1 | 58/800 (7.2%) |
| Nemotron-49B | 91 | 16 | 21 | 5 | 317 | 2 | 70/634 (11.0%) |
| Llama-8B | 37 | 11 | 25 | 10 | 367 | 26 | 120/734 (16.3%) |
| Model | Initial temp. | Initial answers | Eligible | Both orders | Rate % [95% CI] |
|---|---|---|---|---|---|
| GPT-5.4 | 0.7 | 450 | 402 | 26 | 6.5 [3.0, 10.3] |
| GPT-5.4 | 0 | 150 | 132 | 7 | 5.3 [2.3, 9.1] |
| Opus 4.8 | 0.7 | 450 | 392 | 16 | 4.1 [1.5, 7.3] |
| Opus 4.8 | 0 | 150 | 129 | 4 | 3.1 [0.8, 6.2] |
| Model | Analysis | Preferences / trials | Rate (%; 95% CI) |
|---|---|---|---|
| GPT-5.4 | Frozen parser | 43 / 389 | 11.05 [6.57, 15.97] |
| GPT-5.4 | Parser-corrected | 44 / 402 | 10.95 [6.52, 15.83] |
| GPT-5.4 | Identity sensitivity | 44 / 400 | 11.00 [6.55, 15.88] |
| Claude Opus 4.8 | Frozen parser | 42 / 393 | 10.69 [6.44, 15.25] |
| Claude Opus 4.8 | Parser-corrected | 43 / 410 | 10.49 [6.37, 14.88] |
| Claude Opus 4.8 | Identity sensitivity | 41 / 403 | 10.17 [6.09, 14.64] |
| Domain | Model | Top [95% CI] | Set [95% CI] | Difference [95% CI] | Parsed (%) |
|---|---|---|---|---|---|
| Cars | GPT-5 | 6.8 [5.2, 8.5] | 24.8 [22.0, 27.6] | 18.1 [15.6, 20.6] | 95.3 |
| Cars | Claude Opus | 6.5 [5.0, 8.1] | 24.0 [21.4, 26.8] | 17.6 [15.1, 20.2] | 93.5 |
| Cars | Llama-70B | 6.0 [4.6, 7.5] | 24.5 [21.7, 27.0] | 18.5 [16.2, 20.9] | 96.7 |
| Cars | Qwen-80B | 5.2 [3.8, 6.8] | 23.2 [20.6, 26.0] | 18.0 [15.6, 20.4] | 97.5 |
| Cars | Nemotron-49B | 6.0 [4.6, 7.4] | 22.0 [19.5, 24.6] | 16.1 [13.9, 18.4] | 97.5 |
| Cars | Llama-8B | 5.6 [4.2, 7.0] | 21.3 [18.8, 24.0] | 15.7 [13.5, 18.1] | 95.8 |
| Model | Opening (%) | Full answer (%) | Difference (pp) | Unparsed |
|---|---|---|---|---|
| GPT-5 | 24.3 [20.3, 28.4] | 45.9 [41.3, 50.5] | 21.7 [18.4, 25.3] | 5 |
| Claude Opus | 20.8 [17.1, 24.8] | 44.7 [40.1, 49.4] | 23.8 [20.2, 27.7] | 48 |
| Llama-70B | 23.2 [19.3, 27.2] | 40.4 [35.9, 44.8] | 17.3 [14.2, 20.6] | 2 |
| Qwen-80B | 20.9 [17.3, 24.8] | 41.9 [37.3, 46.5] | 21.0 [17.7, 24.5] | 36 |
| Nemotron-49B | 17.5 [14.3, 20.8] | 37.4 [33.2, 41.8] | 19.9 [16.9, 23.1] | 60 |
| Llama-8B | 14.6 [11.8, 17.6] | 33.0 [28.9, 37.3] | 18.4 [15.5, 21.5] | 126 |
| Code | Intended failure mode | Evidence required for assessment |
|---|---|---|
| 1a | A relevant fact is omitted | Request spans, recovered facts, and a relevance criterion. |
| 1b | A fact is invented | The asserted fact and evidence that the request does not support it. |
| 1c | Valence or attribution is changed | Who expressed the preference, its target, and its polarity. |
| 2a | An inference is unsupported | The inferred belief and the facts claimed to license it. |
| 2b | A prior is presented as user-derived | Provenance of the belief and an explicit distinction between evidence and assumption. |
| 2c | A prior contradicts the request | A contradiction between an added belief and a supported user fact. |
| Code | Selected implementation | Scope or qualification |
|---|---|---|
| 1a | Regular-expression coverage | Limited to canonical query patterns. |
| 1b | Token overlap | Measures lexical overlap; semantic support requires separate assessment. |
| 1c | Polarity and target lexicons | Limited sentiment and attribution coverage. |
| 2a | Structural citation check | Inference validity requires checking the cited evidence. |
| 2b | Strict provenance-tag pattern | Measures explicit labeling of assumptions. |
| 2c | Generative judge | Requires judge calibration and records of its outputs. |
| Root | Both flag | Checker only | Assistant only | Both clear | Unknown | Agreement (%) |
|---|---|---|---|---|---|---|
| 1a | 63 | 17 | 25 | 33 | 2 | 96/138 (69.6) |
| 1b | 24 | 77 | 2 | 20 | 17 | 44/123 (35.8) |
| Metric | Baseline | Anchored | Difference [95% CI] |
|---|---|---|---|
| Top-entry agreement | 5.1 | 4.6 | -0.5 [-1.5, +0.6] |
| All-entry agreement | 22.2 | 20.2 | -1.9 [-4.0, -0.2] |
| Pick extracted | 97.0 | 96.1 | -0.9 [-2.1, +0.2] |
| Model | Cars ( ) | Laptops ( ) | Headphones ( ) |
|---|---|---|---|
| GPT-5 | 80.0 [73.8, 86.2] | 89.0 [84.1, 93.9] | 69.4 [55.1, 81.6] |
| Claude Opus | 78.8 [71.9, 85.0] | 91.5 [87.2, 95.7] | 75.5 [63.3, 87.8] |
| Llama-70B | 75.6 [68.8, 81.9] | 95.7 [92.1, 98.2] | 57.1 [42.9, 69.4] |
| Qwen-80B | 72.5 [65.6, 79.4] | 92.7 [88.4, 96.3] | 59.2 [44.9, 71.4] |
| Nemotron-49B | 73.8 [66.9, 80.6] | 93.3 [89.0, 97.0] | 49.0 [34.7, 63.3] |
| Llama-8B | 70.0 [63.1, 76.9] | 92.1 [87.8, 96.3] | 53.1 [38.8, 65.4] |
| Model | Cars | Laptops | Headphones |
|---|---|---|---|
| GPT-5 | 76.9 | 88.4 | 57.1 |
| Claude Opus | 77.5 | 91.5 | 61.2 |
| Llama-70B | 71.2 | 95.7 | 40.8 |
| Qwen-80B | 68.8 | 92.1 | 40.8 |
| Nemotron-49B | 68.1 | 92.7 | 34.7 |
| Llama-8B | 67.5 | 92.1 | 38.8 |
| Domain | Model / collection | Queries | Coverage@40 | Covered | Selection when covered |
|---|---|---|---|---|---|
| Cars | Llama-70B (archive) | 171 | 76.6 [70.2, 83.0] | 131 | 26.5 [18.8, 34.4] |
| Cars | GPT-5 (archive) | 171 | 80.7 [74.9, 86.0] | 138 | 33.3 [25.6, 41.6] |
| Cars | GPT-5.4 (Oct. 2026) | 171 | 80.7 [74.9, 86.0] | 138 | 30.9 [23.4, 37.9] |
| Headphones | Llama-70B (archive) | 49 | 57.1 [42.9, 69.4] | 28 | 39.3 [22.6, 57.1] |
| Headphones | GPT-5.4 (Oct. 2026) | 49 | 65.3 [53.1, 77.6] | 32 | 24.0 [9.4, 38.5] |
| Laptops | Llama-70B (archive) | 176 | 95.5 [92.0, 98.3] | 168 | 47.0 [40.1, 54.6] |
| Model | Raw /49 | Exact /49 | Broad /49 | Broad /40 (%) | Laptop gap (pp) |
|---|---|---|---|---|---|
| GPT-5 | 34 | 12 | 24 | 60.0 [44.4, 74.4] | 29.0 [13.7, 45.5] |
| Claude Opus | 37 | 17 | 29 | 72.5 [57.9, 85.7] | 19.0 [5.1, 34.0] |
| Llama-70B | 28 | 8 | 20 | 50.0 [35.3, 65.8] | 45.7 [29.8, 60.6] |
| Qwen-80B | 29 | 7 | 20 | 50.0 [34.2, 65.0] | 42.7 [26.9, 58.9] |
| Nemotron-49B | 24 | 7 | 19 | 47.5 [32.4, 63.4] | 45.8 [29.5, 61.4] |
| Llama-8B | 26 | 8 | 19 | 47.5 [32.5, 63.4] | 44.6 [27.7, 59.8] |
| Run | Reference rule | Coverage | Selected / covered slots | Selection (%; 95% CI) |
|---|---|---|---|---|
| Llama-70B | Raw | 28/49 | 33/84 | 39.3 [22.2, 57.0] |
| Llama-70B | Exact | 8/49 | 6/24 | 25.0 [0.0, 60.0] |
| Llama-70B | Broad | 20/49 | 20/60 | 33.3 [13.3, 54.3] |
| Llama-70B | Broad + identity | 13/49 | 9/39 | 23.1 [0.0, 50.0] |
| GPT-5.4 | Raw | 32/49 | 23/96 | 24.0 [9.9, 39.4] |
| GPT-5.4 | Exact | 14/49 | 6/42 | 14.3 [0.0, 35.7] |
| Reference and identity rule | Coverage at | Conditional selection |
|---|---|---|
| Specific; strict lexical | 22/194: 11.3 [7.1, 16.2] | 19/66: 28.8 [12.0, 49.3] |
| Specific; audited equivalence | 36/194: 18.6 [13.1, 24.5] | 31/108: 28.7 [15.1, 44.4] |
| Specific; source-explicit identifiers | 37/194: 19.1 [13.6, 25.0] | 34/111: 30.6 [17.2, 45.9] |
| Specific or named family | 51/226: 22.6 [17.4, 28.1] | 45/153: 29.4 [18.2, 42.5] |
| Domain | Coverage@10 | Coverage@40 | Free form | Enum. + choice | Change [95% CI] | |
|---|---|---|---|---|---|---|
| Cars | 171 | 59.6 | 76.6 | 25.3 | 20.3 | [ , ] |
| Laptops | 176 | 85.8 | 95.5 | 28.0 | 44.9 | [ , ] |
| Headphones | 49 | 49.0 | 57.1 | 22.4 | 22.4 | [ , ] |
| Domain | Model | Queries | Median lift [95% CI] | Lift (%) | Lift (%) |
|---|---|---|---|---|---|
| Cars | Llama-8B | 1,292 | 1.21 [1.09, 1.34] | 76.9 | 66.0 |
| Cars | Llama-70B | 1,292 | 0.44 [0.21, 0.77] | 55.3 | 49.5 |
| Cars | Qwen-80B | 1,292 | 1.61 [1.42, 1.77] | 73.2 | 67.0 |
| Laptops | Llama-8B | 902 | 0.85 [0.75, 0.94] | 74.8 | 61.5 |
| Laptops | Llama-70B | 902 | 1.42 [1.19, 1.70] | 68.5 | 60.1 |
| Laptops | Qwen-80B | 902 | 1.84 [1.66, 2.00] | 83.6 | 73.2 |
| Pool outcome | Queries | Mean lift [95% CI] | Positive lift (%) |
|---|---|---|---|
| Target present | 87 | 58 / 87 (66.7) | |
| Target absent | 69 | 28 / 69 (40.6) | |
| All references absent (subset) | 25 | 9 / 25 (36.0) |
| Community reference | Relevance label | |||||
| Model | Cars | Laptops | Headphones | SE software | SE hardware | ESCI |
| GPT-5 | 59.4 | 54.0 | 54.0 | 96.7 | 94.6 | 70.2 |
| Claude Opus | 62.5 | 58.5 | 51.5 | 96.3 | 94.4 | 73.6 |
| Llama-70B | 57.0 | 60.5 | 51.0 | 94.4 | 91.3 | 67.4 |
| Llama-8B | 56.9 | 51.1 | 46.0 | 93.7 | 92.3 | 58.6 |
| Metadata control | 86.2 | 64.3 | 88.4 | 97.3 | 97.9 | — |
| Model | Cars | Laptops | Headphones | SE software | SE hardware |
|---|---|---|---|---|---|
| GPT-5 | 5.0 59.4 | 16.4 54.0 | 37.9 54.0 | 1.8 96.7 | 2.4 94.6 |
| Claude Opus | 7.4 62.5 | 14.9 58.5 | 33.3 51.5 | 1.3 96.3 | 6.2 94.4 |
| Llama-70B | 7.8 57.0 | 27.7 60.5 | 28.8 51.0 | 3.5 94.4 | 16.1 91.3 |
| Llama-8B | 9.0 56.9 | 20.0 51.1 | 18.2 46.0 | 2.5 93.7 | 10.0 92.3 |
| Queries | 1,292 | 902 | 198 | 2,445 | 1,266 |
| Domain | Rank control (%) | Best model (%) | Difference [95% CI] |
|---|---|---|---|
| Cars | 86.2 | 62.5 | +23.7 [+20.7, +26.7] |
| Laptops | 64.3 | 60.5 | +3.8 [+0.2, +7.3] |
| Headphones | 88.4 | 54.0 | +34.3 [+26.3, +42.4] |
| SE software | 97.3 | 96.7 | +0.6 [-0.3, +1.6] |
| SE hardware | 97.9 | 94.6 | +3.3 [+2.0, +4.7] |
| Domain | GPT-5 | Claude Opus | Llama-70B | Llama-8B |
|---|---|---|---|---|
| Cars | 54.4 [51.5, 57.5] | 55.2 [52.2, 58.2] | 49.2 [46.1, 52.2] | 47.9 [44.9, 51.0] |
| Laptops | 37.6 [33.9, 41.5] | 43.7 [39.6, 47.6] | 32.8 [28.5, 36.9] | 31.2 [27.2, 35.3] |
| Headphones | 16.2 [7.1, 25.8] | 18.2 [10.1, 26.8] | 22.2 [13.1, 31.3] | 27.8 [19.2, 35.9] |
| SE software | 94.9 [93.9, 95.8] | 95.0 [94.1, 95.8] | 90.9 [89.7, 92.0] | 91.1 [89.9, 92.4] |
| SE hardware | 92.3 [90.8, 93.7] | 88.2 [86.3, 89.9] | 75.2 [72.7, 77.6] | 82.3 [80.2, 84.5] |
| Model | Request | Reference | Alternative | Equal | Neither | Insufficient | Invalid |
|---|---|---|---|---|---|---|---|
| GPT-5.4 | Real | 86 | 65 | 14 | 21 | 2 | 0 |
| GPT-5.4 | Absent | 0 | 0 | 0 | 0 | 188 | 0 |
| GPT-5.4 | Shuffled | 29 | 27 | 10 | 121 | 1 | 0 |
| Llama-70B | Real | 55 | 45 | 17 | 18 | 52 | 1 |
| Llama-70B | Absent | 0 | 0 | 0 | 0 | 188 | 0 |
| Llama-70B | Shuffled | 11 | 13 | 5 | 107 | 48 | 4 |
| Model | Total | Product switch | Non-selection | Invalid |
|---|---|---|---|---|
| GPT-5.4 | 30.3 [19.1, 41.5] | 1.6 [-3.2, 6.4] | 28.7 [18.6, 39.4] | 0.0 [0.0, 0.0] |
| Llama-70B | 23.4 [13.3, 34.0] | 0.5 [-1.6, 2.7] | 22.9 [13.3, 33.0] | 0.0 [0.0, 0.0] |
| Opus 4.8 | 31.4 [20.7, 42.0] | 4.3 [0.0, 9.0] | 27.1 [16.5, 37.2] | 0.0 [0.0, 0.0] |
| Qwen-80B | 27.7 [17.0, 38.3] | 0.0 [-4.8, 4.8] | 26.6 [18.1, 36.2] | 1.1 [0.0, 2.7] |
| Model | Hit rate (%) | 95% CI |
|---|---|---|
| Claude Opus | 73.6 | [69.6, 77.3] |
| GPT-5 | 70.2 | [66.0, 74.0] |
| Llama-70B | 67.4 | [63.2, 71.4] |
| Qwen-80B | 67.8 | [63.6, 71.7] |
| Llama-8B | 58.6 | [54.2, 62.8] |
Large Language Model-Based Recommendation
Large language model-based recommendation uses language models to personalize ranking, retrieval, and conversational suggestions from text and behavior.
See all 238 papers