Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It
Organizations: Graduate School of Data Science, Seoul National University
Abstract
As LLM agents decide on users' behalf which product to buy, which hotel to book, or which paper to cite, a preference for items from certain sources (the sites or services they come from) shapes what users receive and which sources are selected. We study source preference in end-to-end search with 12 agent models across three domains. Comparing items from different sources that satisfy the same requirements at the same position, we find that each model prefers some sources and avoids others in every domain, largely agreeing on which. This preference can outweigh how well items satisfy the request: an item satisfying one requirement fewer is selected about two-thirds of the time when it comes from a preferred source and the better one from a dispreferred source, but almost never in the reverse case. The information identifying an item's source affects selection by itself: hiding it weakens the preference, and relabeling an item with a preferred source raises its selection rate. We test two routes to this preference: training that rewards better items can make a source a shortcut for requirement satisfaction, and missing information can trigger preconceptions about the source. Supplying missing information or a prompt countering these preconceptions reduces source preference.
Figures & tables
| Shopping | Accommodation | Scholar | |||||||||||
| Family | Model | Amazon | Walmart | eBay | Target | Tripadvisor | Expedia | Booking.com | arXiv | ScienceDirect | OpenReview | Medium | |
| 12% | 6% | 5% | 3% | 9% | 7% | 4% | 3% | 12% | 4% | 4% | 4% | ||
| Qwen | 3.5-27B | ||||||||||||
| 3-30B-A3B | |||||||||||||
| 2.5-32B | |||||||||||||
| GPT | 5.4-nano | ||||||||||||
| Condition | Title / content | URL |
|---|---|---|
| Hidden | source names masked | hidden |
| URL-only | source names masked | shown |
| Original | original text | shown |
| Model | Shopping | Accommodation | Scholar |
|---|---|---|---|
| GPT-5.4-nano | |||
| GPT-5.6-Luna | |||
| GLM-5.3-Flash | |||
| DeepSeek-v4-Flash | |||
| Gemini-3.7-Flash | |||
| Llama-4-Maverick |
| Pre-DPO | DPO ( ) | |||
|---|---|---|---|---|
| Model | ( ) | Balanced | Aligned | Reversed |
| Qwen3.5-9B | ||||
| Llama-3.1-8B | ||||
| Tulu-3-8B-SFT | ||||
| Base (%) | Change from Base (pp) | |||
|---|---|---|---|---|
| Model | Same Price | General | Designate | |
| GPT-5.4-nano | 68.8 | |||
| Llama-4-Maverick | 95.7 | |||
| Llama-3.1-8B-It | 83.9 | |||
| Qwen3.5-27B | 96.4 | |||
| Tulu-3-8B | 87.6 | |||
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Benchmark | #Requests | Construction |
|---|---|---|---|
| Shopping | WebShop | 1,500 | First 1,500 from shuffled goals |
| Accommodation | HotelQuEST | 786 | All 214 + 572 location-substituted variants |
| Scholar | ScholarGym | 2,536 | All queries |
| Total | 4,822 |
| Agreement | Kripp. | Gwet’s AC1 | ||||||
|---|---|---|---|---|---|---|---|---|
| Domain | # Items | # Features | H–H | H–L | H–H | H–L | H–H | H–L |
| Shopping | 375 | 1,265 | .86 | .88 | .72 | .75 | .72 | .76 |
| Accommodation | 375 | 1,198 | .86 | .89 | .69 | .74 | .75 | .80 |
| Scholar | 100 | 272 | .85 | .85 | .67 | .65 | .72 | .73 |
| Model | Requests | Pairs | Comparisons | Rated | Source | Source | Source |
| Shopping | |||||||
| GPT-5.4-nano | 1,373 | 9,857 | 86,362 | 110 | 5 | 17 | 88 |
| GPT-5.6-Luna | 1,340 | 11,594 | 105,532 | 106 | 12 | 32 | 62 |
| Gemini-3.7-Flash | 785 | 7,053 | 50,866 | 77 | 7 | 8 | 62 |
| GLM-5.3-Flash | 1,318 | 10,887 | 82,077 | 138 | 12 | 21 | 105 |
| Deepseek-v4-flash | 1,094 | 8,232 | 61,039 | 95 | 8 | 8 | 79 |
| Shopping | Accommodation | Scholar | ||||
|---|---|---|---|---|---|---|
| Model | MAR | MAR | MAR | |||
| GPT-5.4-nano | 0.64 | 0.038 | 0.76 | 0.052 | 0.69 | 0.048 |
| GPT-5.6-Luna | 0.72 | 0.028 | 0.72 | 0.056 | 0.65 | 0.052 |
| Gemini-3.7-Flash | 0.66 | 0.028 | 0.67 | 0.055 | 0.67 | 0.052 |
| GLM-5.3-Flash | 0.74 | 0.048 | 0.72 | 0.051 | 0.64 | 0.050 |
| Deepseek-v4-flash | 0.74 | 0.041 | 0.70 | 0.052 | 0.62 | 0.049 |
| Rank | Source | Share | GPT | Luna | Gem | GLM | DS | Mav | Sct | L3.1 | Tulu | Q3.5 | Q3 | Q2.5 | Maj. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Amazon | 12.2% | |||||||||||||
| 2 | Walmart | 6.5% | |||||||||||||
| 3 | eBay | 5.1% | – | ||||||||||||
| 4 | Target | 2.7% | |||||||||||||
| 5 | YouTube | 2.6% | |||||||||||||
| 6 | 1.9% |
| Rank | Source | Share | GPT | Luna | Gem | GLM | DS | Mav | Sct | L3.1 | Tulu | Q3.5 | Q3 | Q2.5 | Maj. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Tripadvisor | 8.9% | |||||||||||||
| 2 | 6.5% | ||||||||||||||
| 3 | Expedia | 4.3% | – | ||||||||||||
| 4 | Booking.com | 3.5% | |||||||||||||
| 5 | 3.0% | ||||||||||||||
| 6 | Hotels.com | 2.2% | – |
| Rank | Source | Share | GPT | Luna | Gem | GLM | DS | Mav | Sct | L3.1 | Tulu | Q3.5 | Q3 | Q2.5 | Maj. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | arXiv | 11.5% | |||||||||||||
| 2 | ScienceDirect | 4.2% | |||||||||||||
| 3 | OpenReview | 3.9% | |||||||||||||
| 4 | Medium | 3.9% | |||||||||||||
| 5 | GitHub | 3.6% | – | ||||||||||||
| 6 | GitHub Pages | 2.9% |
| Domain | Source | Rank | Source (models, ) | Source (models, ) | Source |
| Shopping | Amazon | 1 | 11 models | GPT ( 6) | 0 |
| eBay | 3 | Luna (+23), Gem (+12), GLM (+16), DS (+9), Q2.5 (+8) | GPT ( 18) | 6 | |
| Accommodation | Forbes Travel Guide | 34 | Q3 (+22) | Mav ( 20) | 10 |
| Scholar | GitHub | 5 | GLM (+14), DS (+19), Q3.5 (+12), Q2.5 (+4) | Luna ( 7), Gem ( 10), L3.1 ( 6), Tulu ( 7) | 4 |
| ResearchGate | 8 | Gem (+19), Q2.5 (+4) | GPT ( 11), Luna ( 5) | 8 | |
| Emergent Mind | 12 | Mav (+13), Sct (+6), L3.1 (+4), Tulu (+4), Q3 (+10), Q2.5 (+9) | GPT ( 17), Luna ( 22), Gem ( 15) | 3 |
| Shopping | Accommodation | Scholar | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | |||||||||
| GPT-5.4-nano | 3 (2–5; 2,021) | 20 (15–25; 1,646) | 54 (46–62; 1,148) | 7 (3–12; 1,226) | 26 (21–31; 3,142) | 72 (56–84; 380) | 1 (1–1; 20,538) | 19 (14–25; 1,770) | 68 (64–73; 6,489) |
| GPT-5.6-Luna | 0 (0–1; 7,313) | 18 (7–33; 369) | 92 (85–97; 644) | 3 (1–6; 899) | 14 (10–18; 5,310) | 69 (52–83; 222) | 2 (2–3; 24,847) | 20 (16–25; 3,995) | 69 (65–73; 10,597) |
| Gemini-3.7-Flash | 0 (0–0; 926) | 19 (7–34; 447) | 97 (85–100; 39) | 0 (0–0; 44) | 10 (8–13; 8,288) | – | 2 (1–3; 16,286) | 9 (5–13; 1,428) | 48 (42–53; 5,063) |
| GLM-5.3-Flash | 1 (0–1; 3,147) | 22 (16–29; 1,534) | 71 (55–88; 334) | 1 (0–2; 1,052) | 16 (13–19; 7,783) | 85 (70–96; 201) | 1 (1–2; 12,336) | 15 (11–18; 3,784) | 63 (56–69; 2,737) |
| Deepseek-v4-flash | 1 (0–1; 1,818) | 18 (14–23; 1,378) | 77 (63–89; 300) | 2 (1–5; 790) | 19 (15–23; 5,823) | 50 (30–75; 133) | 1 (1–2; 9,527) | 17 (14–21; 3,545) | 56 (49–64; 2,163) |
| Shopping | Accommodation | Scholar | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | ||||||||||||
| GPT-5.4-nano | 36 (28–45) | 40 (33–46) | 7 (5–10) | 15 (11–20) | 43 (28–59) | 57 (51–64) | 7 (5–9) | 10 (5–17) | 34 (29–38) | 43 (35–52) | 5 (3–8) | 6 (5–7) |
| GPT-5.6-Luna | 30 (19–42) | 78 (56–91) | 1 (0–2) | 4 (1–7) | 43 (30–55) | 34 (27–42) | 4 (3–6) | 9 (4–16) | 35 (31–39) | 41 (34–47) | 7 (5–10) | 8 (6–9) |
| Gemini-3.7-Flash | 27 (12–42) | 81 (42–100) | 1 (0–4) | 8 (4–14) | 57 (0–100) | 59 (46–72) | 1 (0–2) | 2 (0–12) | 21 (17–25) | 26 (19–34) | 5 (3–7) | 4 (2–5) |
| GLM-5.3-Flash | 32 (25–39) | 84 (73–93) | 1 (0–1) | 9 (6–12) | 38 (25–51) | 58 (49–67) | 3 (2–5) | 5 (3–8) | 34 (30–38) | 35 (27–45) | 4 (2–6) | 6 (5–7) |
| Deepseek-v4-flash | 32 (25–40) | 74 (57–87) | 3 (1–6) | 12 (9–16) | 34 (22–47) | 48 (40–57) | 5 (3–7) | 6 (3–9) | 31 (28–36) | 33 (25–42) | 4 (2–6) | 7 (6–9) |
| Model | WebShop | Accommodation | Scholar |
|---|---|---|---|
| GPT-5.4-nano | (446) | (158) | (1,846) |
| GPT-5.6-Luna | (397) | (154) | (1,979) |
| GLM-5.3-Flash | (248) | (106) | (990) |
| DeepSeek-v4-Flash | (189) | (110) | (890) |
| Gemini-3.7-Flash | (161) | (15) | (1,460) |
| Llama-4-Maverick | (289) | (248) | (1,138) |
| Setting | Value |
|---|---|
| Unique training pairs / rendered examples | 5,000 / 10,000 |
| Data-assignment and training seed | 17 |
| Optimizer updates | 5,000 |
| Microbatch size / accumulation steps | 4 / 2 |
| Effective batch size | 8 |
| Optimizer | Fused AdamW |