How Web Research Agents Interpret User Priorities
Organizations: Independent Researcher Burlington, VT, USA · University of Vermont Burlington, VT, USA
Abstract
Web research agents gather information online and write cited reports, including financial analysis. Users can state a priority to guide this research, but a priority may not say how to judge the options, leaving the agent to choose the criteria. We examine how two such agents, built on OpenAI and Anthropic models, interpret a stated priority. In 36 runs, the agents compared Microsoft and Alphabet as businesses. Prompts asked for special attention to growth or resilience, or gave no priority. We analyzed reports, searches, returned sources and cited sources. Of the 24 reports given a priority, 21 focused on it, but their criteria varied. Under the growth priority, OpenAI reports that ranked the companies accounted for risk or predictability and favored Microsoft, while the Anthropic ones judged growth by its pace and favored Alphabet. Under the resilience priority, reports used partly different criteria, and most named Microsoft as the more resilient company. Of the search and source measures tested, only the Anthropic agent's search wording differed detectably by condition. These findings suggest evaluations should examine the criteria behind a report's judgements, not only its focus or accuracy. Interfaces could let users see and adjust those criteria.
Figures & tables
| Label | Applied to | Values | Records |
|---|---|---|---|
| Report orientation | Reports | Growth, resilience, balanced or unclear | The report’s focus, the priority that organized it (RQ1) |
| Overall preference | Reports | Microsoft, Alphabet, no overall choice stated or unclear | Which company the report favored overall (RQ1) |
| Growth comparison | Reports | Microsoft stronger, Alphabet stronger, not stated or unclear | Which company the report judged stronger on growth (RQ1) |
| Resilience comparison | Reports | Microsoft stronger, Alphabet stronger, not stated or unclear | Which company the report judged more resilient (RQ1) |
| Microsoft safer / predictable | Reports | Yes, no or unclear | Whether the report called Microsoft safer or more predictable (RQ1) |
| Alphabet higher upside | Reports | Yes, no or unclear | Whether the report said Alphabet had more upside (RQ1) |
| Condition | OpenAI focus | OpenAI overall pick | Anthropic focus | Anthropic overall pick |
|---|---|---|---|---|
| Neutral | 6 balanced | Microsoft, 2 of 6 | 6 balanced | No overall pick |
| Growth | 5 growth, 1 balanced | Microsoft, 3 of 6 | 4 growth, 2 balanced | No overall pick |
| Resilience | 6 resilience | Microsoft, 1 of 6 | 6 resilience | No overall pick |
| OpenAI | Anthropic | |
| Sources cited per report (average) | 7 | 35 |
| Different sources cited across 18 runs | 21 | 391 |
| Citations to sources cited under all three conditions | 94% | 24% |
| Source overlap between two reports | 44% | 5% |
| Search-word overlap, same vs. different prompt | 43% vs. 43% | 41% vs. 33% |
| Search-word comparison across all three conditions, adjusted | 1 | .009 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Measure | Compared | OpenAI | Anthropic |
|---|---|---|---|
| Report words | All | .0008 / .035 | .0004 / .018 |
| G vs R | .0065 / .24 | .0022 / .09 | |
| G vs N | .089 / 1 | .15 / 1 | |
| R vs N | .0022 / .09 | .0043 / .16 | |
| Query words | All | .52 / 1 | .0002 / .009 |
| G vs R | .14 / 1 | .0022 / .09 |
| Run | Overall | Growth | Resilience | Criterion | Quote |
| OpenAI agent | |||||
| Neutral 01 | – | – | – | – | |
| Neutral 02 | Microsoft | – | – | – | |
| Neutral 03 | Microsoft | – | Microsoft | Not classified | “If assessing business resilience and predictability, I would favor Microsoft.” |
| Neutral 04 | – | – | – | – | |
| Neutral 05 | – | – | – | – | |
Web Agents
Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.
See all 191 papers