cs.CLSep 24, 2026
SaveThe Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge
Organizations: University of Utah
Abstract
Existing computer-use agent benchmarks do not fully evaluate agents acting as assistants. A useful assistant retrieves information across complex, multi-step workflows, synthesizes it into artifacts (documents, presentations, spreadsheets), and navigates program interfaces to produce a coherent final product. Such workflows demand reasoning and synthesis, decomposition of complex tasks, as well as visual and spatial understanding. To study agents on workflows like these, we introduce KNOWS, a benchmark of open-ended, complex, browser-based tasks that jointly evaluate these capabilities, with each task culminating in a produced artifact. To write tasks, we develop a task design rubric and a protocol for ensuring that tasks meet the requirements. Each task is paired with an evaluator, a program that combines deterministic checks with LLM judgments to balance the richness, reliability, and automation tradeoff inherent to agent evaluation. We evaluate and analyze frontier computer-use agents and browser-based harnesses. They achieve moderate scores on partial-success metrics, but the best performer fully succeeds in fewer than 3% of our complex, long-horizon tasks. Failures on visual steps render the resulting artifacts unusable, even when agents complete more than 50% of other evaluation steps. Our results expose limitations of current agents acting as end-to-end assistants, and call for progress on tool use, visual understanding, and long-horizon reasoning.
Figures & tables
Figure 1: An overview of the knows benchmark.
Figure 2: An overview of the construction of the knows benchmark.
| Statistic | Value |
|---|---|
| Total Tasks | 110 |
| Prompt length | |
| Words (mean / median) | 204.4 / 165.0 |
| Sentences (mean / median) | 13.7 / 12.0 |
| Checkpoints per task | |
| Mean / Median | 5.9 / 5.0 |
Table 1: Core statistics of the knows dataset.
| Step Type | # Steps | % Steps | % Inst. |
|---|---|---|---|
| Structural | 835 | 30.7 | 90.9 |
| Info. Retrieval | 748 | 27.5 | 100.0 |
| Formatting | 332 | 12.2 | 78.2 |
| Content | 229 | 8.4 | 54.5 |
| Spatial | 199 | 7.3 | 59.1 |
| Web Visit | 189 | 7.0 | 68.2 |
Table 2: Distribution of types of checks knows steps evaluate. % Steps is the share of all 2,716 evaluation steps; % Inst. is the share of 110 tasks containing at least one step of that type.
| Method | Docs | Sheets | Slides | Overall | ||||||||||||
| SR | ASC | ACF | SF | SR | ASC | ACF | SF | SR | ASC | ACF | SF | SR | ASC | ACF | SF | |
| Textual (Accessibility Tree) | ||||||||||||||||
| GPT-5.5 | 0 | 18.0 | 38.5 | 36.6 | 0 | 26.8 | 46.8 | 44.4 | 0 | 15.1 | 38.6 | 31.7 | 0 | 20.6 | 41.9 | 38.0 |
| Claude Opus 4.7 | 0 | 7.7 | 19.9 | 21.9 | 0 | 3.0 | 12.3 | 10.0 | 0 | 9.3 | 20.3 | 12.5 | 0 | 6.4 | 17.0 | 13.6 |
| DeepSeek V4 Pro | 0 | 17.0 | 39.3 | 41.1 | 0 | 8.7 | 23.7 | 18.6 | 0 | 14.2 | 34.3 | 26.1 | 0 | 12.6 | 31.1 | 26.4 |
| Multimodal (Accessibility Tree + Screenshot) | ||||||||||||||||
Table 3: Results on the knows dataset. SR, ASC, ACF, and SF are metrics described in § 4.1 , with values reported as percentages (%). Baseline agents that are not AI Browsers use the BrowserGym-AgentLab harness.
Figure 3: Output produced by the best-performing agent, Comet, on a Slides task. It received moderate partial scores, ACF (0.54) / SF (0.59), but the output remains practically unusable (indicated by a low ASC of 0.14).
Figure 4: Breakdown of Comet’s verdicts across all 110 task instances: (a) on 2,508 steps, excluding web-visit steps; (b) on all 2,697 steps. Bar height is the number of evaluation steps, split into passed (hatched) and failed steps. The label above each bar gives the pass rate (%).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Use Case | Modalities | Tasks | Source |
|---|---|---|---|---|
| Letters | Business Letter | Docs | 5 | Taxonomy |
| Personal | Recipes | Docs, Sheets | 10 | Taxonomy |
| Education | Lesson Plan | Docs, Slides | 10 | Taxonomy |
| Education | Class Notes / Reference List | Docs | 5 | Taxonomy |
| Education | Research Report | Docs | 5 | Taxonomy |
| Education | Book Report | Slides | 5 | Taxonomy |
Table 4: Domain coverage of the knows benchmark. Each row is a distinct use-case domain identified by (category, use case). Domains are drawn from a Google Workspace use-case taxonomy defined in the task creation colab notebook (Appendix A.1 ). 11 fall within the pre-defined taxonomy (“Taxonomy”) and 9 are extensions we introduced to capture live web-heavy use cases the original design under-specified (“Extension”). Every domain is represented by at least 5 individual tasks via our template instance structure.
| Action | Description |
|---|---|
| click | Single left-click on an element |
| dblclick | Double-click on an element |
| drag_and_drop | Drag an element to a target location |
| fill | Clear and fill a text input |
| go_back | Navigate to the previous page |
| go_forward | Navigate to the next page |
Table 5: Action space of knows .
| System | Configs | Total |
| Usage-based (API) | ||
| Claude Opus 4.7 | AXT, AXT+SS | $2,200 |
| GPT-5.5 | AXT, AXT+SS | $1,700 |
| DeepSeek V4 Pro | AXT | $80 |
| Flat-rate (subscription) | ||
| Comet | — | $200 |
Table 6: Cost of one full evaluation pass over all 110 tasks. Totals are not comparable across rows: the API systems differ in the number of configurations run, and subscription tiers are billed monthly and are independent of usage volume.
| Step type | Passed / total steps | Share of failed steps (%) | ||||||
|---|---|---|---|---|---|---|---|---|
| Docs | Sheets | Slides | All | Docs | Sheets | Slides | All | |
| exact match | 74/129 (57%) | 500/566 (88%) | 176/206 (85%) | 750/901 (83%) | 30.4 | 23.7 | 8.7 | 18.8 |
| tolerance match | 43/60 (72%) | 130/207 (63%) | 53/80 (66%) | 226/347 (65%) | 9.4 | 27.7 | 7.9 | 15.1 |
| model judgment | 65/97 (67%) | 188/280 (67%) | 157/218 (72%) | 410/595 (69%) | 17.7 | 33.1 | 17.8 | 23.1 |
| geometric | 20/20 (100%) | 28/36 (78%) | 70/125 (56%) | 118/181 (65%) | 0.0 | 2.9 | 16.0 | 7.9 |
| tree query | 23/51 (45%) | 101/104 (97%) | 68/161 (42%) | 192/316 (61%) | 15.5 | 1.1 | 27.1 | 15.5 |
Table 7: Comet’s evaluation steps by scoring mechanism and artifact type, excluding the 189 web-visit steps. Left: passed / total steps (pass rate). Right: share of each artifact type’s failed steps decided by each mechanism. Last row gives number of failed steps.
| Failure class | Steps | % | Docs | Sheets | Slides |
|---|---|---|---|---|---|
| Check ran and rejected the artifact | 667 | 71.0 | 138 | 256 | 273 |
| Requires the browsing trace | 114 | 12.1 | 10 | 104 | 0 |
| Execution error / dependency skip | 158 | 16.8 | 49 | 32 | 77 |
| All failed steps | 939 | 100.0 | 197 | 392 | 350 |
Table 8: Attribution of Comet’s 939 failed evaluation steps. Only the first class is a verdict on the artifact. The second is unscorable without the browsing trace, and the third never reached their intended check.
| Category | Deciding mechanism |
|---|---|
| Exact Match | Exact or normalized comparison: string equality, regular expressions, exact numeric equality, pixel-exact image match, API-reported formatting flags. |
| Tolerance Match | Tolerance-based, non-geometric comparison: fuzzy string matching, perceptual image hashing, numeric comparison within a relative or absolute tolerance, color-distance thresholds. |
| Model Judgment | An LLM or VLM verdict decided the step (content relevance, semantic equivalence, image–description agreement). |
| Geometric | A geometric test over rendered coordinates decided the step: bounding-box containment or overlap, page-region position, area coverage, OCR-localized text position. |
| Tree Query | An artifact structural test decided the outcome of a step: element counts, ordering, position within the document tree, slide or column structure. |
| Trace Match | A string or pattern match over the agent’s browsing history decided the step. These check that the agent visited the websites a task requires. |
Table 9: Failure-category taxonomy. Each evaluation step records the category of the mechanism that decided its outcome.
| Task | MS 365 | Google Workspace | |
|---|---|---|---|
| docs_1_formal_letter | 8/10 | 8/10 | |
| docs_31_education_lesson_plan | 99/145 | 132/145 | |
| slides_29_buy_car_pres | 57/68 | 62/68 | |
| slides_42_personal_none_product_comparison | 43/57 | 26/57 | |
| slides_51_event_announcement_poster | 175/185 | 100/185 | |
| Combined | 382/465 (82%) | 328/465 (71%) |
Table 10: Cross-platform comparison of agent performance on five knows tasks implemented in both Microsoft 365 and Google Workspace. Scores are evaluation steps scores out of total points available for each task. is the difference in points earned (MS GWS).
Figure 5: Output produced by the best performing agent, Comet, on a Sheets task. The agent makes several errors, including formula errors, inconsistent table structure, misplaced formulas, and an inability to generate visualizations correctly.
Figure 6: Output produced by the best performing agent, Comet, on a Docs task. The agent generates deeply nested bullet points, making the document difficult to navigate and unpresentable.
| Task | Time |
|---|---|
| docs_5_influential_papers | 24m |
| slides_20_Illustrated_Book_Report | 44m |
| docs_31_education_lesson_plan | 55m |
| sheets_6_investmenttracker | 1.4h |
| docs_1_formal_letter | 1.6h |
| slides_39_Personal_Lookbook_PaintColors | 1.6h |
Table 11: Human completion times for each task template (instance 1), measured from Google Drive Activity API edit logs. Times are sorted in ascending order.
| Shortname | Task Instruction |
|---|---|
| docs_1_formal_letter | Structure a formal letter written in my name using Google Docs. Use my website for my info: [redacted]. Include my employers logo in the upper left corner if available online. Include the following information in the letter header: - My Name - My Email Address - My Job/Position Title Take the photo of my signature from the drive folder here: [redacted] and add it to the end of the letter. |
| docs_11_personal_recipe_ocr | Create a recipe from the Recipe Coral template in Google Docs. First, replace the name of the recipe with the correct recipe name by looking at the image saved in the drive at [redacted]. Then, copy the ingredient list into the template and paste the instructions into the corresponding field. Get tips from an online source, and make sure to cite source urls for tips below the tips entries. Use direct quotes for all tips. Replace the default image with a cropped image from the original drive link that shows the recipe text clearly. Fill out the rest of the template with relevant information from the recipe. Then search the internet for at least 3 similar recipes and put them on separate pages with the same format as the first recipe. Put the link for the source of each additional recipe on a line below the title fitting with the formatting. Make sure each page of recipes is visually distinct from the others. Finally, check the other online recipes and fill in the tips with some useful information that you found. |
| docs_31_education_lesson_plan | I am an Elementary School educator, and I have a free class to teach a fun topic. My class is on Deep Learning. Can you help me make a lesson plan document appropriate for my students? First, come up with 2-3 fun topics in Deep Learning that the class might enjoy but is still related and list those as the main bullets. Then, under each of these, find 4 unique websites that have information about the topic and put their URLs as sub-bullets. Then, for each of these websites, list 5 facts as sub-bullets to the url. Finally, list all of the facts below the bullets and color code them by which topic they are about with the highlighter. Use a different color for each topic. Write a summary of the lesson using the facts collected. Finally, go online to find images related to each of the topics. Put them all side-by-side at the bottom of the last page of the document. |
| docs_37_reference_list | Create a Google Doc reference list from this course webpage: [redacted]. For each lecture, add a Heading 3 with ’Module N: Lecture title’, where N is the module number. Things like exams and holidays should not appear in the reference list. Under each title, create a separate bullet list for tutorials, textbooks, and videos. Each bullet list should start with "Tutorials", "Textbooks", or "Videos" in bold, normal font. If a given type has no hyperlinks, delete that category. Each bullet is a hyperlink to a resource referenced in the lecture slides, using the resource title as the anchor text instead of the raw URL. Include author(s) in parentheses after the hyperlink, formatted as "(First author)". Next to the hyperlink, add "Slide: <list of slide numbers separated by commas>" listing the slides where the link is referenced. No duplicates: if something is referenced multiple times, then add all slide numbers. If a link is referenced on more than one slide, bold that hyperlink (and only the hyperlink) in the corresponding bullet list, italicize it, color it ’dark cyan 1’, and set its font size to 12pt. If the link is inactive (dead link) do not include it in the google doc. |
| docs_5_influential_papers | Create a Google Docs with five influential papers on parameter-efficient fine-tuning (PEFT). Include links to arxiv.org for each of the papers. The papers should have at least 100 citations, have been first published within the last three years, and must have an active page on semanticscholar.org. Include the name of the paper, a link below the name, and the abstract of each of the papers below the link. |
| sheets_10_paper_sorting | Create a new google sheets document for all the papers in my drive folder [redacted]. Make a blank google sheets table and for each paper, in column A put the title of the paper, in column B put the paper authors, in column C put the abstract, in column D put a link to the arxiv abstract website, in column E put the links to the google drive pdfs. Additionally, search the web for at least three more papers on arxiv where the first author of each existing paper is also listed as the first author (or less if they do not have at least 3 first-authorships on arxiv) and put the same information. For new papers, download the PDFs and upload them to the pdfs folder in this drive folder: [redacted] and link them in column E. Extract or take a screenshot of Figure 1 for each paper, upload the image to the figures folder in this drive folder: [redacted] and display it in column F using =IMAGE() with the Drive URL. In column G put a checkbox and check for each paper that is a new paper. Highlight papers that have a related works sections which mention dark energy in yellow, and make sure to group these papers at the top rows of the sheet. Ensure all of the text in the table is visible and not cut off. |
Table 12: Example task prompts from KNOWS. One randomly chosen instance per task; URLs replaced with [redacted] .