Computer-use agents are limited by the interfaces humans give them. An agent may type a spreadsheet cell by cell when a single line of code would do, or print all two thousand lines of a file through a shell to find one function. Both the interfaces and the knowledge of when to use each are built by hand, which cannot keep pace with new applications, workflows, and models. We argue that interface design should itself be an agent capability. We introduce Agent S4, an autonomous computer-use scientist that builds Evolving Agent Interfaces from task experience. A meta-agent examines task agents' trajectories on synthetic training tasks, without success labels, and constructs and revises interfaces and usage knowledge in a repository. Population-based evolution then selects which repositories to develop further by their performance on validation tasks the meta-agent never sees, favoring changes that generalize beyond the seen tasks. On OSWorld V2, Agent S4, with Qwen3.8-27B and the evolved repository of interfaces, sets a new state of the art in open-source computer-use agents, reaching 21.3% binary score, 3.8 times the same model with fixed interfaces. This is above both GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their respective costs. On ScienceBoard, the evolved interfaces improve the success rate from 41.4% to 58.6% with the same model. These results suggest that useful interfaces can be evolved from experience, making autonomous interface design a source of improvement complementary to advances in models and harnesses. We open-source our code at https://github.com/simular-ai/agent-s.
Figures & tables
Figure 1: Conventional computer-use agents execute tasks through fixed, human-designed interfaces. Agent S4 evolves interfaces as software repositories through experience. A task agent attempts synthetic tasks, and a meta-agent studies its trajectories to construct and refine the repositories. Each repository combines interfaces, such as GUI interactions, file operations and application-specific workflows, with knowledge of when to use them.
Figure 2: With Evolving Agent Interfaces built by Agent S4, Qwen3.8-27B exceeds GPT-5.5 and Opus 4.8 in binary success on the challenging OSWorld V2 benchmark at a small fraction of their cost.
Figure 3: Agent S4 turns task experience into an evolving repository of interfaces and knowledge. Starting from parents selected by held-out validation, the meta-agent studies their task trajectories to identify limitations, then revises one repository through mutation or combines ideas from two through crossover. Task agents then use the descendant to produce trajectories on synthetic tasks and a separate score on held-out validation tasks. Each descendant joins the population alongside its parents, allowing earlier designs to be revisited and recombined over successive iterations.
Benchmark
Search
Scratch VM
Train
Val.
Descendants
OSWorld V2
Population
No
30
15
30
OSWorld V2
Population
Yes
30
15
30
OSWorld V2
Linear
Yes
30
15
30
ScienceBoard
Population
No
48
15
30
Table 1: Repository-evolution runs. Training tasks are synthetic and ungraded, and validation tasks are official. Descendant counts exclude the seed. Scratch VM access applies to the meta-agent during construction.
Model / Method
Interface Type
Parameters
Binary (%) ↑
Partial (%) ↑
Cost/task ()\downarrow$
Open-source models
MiniMax M3 (single)
Fixed
428B (23B)
4.60
22.30
2.40
Kimi K2.6 (single)
Fixed
1T (32B)
4.60
22.10
6.60
Qwen CUA (Qwen3.8-27B)
Fixed
27B
5.56
20.69
1.39
Evolved repo. (w/ scratch VM)
Evolving
27B
19.44
51.06
1.27
Evolved repo. (w/o scratch VM)
Evolving
27B
21.30
46.90
1.10
Table 2: Results on the full OSWorld-v2 Benchmark. With the same Qwen3.8-27B model, repositories of interfaces evolved by Agent S4 outperform the fixed interfaces of the Qwen CUA at lower cost. Evolved rows use the best repository from population evolution with and without a scratch VM for the meta-agent. Section 7.1 compares them with the seed repository and other evolution strategies.
Method
Success (%) ↑
Qwen CUA
41.4
Agent S4 Seed repository
55.03
Agent S4 Evolved repository
58.58
Table 3: Results on the full ScienceBoard benchmark. Agent S4 rows use Qwen3.8-27B in Prime Agent with the seed or the evolved repository.
Condition
Binary (%) ↑
Partial (%) ↑
No Repository
15.74
44.93
Agent S4 Seed Repository
19.44
44.90
Evolution strategy
Linear, scratch VM
19.44
47.17
Population, scratch VM
19.44
51.06
Population, no scratch VM
21.30
46.90
Table 4: Effect of the seed repository and comparison of evolution strategies on the full OSWorld V2 benchmark. All rows use Qwen3.8-27B in Prime Agent. Each evolution row reports the best repository selected by that strategy. Appendix A.2 lists the contents of the seed repository. With a scratch VM, the meta-agent can open applications and test interfaces while building the repository.
Figure 4: Interface discovery and extension across 30 candidates per population run. a , New interfaces (blue): distinct coherent interfaces absent from the seed, labeled by their implemented scope across the run. b , Interface extensions (orange): newly introduced operations first recorded after their interface’s introduction, each labeled independently by its own behavior.
Figure 5: Human-curated versus Evolved Interface. Top: the Qwen CUA baseline uses a fixed, human-designed interface prioritizing frequent screenshots with mouse and keyboard actions. It reads the receipts and searches the card charges through repeated GUI interactions (steps 21–111). Bottom: Prime Agent with the repository evolved by Agent S4. It reads the mailbox and the bank records through the evolved browser interface, whose browser.api sends requests from inside the already logged-in tab and returns the data as text (calls 17 and 44). It fetches the attachments with curl , and reads them with pdftotext .
Figure 6: Repeated validation separates a high first score from consistent performance. Red shows the best first-evaluation score on ScienceBoard and blue the best mean over repeated evaluations.
Read and write GNOME settings, and list the parts and extract the text of Office files
cli.py
capabilities , doc , call
List interfaces, show the documentation of one, and call one
setup.sh
—
Install scrot , xdotool , xclip , curl and wmctrl
Appendix
Table 5: Contents of the seed repository shared by all evolution runs.
Figure 7: Useful repositories can emerge by returning to an earlier branch. OSWorld V2 lineage over 30 construction steps with scratch-VM access. Blue nodes denote mutation and orange nodes crossover; solid and dashed edges identify the first and second parents. Dashed node outlines mark eliminated variants. The bars report mean partial score and binary success on 15 validation tasks. Step 24 combines steps 2 and 6 and reaches the highest partial score ( 50.2% ), while tying the best binary success ( 26.7% ). Vertical dashed lines mark the seed scores; the shaded band uses the ±1 standard error estimated from the same validation set and model in the run without scratch-VM access.
Figure 8: ScienceBoard evolution combines branches under repeated validation. Repository lineage over 30 construction steps with a 60-minute episode limit. Colors and edge styles follow Figure 7 . Bars show each repository’s mean success across its available evaluations on 15 validation tasks, including repeats where available; they need not equal the scores available when the repository was selected as a parent. The dashed line marks the seed’s initial 53.3% success. The crossover at step 16 has the highest mean, 71.7% over four evaluations.
Task subset
Seed
Evolved ( if-011 )
All tasks
163.2 ( n=108 )
162.1 ( n=108 )
Each repository’s full-credit tasks
111.6 ( n=21 )
103.7 ( n=23 )
Appendix
Table 6: The evolved repository uses fewer calls on its full-credit tasks, while the all-task average remains similar. Mean executed IPython tool calls across 108 tasks and each repository’s full-credit subset. Parentheses give task counts. Full credit means a score of 1.0; the subsets contain different tasks. Counts include failed calls and use the final retained attempt, including the interrupted evolved attempt on task 063, but exclude earlier infrastructure retries.
Figure 9: How the task agent uses an evolved repository. (a) On an OSWorld V2 task, the task agent lists the available interfaces, reads the usage notes of one and calls it from its code. Outputs are shortened. (b) The repository selected without a scratch VM (Table 2 ) is a Python package. Each interface is a module of functions whose docstrings say how and when to use them, including lessons from earlier trajectories. Docstrings are shortened. run_batch , api and the lofile module were added during evolution.
Interface
What the agent observes
What the agent can do
Desktop control and background jobs
Windows and their accessible controls; the status and logs of programs running in the background
Run a list of GUI actions and get back how many ran and any error; launch, monitor and wait for long jobs such as video renders
Web-app access in the browser
The text and data of pages in the user’s logged-in session, rather than their pixels
Run code inside a page; send requests from the logged-in tab, reused across email, banking and chat sites
Office-document editing (LibreOffice)
Cells, formulas, paragraphs, slides and shapes of an open document
Write a block of cells at once; edit document objects; export to PDF
Document reading without the application
Text, metadata and rendered pages of a file, without opening the application that made it
Extract and search content; check whether an exported PDF is complete
Score editing (MusicXML)
Parts, measures and notes with pitch and duration
Add notes and rests; change pitch and duration; add dynamics; validate the score
Molecular-viewer control (ChimeraX)
Which atoms and residues a selection or command affects
Send viewer commands directly, such as coloring one residue of one chain
Appendix
Table 7: Examples of interfaces created or extended by evolution. Each changes what the agent observes about part of the computer and what the agent can do to it in one call. The desktop-control and web-app interfaces begin in the seed with basic operations and are extended during evolution, and the others are new. A fixed-interface agent observes all of these through screenshots and acts on them through clicks and keystrokes.
Figure 10: Reusable browser helpers expose the instructions needed for the rest of the task. Cropped frame from the evolved task-046 recording showing the TeamChat direct message. It asks the agent to check payments and deliver photos or previews accordingly. The box abbreviates the helper names and arguments passed through the CLI in tool cell 11. The frame illustrates the page content; operation timing is measured from the trajectory, independently of the recording.
Figure 11: Reusing CDP access reduces the setup needed to read the same message. Selected milestones from task 046 compare the seed’s client implementation and debugging with the evolved repository’s existing browser helpers. Both retrieve the requested direct message, at tool cells 37 and 11 respectively. Times run from the initial task message to the tool result containing that message. Boxes summarize recorded events and are not proportional to elapsed time; the evolved agent also encounters and corrects an initial JavaScript error.
Measure
Seed
Evolved ( if-011 )
Executed tool cells
299
167
Image-bearing tool results
68
14
Recorded duration (min)
97.5
51.1
Task score
0.50
1.00
Appendix
Table 8: Observed trajectories for the same client photo-delivery task (task 046). Tool cells count executed IPython calls; image-bearing results count cells returning images. Recorded duration spans the initial task message to the last recorded message. Each column represents one trajectory.
Figure 12: Reading a mailbox in MailHub ( s_010 ). The seed agent reads the case emails through the GUI in 60 calls (8–67). Its first six clicks land above the email list because it uses coordinates from its 1200-pixel screenshots as screen coordinates. A click meant for the inbox opens Starred, and its attempt to read the page from code is refused with 403 Forbidden . With if-011 , page_text lists the inbox and one cdp_eval call fetches the mailbox with full email bodies inside the logged-in tab, in 12 calls and without a screenshot.
Figure 13: Exporting an XMind map ( s_005 ). The seed agent tries to export a PNG through XMind’s menus in calls 249–299. Its click on the Export submenu lands between PNG and JPEG, and an “Export to JPEG” panel opens. Its clicks on Cancel miss, and the episode ends at the call limit without an image. With if-029 from a separate evolution run, the agent builds the map with xmind.build_map and save_map , and one call to xmind.render_png writes the image.
Figure 14: Reading a Zotero library ( s_007 ). The seed agent checks items and notes through the GUI in 42 calls (59–100). Nine clicks meant for the Cancel button of a plugin dialog miss (calls 61–79), and the agent closes the dialog with an Escape key press at call 82. Later it clicks outside the screen and asks a vision model to locate the Notes tab. With if-029 from a separate evolution run, zotero.list_collections and list_items return the library as data in one call. The interface covers reading only: on this build the API rejects writes.
Figure 15: Submitting a review in ReviewSphere ( s_022 ). The seed agent’s clicks meant to close two popups open a software-store window, and its click on Write Review misses until it rescales the coordinates. It then writes its own Chrome client to fill the form; Chrome first refuses the connection with 403 Forbidden , and the form is filled at call 37. With if-026 from a separate evolution run, calls 5–26 read the page, open the form, fill six fields, check them, and submit, mostly through browser.chrome_js , with no failed calls.
Figure 16: Resolving tracked changes in a Word document ( s_026 ). The seed agent reads the raw document XML and rebuilds insertion and deletion elements by hand in calls 17–28; its comments XML fails to parse at call 24. With if-030 from a separate evolution run, the agent lists the changes with redline_list , then accepts eight and rejects six with accept_redlines and reject_redlines in one call.
Figure 17: Accessing a TeamChat tab (OSWorld V2 task 038 ). The bare agent repeatedly clicks the MailHub tab while trying to activate TeamChat, then hits TeamChat’s close control. With if-024 , call 14 uses browser.chrome_cdp with a tab ID to read the general-channel rules without switching tabs through the screen. Later pinned-rule checks and CRM updates remain. The full episodes take 94 and 95 model steps and receive task scores of 0.6517 and 0.70, respectively.
Figure 18: Plotting a function in KAlgebra ( KAlgebra:A-02 ). The seed agent’s Add click lands in the graph, and repeated clicks target the color control instead of the expression input. It eventually finds the input and plots y=ln(x)2 in calls 64–67. With if-016 , call 8 uses rest.kalgebra_add_2d to add the curve and rest.kalgebra_functions_2d to read it back; call 10 shows the resulting plot. Both episodes pass, in 68 and 11 model steps, respectively.
Figure 19: Storing a calculation in KAlgebra ( KAlgebra:C-01 ). The seed agent repeatedly clicks above the calculator input and sends test characters before correcting the coordinates and storing log2(3) in p . With if-016 , kalgebra.evaluate_lines enters the expression through the GUI and checks the variable. Its first attempt reports p missing; the agent selects the Calculator tab and retries, confirming p=1.58496250072 at call 17. Both episodes pass, in 40 and 21 model steps, respectively.
Figure 20: Computing raster statistics in GRASS GIS ( GrassGIS:B-12 ). The seed agent locates the GUI console, corrects its typing method, and retries Return until it focuses the command input and obtains the statistics. With if-016 , call 8 runs r.univar map=lakes through grass.grass_run against the live mapset and returns the statistics as text, including n=263975 and sum=263975 . The evolved agent also runs the command through the GUI in calls 15–18. Both episodes pass, in 40 and 21 model steps, respectively.
ID
Task
Applications
s_001
Update a visitor map and event timetable.
LibreOffice Impress, LibreOffice Writer
s_002
Edit a thank-you video with captions and music.
Shotcut
s_003
Assign repair jobs using current staff, parts and equipment.
TeamChat, LibreOffice Calc
s_004
Mix several audio tracks and control volume changes.
REAPER
s_005
Build a decision map for choosing seed sources.
XMind
s_006
Correct cooling-test measurements and plot the results.
LabPlot, Text editor
Appendix
Table 9: Synthetic tasks provide the experience from which Agent S4 discovers interfaces. The 30 tasks span desktop applications and local web workflows, often requiring the agent to work across applications. Their trajectories expose interaction failures and inefficient workflows that the meta-agent can study to revise interfaces and usage knowledge, without ground-truth success labels.
ID
Task
Application
sb_celestia_001
One Moon, two viewpoints.
Celestia
sb_celestia_002
A fixed coastline under changing daylight.
Celestia
sb_celestia_003
Return to a ring-shadow observing point.
Celestia
sb_celestia_004
Enlarge Mars by magnification and by travel.
Celestia
sb_celestia_005
Compare the surface speeds of two slow-turning planets.
Celestia
sb_celestia_006
Date the light received from Neptune.
Celestia
Appendix
Table 10: ScienceBoard synthetic training tasks. The 48 tasks span six scientific applications and give the meta-agent trajectories to study, without ground-truth success labels.
Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were designed for human end users or application developers, leaving agents to interpret visual controls or discover and compose low-level software operations. This mismatch can hinder even capable agents, yet it remains unclear what interface properties make desktop software easier for agents to use. To study this question, we treat software evolution as a natural source of interface variation and introduce DeskIF-Bench, a benchmark of 255 artifact-verified tasks across six desktop applications. By holding the task, agent, and verifier fixed while varying the software release, we measure how interface changes affect agent performance. Across two models and six applications, the newest release consistently outperforms the oldest. Trajectory analysis identifies three relevant interface properties: discoverability, granularity, and observability. Guided by these properties, we develop ANIS, a framework that synthesizes agent-native interfaces from the capabilities of installed applications. On an expanded evaluation of 368 tasks across eight applications, atomic interfaces improve success over native SDK use by 3.8 and 3.5 percentage points with DeepSeek V4 Flash and Claude Haiku 4.5, respectively, while reducing online token usage by approximately 73%. Goal-level commands further raise success to 74.5% and 67.4%, respectively.
Agentic search depends on the interface through which a language-model agent accesses and explores a corpus. Existing interface designs are fixed by hand and applied unchanged to every corpus, although effective designs may depend on the corpus. We introduce SearchWeaver, a framework for automatic search interface optimization guided by downstream search performance. A proposer uses execution trajectories and task rewards to jointly revise corpus artifacts, executable tools, and usage instructions. This pro cess optimizes what information is prepared in advance, which operations tools perform, and which decisions remain with the search agent. With GPT-5.4-nano, the optimized interfaces improve answer accuracy from 71.6% to 78.4% on BrowseComp-Plus and average nDCG@10 from 45.7 to 55.0 across four BRIGHT subsets over Direct Corpus Interaction. These findings highlight the value of jointly optimizing corpus artifacts, executable tools, and usage instructions to develop effective search interfaces.
Computer-use agents increasingly operate software designed for people, but interfaces often leave actions or task state unclear to machine readers. We present Affora, a design system that supports both readers while preserving visual freedom and familiar human workflows. Three controlled studies examine component implementations, visual variation, and interaction-design principles. Their findings inform guidance from individual components to complete sites, supported by reusable implementations and executable checks. Agent performance depends on the interaction meaning available through its interface representation; substantial visual variation remains possible when that meaning is preserved. Evaluation on independently authored interfaces shows gains where Affora addresses existing deficits, but limited effects where those deficits are absent or outside its coverage. A workflow case provides preliminary evidence of reduced interaction cost. Affora connects user experience and agent experience through a shared interface rather than a separate agent-only surface.