Gym-Anything: Turn any Software into an Agent Environment

Pranjal Aggarwal; Graham Neubig; Sean Welleck

cs.LGApr 7, 2026

Gym-Anything: Turn any Software into an Agent Environment

Authors: Pranjal Aggarwal, Graham Neubig, Sean Welleck

Organizations: CMU

Abstract

Computer-use agents hold the promise of assisting in a wide range of digital economic activities. However, current research has largely focused on short-horizon tasks over a limited set of software with limited economic value, such as basic e-commerce and OS-configuration tasks. A key reason is that creating environments for complex software requires significant time and human effort, and therefore does not scale. To address this, we introduce Gym-Anything, a framework for converting any software into an interactive computer-use environment. We frame environment creation itself as a multi-agent task: a coding agent writes setup scripts, downloads real-world data, and configures the software, while producing evidence of correct setup. An independent audit agent then verifies evidence for the environment setup against a quality checklist. Using a taxonomy of economically valuable occupations grounded in U.S. GDP data, we apply this pipeline to 200 software applications with broad occupational coverage. The result is CUA-World, a collection of over 10K long-horizon tasks spanning domains from medical science and astronomy to engineering and enterprise systems, each configured with realistic data along with train and test splits. CUA-World also includes CUA-World-Long, a challenging long-horizon benchmark with tasks often requiring over 500 steps, far exceeding existing benchmarks. Distilling successful trajectories from the training split into a 2B vision-language model outperforms models 2 $\times$ its size. We also apply the same auditing principle at test time: a separate VLM reviews completed trajectories and provides feedback on what remains, improving Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%. We release all code, infrastructure, and benchmark data to facilitate future research in realistic computer-use agents.

Figures & tables

Figure 1: Built with Gym-Anything , CUA-World covers all major occupation groups and industries, spanning over 10K+ long-horizon tasks and environments across 200 software applications, dramatically expanding the scope of computer-use agent evaluation and training.

Figure 2: Overview of the Gym-Anything pipeline. Phase 1: We select $\sim$ 200 economically important software applications grounded in GDP data, balancing high economic impact with broad coverage across occupations, industries, and software categories. Phase 2: Each software is converted into an interactive environment via a creation-audit loop, in which a creation agent iteratively builds and verifies the environment, while an audit agent checks quality over multiple iterations. The creation agent writes its learnings into a memory, allowing it to improve over time. Phase 3: Tasks are scaled with a propose-and-amplify pattern, in which an expensive agentic model creates high-quality seed tasks (e.g., 5 per software), then a cheaper language model generates more tasks (e.g., 75 $\times$ ) using the seeds as in-context examples. Phase 4: Agents are evaluated on CUA-World using a checklist-based VLM verifier with privileged information and fine-grained rubric scores.

Figure 3: GDP-grounded software selection pipeline. Starting from U.S. occupational data, we estimate per-software GDP, filter to sandboxable candidates, and apply tiered selection to yield 200 software applications.

Figure 4: The Gym-Anything creation-audit loop. A Creation Agent writes setup scripts and produces evidence documents (screenshots, logs, etc.) while an Audit Agent evaluates this evidence against quality checklists and returns feedback. Learnings accumulate in a shared memory $M$ , which a Summarization Agent periodically condenses so that newer environments are created faster.

Software	Task Description	Privileged Information	VLM Checklist
AstroImageJ	Analyze the WASP-12 (RA: 06:30:32.79, Dec: +29:40:20.4) astronomical image sequence from January 5-6, 2016, to identify evidence of a planetary transit. If a transit is detected, determine the transit depth, mid-transit time (BJD_TDB or JD), and transit duration in hours. Using a host star radius of 1.599 solar radii, estimate the planet’s radius in Jupiter radii. Save your findings and uncertainties to ~/Documents/transit_analysis.txt .	Target: WASP-12b. Expected Transit Depth: ~1.4% (0.014 relative flux). Expected Duration: ~2.7 hours . Expected Planet Radius: ~1.79 Jupiter radii (calculation: sqrt(0.014) * 1.599 R_sun * conversion factors). The dataset is real ground-based imagery from Jan 5-6, 2016, so the light curve will have noise but the transit dip should be clearly visible.	1. The agent loads the sequence of astronomical images into AstroImageJ. 2. The agent selects the target star (WASP-12) and appropriate comparison stars for differential photometry. 3. The agent generates and displays a light curve plot showing the star’s flux over time. 4. The agent fits a model or trend line to the data to characterize the transit. 5. The agent reports the measured transit depth, mid-transit time, and duration. 6. The agent calculates and reports the planet’s radius in Jupiter radii.
Apache OpenOffice Writer	You are a Clinical Research Associate (CRA) performing an Interim Monitoring Visit (IMV) for Protocol ZN-994 at Site 142. Using the visit data in /home/ga/Documents/visit_notes.json , create a formal IMV Report in Apache OpenOffice Writer saved as /home/ga/Documents/IMV_Report_Site_142.odt . The report must include a document header with the Protocol and Site numbers, page numbers in the footer, and sections using ‘Heading 1’ style for ‘Visit Details’, ‘Subject Enrollment’, ‘Protocol Deviations’, and ‘Action Items’. […]	Protocol Number: ZN-994 . Site Number: 142 . Enrollment counts from the input file: Screened=18, Randomized=14, Completed=3, Discontinued=2 . The calculated ‘Active’ count must be exactly 9 (14 - 3 - 2 = 9). The Action Items table contains items AI-02 and AI-03 which have an ‘Open’ status and must be formatted with yellow highlight or red text. The four required sections are ‘Visit Details’, ‘Subject Enrollment’, ‘Protocol Deviations’, and ‘Action Items’.	1. Verify the document has a header with Protocol and Site numbers, and a footer with page numbers. 2. Verify the four required sections are present and use the ‘Heading 1’ style. 3. Verify the Subject Enrollment section contains a table with the correct base counts. 4. Verify the ‘Active’ count in the Subject Enrollment table is correctly calculated. 5. Verify the Protocol Deviations list and Action Items table structure. 6. Verify that Action Items with an ‘Open’ status are conditionally formatted.
Aerobridge	Calculate the total flight duration in minutes for all flight plans belonging to the operator ‘SkyHigh Surveyors’ that occurred in October 2023. Save the total number of minutes to /home/ga/Documents/utilization_report.txt .	The correct total flight duration for ‘SkyHigh Surveyors’ in October 2023 is 135 minutes (acceptable tolerance: 133-137 minutes). This is calculated from two flights: Flight 1 on Oct 5 (10:00-10:45, 45 mins) and Flight 2 on Oct 12 (14:00-15:30, 90 mins). Distractor flights (wrong operator or wrong month) must be excluded.	1. Locate flight plans belonging to the operator ‘SkyHigh Surveyors’. 2. Identify the flight plans that occurred in October 2023. 3. View the details or times of the relevant flights to calculate duration. 4. Open or create the utilization report file. 5. Save the correct total flight duration to the report file.
Liverpool Cancer iChart	Using the Liverpool Cancer iChart Archive app, determine the drug-drug interaction between Dabrafenib and Ketoconazole (located in the Antifungal agents category). Leave the application on the screen displaying the interaction result.	Dabrafenib is a cancer drug (BRAF inhibitor) and Ketoconazole is an antifungal agent. The interaction between them is clinically significant due to potent CYP3A4 inhibition by ketoconazole, which increases dabrafenib AUC by 71% and Cmax by 33% . The VLM should look for an interaction result indicating a severe/red warning or mentioning CYP3A4 inhibition.	1. The Liverpool Cancer iChart Archive application is opened. 2. Dabrafenib is selected as the cancer drug. 3. The Antifungal agents category is accessed. 4. Ketoconazole is selected as the comedication. 5. The interaction result between Dabrafenib and Ketoconazole is displayed on the screen.

Table 1: Full-text examples of task descriptions, privileged information, and VLM checklist items. ; color only highlights the most important privileged information.

		Environment		Scale		Task Properties		Infrastructure
Benchmark	Agent	Interactive	Platform	# SW	# Tasks	Long-Horizon ⋆	Econ. Cov. §	Auto-Create	Train Split
Static Datasets
Mind2Web Deng et al. (2023)	Web	$\times$	Web	137	2,350	—	7/22	$\times$	$\checkmark^{\!*}$
AITW Rawles et al. (2023)	CUA	$\times$	Android	357+	715K	—	4/22	$\times$	$\checkmark^{\!*}$
AndroidControl Li et al. (2024)	CUA	$\times$	Android	833	15,283	—	7/22	$\times$	$\checkmark^{\!*}$
OmniACT Kapoor et al. (2024)	CUA	$\times$	Lin / Win / macOS / Web	65	9,802	—	6/22	$\times$	$\checkmark^{\!*}$

Table 2: Comparison of CUA-World with datasets and environments for computer-use agents. ✓ yes;

\times

no;

\checkmark^{\!*}

partial or with caveats; — not applicable. ⋆ Benchmark allows or requires >100 agent steps per task. ∗ Offline human demonstrations only (not interactive verified trajectories). § Number of 2018 SOC (Standard Occupational Classification) major occupation groups (out of 22 civilian groups) whose workers would routinely use the benchmark’s applications; counted conservatively such that a group is included only if tasks directly simulate work in that occupation.

Figure 5: Quantitative comparison of CUA-World against existing benchmarks across four dimensions. The first two axes use a log scale.

Large Models
Model	Avg. Score	Pass Rate
Gemini 3 Flash	50.1	22.6
Kimi-K 2.5	37.1	12.8
Small Models
Qwen3-VL-2B	12.7	1.6
Qwen3-VL-4B	19.3	3.9

Table 3: Model performance on CUA-World -Test. Our 2B distilled model outperforms open-source models up to 2

\times

its size.

Max 500 steps, $5 cost cap
Model	Avg. Score	Pass Rate
Gemini 3 Flash	36.2	7.5
GPT-5.4	22.7	3.0
Sonnet 4.6	20.5	6.0
Kimi-K 2.5	33.9	5.5
Max 2000 steps, no cost cap

Table 4: Performance on CUA-World -Long.

Panel of Figure 6 — Figure 6: Scaling behavior on CUA-World . (a) Training data scaling on CUA-World -Test: varying the number of software (50, 100, 200) or the fraction of tasks (25%, 50%, 100%). Both axes improve with scale, following a roughly log-linear trend. (b) Test-time compute scaling on CUA-World -Long: pass rate as a function of average steps taken per task, where each point corresponds to a different maximum step budget (50, 100, 200, 500, 2,000 steps). The star indicates Test-Time Auditing (TTA, § 6.2 ) under the same 2,000-step cap.

Figure 7: Generalization to seen (IID) vs. unseen (OOD) software. We train models on 25% (left) and 50% (right) of the 200 software applications, and evaluate on the training software applications (IID) and the held-out software applications (OOD). Each bar spans from the untrained baseline (bottom) to the model trained on all software (top). The solid portion shows the gain recovered by the model trained on the subset; the hatched portion shows the remaining gap. Training on a subset recovers 65-87% of the gain on IID software but only 22-27% on OOD software, indicating that generalization to unseen software is limited and scaling to diverse software is important.

Figure 8: Behavioral patterns in passed vs. failed trajectories across Gemini-3-Flash evaluations on CUA-World . See Appendix for the full set of 15 patterns.

Panel of Figure 9 — Figure 9: Properties of CUA-World -Long. (a) Distribution of average steps per task. The y-axis is broken to accommodate the spike at the 500-step cap. (b) Distribution of per-task average checklist scores for Gemini 3 Flash and Kimi-K 2.5.

Figure 10: Pass rate on CUA-World -Test by software category. (a) Visual complexity and (b) domain knowledge. See Appendix I for category definitions and assignment of software to categories.

	Teacher	Student Score
Teacher	Score	Q3-VL-2B	Q2.5-3B
Opus 4.5	53.5	19.3	8.5
Sonnet 4.5	45.5	17.5	9.8
Gemini 3 Flash	44.0	16.3	8.3
Kimi-K 2.5	39.8	25.3	15.8
Gemini 3 Pro	39.3	15.8	7.0

Table 5: Teacher model selection on 4 software applications.

No distillation
Train	GIMP	G. Earth	OpenEMR	Slicer 3D	Avg.
200 steps	60.1	13.4	18.2	3.3	23.8
50 steps	64.2	13.4	8.5	4.5	22.7
Q3-VL-2B	47.0	7.2	0.0	0.0	13.6

Table 6: Effect of training trajectory length.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Tier	Budget	Strategy
k1 (Economic Core)	100	Highest GDP products overall
k2.1 (Strategic)	100	Healthcare, Education, Protective Services, Transportation—20 per domain
k2.2 (STEM)	100	Architecture/Engineering, Computer/Math, Life/Physical/Social Science
k3 (Domain Diversity)	116	Round-robin across all 22 SOC major groups ( $\sim$ 5 per group)
k4 (Niche)	44	Products unique to single occupations or domains
k5 (Category Fill)	40	Uncovered software categories, largest GDP first

Appendix

Table 7: Tiered selection budget. Each tier iterates occupations by GDP (descending) and applies substitution for non-selectable products.

Metric	Value
Occupations covered	894
Software categories	$\sim$ 1,400
Products in catalog	$\sim$ 16,600
Products passing all filters	$\sim$ 3,400
Products selected	$\sim$ 488
Substitutions made	$\sim$ 429

Appendix

Table 8: GDP-grounded software selection pipeline summary statistics.

Runner	Use case
DockerRunner	Single-machine development; requires Docker daemon
QemuApptainerRunner	HPC/SLURM clusters; runs QEMU VMs inside rootless Apptainer containers
QemuNativeRunner	Bare-metal Linux or macOS; supports Apple Silicon via HVF
AVFRunner	macOS on Apple Silicon; uses Apple Virtualization Framework with Rosetta 2
AVDApptainerRunner	Android apps; wraps Android emulator in Apptainer
AVDNativeRunner	Android apps; runs emulator directly on host (macOS HVF or Linux KVM)

Appendix

Table 9: Execution backends. The same env.json runs on all runners without modification. New backends can be added by implementing the BaseRunner interface.

#	Check	Criterion
1	Software running	Application is visible and in the correct starting state
2	Data files valid	All project/data files exist with non-zero sizes
3	Permissions correct	Output directories are writable by the agent user
4	Stale outputs cleared	No leftover result files from prior runs
5	Timestamp recorded	/tmp/task_start_timestamp exists
6	Data elements present	All precondition data (bugs, records, assets) verified

Appendix

Table 10: Audit checklist applied by

\text{Agent}_{\text{audit}}

to every environment.

Environment	Round 1 Finding	Round 2 Outcome	Checklist
PEBL	Task description specifies wrong response keys; task is uncompletable	Description rewritten; correct keys are now inferred from on-screen instructions	(a)
SeisComP	evidence_docs/ is completely empty; zero screenshots	Full screenshot evidence produced; real USGS earthquake data verified	(e)
Graphite	Task 2 description is a step-by-step UI walkthrough	Description rewritten to state only the objective	(a)

Appendix

Table 11: Summary of cross-round audit corrections. Checklist column refers to the audit criterion from § E.1 that identified the issue.

Score	Label	Description
1	Not Similar	Completely unrelated tasks (different goals, different domains).
2	Somewhat Similar	Minor thematic overlap but fundamentally different tasks.
3	Some Steps Similar	Tasks share some common substeps (e.g., both navigate to a location) but have distinctly different end goals. This is common and acceptable.
4	Very Similar	Tasks are extremely similar—knowing how to do one would directly help with the other. Nearly interchangeable.
5	Same Rephrased	Essentially the same task with different wording.
6	Duplicate	Identical or near-identical tasks.

Appendix

Table 12: Similarity scale used by the LLM judge. Scores

\geq 4

are flagged as potentially contaminating.

Score distribution
Metric	Value
Total pairwise comparisons	434,699
Flagged pairs (score $\geq 4$ )	2,847 (0.65%)
Score 1–2 (not similar)	1.1%
Score 3 (some steps similar)	98.2%
Score $\geq 4$ (contaminating)	0.65%

Appendix

Table 13: Contamination filtering statistics for CUA-World .

Pipeline Stage	Model	Harness	Section
GDP-Grounded Software Selection
Category extraction	GPT-5-High	–	§ 2.2
Category deduplication	Gemini-3-Flash-Preview	–	§ 2.2
Product enumeration	GPT-5-High	–	§ 2.2
Access-barrier evaluation	Gemini-3-Pro-Preview	–	§ 2.2
GDP attribution	GPT-5-High	–	§ 2.2

Appendix

Table 14: Models and harnesses used at each stage of the Gym-Anything pipeline. Empty cells indicate no harness was used.

Figure 11: Step-weighted pattern intensity across all 15 discovered behavioral patterns. For each pattern, bars show the average fraction of trajectory steps exhibiting that pattern, split by passed vs. failed trajectories. Patterns are sorted by the gap between failed and passed (largest gap on the left).

Figure 12: Pattern presence rate across all 15 discovered behavioral patterns. For each pattern, bars show the fraction of trajectories that exhibit the pattern at least once. Patterns are sorted by the gap between failed and passed.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 153 papers

Oct 8, 2026

SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments

Computer-use agents (CUAs) are rapidly improving, yet synthesizing diverse and reliable software environments at scale remains challenging. Native application environments require substantial construction and integration effort, which limits application coverage and increases the risk of application-specific overfitting. Browser-hosted environments provide limited interaction depth and workflow complexity. These limitations motivate SoftwareGym, an automated pipeline for synthesizing GUI software environments with diverse interfaces, executable functionality, and runtime-verified behavior. It generates structurally diverse designs, realizes them as runnable applications, and uses GUI evaluation with iterative repair to ensure functional reliability. Using SoftwareGym, we synthesize 182 applications across twelve categories and achieve a mean GUI task pass rate of 91.5%. We construct SoftwareGym-Bench, a 120-task evaluation suite spanning the same twelve categories, and use trajectories collected from these environments for CUA training. The resulting model improves Pass@3 from 41.39% to 56.97% on the strictly out-of-domain OSWorld benchmark and achieves 40.0% on SoftwareGym-Bench. These results demonstrate the potential of synthesized software environments as scalable and transferable supervision for CUA training.

Tian Jin, Wenhao Wang, Jing Kang +4

Shanghai Jiao Tong University · Zhejiang University

Computer-Use Agents LLM Agent Training Synthetic Benchmark Generation Computer-Use Agent Benchmarks

Ask PDF

May 25, 2026

CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

Reinforcement learning with verifiable rewards (RLVR) has driven breakthroughs in domains such as math, tool-use, and software engineering, yet its extension to computer-use agents (CUAs) has been bottlenecked by the scarcity of scalable training data with deterministic rewards. Constructing such data for CUAs requires consistent task instruction, executable environment, and verifiable reward. However, hand-curated benchmarks achieve high reward fidelity but cover few applications and LLM-as-judge-based datasets scale broadly but lack reliable verification. We present CUA-Gym, a scalable pipeline that co-generates task instructions, environment states, and reward functions. Concretely, a Generator agent constructs the initial and golden environment states, and a separate Discriminator agent writes the reward function from the task specification. An orchestrator agent drives the two through iterative rounds upon execution. Generated tuples then pass a final filter combining LLM majority voting and agent rollouts, ensuring quality beyond the per-task adversarial loop. To address the scarcity of training environments, we further synthesize CUA-Gym-Hub, a broad suite of high-fidelity mock web applications grounded in real-world software-use distributions, expanding the scale of CUA RLVR data by magnitude. Using this pipeline, we construct CUA-Gym, a dataset of 32,112 verified RLVR training tuples grounded in 110 environments. Trained with GSPO on CUA-Gym, our CUA-Gym-A3B and CUA-Gym-A17B achieve 62.1% and 72.6% on OSWorld-Verified, outperforming prior open-source CUAs at comparable scales, with performance scaling smoothly in both data volume and environment diversity. The same checkpoints also improve on the held-out WebArena benchmark, indicating transfer beyond the training environments. We will open-source the full synthesis pipeline, dataset, CUA-Gym-Hub environments, and models.

Bowen Wang, Dunjie Lu, Junli Wang +11

The University of Hong Kong · Qwen Team, Alibaba Inc. · University of California, San Diego +1

Computer-Use Agents Synthetic Data Generation AI Agent Benchmarks Reinforcement Learning with Verifiable Rewards LLM Agent Training

Ask PDF

Jun 28, 2026

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

Existing computer-use benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier agents. We introduce OSWorld 2.0, a benchmark of 108 long-horizon computer-use workflows across everyday and professional tasks, designed to capture complex and challenging real-world phenomena. Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete and requires an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWorld 1.0. OSWorld 2.0 targets challenge phenomena that are common in real workflows yet underrepresented in prior benchmarks, spanning interaction-design challenges such as streaming interaction and dynamic environments, as well as agent-pattern challenges such as cross-source reasoning, implicit-state inference, and visual-spatial precision. Tasks are grounded in authentic input artifacts and cross-referenced against realistic stateful user profile data, and include separate safety reports auditing safety-sensitive execution. Under our primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score; GPT-5.5 is far more token-efficient yet plateaus near 13%. These results show that current agents are still far from professional-level computer use: rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover.

Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong +33

XLANG Lab

Computer-Use Agent Benchmarks Computer-Use Agents Long-Horizon Agent Evaluation Long-Horizon Agent Tasks Computer-Use Agent Evaluation

Ask PDF

Abstract

Computer-Use Agents

Explore similar work

SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments

CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks