CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning

Xin Yan; Zhengbo Jiao; Jiaqi Liu; Zhenglin Wan; SiYuan Ma; Xuliang Yu; Tianyi Jiang; Chubin Zhang

cs.AISep 26, 2026

CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning

Authors: Xin Yan, Zhengbo Jiao, Jiaqi Liu, Zhenglin Wan, SiYuan Ma, Xuliang Yu, Tianyi Jiang, Chubin Zhang,

Organizations: A*STAR · BNU · HKUST · NUS ·

Abstract

Reinforcement learning enables computer-use agents to improve through interaction with real software environments, including websites and desktop applications. However, conventional deployments replicate an initialized runtime for each independent rollout, even when trajectories use the same software, incurring repeated memory and initialization costs as the number of parallel environments grows. Does an independent computer-use environment require an independent execution runtime? Our key observation is that trajectories require independent mutable state, while initialized application runtimes can be reused across concurrently evolving environments, making state the natural unit of environment independence. Guided by this observation, we introduce CUA-Sandbox, which separates private state capsules from shared runtimes through state-scoped execution and transactional lifecycle operations, including resets and branches, while retaining the original software interfaces and task evaluators. Experiments show comparable or improved task success relative to Docker, while substantially reducing rollout and resource costs. CUA-Sandbox achieves up to a 6.20x increase in rollout throughput, a 9.2x reduction in per-environment memory, and a 504x reduction in incremental storage.

Figures & tables

Figure 1: Environment independence through runtime–state separation. (a) Conventional sandboxes allocate a complete initialized runtime and mutable state to each rollout, repeating process memory and lifecycle work. (b) CUA-Sandbox shares initialized processes while storing each environment’s mutable resources in a private state capsule. State-scoped bindings route actions to the correct capsule, allowing trajectories to evolve independently. Reset, clone, and fork operate on capsule state instead of rebuilding the runtime. Scenario adapters retain the original web, visual-web, and desktop interaction interfaces, so runtime reuse preserves execution in real software.

Figure 2: CUA-Sandbox architecture and execution flow. (a) Task-aware construction turns benchmark setup and resource contracts into capsules of authoritative, derived, and ephemeral state. Environment IDs and binding tables route actions through shared initialized processes to produce observations and rewards. Reset, clone, and fork freeze and drain ongoing work, stage successor capsules, and atomically publish their bindings while processes remain live. (b) The state plane manages capsule backends; the execution plane binds and executes actions; the control plane schedules environments and coordinates lifecycle transitions. Scenario adapters expose this infrastructure to web, visual-web, and desktop tasks.

Model	Backend	WebArena	VisualWebArena	OSWorld
GPT-5.6-Terra	Docker	21.89	28.34	67.29
	CUA-Sandbox	22.43	29.06	70.62
Qwen3.5-9B	Docker	9.67	10.52	24.17
	CUA-Sandbox	10.49	13.54	27.66
Muse-Glimmer-30B	Docker	23.14	25.36	24.40
	CUA-Sandbox	25.63	25.80	34.39

Table 1: Inference performance with matched execution backends. Success rates (%) are unweighted site/category means. Shaded rows show CUA-Sandbox; bold marks pairwise maxima. Site-level scores appear in Appendix B .

Benchmark	Training backend	Base	SFT	RL	$\Delta$ (pp)
WebArena	Docker	8.25	9.85	12.93	+3.08
	CUA-Sandbox	8.25	10.10	13.05	+2.95
VisualWebArena	Docker	18.13	23.84	27.03	+3.19
	CUA-Sandbox	18.13	24.07	27.03	+2.96
OSWorld	Docker	14.96	21.61	25.48	+3.87
	CUA-Sandbox	14.96	22.16	26.04	+3.88

Table 2: Qwen3.5-4B performance before and after training. Success rates (%) follow Base

\rightarrow

SFT

\rightarrow

RL under each training backend. Final checkpoints share the reference evaluation pipeline. Bold marks final RL scores;

\Delta

is the RL gain over SFT in percentage points.

Figure 3: Rollout efficiency with eight concurrent environments. (a) Preparation and batch wall-clock time. (b) Per-environment memory, throughput, and incremental storage. Purple hatched bars show CUA-Sandbox; gray bars show the Docker baseline (Official Sandbox). Axes are logarithmic. Lower time and resource costs and higher throughput are better. Table 6 lists the measurements.

	WebArena			VisualWebArena			OSWorld
Metric	Shopping Admin	Shopping	GitLab	Shopping	Reddit	Classifieds	Desktop
Memory	20.6	46.5	51.2	1.92	22.0	3.70	1.45
Storage	60.1	543.6	6.3	391.2	9,535	3.09	3.67
Create	1.70	1.15	8.82	15.17	8.75	6.89	2.98
Reset	1.20	1.03	5.51	4.99	7.29	2.66	3.17
Restore	1.20	1.21	6.62	5.78	7.39	2.52	2.90

Table 3: Where runtime sharing saves resources and lifecycle work. Entries are conventional-environment costs divided by CUA-Sandbox costs (

\times

); larger is better. The first two rows report resource reductions, and the remaining rows report operation speedups. Shaded bold cells mark the largest factor per metric.

Panel of Figure 4 — Figure 4: Component ablations separate efficiency costs from lifecycle correctness. Thin purple bars show CUA-Sandbox; wide gray bars show the named ablation. Bar lengths use a shared linear scale within each panel; labels give exact values. (a) Copy-on-write on a 3.6 GB Shopping capsule. (b) Non-DB state management on read-only WebArena task 21. (c) Publication events over 100 trials; purple origin markers denote zero events, and N/A denotes an inapplicable comparison.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Launch (s)	Storage	Memory (GiB)
WebServ Incus	1.781	28.01 MiB	1.74
CUA-Sandbox: fresh Docker	12.323	6.621 GiB	1.795
CUA-Sandbox: DB/reflink	1.730	12.50 MiB	0.201 †

Appendix

Table 4: Absolute costs of additional execution configurations. Launch latency is the mean of eight subsequent runs. Storage is per environment; † marks marginal memory growth, while the other memory entries are working-set measurements.

(a) WebArena-Lite
Model	Shopping	CMS	GitLab
GPT-5.6-Terra	20.86 / 21.98	20.90 / 20.90	23.90 / 24.40
Qwen3.5-9B	8.02 / 8.79	9.89 / 9.89	11.11 / 12.78
Muse-Glimmer-30B	24.60 / 27.22	17.03 / 18.13	27.78 / 31.55
Gemma-4-31B-IT	25.13 / 27.81	20.88 / 22.32	28.89 / 28.89

Appendix

Table 5: Detailed task success by site and category. Each cell reports Docker / CUA-Sandbox success rates (%) for the same model and evaluation protocol. Benchmark means appear in Table 1 ; bold distinguishes the CUA-Sandbox result in each pair.

Metric	WebArena		VisualWebArena		OSWorld
	Docker	CUA	Docker	CUA	Docker	CUA
Prepare (s)	68.85	25.68	6.964	0.633	20.75	5.84
Batch (s)	69.20	31.06	9.104	2.138	27.48	8.94
Throughput (env/s)	0.116	0.258	0.220	0.936	0.291	0.895
Memory (MiB)	2102.8	463.9	328.9	188.4	2653.8	837.3
Storage (MiB)	6757.1	13.1	4879.05	12.52	62.7	29.1

Appendix

Table 6: Exact rollout measurements for Figure 3 . All settings use eight concurrent environments. Memory and incremental storage are per environment; CUA denotes CUA-Sandbox. Preparation and batch wall-clock time are in seconds.

Benchmark	Backend	Concurrency	Env. prep. avg./env (s)	Batch wall (s)	Throughput (env/s)	RAM / env (MiB)	Storage / env (MiB)
WebArena	CUA-Sandbox	4	20.360	21.340	0.188	492.9	13.00
WebArena	Docker	4	44.600	44.800	0.089	2112.3	6757.10
WebArena	CUA-Sandbox	16	44.238	49.372	0.324	436.1	13.05
WebArena	Docker	16	120.771	121.483	0.132	2082.6	6778.84
VisualWebArena	CUA-Sandbox	4	1.434	5.735	0.697	15.9	0.44
VisualWebArena	Docker	4	6.208	24.831	0.161	514.8	4880.04

Appendix

Table 7: Concurrency scaling measurements at 4 and 16 active environments. Lower is better for preparation time, batch wall time, RAM, and storage; higher is better for throughput.

Configuration	Active runtime containers	Runtime RAM (MiB)	Agent/Browser PSS (MiB)	Writable storage (GiB)
Official Docker	8 Web	15,560	6,998	52.99
CoW + independent runtime	8 Web + 8 MySQL	13,882	7,045	24.65
CoW + shared runtime	1 Web + 1 MySQL	3,261	7,291	23.19

Appendix

Table 8: Ablation of copy-on-write (CoW) and runtime sharing for eight logical environments. Runtime RAM measures the memory occupied by the application runtime, while Agent/Browser PSS reports the proportional-set size of the agent and browser processes. Writable storage includes environment-specific writable state. Lower is better.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 161 papers

EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents
Jul 7, 2026Mianqiu Huang, Taofeng Xue, Chong Peng +12RL for GUI Agents Partially Observable RL
1Meituan · 2Fudan University · 3Shanghai Jiao Tong University +1
DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback
May 21, 2026Yunpeng Dong, Jingkai He, Shiqi Liu +7LLM Agents
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Engineering Research Center for Domain-specific Operating Systems, Ministry of Education, China · Huawei Technologies Co., Ltd.
SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL
Jul 13, 2026Bowen Lv, Xiao Liu, Yanyu Ren +7Computer-Use Agents Reinforcement Learning with Verifiable Rewards
Tsinghua University

Jul 7, 2026J/K move · Enter open · S save

EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

Mianqiu Huang, Taofeng Xue, Chong Peng, Jinrui Ding+11

1Meituan · 2Fudan University · 3Shanghai Jiao Tong University+1

RL for GUI Agents Partially Observable RL Computer-Use Agents Curriculum RL RL for LLM Agents

Open paper AskPDF arXiv

Computer-use agents must solve long-horizon tasks through repeated interaction with partially observable, multimodal desktop environments. Although imitation learning and offline trajectory refinement provide strong priors, static traces cannot cover the causal feedback loop of real computer use: each action changes the screen state, future action space, and recovery options. EvoCUA-1.5 extends self-evolving computer-use agents from offline experience learning to online reinforcement learning, where policies interact with executable sandbox environments and improve from verifiable task outcomes. Online RL in this setting requires more than directly reusing single-turn language-RL recipes. Multi-turn interaction introduces context-managed observations, sparse terminal rewards, variable-length trajectories, and slow environment feedback. EvoCUA-1.5 addresses these challenges with Step-Level Policy Optimization (STEPO), which preserves trajectory-level advantage balance after decomposition into step-level samples; policy-aware filtering and pass-rate calibration over verifiable synthesized tasks; Dynamic Tri-Adaptive Curriculum (DTAC), which combines learnable tasks, difficult positive replay, and controlled infeasible-task exposure; and a fully asynchronous RL infrastructure with staleness control and mini-group batching. Experiments show that these components improve training stability and downstream performance. EvoCUA-1.5 achieves 63.2% success on OSWorld-Verified, outperforming comparable 32B/35B-scale open-weight baselines and even approaching models with significantly larger parameter counts. Overall, EvoCUA-1.5 provides a practical framework for scaling online RL in multi-turn computer-use agents.

Abstract

Computer-Use Agents

Explore similar work

EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback

SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL