Sandman: Building and Operating Sandboxes for Agentic Reinforcement Learning on Shared Kubernetes

Yiming Yan; Hongyang Wang; Wenbo Chen; Xuan Wang; Jingguo Liu; Xian Li; Chen Qi; Leo Fan

cs.DCOct 29, 2026

Sandman: Building and Operating Sandboxes for Agentic Reinforcement Learning on Shared Kubernetes

Authors: Yiming Yan, Hongyang Wang, Wenbo Chen, Xuan Wang, Jingguo Liu, Xian Li, Chen Qi, Leo Fan,

Organizations: Amazon

Abstract

Agentic reinforcement learning (RL) often requires repeated tool calls in task-specific environments. A training job may create thousands of these environments in bursts while sharing a cluster with other jobs. We present Sandman, a sandbox service built and operated at Amazon for agent tool execution and evaluation. It serves multiple production agentic RL workloads on shared Kubernetes. A training framework supplies a task image and uses commands and file operations in the returned sandbox. Files and processes persist across calls until the framework releases the environment. Sandman manages creation, readiness and cleanup. Each sandbox is a Kubernetes pod with a small daemon mounted into the unmodified image. In a scale test, Sandman held 11,858 concurrent tiny sandboxes. Four production SWE-bench Verified training campaigns created 3.1 million sandboxes in October 2026. We report the design trade-offs and operational lessons from handling creation bursts, preparing task environments, reclaiming abandoned sandboxes, and running evaluation under controlled conditions. We will open-source Sandman's daemon, provider and pool.

Figures & tables

Figure 1: What a sandbox service for agentic RL faces (production records). (a) Creates per second around campaign A’s largest burst, when two runs started together. (b) One campaign-D rollout sandbox of median life and tool-call count. (c) Where rollout-sandbox time went in campaigns C and D (sampled traces). (d) Cumulative fraction of gaps between requests for one image in a run of campaign A.

Panel of Figure 2 — Figure 2: Sandman’s components and the path of one SWE rollout (conceptual), colored by code ownership; Sandman’s provider and pool run inside the training job, which captures patches, runs graders and applies the grading policy. Solid arrows are defaults and dashed ones options; in our SWE job defaults, both guards apply to every sandbox. A task goes to an agent worker (1) that acquires a sandbox (2), alternates model and tool calls (3), captures the patch (4), has it graded in a separate pod (5), releases both (6) and sends the learner its reward (7).

Component	Calls
Pool (one Ray actor per job)	acquire (image, base commit, spec) $\rightarrow$ sandbox id, address; release (id); prewarm (images); drain ()
Provider (library)	create (spec) $\rightarrow$ handle; destroy ; health
execd (HTTP, one per pod)	execute_command , read_file , read_file_bytes , write_file , env , reset , state , health

Table 1: Calls used by the SWE trainer integration. Our SWE jobs run the optional pool with no warm pods.

Case	Why it is missed	Who owns it now	Windows that left a pod
Caller cancelled or timed out mid-create	the caller has left	provider finishes the create; pool keeps or deletes the late pod	12/0/0 of 15
API answers an error after storing the pod	an error is not proof of no pod	provider reads it by name; deletes it on a nonce match	3/2/0 of 3
Delete fails or its reply is lost	a sent delete is not a done delete	provider retries until confirmed; owner reference at job end	6/5/0 of 6

Table 2: Three ways a pod outlives its caller. Windows: stand-in-API tests that left a pod without an owner, before the fix / after the first (cancellation) fix / now, 20 repeats each (Table A21 ); on the cluster, 25 of 25 trials left no pod.

Boundary	Protected	Not protected
Execution	runc container, not privileged; guarded pods drop raw sockets	shared kernel; root user; no seccomp profile, user namespace or read-only root
Outbound	network policy denies egress; in-pod rules block all outgoing traffic but loopback and replies, failing closed	–
Inbound	network policy admits only trainer pods; per-sandbox token on every call but health checks; in-pod allowlist of the node and consumers	past use (no request logs)
Metadata	one-hop limit (not re-measured)	–

Table 3: What each boundary protects and what it does not.

Figure 3: The creation median held at every scale; the tail coincided with pod-address assignment failures. All panels are controlled. (a) Create-to-Ready p50 and p99 of all 30 steps, arms and controls included (tiny pods, cached image); filled: inside the address-failure window. (b) One sequential pair of 1,000-pod batches at 10 creates/s: create-to-Ready and wait from submission, pacer queue included. (c) Acquire to first successful execd call, 99 sandboxes per arm (dot p50, whisker p90);

\dagger

overlapped the failure window.

Mechanism	Production	RL experiments	Now
Agents call pods by address, no Service	–	p	✓
Paced and spread creates	–	p	✓
Create keeps an owner on cancel	–	p	✓
Outbound guard on every sandbox	C–D	✓	✓
Token, inbound guard, network policy	–	p	✓
Condition probe, task preflight	–	p	✓

Table 4: Where key mechanisms ran (all 16: Table A4 ). Production: campaigns A–D; Now: defaults of our code and SWE job templates; p: in part; o: option.

Figure 4: Where a start spends its time, what a warm pod saves and what it costs. (a) Per-phase means adding up to the mean create-to-healthy time, SWE-Gym images (records of controlled runs; cache state observed, not set; axes differ); markers: p50, p90. (b) Grading-pod acquisition, three pairs of runs, pool off and on (controlled). (c) Idle warm vCPU-hours and hit rates per run by idle timeout (replayed; measured: diamonds, hit rates in parentheses); below, the measured step-interval difference (95% CI).

Figure 5: Where learner-step time went on the critical path, over 3,734 traced steps (production: campaign A; the rest controlled). Campaign A ran up to two test suites, and two of its arms re-ran suites when a batch went to the learner; the reuse A/B’s failed attempts fell in incident windows. Grading acquisition at right; the pre-registered arms ran with and without the grading change.

Workload	When	Runs and scale	Tasks	Sandboxes and notes	See
Production
Campaign A	3–4 Oct, 30.1 h	32 runs of 50 learner steps, 16 GPUs each; 64 trajectories in flight per run	24 Verified, 27 held out	79,861 rollout, 113,407 grading; no outbound guard on any pod	B , §5.1
Campaign B	5 Oct, 10.6 h	16 runs of 50 steps, 16 GPUs; 64 in flight	Verified	40,136 and 59,164; no outbound guard on any pod	B , §5.1
Campaign C	10–13 Oct, 57.3 h	35 runs of 50 steps, 64 GPUs; 256 in flight	88 Verified	396,810 and 568,930; outbound traffic denied on all pods	B , H , §5.1
Campaign D	13–17 Oct, 82.7 h	39 runs of 300 steps, 16 GPUs; 64 in flight; up to 27 runs (432 GPUs) at once	88 Verified (sphinx-8475 removed at start)	720,332 and 1,147,595; outbound traffic denied on all pods	B , H , §5.1
Extension of D	17–20 Oct	48 runs of 16 GPUs, in two attempts	as D	incident list only	M

Table A1: Workloads behind the paper’s numbers. Sandboxes: rollout and grading sandboxes created; GPUs are B200s. See: where the workload is used (§: the paper).

	Tasks	Runs $\times$ GPUs	In flight per run	Rollout pods	Grading pods	Creates/s (mean)	Rollout life p50 / p99	Outbound guard
A	24	32 $\times$ 16	64	79,861	113,407	1.8	196 / 443 s	none
B	24	16 $\times$ 16	64	40,136	59,164	2.6	–	none
C	88	35 $\times$ 64	256	396,810	568,930	4.7	136 / 457 s	all pods
D	88	39 $\times$ 16 ( $\leq$ 27 at once)	64	720,332	1,147,595	6.3	207 / 424 s	all pods

Table A2: Production campaigns: pods created, not gradings (a trajectory can use two grading pods); lifetimes for C and D from sampled traces. Table A5 gives availability, create latency and waiting share.

Component	Changes, with dates
execd	first version, reads of at most 2,000 lines (21 Jul); syntax check of the wrapped command (4 Aug); own process group (12 Aug); byte-exact whole-file reads (19 Aug); group kill by original id, output capped while draining, complete-bytes reads (6 Oct); strict exit-status frames (7 Oct); per-sandbox token (20 Oct); state-check routes (19–21 Oct)
provider and pool	owner references (21 Jul); central drain (29 Jul); tags pulled “always” (10 Sep); outbound guard (campaigns’ patched copy 6 Oct; on every sandbox, late Oct); per-sandbox token and inbound guard on every sandbox (late Oct); soft topology spread and paced creates by default (late Oct); eight-field probe before every grading and retirement of contradicted pods (late Oct); Service variables off (campaigns’ patched copy 10 Oct 20:06, from campaign C on; default branch 13 Oct); creates shielded from cancellation (14 Oct); pacing (15 Oct); warm budget, recycling, Services paced with pods (20 Oct); readiness budget per job (21 Oct); error-path fix: create nonce, read-back after an ambiguous error, owned retries of failed deletes (27 Oct)
trainer and harness	our trainer: fresh grading pod, rollout sandbox kept until grading completes (28 Jul); test-configuration files left out of patches (29 Jul); agent calls bound to pod addresses (13 Oct; capture and grading always used pod addresses); reference/empty-patch check of the task set before each job (late Oct); gradings without a verdict retried, then resampled, never scored 0 (27 Oct). The campaigns’ trainer (every campaign and RL experiment): rollout sandbox released before grading (9 Sep); gradings without a verdict invalid and resampled, never scored 0. Harness: reads checked by length and SHA-256 (6 Oct; on by default, late Oct); editor refuses files it cannot write back unchanged (27 Oct)
cluster	task images kept 90 days (13 Oct); reboot-safe NVMe image store (18 Oct); metadata responses limited to one hop on the CPU nodes (late Oct); eight warm pod addresses per CPU node and a network policy on sandbox pods (late Oct)
What each workload ran
Campaigns A–B	the campaigns’ trainer; the provider shipped in the runtime image (15 Sep), from before the 14 Oct fix, with no outbound guard and Service variables on; execd of 19 Aug, copied into each pod from a carrier image; no warm pool

Table A3: Code changes that matter for the measurements, and the code each workload ran. Provider and pool are Sandman’s; the trainer and agent harness are our RL trainer’s.

Mechanism	A–B	C–D	WP	RU	PR	VR	Now
Agent calls by pod address, not a Service	–	–	–	–	p	p	✓
Service environment variables off	–	✓	✓	✓	✓	✓	✓
Soft topology spread over nodes	–	–	–	–	✓	✓	✓
Paced creates	–	–	–	g	g	g	✓
Outbound guard on every sandbox	–	✓	✓	✓	✓	✓	✓
Network policy on sandbox pods	–	–	–	–	–	–	✓

Table A4: What ran where. A–B, C–D: production campaigns; WP, RU: the warm-pool and reuse A/Bs; PR: pre-registered package arm; VR: validation run; Now: defaults of our code and SWE job templates at submission. ✓ ran; – did not; o option, off by default; g grading pods only; p in part (Table A3 gives the variant).

	Span		Available (%)		Create (s)		Wait	Sandbox-h
	(h)	Attempts	attempt	traj.	p50	p99	(%)	per GPU-h
A	30.1	81,102	99.73	99.98	4.31–5.65	6.8–175.8	6.4	3.30
B	10.6	42,409	96.98	99.65	4.14–7.01	17.3–258.4	9.8	3.75
C	57.3	453,155	91.16	$\geq$ 96.69	5.11–5.22	8.2–134.2	9.9	3.56
D	82.7	1,036,028	69.5	–	5.12–5.29	18.4–35.7	6.5 ∗	–

Table A5: Production campaigns in operation. Attempts: agent-session starts, retries included; available: share of attempts, and of trajectories after one retry, that got a working sandbox (D: a pod); create: range of per-run p50 and p99; wait: waiting share ( ∗ sampled traces); sandbox-h: per B200 GPU-hour.

Workload	Peak live (typical)	Runs at once	Creates/s mean	Per minute p99 / max	In 1 s p99 / max	Destroys/s mean / p99	Ended early
Campaign A	448 (211)	7	1.84	3.78 / 5.02	8 / 132	1.82 / 3.70	0.57%
Campaign B	$\leq$ 512	8	2.61	–	–	–	–
Campaign C	$\leq$ 1,024	4	4.68	–	–	–	–
Campaign D	$\leq$ 1,728	27	6.27	–	–	–	–
Pool/reuse A/Bs	139–319	2	1.80	3.39 / 5.40	–	1.78 / 2.98	15.6%
Pre-registered comparison	712 (389)	10	2.93	5.77 / 6.55	9 / 130	2.92 / 5.79	0.05%

Table A6: Creates arrive at a few per second on average and in bursts at run starts. Per-minute windows; creates count rollout and grading pods. B–D: peaks are upper bounds (runs at once times trajectories in flight) and rates are totals over the span; A/Bs: peaks per run; pre-registered comparison: the training runs’ rollout, grading and warm pods. Ended early: aborted, timed out or failed rollouts.

Figure A1: Rollout sandboxes live for minutes and grading sandboxes for seconds; a rollout sandbox is idle while the model generates for 74–91% of its life. (a) Lifetime, create request to release. Solid: campaign A (production; grading lifetime estimated per pod from each trajectory’s grading window). Dashed: the pre-registered comparison (controlled; grading on fresh pods). (b) Share of rollout-sandbox lifetime by phase, from per-turn traces.

Population	$n$	p50	p90	p99	max	Basis
Rollout, campaign A	79,869	196	317	443	–	start and end records
Rollout, campaign D (sampled)	9,723	207	348	424	805	per-turn traces
Rollout, campaign C (sampled)	4,054	136	247	457	740	per-turn traces
Rollout, pool/reuse A/Bs	49,043	168	360	550	1,885	start and end records
Rollout, pre-registered comparison	117,146	229	358	436	1,533	start and end records
Rollout, SWE-Gym runs	3,169	327	566	782	1,187	per-turn traces

Table A7: Rollout sandboxes live 136–327 s at the median and a fresh grading pod 13.5 s; pooled pods live longer because they also wait warm. Seconds.

	Create, set up	Run commands	Idle	Tool calls per rollout p50 / p90 / p99	Command p50 / p99	Commands $\geq$ 10 s: share of calls / time	Installs among commands $\geq$ 10 s
Campaign D (sampled)	3.62%	4.85%	91.38%	37 / 61 / 86	12 ms / 7.8 s	0.28% / 19%	93%
Campaign C (sampled)	6.20%	19.24%	74.35%	31 / 46 / 56	19 ms / 15.4 s	1.54% / 29%	97%
Pool/reuse A/Bs	5.48%	16.97%	77.25%	30 / 45 / 57	28 ms / 15.7 s	2.38% / 36%	70%
Pre-registered comparison	3.53%	8.81%	87.37%	35 / 51 / 67	9.5 ms / 8.3 s	0.72% / 23%	98%
SWE-Gym runs	7.91%	13.95%	78.06%	50 / 76 / 97	9.5 ms / 12.8 s	1.70% / 39%	85%
Campaign A	–	–	–	30 / 45 / 58	–	–	–

Table A8: Rollout sandboxes are mostly idle, and their few long commands are mostly installs and downloads that the outbound guard blocks. Time-weighted shares of sandbox seconds; capture and release take the remaining 0.08–0.30%, and the longest command was 300–301 s, execd ’s limit. Campaign A records turns only.

Image set	Images	Total	Mean	p50	p90	Max
Campaign A tasks	24	33.6 GB	1.40 GB	1.20 GB	2.03 GB	3.15 GB
Campaign D pool	89	107.6 GB	1.21 GB	1.15 GB	1.21 GB	3.17 GB
SWE-Gym runs	306	871 GB	2.85 GB	2.07 GB	6.19 GB	8.43 GB
All of SWE-bench Verified	500	699.8 GB	1.40 GB	1.24 GB	2.05 GB	3.33 GB
All SWE-Gym tags	2,424	7,276 GB	3.00 GB	2.94 GB	6.19 GB	8.50 GB
Re-requests	in a run: $<$ 10 s / $<$ 60 s		gaps $\geq$ 60 s: p10 / p50 / p90		any run: p50 / p90 / $<$ 5 min

Table A9: Task images average 1.2–1.4 GB on Verified and 2.85–3.00 GB on SWE-Gym, and an image is usually requested again within seconds because a GRPO group starts together. Upper: compressed sizes. Lower: time between consecutive requests for the same image.

Population	Pods	Requests	CPU use / request	CPU p99 per pod-minute	Peak memory p50 / p99 / max
Rollout, campaign D	1,089	2 vCPU, 8 GiB; 3.6% at 16 GiB	0.43%	0.05 cores	20 MiB / 0.08 / 0.76 GiB
Rollout, campaign C	813	as above; 3.0% at 16 GiB	1.35%	0.15	31 MiB / 0.09 / 1.16 GiB
Rollout, pool/reuse A/Bs	2,967	as above; 4.5% at 16 GiB	1.05%	0.14	32 MiB / 0.12 / 1.36 GiB
Rollout, pre-registered	1,061	as above; 4.0% at 16 GiB	0.84%	0.10	27 MiB / 0.10 / 0.23 GiB
Rollout, SWE-Gym runs	780	2 vCPU, 8 GiB	2.80%	1.24	55 MiB / 2.22 / 6.69 GiB
Sandbox pods, all users	198	2/8 for 137 pods; 23 at 0.5/0.75; 15 at 1.5/2; 23 other	10.5%	–	47 MiB / 3.56 / 11.19 GiB

Table A10: Rollout pods use under 3% of their CPU request; grading pods keep about one core busy. Rollout: one-minute metrics of sampled pods; all users: 198 pods over 14 days, weighted by pod-minutes; grading: cgroup counters.

Step (start)	$N$	Creates/s	Ready	Not Ready in 120 s	Address failures	Create-to-Ready (s) p50 / p90 / p99	API call p99 (ms)
clean (08:18)	99	20	99	0	0%	5 / 7 / 8	53
ramp	250	20	249	1 (0.40%)	21.2%	7 / 34 / 73	383
ramp	500	20	496	4 (0.80%)	23.4%	6 / 35 / 95	69
ramp	1,000	20	997	3 (0.30%)	21.5%	6 / 36 / 87	272
ramp	2,000	20	1,973	27 (1.35%)	24.5%	6 / 46 / 94	1,173
ramp	4,000	25	3,947	53 (1.32%)	18.9%	6 / 42 / 99	571

Table A11: The median stayed at 5–7 s at every size; the tail followed address-assignment failures, which hit 19–25% of pods in every ramp step of a window of about 2.2 hours (08:40–10:54). Pod-IP routing, no warm pods, API timestamps (1 s). The last two steps are a control after 20 minutes idle and one right after it.

Routing and warm pods	$N$	Ready	Acquire (s)	First call (ms)	Retried	Service create p50 / p99 (ms)	Acquire to first success (s)
pod IP, no warm pods	99	99	8.475	4	0	–	8.48
Service, no warm pods	99	99	9.153	1,063	39	235 / 845	10.44
pod IP, warm pods	99	99	0.041	6	0	–	0.05
Service, warm pods	99	99	0.017	2,267	99	17 / 29	2.28
Service, warm pods, acquires spread out	500	500	0.017	2,263	500	14 / 43	2.28
Service, no warm pods, Services unpaced	1,000	759	26.619	1,064	234	1,839 / 4,865	27.81

Table A12: A per-agent Service added one to two seconds before the first successful call and gave back most of the warm pool’s gain; unpaced Service creates were throttled. Medians; retried: first calls that needed more than one attempt.

Step (start)	$N$	Max pods per node	Creates/s	Address failures	By node load: $<$ 10 / 10–19 / $\geq$ 20	Not Ready in 120 s	Create-to-Ready (s) p50 / p90 / p99	Others, before $\rightarrow$ during
unspread (08:50)	500	26	20	17.6%	0.0 / 18.9 / 54.8%	0	6 / 33 / 55	–
unspread (09:05)	2,000	45	20	24.4%	0.0 / 7.5 / 39.1%	0	7 / 36 / 55	–
unspread (09:28)	5,000	83	20	28.4%	0.0 / 9.8 / 31.3%	3	7 / 43 / 74	–
unspread (10:11)	500	17	20	8.8%	6.1 / 16.9 / –%	0	6 / 8 / 61	79 $\rightarrow$ 138
unspread, aborted (10:37)	5,000	70	20	31.8% a	15.5 / 11.2 / 34.8%	59	7 / 54 / 100 b	172 $\rightarrow$ 643
realistic, spread (11:49)	1,000	4	10	0.2%	–	0	2 / 2 / 3	409 $\rightarrow$ 7

Table A13: Unspread, the affected share grew with

N

and concentrated on nodes that received many of a step’s pods; spread, the tail was gone at the same size and rate. The spread steps ran 1.5–3.5 h later, so this is an association. By node load: affected share on nodes that received

<

10, 10–19 and

\geq

20 of the step’s pods. Others: other tenants’ failures per minute on the shared nodes, before and during the step.

Stage, date	Limit we hit	Measure taken	How checked
Node pool, 14 Oct	Workers pinned to one instance type found its 115 nodes full of other tenants’ pods; 9 jobs never started.	Select the pool label, not a type.	P: 9 of 9 reran and finished.
Admission webhook, 14 Oct	One replica (115 restarts) failed 84.9% of an evaluation’s 6,144 trajectories (4,096 concurrent).	3 replicas, 1 GiB; paced creates.	C: 0 admission errors in 38,377 paced creates, consistent with the fix, not proof.
Per-agent Services, 3–18 Oct	3,255 Services in the namespace (10 Oct); campaign A’s median bash call 0.63 $\rightarrow$ 2.12 s; 241 of 1,000 unpaced Service creates failed.	Service variables off; pod-address routing; paced creates.	P: median bash call 16–22 ms in a production smoke test with links off. C: paced, 0 failed.
Pod address assignment, 14–27 Oct	Address assignment failed for 0.64% of creates in one production hour, and for 19–25% of the ramp’s pods in a 2.2-hour window.	Spread creates over nodes.	C: 0.35–0.45% spread vs 24% unspread (20 Oct); 0 vs 25.4% back to back (27 Oct).
Subnet headroom, 18 Oct	Free pod addresses fell from 33.6k to 26.5k at 12,000 sandboxes.	Stop at 12,000; watch free addresses.	C: counts before and after.

Table A14: Limits we met while scaling, what we changed and how we checked it: separate observations, not one curve. P: production observation; C: controlled experiment on the shared cluster.

Unit	$n$	Sandbox and network	Pull p50 / p90 / p99	Container start	Rest p50	Create p50 / p90 / p99
On demand, image pulled	422	1 s	13.4 / 69.4 / 85.1 s	2 s	1.47 s	17.4 / 74.3 / 90.5 s
On demand, image present	230	1 s	–	2 s	1.16 s	4.11 / 5.12 / 5.54 s

Table A15: A pulled SWE-Gym image made an on-demand create about four times slower at the median and set its tail. Re-grading sandboxes; phases at 1 s resolution; rest: create time minus scheduled to last container started.

Blocked by	Blocks	p50 / p90	Range	Size (GB) or type	$n$	Pull p50 / p90 / max
same image	39	18.2 / 71.2 s	6.0–140.6 s	1.29–1.42	398	8.2 / 13.0 / 28.9 s
same node	101	14.1 / 51.6 s	4.7–81.3 s	1.42–3.11	393	14.8 / 46.2 / 70.9 s
same 30-min block	6	12.4 / 15.5 s	10.1–16.7 s	3.11–8.41	413	64.1 / 94.2 / 153.3 s
				m8id.24xlarge	389	10.3 / 54.5 s
				m6id.24xlarge	342	11.8 / 64.1 s
				m5d.24xlarge	453	14.2 / 82.9 s

Table A16: The cold penalty appeared within each image, each node and each half hour (blocked separately), and grew with image size. Left: pulled minus present median create, across blocks with both units. Right: pull time by size tercile and by instance type (an observation: image mix and timing differ by type).

Grading acquisition p50 (ms)	5,159	5,917	7,454	26	85	247	$-6{,}057$ [ $-8{,}680$ , $-3{,}434$ ]
	Pool off, pairs 1 / 2 / 3			Pool on, pairs 1 / 2 / 3			On $-$ off [95% CI]
Warm-hit rate	0	0	0	0.932	0.962	0.931	–
Rollout create p50 (ms)	5,251	6,114	7,470	5,250	5,736	7,218	$-210$ [ $-686$ , $+266$ ]
Waiting share	7.04%	6.26%	7.62%	4.58%	3.38%	4.57%	$-2.80$ pp [ $-3.56$ , $-2.04$ ]
Learner step interval, steps 2–24 (s)	172.5	145.9	163.7	150.5	147.0	158.4	$-8.7$ [ $-38.4$ , $+21.0$ ]
Wall clock to step 24 (h)	1.351	1.186	1.302	1.235	1.083	1.158	$-0.121$ [ $-0.174$ , $-0.068$ ]

Table A17: The warm pool cut grading acquisition by 97–99.5% at the median in every pair and used 2.46 times the pool-off runs’ sandbox vCPU-hours (rollout and grading pods), mostly in idle warm pods; its effect on the steady-state learner step interval remained inconclusive in three paired runs. Last column: pool on minus off, 95% CI; acquisition: per-run medians.

	Control, pairs 1 / 2 / 3			Checked reuse, pairs 1 / 2 / 3			Difference [95% CI]
Pool creates per grading	1.306	1.287	1.340	0.934	0.946	0.911	ratio 0.710 [0.640, 0.780]
Gradings on reused pods	0	0	0	35.5%	34.0%	42.7%	–
Pool hit rate	0.984	0.971	0.959	0.988	0.985	0.979	$+0.012$ [ $-0.007$ , $+0.031$ ]
Acquisition p50, warm pod (ms)	30.6	121.7	234.7	31.3	122.9	238.1	$+1.8$ [ $-1.7$ , $+5.2$ ]
Acquisition p50, on demand (ms)	4,120	5,619	7,338	4,748	5,946	7,687	$+435$ [18, 852]
Mean idle warm pods	78.2	77.7	69.4	86.6	89.9	83.6	$+11.6$ [4.3, 18.9]

Table A18: Checked reuse cut pod creates per grading by 26.5–32.0% in every pair (ratio 0.710), with no detectable change in hit rate and an inconclusive effect on step time; reused pods idled in the same buffer, and the budget discarded most of them unused (one minus reused hits over checks). Last column: treatment minus control, or ratio, 95% CI.

Trace	Measured setting: hits, replayed (measured)	2 per image, 300 s: hits / idle	Global budget 16 / 32 / 64: hits	Budget 96, $\geq$ 2 per key, 300 s: hits / idle	Images graded per 5 minutes
Pair 1	96.7% (93.2%)	96.4% / 0.50 $\times$	44 / 70 / 81%	96.6% / 0.50 $\times$	35
Pair 2	98.0% (96.2%)	97.7% / 0.50 $\times$	46 / 73 / 85%	97.7% / 0.50 $\times$	35
Pair 3	95.1% (93.1%)	94.5% / 0.48 $\times$	43 / 68 / 77%	94.5% / 0.48 $\times$	31

Table A19: A per-image pool with a 300 s idle timeout kept 94.5–97.7% hits at about half the idle pod-hours; global budgets of 16–64 pods did far worse. Idle pod-hours relative to the replayed measured configuration, where the replay overstated hits by 1.8–3.5 points.

		Pod-hours			Creates per grading		On demand acquisition p50 / p99	Acquisition p99, warm pool / reuse
Trace	Gradings	on demand	warm pool	reuse	warm pool	reuse
Warm-pool A/B, pool off	1,826	10.9	110.8	136.5	1.312	0.867	5.16 / 57.94	5.84 / 5.70
Warm-pool A/B, pool on	1,879	10.9	112.6	130.8	1.308	0.965	5.16 / 64.52	5.65 / 5.73
Checked-reuse A/B, control	3,700	16.2	182.3	238.4	1.362	0.769	4.11 / 40.01	4.73 / 5.05
Checked-reuse A/B, reuse	3,872	18.3	185.3	242.4	1.354	0.732	4.11 / 38.77	5.11 / 5.27
Natural-pool SWE-Gym run	1,475	44.4	179.1	286.2	1.313	0.798	11.18 / 94.53	36.48 / 13.83

Table A20: Creating each grading pod on demand used the fewest pod-hours on all eight traces; checked reuse cut creates but raised pod-hours, because reused pods wait idle like warm ones. Acquisition in seconds.

#	Window and trigger	Before the fix (13 Oct)	Cancellation fix (21 Oct)	Error-path fix (27 Oct)	Caller / deleted (ms)
Cancel before the API stores the pod
1	cancel while the create waits in the client’s work queue	no pod (withdrawn)	request sent anyway, then deleted	$=$	50.8 / 50.7
2	cancel during admission; the pod is then stored	without an owner	deleted	$=$	50.8 / 50.6
3	cancel during admission; the create is then rejected (403)	no pod	no pod	$=$	50.5 / –
4	pool acquire times out while its create waits for a create slot	no pod	sent when the slot frees, then deleted	$=$	0.3 / –
Cancel after the API stores the pod, before the caller has it

Table A21: Before the fix, every window in which the API stored the pod left it without an owner. With the cancellation fix a cancellation alone never does, but two error paths still lose a pod (windows 16, 18, 19 and 21–24); the error-path fix closes both and leaves another tenant’s pod alone (window 25). Stand-in API, 20 repeats per window;

=

: as with the cancellation fix. Timing, with the cancellation fix: median ms after the trigger until the caller’s error and the delete, with 50 ms holds, so it shows who waits for what.

Scenario	Children running before	21 Jul	19 Aug	7 Oct (= 21 Oct)
timeout: plain background child	2	2 / 2	0 / 0	0 / 0
timeout: background child ignores the termination signal	2	2 / 2	1 / 1	0 / 0
timeout: foreground child ignores the termination signal	1	1 / 1	1 / 1	0 / 0
shell killed, 2 children	2	2 / 2	2 / 2	0 / 0
shell exits, 1 child	1	1 / 1	1 / 1	0 / 0
shell killed, child still holds the output pipe	1	1 / 1	0 / 0	0 / 0

Table A22: Child processes: before the October changes, a timed-out or crashed command left up to two children running past a restart; now none survives a reset in seven of eight scenarios. Cells: children alive after execd answers / after a reset. Condition hit: each child wrote its process-id file once running. Timeout 3 s.

Agent tools / execd	Edit placed	Files	Read incomplete	File rewritten from that read	Bytes outside the edit changed	Refused	Exact
30 Jul / 21 Jul	past line 2,000	408	408	0	0	408	0
30 Jul / 21 Jul	lines 1,000–2,000	408	408	408	408	0	0
30 Jul / 19 Aug	both	816	0	0	0	0	816
6 Oct (exact reads on) / 7 Oct	both	816	0	0	0	0	816
13 Oct (exact reads on) / 21 Oct	both	816	0	0	0	0	816
27 Oct (on or off) / 21 Oct	both	816	0	0	0	0	816

Table A23: An edit through the display view cut every long file to 2,000 lines; complete reads made all 816 edits exact. Condition hit: the proxy observed each read, rewrite and changed byte.

File category	Files	30 Jul / 21 Jul	30 Jul / 19 Aug	6 Oct / 7 Oct	13 Oct / 21 Oct	27 Oct / 21 Oct
LF-only Python files (random 500)	500	500 / 0 / 0	0 / 0 / 500	0 / 0 / 500	0 / 0 / 500	0 / 0 / 500
CRLF only	385	385 / 0 / 0	0 / 0 / 385	0 / 0 / 385	0 / 0 / 385	0 / 0 / 385
UTF-8 with byte-order mark	3	3 / 0 / 0	0 / 0 / 3	0 / 0 / 3	0 / 0 / 3	0 / 0 / 3
not valid UTF-8	2	2 / 0 / 0	0 / 2 / 0	2 / 0 / 0	2 / 0 / 0	0 / 2 / 0
synthetic: mixed CRLF and LF	1	1 / 0 / 0	1 / 0 / 0	1 / 0 / 0	1 / 0 / 0	0 / 1 / 0
synthetic: bare CR inside a line	1	1 / 0 / 0	1 / 0 / 0	1 / 0 / 0	1 / 0 / 0	0 / 1 / 0

Table A24: The text-mode read changed bytes outside the edit in all 892 sampled files; byte-exact reads kept LF, CRLF and byte-order-mark files intact, and the editor of 27 October refuses the files it cannot write back unchanged (last column, with exact reads on or off). Cells: changed / refused / exact. Condition hit: the proxy showed each old read inexact.

Case	Condition hit	19 Aug reports	6 Oct reports	7 Oct reports
empty status in the frame (1)	forced	0, no error; the next failing command also 0	0, no error; next also 0	1 and “invalid frame”; shell restarted
status arrives in a second read (1)	forced	0; the late “1” leaks into the next output	1	1
shell dies after the marker, before the status (1)	forced	0, no error	1, “shell exited”	1, “incomplete frame”
shell dies after “12” of status 127	forced	12, no error	1, “shell exited”	1, “incomplete frame”
4,053 bytes of output, then a failing command, one core (1)	200 of 200 trials	0 in 200 of 200	right in 200 of 200	right in 200 of 200
same, not pinned, three versions at once (1)	73, 101 and 83 of 200	0 in 73 of 200, exactly the hit trials	right in 200 of 200	right in 200 of 200

Table A25: The old parser reported success, or a wrong status, for a failing command whenever the status frame was empty, split or cut off; the strict parser reported the real status, or an error and a restart. Real status in parentheses.

Command	21 Jul	4 Aug	7 Oct
heredoc into a file without its terminator, two forms seen in a real rollout	300.010 / 300.004 s, exit 124	4 / 4 ms, exit 2	6 / 3 ms, exit 2
unbalanced quote nesting in a search command (seen)	300.005 s, exit 124	18 ms, exit 2	3 ms, exit 2
an inline Python program left open (shape seen)	300.004 s, exit 124	4 ms, exit 2	5 ms, exit 2
an unterminated single quote (synthetic)	300.005 s, exit 124	9 ms, exit 2	8 ms, exit 2
a valid one-word echo, 200 runs, p50 / p90	1.12 / 1.23 ms	3.30 / 4.48 ms	–

Table A26: Unterminated input: a syntax check turns a silent 300 s stall into a 3–18 ms refusal that keeps the shell’s state, at about 2 ms per valid command. Condition hit: the old version stalled to the timeout; the check fired.

Output	19 Aug	7 Oct
16 MiB	68.8 MiB, 0.13 s	21.5 MiB, 0.09 s
64 MiB	213.2 MiB, 0.47 s	21.5 MiB, 0.19 s
256 MiB	791.4 MiB, 1.80 s	21.5 MiB, 0.48 s
1,024 MiB	3,106.2 MiB, 7.33 s	21.5 MiB, 3.20 s
endless, 20 s timeout, 2 GiB memory limit	killed for memory after 9.63 s; no response	normal timeout at 20.01 s; 5,264 MiB drained

Table A27: Output floods: buffering grew execd ’s memory to about three times the output, and an endless printer got it killed within 10 s; capping while draining kept it at 21.5 MiB. Condition hit: every flood exceeded the 100 KiB cap.

	Factor and “ $-$ ” level	“ $+$ ” level	How it is applied
Q	CPU quota: none (request 1, may burst)	a limit of 1 CPU	pod spec
K	CPU affinity: every node CPU	test process pinned to one core	affinity set in the test’s process, on the idlest core
M	memory limit: 4 $\times$ request (8 or 16 GiB)	equal to the request (2 or 4 GiB)	pod spec
N	egress: open	all outbound traffic denied	the outbound guard
U	user: root	uid and gid 1000	workspace and home given to that user; the test drops to it
S	shared memory: 64 MiB default	a 1 GiB memory-backed volume	pod spec

Table A28: The eight run-condition factors.

Task	Factor	Mechanism	Failing / other / reference
matplotlib-20488	egress (N)	a test reads a remote image and gets a DNS failure	16/16 fail; 16/16; 4/4
requests-1724	egress	the tests call a public HTTP test service and get DNS failures	16/16; 16/16; 4/4
requests-1921	egress	as above	16/16; 15/16; 4/4
requests-2317	egress	as above; a connect test to a non-answering address is refused	16/16; 14/16; 4/4
requests-2931	egress	that connect test is refused instead of timing out	16/16; 15/16; 4/4
requests-5414	egress	as above	16/16; 16/16; 4/4

Table A29: Denied egress explained 11 of the 15 flips, one-core pinning or a CPU quota the other 4. Gradings that failed at the failing level; passed at the other; passed in the reference configuration, unless a row states otherwise.

Pool and run	Labels	Patches	Cells	Open egress: patches 0 $\rightarrow$ 1 (repeat)	Net labels, and other cells
Pair 1 of both A/Bs (88 Verified tasks)	11,399	5,201	egress open; 1 vCPU, 1 core; visible CPUs = quota; time zone; rebuilt image; memory R; uid 1000	1 (the same 1)	0 labels net in every cell except uid 1000: 12 labels (0.105%)
Natural SWE-Gym task set, 60 steps (230 tasks drawn uniformly, 14 dependent)	1,776	1,273	egress open; 1 core with memory R and uid 1000; memory R; 1 vCPU, 1 core; hash seed 0	22 of 1,273 (1); none 1 $\rightarrow$ 0	25 labels, 24 net of the repeat: 1.35% of the 1,776 (task-cluster bootstrap 95% CI 0.17–2.93%); other cells changed only validity
Enriched SWE-Gym task set, 20 steps (115 dependent, 115 matched stable tasks)	1,447	834	10 cells: the above, uid 1000, visible CPUs = quota, 16 GiB	56 of 822 (1)	71 labels net of the repeat: 4.9% of the 1,447; memory R invalidated up to 22 of 655 patches

Table A30: On a task set without condition-dependent tasks no condition but uid 1000 (12 labels) moved a label beyond the repeat; on SWE-Gym task sets, open egress turned 1.35% and 4.9% of labels from 0 to 1, net of the repeat. Training configuration: 2 vCPU, 8 GiB, egress denied, root. Flips are counted per distinct patch; a patch carries one or more labels.

Campaign	Task	Consumed positives	Wrong	Wrong %	Run bootstrap 95%	Lower bound %
D	django-7530	4,233	749	17.7	14.2–21.5	17.7
D	django-10097	1,144	484	42.3	39.5–45.1	23.8
D	both	5,377	1,233	22.9	18.7–26.9	19.0
C	django-7530	1,749	50	2.9	2.0–3.8	2.9
C	django-10097	168	72	42.9	36.2–49.7	31.0
C	both	1,917	122	6.4	5.3–7.5	5.3

Table A31: 19.0–22.9% of campaign D’s rewards on the two vacuous tasks went to wrong patches. Bootstrap: runs resampled within each campaign. Wrong: fails the restored official tests (django-7530) or the hardened suite (django-10097). Lower bound: the restored official tests for both.

Pool	Tasks	Configurations	Gradings	Pod-hours	Result
Checked-reuse A/B (Verified)	88	fresh, reuse	1,408	8.1	rejected the two vacuous tasks
Frozen deployment, SWE-Gym	230; 280	fresh, reuse	2,937 pods	134.7	admitted 218 and 241
Frozen deployment, Verified	115	fresh, reuse	577 pods	9.09	admitted 113
Pre-registered, training task set	89	fresh, reuse	708	7.59	admitted 86; rejected the two vacuous tasks and sphinx-8475
Pre-registered, held-out set	112	fresh	448	6.41	admitted all 112
Validation run	24	fresh, reuse	188	1.82	admitted 21; rejected the same 3

Table A32: The check cost a few pod-hours per Verified task set; pod-hours include pod creation.

Runs	Checks	Clean	Rolled back	Fresh fallback	Check p50 (ms)	Gradings on reused pods	Shadow agreement
Checked-reuse A/B, 3 runs (Verified)	10,779	0	10,777	2	459–505	34–43%	3,642 / 3,646
Frozen deployment, 3 runs (SWE-Gym)	2,505	553	1,359	593	231–282	1,294 / 2,557	1,290 / 1,290
Pre-registered, 6 runs (Verified)	52,639	0	52,633	6	338–350	8,838 / 55,011	8,816 / 8,819
Validation run, 1 run (Verified)	781	0	781	0	321	348 / 789	347 / 347

Table A33: The fallback rate is a property of the task set’s graders: on Verified task sets the check fell back to a fresh pod in 8 of 64,199 checks, on SWE-Gym in 23.7% of checks. Shadow: the reused pod’s verdict against a never-used pod’s.

Condition	Gradings per arm	Instances	Plain reuse wrong
real agent patches, up to 8 per pod	679	86	1 (an unstable test)
poisoned tenant, before the poison	30	30	0
poisoned tenant, after the poison	60	30	60
one task’s agent tenants (scikit-learn-14496)	16	1	1
that task, a tenant and then poison	2	1	2
total	787	86	64

Table A34: Plain reuse graded 64 of 787 gradings wrongly, all 60 after a poisoned tenant among them; checked reuse graded none of the 787 wrongly.

Rule	Retired early: defective / healthy	Wrong reused labels prevented (of 3)	Gradings moved	Extra creates, SWE-Gym / Verified
no verdict, or a contradiction (the guard’s rule)	2 / 3	3	9	0.12% / 0.009%
any apply failure or no verdict; two identical failures	3 / 135	3	542	18.7% / 0.094%
two identical failures	2 / 72	2	237	6.8% / 0.092%
the guard’s rule plus two identical failures	2 / 75	3	242	6.9% / 0.098%

Table A35: The contradiction rule retired 2 of 3 defective pods early and prevented all 3 wrong reused labels at almost no extra creates; stricter rules retired many healthy pods on SWE-Gym. Replay of 69,812 gradings of 13 runs; early: before the pod’s last grading.

Workload	Steps	Rollout acq.	Grading acq.	Failure, retry	Tests	Generation / commands	Learner	Sandbox waiting
Campaign A, all arms	1,568	0.0	9.5 a	0.0	47.9 a	loop 19.8	22.8	9.6
Campaign A, arms without re-runs	784	0.0	6.8	0.1	37.9	loop 30.0	25.1	6.9
Warm-pool A/B, pool off	72	0.0	6.0	0.0	7.8	20.7 / 2.3	62.8	6.0
Warm-pool A/B, pool on	72	0.0	2.0	0.0	10.0	30.7 / 3.4	53.2	2.0
Checked-reuse A/B, warm-pool arm	117	0.9	1.0	6.7	9.1	36.3 / 8.7	35.6	8.6
Checked-reuse A/B, reuse arm	117	0.5	1.1	6.4	8.2	36.1 / 12.6	33.5	7.9

Table A36: Rollout-sandbox creation was almost never on the learner’s critical path; grading acquisition and tests at the end of the slowest trajectory were. Share of total step time (%). Generation and commands: where per-turn traces exist, generation / commands; “loop” is the untraced agent loop. Other (setup, gaps, admission; at most 1.7%) omitted.

Campaign	B200-h	Analyzed B200-h	Sandbox waiting on the critical path, B200-h (% of analyzed)	Estimated reduction (%)	Note
A (traced)	1,773	1,610	153.8 (9.6)	11.0	81.6 of the 153.8 from re-run test-suite creates
B (estimated)	966	966	105 [70–145] (10.9)	14.3	includes 1,282 failed attempts that held the 300 s budget
C (estimated)	5,902	5,902	656 [433–902] (11.1)	11.4	64 GPUs per run
D (estimated)	12,766	12,617	911 [601–1,253] (7.2)	6.5	$f$ from sampled traces (6.46%)
A–D	21,407	21,095	$\approx$ 1,830 [1,260–2,450] (8.7)	$\approx$ 8.5

Table A37: Estimated step-time reduction from removing every sandbox wait, per campaign, under the fixed-concurrency accounting model (a model output in every row). Both percentages are of analyzed B200-hours, i.e. of analyzed run time (each run’s first step and start-up excluded; B and C: whole runs, start-up included). Critical-path waiting is traced for A and estimated from per-run totals for B–D (brackets:

\kappa

= 0.74–1.54).

Pair	Package	No package	Difference
1	0.4286	0.4810	$-5.25$
2	0.4721	0.4721	0.00
3	0.5112	0.5223	$-1.12$
4	0.4487	0.4810	$-3.24$
5	0.4531	0.4766	$-2.34$
6	0.4531	0.4732	$-2.01$

Table A38: The package scored lower in five of six pairs and tied in one; the registered reading is “no package higher”. Held-out resolve rate at step 150 on 112 tasks; difference in points.

	Package arm (6 runs)	No package (6 runs)
Trajectories / graded labels / consumed graded labels	59,535 / 54,980 / 25,227	58,340 / 53,819 / 25,246
Learned groups; B200-hours (estimate)	3,604; 1,028	3,600; 1,010
Gradings: reused / warm or on demand / direct	8,838 / 43,812 / 2,361	0 / 0 / 53,851
Creates per grading; grading pod-hours	1.21; 938 (pool pod lifetimes)	1.00; 323 (estimate from grading times)
State checks: rolled back / fresh fallback / errors	52,633 / 6 / 0	–
Check latency p50 / p99	338–350 / 756–764 ms	–

Table A39: The package’s mechanisms ran by default in six runs, and the arm without it trained on 623 labels from the three rejected tasks. Grading pod-hours come from different methods in the two arms and are not comparable.

Observed problem	Fix	Measured after the fix	Remaining
Create left without an owner after a cancel or an ambiguous error; failed delete forgotten	create shielded; nonce read-back; delete retries	stand-in API, 25 windows $\times$ 20 repeats: no window left a pod (the old code leaked wherever the API had stored it); cluster: 25 of 25 trials clean	pending deletes die with their actor
Command outlives its timeout; exit status misread at a 4 KiB read boundary	group kill; output cap; syntax check; strict status frames	reproductions: none of the four defects recurs after the fixes	own-session child survives a reset
Trimmed read written back as the file	byte-exact reads; length and SHA-256 check	816 of 816 long-file edits exact; on 892 sampled files every exact-match edit exact or refused	approximate-match edits
Planted test configuration in a patch	configuration files left out	original case reproduced: file left out; campaigns: 10 positives, none verdict-changing	other planted files
Residue in a reused pod	baseline state check	wrong on 0 of 787 gradings (plain reuse: 64); 3,642 of 3,646 shadow re-grades agreed	state graders were not seen to change
Verdicts that depend on run conditions	declared conditions; probe of the readable fields	on 240 pods the declared configuration passed and two wrong declarations were refused on all 240; stored readings exposed 85 pods with undeclared memory; 789 of 789 validation-run acquisitions passed	image, test suite; unreadable fields

Table A40: Failure paths. Window tests used a stand-in API, reproductions one host; others are controlled runs or campaign data.

What happened	How found	Cause	Fix	How verified	Effect
Managed service throttled session churn (July, before Sandman). On a managed code-interpreter service, a batch of trajectories finished together and recycled its sessions at once.	Errors in our run; the service team’s analysis.	Shared managed service: default limits of 30 calls per second on session starts, stops and invokes, of which we had starts and invokes raised and a shared per-account storage partition. Trigger: our workload (synchronized recycling).	Built Sandman on our own pods: per-task images, configurable sizes, our own create and stop rates.	— (motivation for Sandman).	Stop calls reached about 117 per second, inflated by our own retries (one session stopped 8–13 times); session starts were throttled and some faulted; trajectories timed out.
Admission webhook collapse (14 Oct). An evaluation with 4,096 concurrent trajectories (up to 5,120 actors) created sandboxes without pacing; the policy admission webhook answered pod creates with HTTP 500, other jobs’ creates included.	The evaluation’s failure counts; re-derived from the API server’s audit log.	Cluster configuration: the webhook ran one replica with a 384 MiB limit and had restarted 115 times since early September, last by an out-of-memory kill (the cluster’s change record, which the sandbox service’s role cannot read). Trigger: our workload (the burst).	Cluster: three webhook replicas of 1 GiB on separate nodes, with a disruption budget. Sandbox: paced creates.	Controlled: the scale ramp’s 38,377 paced creates had 0 admission errors. Load, pacing, pod spec and day differed, and the replica count was unreadable, so this is consistent with the fix, not proof.	First rollout: 84.9% of 6,144 trajectories (4,096 concurrent) failed (5,217); second: 5,249 (85.4%). Within 38 s, 6,558 of the evaluation’s creates and 6,126 of other jobs’ were rejected; all 21,517 HTTP-500 creates of 19:00–22:10 carried the webhook’s message. GPU-h (cluster or shared service): $\leq$ 973–1,145 B200-h (allocated: about 784 B200 for 1.46 h).
Webhook outages in campaign C (10–13 Oct). Pod creates failed with HTTP 500 from unreachable mutating webhooks: in a sample of messages, about 90% the policy webhook and 10% the queue webhook.	Failure-cause tally over the logs of campaigns A–C (83 runs).	Cluster or shared service.	Retry in a fresh session (existing behaviour); a highly available webhook.	Not re-measured under the same load.	39,834 of the campaign’s 40,039 failed session attempts; 8.79% of its 453,155 attempts, within 13 of about 57 hours. Per-attempt success 91.16%; per trajectory after one retry $\geq$ 96.69%. All 35 runs exposed. GPU-h (cluster or shared service): $\leq$ 24–39 B200-h (stall; 24 noise-adjusted, 39 raw).
Scheduling tail in campaign B (5 Oct). In the eight runs of the campaign’s earliest window, per-run create p90 reached 37–87 s, and attempts failed with pods still Pending.	Per-run create latency and failure tally.	Not established.	None.	—.	1,282 failed attempts in 8 runs; campaign per-attempt success 96.98%. GPU-h (not established): $\leq$ 177 B200-h (stall: the window’s 8 runs of 16 GPUs took 4.46 h against 3.08 h).
Pod-address assignment tail (production 14 Oct; ramps 18 and 20 Oct). The VPC CNI plugin failed to assign some pods an address and the kubelet retried, so those pods waited tens of seconds. With two warm addresses per node and a 30 s cooldown, a burst on one node can use up its warm addresses.	Per-pod condition timestamps (scale ramp); network-plugin failure events.	Not established: associated with pods per node (placement follow-up) and with a cluster-wide, time-varying condition (scale ramp: 21% at about 2 pods per node). Trigger: our and other tenants’ bursts.	Soft topology spread across nodes; watch free subnet addresses.	Controlled: spread 2,000-pod steps later that day (10 and 20 creates/s) had 0.35% and 0.45% affected, p99 8 and 7 s, an association; back to back (27 Oct), 0 of 1,000 spread pods against 254 of 1,000 default ones (one pair).	One production hour: 214 of 33,236 sandbox creates (0.64%). Scale ramp: 18.9–24.7% of pods per ramp step inside a window of about 2.2 h (08:40–10:54); 0.3–1.35% per step not Ready within 120 s. Unspread 2,000-pod step: 487 of 2,000 (24.35%).

Table A41: Recorded cases (1 of 8): the managed service, admission, scheduling and pod addresses.

What happened	How found	Cause	Fix	How verified	Effect
Another workload’s run failed during our benchmark (18 Oct). During the scale ramp’s 500-pod steps a run of the extension failed; other tenants’ pod-sandbox failures averaged 813 per minute while the ramp was active and 311 while it was idle (Pearson $r$ 0.50).	The benchmark’s co-tenancy watchdog; per-minute event counts.	Cluster configuration (address and network-plugin pressure). Trigger: our ramp. A pod-count co-tenancy rule held but does not bound address pressure.	Post-ramp extra steps capped at 1,000 pods and 20 creates/s; watch address and plugin signals.	Controlled: the placement follow-up’s guard aborted an unspread 5,000-pod step when others’ failures on shared nodes rose from 172 to 643 per minute (peak 1,422); they fell to 0–28 within a minute of our teardown.	1 run lost (one arm of one seed pair). GPU-h (cluster or shared service): $\leq$ 23 B200-h (lifetime).
Node-pool pinning (14 Oct). Job generators pinned agent workers to one instance type; other tenants had filled all 115 such nodes, while 260 nodes of other families had room for about 2,750 workers.	Pending pods reporting insufficient memory.	Sandbox implementation (the pin). Trigger: other tenants’ load (2,946 pods and 1,272 pending on those nodes).	Select the CPU pool by its label, not by an instance type.	Production: the driver recreated the 9 jobs under its never-started rule; all 9 completed.	6 training runs and 3 evaluations of campaign D never started. GPU-h (sandbox implementation): 0–122 B200-h (lifetime; lower end 0).
Event flood stalled the campaign driver (14 Oct). About 63,000 warning events in one hour (45,700 address-assignment failures on 77 CPU nodes, 8,700 scheduling failures; 5,706 sandbox pods in the namespace) made listing events take about 180 s.	Driver ticks exiting.	Cluster or shared service (namespace event volume).	A longer call timeout (75 s to 360 s).	Later ticks completed.	Every driver decision blocked from 18:58 until the timeout was raised.

Table A42: Recorded cases (2 of 8): co-tenant load, node-pool pinning and event volume.

What happened	How found	Cause	Fix	How verified	Effect
Service environment injection (3–4 and 10 Oct). Kubernetes injects variables for every Service in a namespace into every new container. Other jobs’ 3,255 Services, 3,187 of them per-agent, produced about 23,400 variables per container, slowing every shell and failing Ray readiness probes.	Tool-latency analysis across 35 run directories; Ray pods not becoming Ready.	Sandbox implementation (a Service per agent slot) plus a Kubernetes default (service links on). Trigger: other jobs’ Services in the shared namespace.	Service links off on sandbox pods.	Production smoke test with links off: median bash call 0.016–0.022 s.	Median bash tool call 0.63 s $\rightarrow$ 2.12 s in campaign A (later 3.4 s); bash -c about 2.3 s. GPU-h (sandbox implementation and a Kubernetes default): $\leq$ 123–135 B200-h (stall, campaign A, our recompute; an earlier estimate gave 60–143).
Synchronized-start stall (17 Oct). 48 runs of the extension started together and created about 4,600 per-agent Services in 6 minutes. Within about 2 minutes every run reported execd unreachable through Service names, and in-flight tool calls failed.	Campaign driver logs.	Not established. Our status note blamed per-agent Services, the unstaggered start and Kubernetes DNS defaults (search path with ndots 5, no node-local cache); the same window also had API-server timeouts, queue-webhook failures and slow DNS seen by another study.	Staggered run starts (at most 32 runs, 4 per 15 minutes); routing by pod IP.	The staggered restart did not show this signature (observation). The scale ramp later added up to 1,000 Services at a time with DNS p99 $\leq$ 15 ms (a different scale).	All 48 runs affected: 15 learner steps in 95 minutes across them; 20 died at the 7,200 s collect timeout, 9 made no learner step within the start timeout, and the rest stalled 57–132 minutes. GPU-h (not established): $\leq$ 1,170 B200-h (lifetime of these 29 runs and one 8-GPU evaluation, all ended in the window).
Service routing failed under churn (18 Oct; a training run on an enriched SWE-Gym task set). Pods answered by IP, but calls through their per-agent Services timed out after 10 s.	The job’s trajectory statuses.	Sandbox implementation (Service routing) plus a Kubernetes default (endpoint propagation).	A loopback forwarder to the pod IP; our trainer binds agent calls to pod addresses.	Controlled: a third attempt with the forwarder had 0 transport failures.	45–78% of trajectories in the first attempt and 80–86% in the second; 0 learner steps in 63 minutes; 63 actor replacements, each with a new Service.
Unpaced Service creates throttled (18 Oct; the scale ramp’s routing arm). 1,000 Service creates at once drew HTTP 429 from the API server’s priority-and-fairness limits (retry after 24 s).	The step’s create failures.	Sandbox implementation plus a Kubernetes default (API priority and fairness).	Pace Service creates with the same token bucket as pods.	Controlled: the same step, paced: 1,000 of 1,000 sandboxes succeeded; Service create p50 14 ms, p99 32 ms.	237 HTTP 429s on Service creates and 4 on pod creates; 241 of 1,000 sandboxes failed. No GPUs involved.
Possible Service-name collision (about 11 Oct; another team). Another team suspected that concurrent jobs shared per-agent Service names.	Branch history.	Not established (a hypothesis).	Job-unique actor identifiers, on that team’s branch.	Not verified.	Unknown.
Readiness budget included image pulls (21 Oct; a deployment run with frozen code). Cold pulls of large task images missed the provider’s fixed 120 s create-to-Ready budget.	Tasks with no verdict after a retry.	Sandbox implementation.	A readiness budget set per job.	Controlled: every grading that ran gave the expected reference-pass and empty-fail results; 7 of the 9 tasks were fully confirmed in a later hold-out.	9 admissible tasks of a 280-task set (3.2%) got no verdict; 1 shadow grading missed; cold pulls missed the budget in 4–6 of 10 gradings of those tasks.

Table A43: Recorded cases (3 of 8): per-agent Services and images on the creation path.

What happened	How found	Cause	Fix	How verified	Effect
Commands that never returned (4 Aug). A command with an unterminated heredoc or quote made bash read execd ’s closing brace and end marker as heredoc text, so the call waited out the 300 s command timeout and returned no output.	A per-run metric: maximum tool time 302 s in early test runs (August).	Sandbox implementation (the command wrapper). Trigger: agent-written shell.	Parse the whole wrapped command with bash -n first (failing open) and return bash’s own error.	Reproduction (Table A26 ): the observed heredocs refused in 4 ms instead of a 300 s stall, about 2 ms per valid command; the next test run’s maximum tool time was 6.2 s.	4 of 16 trajectories in one run, about 300 s each; the agent could not tell what went wrong.
Leftover child processes wedged sandboxes (August). The command timeout killed bash but not its children; they were reparented to init, and the health route checked only that bash was alive, so a wedged pod kept being served.	Inspection of one 19-hour pod: 21 zombies among 44 processes and a file-system walk blocked for 6,089 s.	Sandbox implementation.	Run bash in its own process group, signal and reap the group, and make the health route run a trivial command within 2 s.	Production-scale runs before and after, not concurrent: on the same 293-task workload, hangs fell from 0.47 to 0.26 per trajectory and timeouts from 32.6% to 22.4%; 0 zombies after 2 h. Doubling CPU had not changed timeouts (33.3% vs 32.6%).	0.47 hangs per trajectory and 32.6% of trajectories timed out on a 293-task SWE-bench run.
kubectl silently truncated results (15 Oct). Over the SPDY transport, kubectl returned exit 0 with a probe’s 224 KB result cut at a stream-buffer boundary (65,536 or 163,840 bytes).	The campaign driver detected a short result.	Tooling default (transport).	The WebSocket transport, which reports a dropped stream as an error.	Reproduction on the driver’s read path: 2 of 5 reads complete over SPDY, 4 of 4 over WebSocket.	None counted.
Per-task working directory overridden (about 17 Oct; another team’s evaluation). The provider appended the job-level workspace variable after the per-sandbox one, so the job default won, and agent file operations on the task’s repository were rejected.	The other team’s analysis.	Sandbox implementation (environment precedence in the provider).	The per-sandbox value takes precedence.	—.	513 of 2,683 tasks affected; their solve rate was 0.014 against 0.150 for the others; 468 healthy tasks were dropped as broken.
Tool-call transport failures in a cluster episode (18 Oct, 35 minutes; the checked-reuse A/B). 3–4% of tool calls failed (“fetch failed”, execd unreachable), pod-create p90 rose to 44–73 s and 10 creates got HTTP 429; grading pods reached by IP were not affected.	The runs’ error records.	Not established (consistent with API-server pressure).	None.	—.	847 and 710 failed tool calls and 904 and 774 error trajectories in the two arms.
Rollout actors exceeded agent slots (22–23 Sep; tool-use benchmark runs). The trainer started 1.25 actors per concurrent trajectory, more than the 16 agent slots; some actors stayed pending, and the pool waited for all of them without logging an error.	Ray actor listing.	Sandbox implementation (trainer sizing).	Keep 1.25 times the concurrency within the agent slots.	The next run dispatched rollouts.	0 rollouts dispatched.

Table A44: Recorded cases (4 of 8): execution, files and access boundaries.

What happened	How found	Cause	Fix	How verified	Effect
Planted test file passed a non-fix (July). Patch extraction with git add -A picked up a test configuration file that the agent had planted, so a trajectory that never fixed the bug passed.	Review of a passing trajectory.	Sandbox implementation (extraction). Trigger: the agent’s own output.	Extraction leaves test-configuration files out (our trainer, 29 Jul; not the campaigns’ grader).	Reproduction on the same case.	1 observed case.
Grader cut test runs at 300 s (18–19 Oct; the enriched-pool training run). A SWE-Gym grader revision sent no per-command timeout, so execd ’s 300 s default cut long test runs, which scored 0.	Re-grade analysis of the run (cut logs).	Sandbox implementation.	Send the declared timeout; the in-pod probe refuses a grader that sends none.	Controlled: re-grading at 1,800 s turned 24 reference failures into passes; a re-probe refused 240 of 240 pods for a grader without a timeout.	1,191 of 7,090 agent-patch logs (17%) cut; 26 training zeros (1.8% of the run’s labels) on patches that pass at 1,800 s. The SWE-bench Verified campaigns sent 1,800 s and were not affected.
Egress-denied label without the guard (18 Oct). The provider set the egress-deny label without installing the egress guard, which existed only in the patched provider copy that campaigns C and D used.	A live run of the reference/empty-patch check refused the deny configuration: 6 of 6 pods reached the internet.	Sandbox implementation.	The provider installs the guard and refuses the label without it.	Controlled: 20 of 20 guarded pods had no TCP and no DNS (32 of 32 unguarded pods were open); 6 of 6 network-dependent tasks were rejected.	Latent: all 2,540 deny-labelled pods then in the namespace had the guard.
Agent-backend error bursts in campaign D (14 Oct). 281,954 of the campaign’s 1,033,764 trajectory end events were agent-backend errors before any grading. In every traced hour with a burst of HTTP 500s, the logs also showed admission-webhook failures.	Trajectory end records; a trace of 6 run directories.	Not established (associated with shared webhook failures; not established per hour).	Retry in a fresh session; such trajectories are never scored 0.	—.	321,496 of 1,033,764 end events (31.1%) were not valid labels, concentrated in the earliest 12 runs (19–84% of their end events, against 0.6–1.1% in the other 27). They held 1.02% of the campaign’s 50,452 agent-sandbox hours. Of the 321,496, 317,602 (98.8%) fell in those 12 runs; 281,954 were agent-backend errors and 37,686 trajectories the trainer aborted, both before any grading; 1,855 (0.58%) were gradings or captures without a verdict (1,318 failed patch captures, 526 gradings without a verdict after retries, 11 without a pinned grader version); 1 timed out.
Job controller lagged under one Job per grading (16–17 Oct; the grading-condition study). The shared Job controller started each Job’s next repetition 15–27 minutes late (6,819 Jobs; finalizers lagging 18 minutes or more).	Repetition timestamps.	Cluster or shared service plus the sandbox implementation (one Job per grading).	Harvest and delete Jobs; a 900 s time-to-live; at most 300 live Jobs.	Qualitative.	Repetitions of some blocks ran together.

Table A45: Recorded cases (5 of 8): grading.

What happened	How found	Cause	Fix	How verified	Effect
Sandbox reset deleted the task repository (17 July). execd ’s reset wiped every directory under the workspace, which for SWE tasks was the repository itself, so every reuse attempt failed and fell back to a new pod.	Reuse counters in a run of 23–30 trajectories.	Sandbox implementation.	Wipe the workspace only on request; SWE pods keep the repository, which is reset with git.	Reproduction: the next run reused pods for 11 of 30 trajectories with 0 reset failures.	Reuse in 0 of 15–23 eligible trajectories.
Warm pods built without the requested resources (18 Oct). A prewarm call without the request’s spec built warm pods with no CPU or memory requests, that is, a different grading environment.	Code review during setup.	Sandbox implementation.	Hand out a warm pod only if CPU, memory and workspace match the request; a prewarm without a spec builds nothing.	Controlled: all 90 initial warm pods carried 2 vCPU and 8 GiB.	Caught before any measured run.
Warm budget discarded recycled pods (18 Oct; the checked-reuse A/B). At acquire the budget saw a key one pod short and started a replacement; the released pod came back from its check 10–20 s later, over budget, and was trimmed from the front of the buffer, where recycled pods sit.	The pool’s event log.	Sandbox implementation (pool design).	Count pods under check toward their key; trim the newest never-used pods first.	Unit tests only; not re-measured on the cluster.	58–66% of recycled pods per run reaped unused (65.0% in the original run), adding idle pod-hours.
State check missed edits to files that the image ships modified (21 Oct; a deployment run with frozen code). Restore reset such files to the repository head instead of their baseline bytes, and the diff compared only git status codes, so a further edit kept the same status and stayed invisible.	Comparing each recycled grading’s patch-apply step with its fresh shadow; a reproduction.	Sandbox implementation.	Fingerprint tracked files by content at baseline; restore baseline bytes.	Reproduction: 6 of 6 failures reproduced on the old build; with the fix, all 9 gradings matched fresh pods, and on a second image 27 of 27 verdicts were equal.	6 of 1,290 recycled gradings failed to apply patches that fresh pods applied (labels unchanged); extra fallbacks to fresh pods.
A defective never-used pod was reused three times (22 Oct; a replication of the checked-reuse A/B). A never-used pod failed to apply a 475-byte patch that applied 8 times on 5 other pods; each time its state check found and removed one added file and returned it to service.	Shadow-grading disagreements.	Not established (the pod was reaped before inspection, and the grader kept exit codes only).	Retire a pod after a grading with no verdict, or with a result that contradicts an earlier one for the same patch and tests.	Replay of 13 runs and 69,812 gradings: the rule retires 2 of the 3 defective pods early, and 3 healthy pods after a flaky test.	4 wrong zero labels in one learned group of 8 (advantages $\pm$ 0.935).

Table A46: Recorded cases (6 of 8): warm and reused sandboxes.

What happened	How found	Cause	Fix	How verified	Effect
Pods left behind at job teardown (16–17 July). The per-actor teardown hook did not reliably run, so sandbox pods outlived their job.	Counting leftover pods after runs.	Sandbox implementation.	Owner references to the trainer pod, a central drain, and the campaigns’ trainer’s reaper of failed actors’ pods.	Production counters, 122 runs of A–D: every quarantined actor shut down cleanly, so the reaper never ran; pods lost on other paths (a create answered with an error, a failed cleanup delete) are not counted.	5–20 pods per run.
Cancelled creates leaked pods (14 Oct, about 2.5 h). In a 1,536-concurrency stress test by a member of our team, the pool’s 120 s acquire timeout cancelled creates that were still running. The provider’s cleanup caught errors but not cancellation, so the accepted pod was never deleted, and the actor then made a second, direct create. Leaked 8 GiB pods filled the single node type the job selected, and creates slowed further.	Progressive collapse in the job’s logs; re-derived from the API server’s audit log.	Sandbox implementation.	Shield the create; clean up on any cancellation; wait for the create call to settle before deleting; keep late pods warm; no second create; a 600 s acquire timeout.	Controlled: the scale ramp (tiny pods, other nodes, paced, newer code, another day) had 83 acquire timeouts at 12,000 sandboxes; 48 late creates were kept warm, the rest deleted, and 0 pods were left after any step. This supports the fix but does not show that every cancellation window was hit. The 600 s timeout alone, as a workaround, gave 2.7% errors and 34 timeouts.	9,570 serve attempts, 72.3% ended in error; 9,881 acquire timeouts; 13,516 rollout pods created, 5,904 removed only by garbage collection at teardown; peak 5,920 live, at least 4,181–4,189 beyond one per actor. GPU-h (sandbox implementation): $\leq$ 974–1,347 B200-h (allocated: 528 B200 for 2.55 h).
Fallback creates bypassed the pod cap (19 Oct; a check run of merged code). Lookahead prewarm fired late under asynchronous rollout and held 46 idle pods; the pool refused 14 creates at its cap, and the trainer’s direct-create fallback bypassed the cap 9 times: our pods went from 56 to 88 to 96 in 35 s.	Our pod-count guard.	Sandbox implementation.	Planned: route the fallback through the pool, and trigger lookahead at rollout start.	Not implemented.	Our 90-pod cap was exceeded for one guard tick, and the guard deleted the job.
Pool not drained at job end (19 Oct; the same check run). The detached pool actor received no drain call, so its pods stayed until owner-reference garbage collection about 10 minutes later.	Pool versus cluster pod accounting.	Sandbox implementation.	Not fixed in our records.	—.	18 pods for about 10 minutes.

Table A47: Recorded cases (7 of 8): cleanup and ownership.

What happened	How found	Cause	Fix	How verified	Effect
Node-group rolling updates under running jobs (18 Oct). A GPU node group was rolled (294 of 336 nodes cordoned, 72 reporting status unknown) and a CPU node group; head pods were killed and jobs failed.	Campaign driver logs.	Cluster configuration (a node-group update).	None on our side.	—.	The extension’s first attempt lost 19 runs (9 with a node-loss signature, 10 without); its second attempt, 8 jobs. GPU-h (cluster or shared service, outside the sandbox path): $\leq$ 1,155 + 154 B200-h (lifetime).
Fleet-wide stall in the extension (18 Oct). 16 of 31 runs stopped progressing; 10 hit the 100-minute stall timeout.	Campaign driver.	Not established: concurrent heavy shared-file-system reads (up to about 27 GB/s), a sandbox burst and image pulls of another study, and a namespace-wide list loop; not attributed to the sandbox path.	Campaign rules: no bulk file-system I/O, image pulls, sandbox bursts or list loops during a campaign.	—.	10 runs deleted and classified as infrastructure failures. GPU-h (not established, outside the sandbox tier): $\leq$ 2,351 B200-h (lifetime).
GPU-side and trainer-side events (October; not the sandbox path). A mixture-of-experts dispatch deadlock at step 1 (2 runs of campaign C); GPU nodes reporting that they needed a reset (one blocked an evaluation, and 13 pods failed on it; two evaluations of the pre-registered comparison were re-applied); one relaunch for a collective-communication hang; a shadow-grading driver killed by a transient file-system error; a file-system access pod killed for memory (the driver stopped for about 10 minutes).	Campaign drivers.	Not the sandbox path.	Progress timeouts; avoiding bad nodes.	—.	Listed for completeness; no GPU-hour total recorded (the progress-timeout rule was sized at about 26 B200-h per hang).

Table A48: Recorded cases (8 of 8): cluster events outside the sandbox path.

Case	Workload	Root-cause class	Basis	B200-h
Sandbox path (Tables A41 – A47 )
Cancelled creates leaked pods	stress test	sandbox implementation	allocated	974–1,347
Node-pool pinning	campaign D	sandbox implementation	lifetime	0–122
Service environment injection	campaign A	implementation and Kubernetes default	stall	123–135
Admission webhook collapse	evaluation	cluster or shared service	allocated	973–1,145
Webhook outages	campaign C	cluster or shared service	stall	24–39

Table A49: The 10 cases with a GPU-hour account, by root-cause class (§6). B200-h are upper bounds; a range spans the case’s bounding methods (lower end weighted by the failed share, noise-adjusted, or 0 for jobs that never started). Each job’s loss is assigned to one case, so rows add. No other case has a GPU-hour record.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 525 papers

Sep 26, 2026

CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning

Reinforcement learning enables computer-use agents to improve through interaction with real software environments, including websites and desktop applications. However, conventional deployments replicate an initialized runtime for each independent rollout, even when trajectories use the same software, incurring repeated memory and initialization costs as the number of parallel environments grows. Does an independent computer-use environment require an independent execution runtime? Our key observation is that trajectories require independent mutable state, while initialized application runtimes can be reused across concurrently evolving environments, making state the natural unit of environment independence. Guided by this observation, we introduce CUA-Sandbox, which separates private state capsules from shared runtimes through state-scoped execution and transactional lifecycle operations, including resets and branches, while retaining the original software interfaces and task evaluators. Experiments show comparable or improved task success relative to Docker, while substantially reducing rollout and resource costs. CUA-Sandbox achieves up to a 6.20x increase in rollout throughput, a 9.2x reduction in per-environment memory, and a 504x reduction in incremental storage.

Xin Yan, Zhengbo Jiao, Jiaqi Liu +11

A*STAR · BNU · HKUST +4

Computer-Use Agents Reinforcement Learning RL for GUI Agents

Ask PDF

Apr 6, 2026

Synthetic Sandbox for Training Machine Learning Engineering Agents

As large language model agents advance beyond software engineering (SWE) tasks toward machine learning engineering (MLE), verifying agent behavior becomes orders of magnitude more expensive: while SWE tasks can be verified via fast-executing unit tests, MLE verification requires running full ML pipelines -- data preprocessing, model training, and metric evaluation -- on large datasets at each rollout step, rendering trajectory-wise on-policy reinforcement learning (RL) prohibitively slow. Existing approaches retreat to supervised fine-tuning (SFT) or offline proxy rewards, sacrificing the exploration and generalization benefits of on-policy RL. We observe that sandbox data size is the primary source of this bottleneck. Based on this insight, we introduce SandMLE, a multi-agent framework that generates diverse, verifiable synthetic MLE environments from a small number of seed tasks, preserving the structural and technical complexity of real-world problems while constraining datasets to micro-scale (each task is paired with only 50-200 training samples). Through extensive experiments, we show that SandMLE reduces execution time by over 13 times, enabling large-scale, on-policy trajectory-wise RL for the first time in the MLE domain. On MLE-bench-lite, SandMLE yields significant gains over SFT baselines across Qwen3-8B, 14B, and 30B-A3B, with relative medal rate improvements ranging from 20.3% to 66.9%. Furthermore, the trained policy generalizes across unseen agentic scaffolds, achieving up to 32.4% better HumanRank score on MLE-Dojo.

Yuhang Zhou, Lizhu Zhang, Yifan Wu +4

Meta AI

LLM Agent Training RL for LLM Agents

Ask PDF

Jul 15, 2026

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes. State-of-the-art algorithms such as PPO, RLOO, and GRPO inherit their rollout topology from RLHF: for each prompt, N independent trajectories are sampled from the initial state, and an advantage is computed by subtracting a group baseline. This design ignores a defining property of agent sandboxes. They are deterministic, snapshottable, and resumable from any intermediate state. We argue that this property enables a fundamentally different rollout topology: rather than N independent trees of depth T, one can construct a single tree of N leaves whose siblings share prefixes, and therefore share variance. We instantiate this idea as Branching Policy Optimization (BPO), a sandbox-native RL algorithm that (i) adaptively snapshots the sandbox at high-entropy decision points along a backbone trajectory, (ii) forks K alternative actions per branch point and rolls out each to termination, and (iii) computes per-step advantages from sibling returns rather than from independent prompts. We prove this estimator is unbiased and has strictly lower variance than the trajectory-level baseline, with the reduction equal to the prefix-explained portion of return variance. On WebShop, ALFWorld, and SWE-bench Verified with Qwen2.5-7B and Llama-3.1-8B backbones, BPO improves success by 3.6--6.1 absolute points over GRPO and RLOO at matched compute, halves gradient-norm variance, and matches the best baseline using 38% fewer policy updates.

Bowei He, Yankai Chen, Xiaokun Zhang +1

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · McGill Univeristy, Montreal, Canada · City University of Hong Kong, Hong Kong SAR.

Reinforcement Learning Policy Gradient Methods Policy Learning Sample-Efficient RL RL for LLM Agents

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning

Synthetic Sandbox for Training Machine Learning Engineering Agents

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning