Efficient batch execution of agent tasks is critical to LLM training and development workflows, including benchmark evaluation, synthetic-data generation, and RL rollouts. Since these tasks alternate between LLM inference and environment interaction, we present coordinated optimizations to reduce batch makespan, the time required to complete all tasks in a batch. For hybrid LLMs, a workload-based resource model guides GPU-memory allocation between key-value (KV) cache and recurrent state and provides a reference sample concurrency to exploit parallelism while limiting recomputation. Trajectory-aware routing and scheduling promote cache reuse while balancing worker load. Under constrained environment capacity, we decouple sample admission from environment execution and introduce Chain-Advancing Action Scheduling (CAAS), which prioritizes long chains using observed progress to shorten the batch tail. Controlled replay combines actual inference with recorded environment responses and durations for ablations of individual optimizations. Live evaluations with Ling-3.0-Flash and Ling-3.1-Flash on the BFCL, SWE-bench Multilingual, WideSearch, and MCP-Atlas benchmarks demonstrate makespan reductions of 10.4% to 65.0%, while evaluation scores remain aligned. Furthermore, these methods have been successfully applied to training and evaluation workflows for the Ling-3.x model series, accelerating the model-development cycle.
Figures & tables
Figure 1: Illustration of batched execution of multi-turn LLM agent tasks. Active tasks are those admitted but not yet completed, including tasks waiting for inference or environment interaction. Each maintains its own context while sharing inference and environment-execution resources. Purple labels indicate where our methods are applied. Capacity separation allows the sample-admission limit CS and environment-execution limit CE to be configured independently. Our workload-aware capacity model guides the GPU memory allocation between KV cache and recurrent state at deployment; the allocation remains unchanged during batch execution.
Figure 2: Overview of the controlled trace-driven agent replay framework. The numbered markers indicate optimization locations. \scriptsize1⃝ Capacity separation: the sample-admission limit CS is set independently of the environment-action concurrency limit CE . \scriptsize2⃝ Inference optimizations: cache allocation, routing-prefix scheduling, and LBCH routing. \scriptsize3⃝ CAAS for job ordering.
Figure 3: Cache allocation and sample-concurrency sweeps on (a) SWE, (b) BFCL, and (c) WideSearch. Cell values are complete-batch makespans in minutes; the panels use separate color scales. Second tick-label lines indicate multiples of rbal or Cref , except for the default ratio. Magenta outlines mark the best measured points. TO denotes a six-hour timeout and is excluded from the continuous color scale.
Figure 4: Cache reuse and cumulative new-prefill work versus sample concurrency at the best measured ratio for each workload: (a) SWE, r=0.135 ; (b) BFCL, r=1.13 ; (c) WideSearch, r=0.15 . Blue circles use the left axis and orange squares the right axis. Every point represents a complete batch. Concurrency configurations are equally spaced on the horizontal axis. Shading highlights the concurrency ranges where cache reuse drops sharply and prefill recomputation increases (SWE: 147–157; BFCL: 644–859; WideSearch: 123–147).
Figure 5: Mean requests in service (left) and output throughput (right) versus sample concurrency at the same cache ratios as Figure 4 : (a) SWE; (b) BFCL; (c) WideSearch. Concurrency configurations are equally spaced on the horizontal axis. Shading highlights the concurrency ranges where output throughput drops despite increasing requests in service (SWE: 147–157; BFCL: 644–859; WideSearch: 123–147).
Figure 6: In-worker scheduling: (a) SWE; (b) BFCL; (c) WideSearch. Cell values are complete-batch makespans in minutes, with separate logarithmic color scales. Magenta outlines mark the minimum makespan for each workload before rounding. The hatched cell marked >13h denotes a timeout for SWE FCFS at CS=289 under the 13-hour limit.
Figure 7: Cache reuse (top) and cumulative new-prefill work (bottom) from the scheduling runs under the configurations in Figure 6 . Each point represents a complete batch; the SWE FCFS timeout at CS=289 is omitted. The SWE new-prefill axis is logarithmic. The LPM, DFS-weight, and routing-prefix curves nearly overlap on WideSearch.
Workload
CS
LPM (s)
DFS-weight (s)
routing-prefix (s)
SWE
289
735.00
1704.25
303.05
BFCL
1950
7.65
172.15
10.91
WideSearch
200
62.82
144.08
39.97
Table 1: Cumulative wall-clock time spent computing scheduling priorities over a complete batch at the highest tested effective sample concurrency for each workload. Bold values mark the minimum in each row.
Figure 8: Cross-worker routing on two TP4 workers: (a) SWE; (b) BFCL; (c) WideSearch. Bars show measured batch makespan in minutes; the panels use separate vertical scales that do not start at zero.
Figure 9: Cache reuse and new-prefill work for the runs in Figure 8 : (a) SWE; (b) BFCL; (c) WideSearch. Bars show new-prefill work on the left axis; the purple line shows cache reuse on the right axis. Workload panels use separate scales for new-prefill work.
Figure 10: Routing load distribution on (a) SWE, (b) BFCL, and (c) WideSearch. Top: time-averaged request-count imbalance over intervals with more than one in-flight request across both workers; each unit of retained time has equal weight. Bottom: output-token imbalance over the complete batch. Each row uses a common percentage scale. Lower values indicate more even distributions of requests over time and generated tokens across workers, respectively.
Figure 11: Capacity separation under FCFS job scheduling: batch makespan (top) and mean in-flight inference requests (bottom). The first point uses unified admission CS=CE . In-flight requests are measured from client request launch to response completion and averaged over the replay makespan.
Figure 12: Completion of the final 5% of trajectories at CS=50 . Both policies use the same absolute replay time axis within each panel. The final-5% tail duration is denoted by L5 .
Makespan
Tokens (M)
Reuse
Imbalance (%)
Model
Workload
Config.
(min)
Input
Output
H (%)
β
γout
Score
3.0
BFCL
Baseline
68.4
141.1
3.9
84.2
10.6
2.9
73.0
Opt
49.2
141.6
3.9
84.5
17.0
3.1
75.65
SWE
Baseline-1
166.9
1286.2
8.5
98.3
17.2
13.0
72.4
Baseline-2
197.7
1371.4
8.7
87.5
17.2
7.7
71.3
Opt
149.5
1347.9
8.7
98.6
16.4
3.5
72.6
Table 2: Live inference-side results. Each metric is calculated per run and averaged equally over three runs per configuration. Input token counts include cached tokens. Token totals are in millions. Imbalance metrics follow Section 5.2.3 .
Model
Configuration
Makespan (min)
Mean in-flight
L5 (min)
L10 (min)
Score
3.0
Baseline
466.1
1.2
38.9
73.7
65.5
Opt-FCFS
168.2
7.3
67.6
79.2
66.6
Opt-CAAS
163.0
7.9
66.0
73.4
65.6
3.1
Baseline
333.5
1.6
26.1
43.5
81.0
Opt-FCFS
140.5
12.5
49.8
54.4
80.9
Opt-CAAS
128.5
11.5
35.3
38.6
81.3
Table 3: Live MCP-Atlas results, averaged equally over three 500-task runs per configuration. Tail durations L5 and L10 are measured from each run’s own 95% and 90% completion points to its final task termination, including both successful and failed tasks. Mean in-flight is the time average of recorded inference calls over the full task-execution window, computed from native trajectory start/end timestamps and averaged equally across runs.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Workload
Recorded trajectories
Inference requests
Output tokens
SWE
289
25,416
6,000,784
BFCL
5,106
17,029
1,441,547
WideSearch
200
5,672
4,736,147
Appendix
Table A1 : Historical inference-only replay workloads. Output totals are from the routing replay runs. Live task counts are reported separately in Table A2 .
Sample concurrency CS
Model
Workload
Tasks
Baseline/B1
B2
Opt
Opt ratio
CE
3.0
BFCL
5,106
100
–
300
1.5
–
SWE
300
100
250
250
0.1
–
WideSearch
200
40
–
100
0.25
–
MCP-Atlas
500
3
–
50
0.14
3
3.1
SWE
300
50
100
100
0.2
–
Appendix
Table A2 : Live task sets and operating points. All baseline cache ratios are 0.9. B1 and B2 denote SWE Baseline-1 and Baseline-2. The two MCP-Atlas Opt variants share the listed settings. A dash in CE indicates that environment execution does not constrain sample admission.
Ling-3.0-Flash
Ling-3.1-Flash
Metric
Baseline-1
Baseline-2
Opt
Baseline-1
Baseline-2
Opt
Sample concurrency
100
250
250
50
100
100
Makespan (min)
166.9
197.7
149.5
234.1
304.1
185.1
Cache reuse (%)
98.3
87.46
98.58
98.2
80.41
98.27
New-prefill (M)
22.2
170.90
18.98
19.3
211.81
17.93
Mean API in-flight
40.9
105.2
76.7
35.5
71.0
63.7
Appendix
Table A3 : Live SWE diagnostics under the tuned baseline, concurrency-matched baseline, and Opt. New-prefill counts uncached input tokens. TTFT and TPOT are averaged across requests within each run, then across runs. Mean API in-flight averages outstanding API requests, including those waiting, over each live run.