SkillHorizon: Constructing Long-Horizon Agent Tasks through Skill Composition

Yuxuan Liu; Zhaochen Su; Hangyu Guo; Shijue Huang; Zhixiong Zhang; Kawai Chung; Zhongwei Xie; Tianshi Zheng

cs.AIOct 10, 2026

SkillHorizon: Constructing Long-Horizon Agent Tasks through Skill Composition

Authors: Yuxuan Liu, Zhaochen Su, Hangyu Guo, Shijue Huang, Zhixiong Zhang, Kawai Chung, Zhongwei Xie, Tianshi Zheng,

Organizations: The Hong Kong University of Science and Technology · Alibaba Group · Shanghai Jiao Tong University

Abstract

Long-horizon agents must complete workflows in which later activities depend on earlier outputs and changes to shared state. A key challenge for evaluation is to expand task requirements and their dependencies while keeping tasks executable and outcomes verifiable. We introduce SkillHorizon, a benchmark that constructs long-horizon tasks through skill composition in seven business environments. Its generators use domain-specific rules to turn reusable skill procedures into concrete business activities. These activities are connected through input-output dependencies, shared resources, and state updates, allowing workload and coordination demands to grow together. Successful reference executions validate task feasibility and measure interaction workload, while programmatic evaluators assess local completion and whole-task success. We evaluate five language models on 1,050 tasks with reference interaction workloads ranging from 64K to 1M tokens. Across these scales, mean local-goal completion falls from 90.4% to 46.8%, as completed work fails to keep pace with growing requirements. At larger workloads, agents more often submit unsuccessful final answers with substantial work unfinished and tool-call allowance remaining. Trace analysis also identifies cases where scripts execute successfully but their outputs fail downstream business checks. SkillHorizon provides a configurable testbed for studying sustained progress, dependency coordination, and complete task delivery in growing workflows. Our code is available at https://github.com/KnowComp/SkillHorizon.

Figures & tables

Figure 1: SkillHorizon constructs long-horizon tasks through skill composition. (a) Environments provide objects, resources, tools, and skills. (b) Tasks grow through dependencies, resource constraints, and updates requiring downstream refresh. Public reference executions validate business completion and measure

L_{\mathrm{ref}}

. (c) Agents execute validated tasks through repeated interactions, changing business state and accumulating actions

a_{i}

and observations

o_{i}

Model	Metric	64K	128K	256K	512K	1M	Overall
Grok 4.6	LGC $\uparrow$	95.5	93.1	89.2	78.7	73.4	86.0
	TS $\uparrow$	90.0	81.4	74.8	60.5	52.4	71.8
Qwen3.8-Flash	LGC $\uparrow$	94.3	83.1	73.1	61.6	48.9	72.2
	TS $\uparrow$	82.9	61.4	45.2	39.5	22.9	50.4
DeepSeek V4.1 Flash	LGC $\uparrow$	88.9	79.1	72.5	63.9	56.1	72.1
	TS $\uparrow$	74.8	53.8	43.8	37.1	26.2	47.1

Table 1: Main results across seven environments and reference-workload bands (%). Local-goal completion (LGC) averages terminal native-goal fractions; task success (TS) measures full business acceptance. Both metrics use the same 210 executions per model and band, balanced across environments. Overall averages all 1,050 executions per model.

Figure 2: Task success across seven business environments and reference-workload scales. The Overall panel pools all seven environments with equal task weight. Colors and markers identify models; each environment point includes 30 assigned tasks and each Overall point includes 210. Table 9 reports task success and local-goal completion.

Figure 3: Growing requirements outpace effective output. (a) Required and completed goal counts across five workload bands, averaged over five models and separately normalized to each environment’s 64K baseline. The gray diagonal marks equal proportional growth. (b) Percentage of failed finals with both measurements that complete less than 25% of goals while leaving more than 50% of the tool allowance unused.

Figure 4: Local completion and global delivery across seven environments and five models. (a) Joint outcomes for all 5,250 executions, with 1,050 per workload band. (b) Task success among executions whose terminal LGC meets threshold

q

, shown separately for each workload band. Table 13 reports the numerators and denominators.

Figure 5: Recovery across workload scales. (a) Task-averaged recovery within 3–7 subsequent checks of the same goal and obligation cycle, shown by model and workload. (b–d) Local-goal completion, task success, and failed-goal recovery along fixed trajectories at

B_{\lambda}=R+\lceil\lambda A\rceil

, using the same 3,374 executions with public failures by

R

and retaining earlier terminal states. Appendix E.3 details the populations and counting rules.

Metric	Retained	Removed
First-check acceptance (%)	84.5	77.2
Ever accepted (%)	91.8	89.8
Additional checks	0.17	0.34

Table 2: Local dependency checks and task-level outcomes across compaction.

Figure 6: Mean and central 90% range across 1,000 family-balanced task orders.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Representative resources	Seed records	Skills
FinanceOps	Financial sources, transactions, identity maps, invoices, and remittances	Generated	12
FieldOps	Devices, service networks, maintenance windows, crews, and spare parts	Generated	9
SocialOps	4,200 posts; 180 campaigns; 900 assets	20,734	8
LogisticsOps	850 passengers; 320 flights; 4,100 bookings	21,150	9
ResearchOps	1,800 papers; 900 authors; 2,000 citations	11,160	7
DocOps	140 projects; 1,600 pages; 6,400 document blocks	22,612	6

Appendix

Table 3: Environment resources and public skill libraries. Seed records include relationship and event records in the underlying databases. Generated resources are instantiated from task parameters. Table 6 reports the business scale of evaluated tasks.

Domain	Fixed construction factors	Scale adjustment
FinanceOps	Workflow family, variant, stage count, and seed	Adjust source width using public reference workload.
FieldOps	Profile and scenario	Calibrate service and device counts with the public reference.
SocialOps	Scenario and dependency pattern	Adjust campaigns, waves, and campaigns in the final wave using the fixed public reference policy.
LogisticsOps	Family, topology, and route legs	Adjust cohort count; use the next planned seed only when the size granularity cannot fit the target interval.
ResearchOps	Family, seed, sections, rounds, aliases, exclusions, and topology	Adjust studies per section and trim terminal studies while retaining at least one original study per topic.
DocOps	Family, seed, and topology	Adjust document count $n$ ; derive depth as $\lceil n/3\rceil$ .

Appendix

Table 4: Environment-specific workload calibration. Fixed factors apply within each calibration search. Appendix A.2 specifies dependency and update semantics.

Band	Lower bound	Upper bound
64K	0	65,536
128K	65,536	131,072
256K	131,072	262,144
512K	262,144	524,288
1M	524,288	1,048,576

Appendix

Table 5: Reference-workload bands, with an exclusive lower bound and inclusive upper bound. K denotes

1024

tokens and M denotes

1024^{2}

tokens. Each instance retains its measured reference length.

Environment	Attribute	64K	128K	256K	512K	1M
FinanceOps	Work packages	24–64	54–132	110–270	224–544	480–1,152
	Logical stages	4	6	10	16	24
	Business calls	126–300	249–622	509–1,228	1,033–2,438	2,221–5,092
FieldOps	Devices	9–33	30–64	49–128	120–294	294–768
	Services	8–20	16–36	32–68	64–196	176–446
	Business calls	139–206	301–411	573–728	1,281–1,765	3,593–4,040

Appendix

Table 6: Business scale across workload bands. Entries are minimum–maximum ranges over 30 tasks per environment and band. Business calls count reference business-execution invocations, including each supplied script invocation as one call.

Model identifier	Reasoning setting	Max. output tokens
grok-4.6	Medium	8,192
qwen3.8-flash	Medium or 4,096 tokens	8,192
deepseek-flash	Medium	32,768
gemini-3.8-flash	Medium	8,192
gpt-5.6-sol	Medium	8,192

Appendix

Table 7: Requested per-stage generation settings. Output limits include reported reasoning tokens where applicable.

Environment	Local-goal unit	Local completion	Additional task requirements
FinanceOps	Source work package	Correct current outputs and required historical views incorporated into stage closures	Stage accounting, release review, publication, and released obligations
FieldOps	Device case	Restored device with passing current tests	Service commissioning, closed isolations, resource accounting, and system acceptance
SocialOps	Campaign brief	Accepted experiment, report, and deployment with valid content, audience, and source evidence	Scheduled updates, shared publication constraints, and business-input integrity
LogisticsOps	Booking	Acceptance by network evaluation	Cargo and freight delivery requirements
ResearchOps	Review round	Accepted review with valid evidence and synthesis	Completion of all rounds and event/state integrity
DocOps	Page	Accepted current page, publication, and matching tracker evidence	Completed change windows and update/state integrity

Appendix

Table 8: Local-goal units, completion checks, and whole-task delivery requirements. TS requires every local goal and the additional requirements in the final column.

Environment	Model	64K	128K	256K	512K	1M
SocialOps	Grok 4.6	98.9 / 96.7	98.0 / 93.3	96.7 / 96.7	96.8 / 93.3	99.3 / 93.3
	Qwen3.8-Flash	100.0 / 100.0	100.0 / 100.0	97.8 / 96.7	95.6 / 90.0	80.7 / 53.3
	DeepSeek V4.1 Flash	100.0 / 100.0	98.7 / 96.7	97.8 / 96.7	96.9 / 86.7	90.3 / 80.0
	Gemini 3.8 Flash	96.7 / 90.0	92.7 / 63.3	90.7 / 76.7	80.4 / 50.0	47.4 / 30.0
	GPT-5.6 Sol	70.0 / 53.3	54.0 / 26.7	46.9 / 13.3	33.5 / 10.0	20.6 / 0.0
LogisticsOps	Grok 4.6	100.0 / 100.0	99.1 / 96.7	100.0 / 100.0	99.7 / 96.7	89.2 / 83.3

Appendix

Table 9: Local-goal completion / task success (LGC / TS, %) by environment and workload. Both metrics use all 30 executions per cell. LGC averages native terminal-goal fractions; TS requires full business acceptance.

SocialOps
Family	Grok	Qwen	DeepSeek	Gemini	GPT
Overlapping conversions	99.1 / 98.0	97.0 / 92.0	96.8 / 92.0	95.4 / 92.0	46.1 / 22.0
Cooldown reactivation	98.0 / 94.0	96.7 / 90.0	96.3 / 90.0	69.8 / 30.0	40.6 / 20.0
Overlap + reactivation	96.6 / 92.0	90.7 / 82.0	97.1 / 94.0	79.5 / 64.0	48.4 / 20.0
LogisticsOps
Carrier disruption	97.5 / 96.0	42.8 / 34.0	58.3 / 44.0	52.0 / 50.0	51.5 / 20.0

Appendix

Table 10: Local-goal completion / task success (LGC / TS, %) by task family, pooling the five workload bands. Each cell gives equal weight to 50 executions, or 75 for each FieldOps scenario. Model labels abbreviate the full names in the experimental setup.

Model	Attempts	Compactions	Util. (%)	$n$	Median $U/R$	P90 $U/R$
	All executions			Successful executions
Grok 4.6	1,224,271	4,182	76.7	754	$1.10\times$	$1.26\times$
Qwen3.8-Flash	952,604	1,818	73.6	529	$1.08\times$	$1.20\times$
DeepSeek V4.1 Flash	1,162,694	3,379	75.9	495	$1.12\times$	$1.27\times$
Gemini 3.8 Flash	583,490	1,578	70.7	451	$1.11\times$	$1.30\times$
GPT-5.6 Sol	1,134,881	2,898	82.3	195	$1.11\times$	$1.35\times$

Appendix

Table 11: Cumulative costs for 1,050 executions per model and reference-relative costs for the successful subset (

n

U

counts charged attempts and

R

reference calls. Compactions count adopted history replacements; utilization is median

100U/B^{\mathrm{run}}

using actual allowances. P90 is the 90th percentile.

Figure 7: Mean completed native units (solid) and required units (dashed), by model and environment. All 30 assigned executions contribute to each cell; required counts are identical across models. Panels retain domain-specific scales.

Workload	Measured/all finals	Events	Measured rate (%)	All assigned (%)
64K	206/208	5	2.4	0.5
128K	406/412	21	5.2	2.0
256K	518/543	79	15.3	7.5
512K	609/653	191	31.4	18.2
1M	706/742	381	54.0	36.3

Appendix

Table 12: Failed finals with LGC below 25% and unused allowance above 50%. Measured/all finals reports joint measurement coverage; all-assignment rates use 1,050 executions per band.

	Task success (%)
Workload	$q=75\%$	$80\%$	$90\%$	$100\%$
64K	88.6	92.1	95.3	99.7
128K	79.7	85.8	93.3	100.0
256K	75.0	80.2	87.3	100.0
512K	74.5	75.9	83.1	100.0
1M	62.2	65.7	71.6	96.3

Appendix

Table 13: Task success among executions with terminal LGC at least

q

, pooling seven environments and five models. The accompanying data retain the successful and eligible counts for all thresholds plotted in Figure 4 .

Public rejection reason	Attempts
Required historical source identities	1,796
Current decayed audience loads	1,222
Due-post time or status	366
Schema / post lookup	2 / 1
Total rejected / matched attempts	3,387 / 24,713

Appendix

Table 14: SocialOps consumer rejection after matching a successful script output. Attempt counts and task-averaged rejection fractions have different denominators.

Model	64K	128K	256K	512K	1M
Grok	68/131	91/375	107/853	118/1443	129/3484
Qwen	122/335	150/709	160/1048	162/2097	166/2251
DeepSeek	135/335	140/897	152/1138	158/1348	170/3432
Gemini	107/333	132/793	143/891	97/426	92/1174
GPT	159/602	174/1283	179/2069	180/3584	180/4234

Appendix

Table 15: Eligible executions / first-failure cycles for the recovery curves. Every retry cutoff uses these same populations; their environment composition varies across cells.

Workload	Executions	Failed goals	Recovered (%)	Rechecked, no pass (%)	No recheck (%)
64K	591	1,736	88.7	5.8	5.4
128K	687	4,057	82.6	11.0	6.5
256K	741	5,999	80.5	12.7	6.8
512K	715	8,898	79.7	14.6	5.8
1M	737	14,575	74.1	15.8	10.1

Appendix

Table 16: Observed outcomes after a public business failure. Percentages are averaged within each execution and then across eligible executions in each workload band. All subsequent checks up to the observed endpoint are included.

		First pass (%)		Ever accepted (%)		Additional checks
Environment	Executions	Retained	Removed	Retained	Removed	Retained	Removed
FinanceOps	342	50.7	47.1	51.1	51.1	0.16	0.28
FieldOps	302	96.2	94.3	99.3	99.2	0.05	0.06
SocialOps	137	95.4	86.8	100.0	100.0	0.08	0.22
LogisticsOps	159	77.1	67.0	97.3	91.7	0.49	1.05
ResearchOps	120	99.7	99.6	100.0	100.0	0.00	0.00

Appendix

Table 17: Business checks with required inputs retained or removed from the active context. Each environment uses executions with observations in both groups; the overall row weights the seven environments equally. Acceptance rates are percentages.

Compaction	Executions	LGC (%)	TS (%)	Mean tool calls
None	1,116	37.0	23.4	303
At least one	1,734	71.3	41.1	1,159

Appendix

Table 18: Observed task outcomes with and without compaction. Both groups receive equal weights over the same 95 environment–workload–model strata.

Gemini 3.8 Flash
Workload	$n$	LGC	Unused allowance	Low progress, high allowance
64K	41	66.7	18.1	0.0
128K	82	66.1	24.3	15.9
256K	112	25.0	44.7	36.6
512K	153	3.2	90.8	67.3
1M	188	1.2	96.1	83.0

Appendix

Table 19: Stopping behavior at unsuccessful agent-submitted final answers. LGC and unused allowance are medians. The last column gives the share with LGC below 25% and unused allowance above 50%.

n

counts executions with both measurements; other values are percentages.

Configuration	Success	Final failure	$B$ exhausted	TS
Harbor + Inspect	9	38	3	18.0
Codex + direct MCP	42	8	0	84.0

Appendix

Table 20: GPT-5.6 Sol on the same 50 FinanceOps tasks, with ten tasks per workload band. Both configurations request medium reasoning. TS is a percentage; other entries are task counts.

Environment	LGC	TS	High-completion failures
FinanceOps	85.0	6.7	21
FieldOps	84.7	30.0	14
DocOps	97.2	66.7	8

Appendix

Table 21: DeepSeek V4.1 Flash at 1M in three environments. Each row includes 30 tasks. LGC and TS are percentages; the last column counts unsuccessful tasks with at least 75% LGC.

Long-Horizon Agent Tasks

Long-Horizon Agent Tasks study how AI agents plan, remember, and act reliably across extended, multi-step interactions.

See all 180 papers

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Jul 9, 2026Zongxia Li, Zhongzhi Li, Yucheng Shi +12Long-Horizon Agent Evaluation AI Agent Benchmarks
Tencent HY LLM Frontier · University of Maryland, College Park · University of Georgia +5
LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks
Aug 3, 2026Ziyu Ma, Hailang Huang, Shun Zou +5Long-Horizon Agent Evaluation Long-Horizon LLM Agents
DreamX Team, Alibaba Group
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Jun 5, 2026Rishi Desai, Jesse Hu, Joan Cabezas +23AI Coding Agents Software Engineering Benchmarks
Abundant · Zenity · Harvard University +14

Jul 9, 2026J/K move · Enter open · S save

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang+11

Tencent HY LLM Frontier · University of Maryland, College Park · University of Georgia+5

Long-Horizon Agent Evaluation AI Agent Benchmarks Long-Horizon Agent Tasks

Open paper AskPDF arXiv

AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.

Abstract

Long-Horizon Agent Tasks

Explore similar work

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks

SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?