CaSKG: Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrieval

Zhiyuan Li; Linyuan Gao; Xuechun Ding; Hongwei Chen; Yuan Wu; Yi Chang

cs.AIAug 26, 2026

CaSKG: Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrieval

Authors: Zhiyuan Li, Linyuan Gao, Xuechun Ding, Hongwei Chen, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University Ant Group Changchun, China · Ant Group Hangzhou, China

Abstract

Reusable skill libraries allow large language model (LLM) agents to reuse procedural knowledge across tasks, but they also turn memory access into a challenging retrieval problem. Full-library prompting preserves coverage at high context cost, vector retrieval returns compact neighborhoods but treats skills as independent text, and graph-based retrieval can recover workflow context only when the edges that carry relevance are reliable. We propose CaSKG, a counterfactual-causal skill graph framework that calibrates procedural relations before retrieval. CaSKG first builds a high-recall directed candidate graph from semantic, lexical, input/output, and structural evidence, with repair evidence and an optional LLM judge further refining candidate scores. It then applies direction-conditioned textual counterfactual probes that remove, substitute, and reorder skill pairs, aggregates the evidence with Bayesian smoothing, and publishes a state-filtered weighted graph for task-conditioned expansion. The graph is constructed offline and used without changing the downstream agent policy or task interface. Across six LLM backbones on ALFWorld ID-140 and ScienceWorld U211, CaSKG achieves the highest task score in all twelve combinations of model and benchmark. Relative to Graph-of-Skills (GoS), it improves the six-model macro-average ScienceWorld score from 72.62 to 80.50 and ALFWorld success from 80.01% to 86.79%, while reducing mean environment steps on both benchmarks. Qualitative and ablation analyses further show that calibrated edges help retrieval preserve prerequisites, state-changing actions, verification routines, and final completion steps. These results position edge-confidence calibration as an effective route to compact and executable skill retrieval at scale\footnote{Code is available at: https://github.com/ZhiyuanLi218/Caskg }.

Figures & tables

Figure 1. Overview of CaSKG. Multiple skill-level signals induce a directed candidate set. A budgeted subset is examined with direction-conditioned removal, substitution, and reordering probes. The resulting evidence assigns edge states and publication weights, producing the published skill graph. Task-conditioned retrieval then propagates query relevance over this graph to return a compact skill context. A four-stage CaSKG pipeline. Candidate relations are induced from skill-level signals, selected edges are evaluated with removal, substitution, and reordering probes, edge states determine graph publication, and personalized PageRank retrieves task-relevant skills from the published graph.

Model	Method	ALFWorld ID-140		ScienceWorld U211
		$R\,(\%)\uparrow$	Steps $\downarrow$	$R\uparrow$	Steps $\downarrow$
MiniMax-M2.7	Vanilla	42.90	22.54	45.90	21.73
	Vector	45.70	22.84	43.21	21.45
	GoS	63.60	19.69	55.85	18.91
	CaSKG	73.57	18.44	68.33	17.45
GLM-5.2	Vanilla	95.00	11.05	75.50	17.03

Table 1. Reward and mean interaction steps at Skill1000 on ALFWorld ID-140 and ScienceWorld U211.

Figure 2. Task-type reward gains of CaSKG over GoS on ScienceWorld U211. All 24 task types are shown individually and ordered by gain; each bar is the unweighted mean reward difference across the six evaluated backbones. Blue and orange indicate positive and negative gains, respectively. Horizontal bar chart of the mean CaSKG score difference over GoS across 24 ScienceWorld task types. CaSKG improves on 21 types, ties on one, and trails on non-living-thing search and temperature measurement.

Model	Skills	$R\,(\%)\uparrow$			Steps $\downarrow$
		CaSKG	GoS	$\Delta R$	CaSKG	GoS
MiniMax-M2.7	200	57.14	50.00	+7.14	20.73	22.21
	500	67.86	45.00	+22.86	19.95	23.07
	1,000	73.57	63.60	+9.97	18.44	19.69
	2,000	70.00	54.29	+15.71	18.71	21.32
Qwen3.5-397B- A17B	200	85.00	76.43	+8.57	14.50	16.25

Table 2. Archived skill-library scale comparison on ALFWorld ID-140 for MiniMax-M2.7 and Qwen3.5-397B-A17B.

Figure 3. Skill-library-size sensitivity on ALFWorld ID-140 for MiniMax-M2.7 and Qwen3.5-397B-A17B. The upper panel reports task success rate and the lower panel reports mean environment steps for CaSKG and GoS at four library sizes. Two-panel line chart comparing CaSKG and GoS for MiniMax-M2.7 and Qwen3.5-397B-A17B at skill-library sizes 200, 500, 1,000, and 2,000. CaSKG has higher success rates and fewer mean environment steps at every size.

Variant	$\|C\|$	$\|F\|$	$\|E_{\mathrm{pub}}\|$	$R\,(\%)\uparrow$	Steps $\downarrow$
Full CaSKG	9,937	500	3,292	73.57	18.44
Semantic-only	3,982	500	2,698	67.14	19.21
w/o LLM judge	9,753	500	3,188	71.43	18.79
Publish all candidates	9,937	0	9,937	71.43	18.74

Table 3. Component ablation of CaSKG’s graph-construction pipeline with MiniMax-M2.7 and the Skill1000 library on ALFWorld ID-140.

LLM Agent Skill Retrieval

Large language model agent skill retrieval finds and selects reusable procedures to help agents complete tasks.

See all 102 papers

Apr 7, 2026

Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills

As LLM agents act across personal applications, web browsers, and other interfaces, their reusable skill libraries can scale to thousands of skills. This scale introduces two challenges. First, loading the full library saturates the context window, driving up token costs, hallucination, and latency. Second, semantic retrieval surfaces topically relevant skills but can miss upstream and downstream prerequisite skills, creating a prerequisite gap that leaves the retrieved bundle insufficient for execution. We present Graph-of-Skills (GoS), an inference-time structural retrieval layer for large skill libraries. GoS constructs an executable skill graph offline from skill packages, then retrieves a bounded, dependency-aware bundle through hybrid semantic-lexical seeding, reverse-aware Personalized PageRank, and context-budgeted hydration. Across SkillsBench and ALFWorld, with three model families (Claude Sonnet 4.5, MiniMax M2.7, and GPT-5.2 Codex), GoS attains the highest average reward in all six model-benchmark blocks, at a fraction of the token cost of loading the full library. On SkillsBench with GPT-5.2 Codex it raises average reward by 7.0 absolute points over full skill loading, a 25.6% relative gain, while cutting total tokens by 56.7%. Ablations isolate the mechanism: replacing reverse traversal with forward propagation costs 9.1 reward points, a larger loss than removing the graph altogether. The gain thus comes from traversing dependencies backwards, not from graph diffusion as such. A budget-matched retrieval study holding seeding, reranking, hydration, and context budget fixed reproduces the same ordering, with dependency-pair co-recovery falling from 0.654 to 0.362. Code is available at https://github.com/davidliuk/graph-of-skills

Dawei Liu, Zongxia Li, Hongyang Du +4

University of Pennsylvania · University of Maryland · Brown University +2

Tool-Augmented Language Model Agents Graph-Based Retrieval LLM Agent Skill Retrieval LLM Agents

Ask PDF

Jun 2, 2026

SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scale

Large skill libraries are now standard equipment for LLM agents, and which subset reaches the prompt decides what the agent can do. Enumeration and top-

K

embedding retrieval both treat skills as independent documents, assuming relevance is a property of one skill at a time. It is not: skills depend on, conflict with, specialize, and duplicate one another, so the closest match to a task can name it while omitting a prerequisite it needs. Graph retrieval encodes these relations but spends them in a fixed pipeline, handing the agent an opaque bundle. We present SkillDAG, which exposes a typed skill graph to the LLM as an agent-callable interface rather than consuming it for the agent: search returns vector matches, typed-edge neighbors, and conflict signals, and a propose-then-commit protocol lets it register execution-backed edges as it works. Across two benchmarks and three model families, SkillDAG outperforms full-library prompting, vector retrieval, and Graph-of-Skills, gaining +14.3 points of ALFWorld success and tying only at ceiling; its ranking holds flat across a

10\times

pool expansion that degrades the fixed pipeline. Inter-skill structure, these results suggest, is worth more to an agent that can query and revise it than to a retriever that consumes it.

Tong Bai, Zhenglin Wan, Pengfei Zhou +3

Fudan University · National University of Singapore · CFAR, A*STAR

LLM Agent Skill Retrieval Dynamic Graph Learning LLM Agents Tool-Using Language Agents

Ask PDF

Aug 3, 2026

SkillTrace: Traversing a Query-Skill Graph for Composable LLM Agents

Large language model agents increasingly solve complex tasks by composing reusable skills from a library. To address this, the key challenge is not merely to retrieve individually relevant skills, but to identify a complete and executable skill composition. In this paper, we argue that this problem can be solved in a graph with three levels: compositional relations among skill queries, similarity between queries and candidates in the skill library, and the dependencies among the selected candidates. We introduce SkillTrace, which organizes the user query into a semantic hierarchy, matches skill queries and candidates, and propagates over the skill dependencies. Experiments on SkillsBench and ALFWorld demonstrate that SkillTrace achieves state-of-the-art performance, reaching a success rate of 53.17% on SkillsBench and 91.43% on ALFWorld. SkillTrace also delivers consistent improvements across different backbone language models, demonstrating the generality and robustness of graph-based skill retrieval.

Yue Yao, Shengyuan Wang, Xin Chen +4

Shandong University · The Australian National University · Tsinghua University +1

Graph-Based Retrieval LLM Agent Skill Retrieval LLM Agents

Ask PDF

Abstract

LLM Agent Skill Retrieval

Explore similar work

Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills

SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scale

SkillTrace: Traversing a Query-Skill Graph for Composable LLM Agents