Knowledge Graphs as Files: An Agent Harness Where Tool Creation and Tool Use Co-Evolve

Yuyang Bai; Yu Wang; Yu Zhang

cs.CLOct 13, 2026

Knowledge Graphs as Files: An Agent Harness Where Tool Creation and Tool Use Co-Evolve

Authors: Yuyang Bai, Yu Wang, Yu Zhang

Organizations: Texas A&M University · University of Georgia

Abstract

Searching text-rich knowledge graphs requires following relations while examining textual evidence. Predefined graph interfaces constrain what evidence agents can access, even when they write code to combine tool calls or process retrieved subgraphs. We propose direct knowledge graph interaction (DKGI), a harness in which language model agents create graph access tools directly over complete node and edge files. The agent writes Python code to follow relations, combine node sets, and select textual evidence, interleaving execution with shell search and file inspection. This lets agents process large node sets in bulk while returning selected text for the model to assess. Execution feedback guides tool creation and revision, while the evolving tools determine what evidence the agent gathers next. On benchmark datasets spanning three domains, training-free DKGI exceeds the strongest published baseline by at least 6.7 Hit@1 points on every graph, even with a minimal harness. Controlled ablations show the benefits of multi-turn programming over shell-only access and predefined graph tools, while trajectories show agents revising code to extend searches or address errors. These findings highlight the value of allowing tool creation and tool use to co-evolve during graph search. Our code and data are available at https://github.com/leopoldwhite/DKGI.

Figures & tables

Figure 1: An example from the STaRK-MAG dataset ( Wu et al., 2024b ) : “Papers on data quality enhancement algorithms affiliated with the University of the Balearic Islands” . (a) A neighbors call can only return 20 (out of 66) researchers affiliated with the university. The agent never encounters the gold paper in this run. (b) In our DKGI, tool creation and tool use co-evolve : execution feedback guides how the agent revises and extends its code, and the revised code gathers further evidence, ultimately isolating the gold paper. Bracketed numbers denote call indices.

	Predefined	Written by the agent
Setting	graph API / retriever	code	revisions	direct file access
①Direct tool use	✓	–	–	–
ToG, Graph-CoT, ARK, GRASP
Graph tools
②Code over a graph interface	✓	✓	(✓)	–
semantic parsing, KG-Agent, Code-on-Graph, AvaTaR

Table 1: Graph access and agent-written code across three settings. Direct file access lets agent-written code access all node and edge files without a predefined graph API or subgraph retriever. “ (✓) ” indicates support in some systems. Italicized entries denote our controlled variants (§ 3.3 ).

Trained or tuned on STaRK
	Amazon			MAG			Prime
Method	H@1	R@20	MRR	H@1	R@20	MRR	H@1	R@20	MRR
AvaTaR ( Wu et al., 2024a )	49.9	60.6	58.7	44.4	50.6	51.2	18.4	39.3	26.7
PASemiQA ( Yang et al., 2025 )	45.9	–	55.7	43.2	–	50.2	29.7	–	31.0
mFAR (All+2) ( Li et al., 2025a )	53.0	66.3	64.3	55.9	74.1	64.3	40.0	72.6	52.0
MoR ( Lei et al., 2025 )	52.2	59.9	62.2	58.2	75.0	67.1	36.4	63.5	46.9

Table 2: H@1, R@20 and MRR on the synthetic test sets of STaRK. Bold marks the best value in each column. Baseline numbers are published values (backbones in Appendix E ), and a dash marks a value absent from the source paper. † R@20 from the complete-pipeline row of its ablation table. ‡ From the H@1 and MRR tables of Yang et al. (2025) . The DKGI rows cover the full test sets (7,108 queries) and share the prompt and the time limit.

			H@1					Overall
Variant	Graph access	Python	Amazon	MAG	Prime	Overall	$\Delta$	R@20	MRR
Full DKGI	node and edge files	✓	59.1	91.0	69.6	75.2	–	78.0	78.9
Shell only	node and edge files	–	48.8	86.1	62.9	68.3	$-6.9$	69.5	72.0
Graph tools + code	predefined graph tools	✓	60.4	90.2	67.1	74.2	$-1.0$	79.4	78.9
Graph tools	predefined graph tools	–	59.8	87.6	66.4	72.8	$-2.4$	79.5	77.6
One program	node and edge files	once	48.2	87.2	63.9	69.0	$-6.2$	70.8	72.5

Table 3: Ablation on the test subset (710 queries) with GPT-6-Luna in Pi, unless marked Codex. Graph access : the graph files or graph tools that the agent can use. Python : ✓ over several turns, “once” for a single program per query, and – for no Python access. The first four rows form the

2\times 2

design, the fifth row limits Python to one program, and the last three rows are controls.

\Delta

: overall H@1 minus that of Full DKGI.

Figure 2: Behavior of Full DKGI (Pi, GPT-6-Luna) on the full test sets. (a) Python commands, rg searches and other calls (mostly read ) by position in the trajectory. (b) Programs by operation type (types overlap). The blue types have counterparts among predefined graph tools. (c) Error types of incorrectly answered queries (Appendix H ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Amazon	MAG	Prime
Node types	4	4	10
Relation types	5	4	18
Nodes	1,035,542	1,872,968	129,375
Candidates	957,192	700,244	129,375
Stored edge rows	5,382,103	19,919,698	8,100,498
Node shards	104	188	13

Appendix

Table 4: Size of the exported graphs (1 GB =

10^{9}

bytes, 1 KB =

10^{3}

bytes). The row “per node” gives the size of the node files divided by the number of nodes. The export total includes schema.json , DATA_FORMAT.txt and candidate_ids.txt . STaRK reports 9,443,802, 39,802,116 and 8,100,498 relations for Amazon, MAG and Prime, respectively. For Amazon and MAG, these counts equal the number of distinct ordered node pairs after every stored row is also added in reverse.

get_schema()

Returns schema.json .

search(query, k=5)

The

k\leq 30

best BM25 matches for query over all nodes, each with ID, name, type, score and a short context around the first matching query word.

get_node(node_id)

The complete JSON record of the node, as stored in the graph export.

neighbors(node_id, relation='', node_type='', query='', k=20)

Appendix

Table 5: Graph tools available to Graph tools, Retrieval only and Graph tools + code. The box gives the description of each tool that the model receives in Graph tools and Retrieval only, in the order of the rows. The tool section of the prompt is given in Appendix B.1 . Retrieval only has all tools except neighbors .

	$ per		Tokens		Time
Configuration	query	Calls	Input	Output	(s)
Full test set (7,108 queries), GPT-6-Luna
Full DKGI	0.0050	10	120,346	2,212	273
Full DKGI (Codex)	0.0082	10	250,080	3,930	170
Full test set (7,108 queries), GPT-6-Astra
Full DKGI	0.42	9	119,106	1,151	139

Appendix

Table 6: Cost and resource use per query.

per query is the mean over queries, and the other columns are medians. Calls are the agent’s actions

a_{t}$ (§ 2 ), input tokens include cached input, and time is wall-clock time. For Graph tools + code, calls are bash calls, each of which may call several graph tools, and for Full DKGI (Codex) they are command executions. The wall-clock time of One program mainly reflects slower model responses during its run, with a median of 87 s per model request compared with 33 s for Full DKGI. Its single Python program ran for a median of 10 s.

	H@1				Overall
	Amazon	MAG	Prime	Overall	R@20	MRR
Pi, Luna	59.1	91.0	69.6	75.2	78.0	78.9
Codex, Luna	58.5	87.6	67.5	73.0	75.5	76.9
Pi, Astra	72.6	93.2	79.6	83.1	84.9	85.9
Codex, Astra	72.6	93.2	79.6	83.1	84.8	86.2

Appendix

Table 7: Full DKGI on the test subset (710 queries). Each row is labeled with the harness and the model, where Luna and Astra denote GPT-6-Luna and GPT-6-Astra. All rows use the same prompt and time limit per attempt.

Full test set
	MAG		Prime
Name probe	right	wrong	right	wrong
Queries	61	2,604	585	2,216
Full DKGI	67.2	88.4	89.7	56.9
Test subset
Queries	4	262	57	223

Appendix

Table 8: H@1 of the Pi variants with GPT-6-Luna (all except One program) on the queries that the name probe answers correctly (right) or incorrectly (wrong).

	Pi		Codex
	Luna	Astra	Luna
Write, run: programs per query	4.0	4.0	5.3
Read: programs with earlier IDs (%)	48.4	59.9	50.3
next turn runs Python (%), after a
truncated rg output	66.8	76.1	–
untruncated rg output ${\geq}$ 4 KB	59.3	40.1	–

Appendix

Table 9: Tool creation and tool use in the full-test trajectories of Full DKGI in Pi with GPT-6-Luna and GPT-6-Astra and in Codex with GPT-6-Luna. A program is one inline Python source, earlier IDs are node IDs shown in outputs of earlier turns, and truncated outputs were shortened by head-and-tail truncation in Pi. Codex truncates outputs by its own rule, and its two truncation cells show a dash. The measures are defined in this appendix.

GPT-6-Luna
		Share of errors (%)
		Ranking	Selection	Search	Empty
Dataset	H@1	error	error	miss	answer
Amazon	57.8	58	1	41	0
MAG	88.0	46	26	23	6
Prime	63.8	55	7	37	0

Appendix

Table 10: Error categories of Full DKGI in Pi on the full test sets (7,108 queries) with GPT-6-Luna and GPT-6-Astra. The four categories divide the incorrectly answered queries, and each share is a percentage of these queries, rounded to a whole number.

Configuration	Rank	Calls	Rank	Calls	Rank	Calls
	Case 1		Case 2		Case 3
	MAG		Prime		Amazon
Full DKGI, GPT-6-Luna	1	21	1	13	1	13
Full DKGI, GPT-6-Astra	1	10	1	5	1	10
Full DKGI (Codex)	1	12	1	9	1	11
Graph tools + code	1	11	1	9	1	4

Appendix

Table 11: Outcome of each configuration on the three case requests. Rank: rank of the first gold node in the returned list (rank 1 is correct under H@1). –: every returned node is incorrect. Calls: the agent’s actions

a_{t}

of § 2 (for Graph tools + code, bash calls, each of which may call several graph tools, and for Full DKGI (Codex), command executions). All rows except the second use GPT-6-Luna.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 286 papers

Schema-Agnostic Graph Reasoning Agent for Hybrid Knowledge Graphs
Aug 16, 2026Marius Dragic, Ruben Ifrah, Alexandre RioLLM Reasoning with Graphs Tool-Using Agents
R&D Department, Oplit
GraphScout: Empowering Large Language Models with Intrinsic Exploration Ability for Agentic Graph Reasoning
Mar 2, 2026Yuchen Ying, Weiqi Jiang, Tongya Zheng +4LLM Reasoning with Graphs Agentic RAG
Zhejiang University, Hangzhou, China · Hangzhou City University, Hangzhou, China · Nanyang Technological University, Singapore, Singapore
Wikidata Search Traces: A Dataset for Training Knowledge Graph Search Agents
Oct 5, 2026Mohamed Chenene, Carlos Rosas-Hinostroza, Anastasia Stasenko +2Multi-Hop QA Knowledge Graph Question Answering
PleIAs · Lattice, ENS-PSL, CNRS, Sorbonne Nouvelle · Université Paris Dauphine-PSL +4

Aug 16, 2026J/K move · Enter open · S save

Schema-Agnostic Graph Reasoning Agent for Hybrid Knowledge Graphs

Marius Dragic, Ruben Ifrah, Alexandre Rio

R&D Department, Oplit

LLM Reasoning with Graphs Tool-Using Agents Knowledge Graph Question Answering Knowledge Graphs Graph-Based Retrieval

Open paper AskPDF arXiv

Tool-calling LLM agents navigate unfamiliar codebases with a handful of generic primitives for listing, reading and searching files (ls, cat, grep). A knowledge graph admits the same interface: listing neighbours, reading node content and searching descriptions are the same operations on a different substrate. Building on this correspondence, we present GRA, a Graph Reasoning Agent that explores hybrid knowledge graphs, whose nodes are either textual concepts or relational tables, with seven generic tools, discovering everything domain-specific at run time. On UFK-M (Unified Factory Knowledge Model), an industrial benchmark of 258 analytical questions whose gold answers are produced by executing validated SQL programs, GRA beats a full-context agent by 5.1 pp (88.4% vs. 83.3%), while reading under a third of its input tokens. A graph-free control shows the gain comes chiefly from selective agentic access rather than graph topology, and that the effect depends on a model able to drive tools reliably. Seeing less, the agent answers better: selective navigation over a structured substrate beats exhaustive context.

Abstract

Tool-Using Agents

Explore similar work

Schema-Agnostic Graph Reasoning Agent for Hybrid Knowledge Graphs

GraphScout: Empowering Large Language Models with Intrinsic Exploration Ability for Agentic Graph Reasoning

Wikidata Search Traces: A Dataset for Training Knowledge Graph Search Agents