Knowledge Graphs as Files: An Agent Harness Where Tool Creation and Tool Use Co-Evolve
Organizations: Texas A&M University · University of Georgia
Abstract
Searching text-rich knowledge graphs requires following relations while examining textual evidence. Predefined graph interfaces constrain what evidence agents can access, even when they write code to combine tool calls or process retrieved subgraphs. We propose direct knowledge graph interaction (DKGI), a harness in which language model agents create graph access tools directly over complete node and edge files. The agent writes Python code to follow relations, combine node sets, and select textual evidence, interleaving execution with shell search and file inspection. This lets agents process large node sets in bulk while returning selected text for the model to assess. Execution feedback guides tool creation and revision, while the evolving tools determine what evidence the agent gathers next. On benchmark datasets spanning three domains, training-free DKGI exceeds the strongest published baseline by at least 6.7 Hit@1 points on every graph, even with a minimal harness. Controlled ablations show the benefits of multi-turn programming over shell-only access and predefined graph tools, while trajectories show agents revising code to extend searches or address errors. These findings highlight the value of allowing tool creation and tool use to co-evolve during graph search. Our code and data are available at https://github.com/leopoldwhite/DKGI.
Figures & tables
| Predefined | Written by the agent | |||
| Setting | graph API / retriever | code | revisions | direct file access |
| ①Direct tool use | ✓ | – | – | – |
| ToG, Graph-CoT, ARK, GRASP | ||||
| Graph tools | ||||
| ②Code over a graph interface | ✓ | ✓ | (✓) | – |
| semantic parsing, KG-Agent, Code-on-Graph, AvaTaR | ||||
| Amazon | MAG | Prime | |||||||
| Method | H@1 | R@20 | MRR | H@1 | R@20 | MRR | H@1 | R@20 | MRR |
| Trained or tuned on STaRK | |||||||||
| AvaTaR ( Wu et al., 2024a ) | 49.9 | 60.6 | 58.7 | 44.4 | 50.6 | 51.2 | 18.4 | 39.3 | 26.7 |
| PASemiQA ( Yang et al., 2025 ) | 45.9 | – | 55.7 | 43.2 | – | 50.2 | 29.7 | – | 31.0 |
| mFAR (All+2) ( Li et al., 2025a ) | 53.0 | 66.3 | 64.3 | 55.9 | 74.1 | 64.3 | 40.0 | 72.6 | 52.0 |
| MoR ( Lei et al., 2025 ) | 52.2 | 59.9 | 62.2 | 58.2 | 75.0 | 67.1 | 36.4 | 63.5 | 46.9 |
| H@1 | Overall | ||||||||
| Variant | Graph access | Python | Amazon | MAG | Prime | Overall | R@20 | MRR | |
| Full DKGI | node and edge files | ✓ | 59.1 | 91.0 | 69.6 | 75.2 | – | 78.0 | 78.9 |
| Shell only | node and edge files | – | 48.8 | 86.1 | 62.9 | 68.3 | 69.5 | 72.0 | |
| Graph tools + code | predefined graph tools | ✓ | 60.4 | 90.2 | 67.1 | 74.2 | 79.4 | 78.9 | |
| Graph tools | predefined graph tools | – | 59.8 | 87.6 | 66.4 | 72.8 | 79.5 | 77.6 | |
| One program | node and edge files | once | 48.2 | 87.2 | 63.9 | 69.0 | 70.8 | 72.5 | |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Amazon | MAG | Prime | |
| Node types | 4 | 4 | 10 |
| Relation types | 5 | 4 | 18 |
| Nodes | 1,035,542 | 1,872,968 | 129,375 |
| Candidates | 957,192 | 700,244 | 129,375 |
| Stored edge rows | 5,382,103 | 19,919,698 | 8,100,498 |
| Node shards | 104 | 188 | 13 |
| get_schema() |
| Returns schema.json . |
| search(query, k=5) |
| The best BM25 matches for query over all nodes, each with ID, name, type, score and a short context around the first matching query word. |
| get_node(node_id) |
| The complete JSON record of the node, as stored in the graph export. |
| neighbors(node_id, relation='', node_type='', query='', k=20) |
| $ per | Tokens | Time | |||
| Configuration | query | Calls | Input | Output | (s) |
| Full test set (7,108 queries), GPT-6-Luna | |||||
| Full DKGI | 0.0050 | 10 | 120,346 | 2,212 | 273 |
| Full DKGI (Codex) | 0.0082 | 10 | 250,080 | 3,930 | 170 |
| Full test set (7,108 queries), GPT-6-Astra | |||||
| Full DKGI | 0.42 | 9 | 119,106 | 1,151 | 139 |
| H@1 | Overall | |||||
|---|---|---|---|---|---|---|
| Amazon | MAG | Prime | Overall | R@20 | MRR | |
| Pi, Luna | 59.1 | 91.0 | 69.6 | 75.2 | 78.0 | 78.9 |
| Codex, Luna | 58.5 | 87.6 | 67.5 | 73.0 | 75.5 | 76.9 |
| Pi, Astra | 72.6 | 93.2 | 79.6 | 83.1 | 84.9 | 85.9 |
| Codex, Astra | 72.6 | 93.2 | 79.6 | 83.1 | 84.8 | 86.2 |
| MAG | Prime | |||
| Name probe | right | wrong | right | wrong |
| Full test set | ||||
| Queries | 61 | 2,604 | 585 | 2,216 |
| Full DKGI | 67.2 | 88.4 | 89.7 | 56.9 |
| Test subset | ||||
| Queries | 4 | 262 | 57 | 223 |
| Pi | Codex | ||
| Luna | Astra | Luna | |
| Write, run: programs per query | 4.0 | 4.0 | 5.3 |
| Read: programs with earlier IDs (%) | 48.4 | 59.9 | 50.3 |
| next turn runs Python (%), after a | |||
| truncated rg output | 66.8 | 76.1 | – |
| untruncated rg output 4 KB | 59.3 | 40.1 | – |
| Share of errors (%) | |||||
| Ranking | Selection | Search | Empty | ||
| Dataset | H@1 | error | error | miss | answer |
| GPT-6-Luna | |||||
| Amazon | 57.8 | 58 | 1 | 41 | 0 |
| MAG | 88.0 | 46 | 26 | 23 | 6 |
| Prime | 63.8 | 55 | 7 | 37 | 0 |
| Case 1 | Case 2 | Case 3 | ||||
|---|---|---|---|---|---|---|
| MAG | Prime | Amazon | ||||
| Configuration | Rank | Calls | Rank | Calls | Rank | Calls |
| Full DKGI, GPT-6-Luna | 1 | 21 | 1 | 13 | 1 | 13 |
| Full DKGI, GPT-6-Astra | 1 | 10 | 1 | 5 | 1 | 10 |
| Full DKGI (Codex) | 1 | 12 | 1 | 9 | 1 | 11 |
| Graph tools + code | 1 | 11 | 1 | 9 | 1 | 4 |
Tool-Using Agents
Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.
See all 286 papers