Grounding LLM Agents in Business Processes: Process-Aware Knowledge Graphs for Reliable Multi-Step Service Invocation
Authors: Saiful Khan, Afrah Farea
Organizations: Scientific Computing, Science and Technology Facilities Council, RAL, OX11 0QX, UK. · Department of Engineering Science, University of Oxford, Parks Road, OX1 3PJ, UK.
Large language model (LLM) agents that call enterprise APIs must select the correct endpoints, follow process order, use valid payload keys, and choose coded values from predefined lists. However, current retrieval-augmented service discovery treats endpoints independently and therefore misses process and schema constraints. We address this limitation by integrating Open Data Protocol (OData) metadata and Business Process Model and Notation (BPMN) reference processes in a typed property graph. A hybrid retriever identifies candidate endpoints using dense embeddings and follows process edges to retrieve subsequent endpoints. We evaluate the method on 42 enterprise services with process-chain ground truth. For queries with low lexical overlap between the query and downstream entities, it improves Recall@3 by 5.7 points (n=74), with consistent gains across embedding models and cut-offs. It matches dense retrieval on other queries and provides no improvement on OpenAPI services without a process layer, indicating that the gains arise from process structure. We also evaluate an LLM agent on 81 multi-step queries. It completes 38.7% of tasks, compared with 10.3% for a chunk-based baseline. In an elicitation probe, providing valid code lists eliminates invented codes but leaves invalid keys in 46% of write runs. Full typed context reduces invalid calls to 3% (3 of 101 runs), with no invented codes. These results show that exposing process and schema constraints during retrieval improves the reliability of agent-driven API calls.
Figures & tables
Term
Meaning
Service (API)
A web interface that lets a program read or write business data.
$metadata
The machine-readable description that every OData service publishes.
EntitySet
A named collection of records, similar to a table (e.g., PurchaseOrder ).
Property
One field of a record (e.g., Supplier ).
Navigation
A link between two EntitySets, with a cardinality such as 1:N.
Value help
The fixed list of valid codes for a property (e.g., NB , UB , FO ).
Table 1: Key terms and metrics used in this paper.
Figure 1: (Left) End-to-end architecture: offline ingestion and indexing feed an online hybrid retriever and an execution-capable ReAct agent. (Right) Property-graph schema: OData structural nodes (purple); BPMN process nodes (orange); ValueHelp enum node (green).
Element
Count
Queries
Count
Source services
42
Total
288
EntitySets
132
Single entity
130
EntityTypes
1,270
Process-chain goal
74
Properties
8,481
Navigation chain
67
Navigation edges
1,499
Multi-step workflow
15
BPMN processes
14
Value-help dep.
2
Table 2: OData-Proc dataset statistics. Structural counts are distinct elements after removing 29 duplicate source rows (Sec. 5.3 ). Most EntityTypes are complex, nested, or parameter types that are never exposed as a top-level collection. The RQ1 process-chain probe ( n=74 ) is exactly the process-chain goal type.
Cond.
R@1
R@3
R@5
R@10
MRR
nDCG@10
lat. (ms)
Full set ( n=138 )
Dense
0.472
0.787
0.870
0.928
0.902
0.868
202.9
Graph
0.132
0.292
0.394
0.617
0.352
0.397
15.6
Hybrid
0.472
0.810
0.885
0.945
0.893
0.875
208.4
Process-chain probe ( n=74 )
Dense
0.318
0.675
0.788
0.869
0.889
0.809
203.9
Table 3: RQ1 retrieval quality on the OData-Proc test split ( n=138 ). Hybrid is our process-aware method; Dense is FAISS-only retrieval; Graph is graph-only retrieval. The lower block shows the process-chain probe subset ( n=74 ); there, Hybrid beats Dense at Recall@3 with p=0.037 (paired test).
Cond.
Runs
Compl.
Calls/run
Invalid/run
Lat. (s)
ChunkRAG
243
0.103
0.89
0.107
5.3
Ours
243
0.387
2.45
0.103
11.5
Table 4: RQ3: mock-execution agent on the 81 multi-step test queries ( 3 runs per query, primary LLM). “Calls/run” is the mean number of tool calls per run, and “Invalid/run” the mean number of invalid calls per run ( GET metadatarequestsarenotcounted).Noruntimedout.Completiondifference+0.284,p<0.001(pairedtest,n=81$ ). Results on an earlier 26-query subset and with other LLMs are in Supplementary Sec. S5.
Cond.
POST runs
Inv. runs
Invalid
Codes
ChunkRAG
127
100/127
218
60
ChunkRAG+VH
128
59/128
120
0
Ours
101
3/101
3
0
Table 5: RQ4: code elicitation on the 56 -query probe ( 3 runs per query; 168 per condition; primary LLM). “POST runs”: runs with at least one POST. “Inv. runs”: POST runs with an invalid call. “Invalid”: all invalid calls (entity set, key, or code). “Codes”: invented codes. ChunkRAG+VH adds the value-help code lists as plain text to the retrieved chunks. No run timed out.
Retriever
R@3
R@5
R@10
Tokens
0.297
0.413
0.614
Hybrid (no BPMN)
0.307
0.392
0.573
Table 6: RQ5: generalization on SOCBench-D (OpenAPI, no BPMN layer), pooled over 11 GICS sectors ( n=110 queries). Hybrid vs. Tokens (paired tests): R@3 +0.010 , p=0.753 ; R@5 −0.021 , p=0.437 ; R@10 −0.041 , p=0.071 .
ChunkRAG
Ours
Retrieval tool
Dense top- k search over EntitySet summaries Pesl et al. (2026)
Dense seeds + bounded process-chain expansion
Retrieval returns
EntitySet, service, and property names
Full typed specification: required keys, navigation cardinality, value-help code lists, process-chain information
Execution
GET and POST through the same mock HTTP layer and validator
Table S1: Tool surface per condition, identical across RQ3 and RQ4.
Structural class
Source
Graph
Coverage
Exactness
EntitySets
132
132
1.000
1.000
EntityTypes
1,270
1,270
1.000
1.000
Properties
8,481
8,481
1.000
1.000
Navigation edges (incl. cardinality)
1,499
1,499
1.000
1.000
Activities (BPMN)
79
79
1.000
1.000
HAS_NEXT (BPMN)
65
65
1.000
1.000
Table S2: Graph-construction fidelity audit. Coverage: share of distinct source elements present in the graph. Exactness: share of graph elements whose attributes all match the source. The 79 activities are the 51 task activities plus 28 start and end events.
Query type
Golds
Seed
Exp.
Displ.
Miss
Single entity
35
33
0
1
1
Value-help dependent
1
0
0
0
1
Navigation chain
49
44
0
1
4
Multi-step workflow
13
6
4
2
1
Process-chain goal
206
120
28
12
46
All
304
203
32
16
53
Table S3: Gold-entity attribution for Hybrid at k=3 (counts over the 304 gold entities of the test split). Seed : in the dense top-3. Exp. : only in the Hybrid top-3, added or moved up by process expansion. Displ. : in the dense top-3 but pushed out by expansion. Miss : in neither. Seed + Exp. equals Hybrid Recall@3 and Seed + Displ. equals Dense Recall@3, micro-averaged over gold entities.
LLM
Cond.
Runs
Compl.
Calls/run
Invalid/run
Timeouts
Primary LLM; difference −0.103 , p=0.461 ( n=26 )
qwen3-coder:30b
ChunkRAG
78
0.295
2.12
0.128
0/78
qwen3-coder:30b
Ours
78
0.192
1.69
0.051
0/78
Sensitivity LLMs
gemma4:31b
ChunkRAG
26
0.192
1.00
0.000
0/26
gemma4:31b
Ours
26
0.115
0.65
0.000
0/26
Table S4: Agent results on the earlier 26-query subset. “Invalid/run” is the mean number of invalid calls per run ( GET $metadata requests are not counted). gemma4:31b runs once per query; the other LLMs run three times. The qwen3.6 rows were collected on CPU-only serving with an earlier version of the system and mainly measure serving instability.
Sector
Hybrid
Tokens
Δ
01 Energy
0.346
0.258
+0.088
02 Materials
0.533
0.432
+0.101
03 Industrials
0.364
0.307
+0.058
04 Consumer discretionary
0.502
0.477
+0.025
05 Consumer staples
0.397
0.394
+0.003
06 Health care
0.339
0.403
−0.064
Table S5: RQ5 per-sector Recall@5 on SOCBench-D (Hybrid vs. Tokens), 10 queries per GICS sector.