Grounding LLM Agents in Business Processes: Process-Aware Knowledge Graphs for Reliable Multi-Step Service Invocation
Authors: Saiful Khan, Afrah Farea
Organizations: Scientific Computing, Science and Technology Facilities Council, RAL, OX11 0QX, UK. · Department of Engineering Science, University of Oxford, Parks Road, OX1 3PJ, UK.
Large language model (LLM) agents that call enterprise APIs must select the correct endpoints, follow process order, use valid payload keys, and choose coded values from predefined lists. However, current retrieval-augmented service discovery treats endpoints independently and therefore misses process and schema constraints. We address this limitation by integrating Open Data Protocol (OData) metadata and Business Process Model and Notation (BPMN) reference processes in a typed property graph. A hybrid retriever identifies candidate endpoints using dense embeddings and follows process edges to retrieve subsequent endpoints. We evaluate the method on 42 enterprise services with process-chain ground truth. For queries with low lexical overlap between the query and downstream entities, it improves Recall@3 by 5.7 points (n=74), with consistent gains across embedding models and cut-offs. It matches dense retrieval on other queries and provides no improvement on OpenAPI services without a process layer, indicating that the gains arise from process structure. We also evaluate an LLM agent on 81 multi-step queries. It completes 38.7% of tasks, compared with 10.3% for a chunk-based baseline. In an elicitation probe, providing valid code lists eliminates invented codes but leaves invalid keys in 46% of write runs. Full typed context reduces invalid calls to 3% (3 of 101 runs), with no invented codes. These results show that exposing process and schema constraints during retrieval improves the reliability of agent-driven API calls.
Figures & tables
Term
Meaning
Service (API)
A web interface that lets a program read or write business data.
$metadata
The machine-readable description that every OData service publishes.
EntitySet
A named collection of records, similar to a table (e.g., PurchaseOrder ).
Property
One field of a record (e.g., Supplier ).
Navigation
A link between two EntitySets, with a cardinality such as 1:N.
Value help
The fixed list of valid codes for a property (e.g., NB , UB , FO ).
Table 1: Key terms and metrics used in this paper.
Figure 1: (Left) End-to-end architecture: offline ingestion and indexing feed an online hybrid retriever and an execution-capable ReAct agent. (Right) Property-graph schema: OData structural nodes (purple); BPMN process nodes (orange); ValueHelp enum node (green).
Element
Count
Queries
Count
Source services
42
Total
288
EntitySets
132
Single entity
130
EntityTypes
1,270
Process-chain goal
74
Properties
8,481
Navigation chain
67
Navigation edges
1,499
Multi-step workflow
15
BPMN processes
14
Value-help dep.
2
Table 2: OData-Proc dataset statistics. Structural counts are distinct elements after removing 29 duplicate source rows (Sec. 5.3 ). Most EntityTypes are complex, nested, or parameter types that are never exposed as a top-level collection. The RQ1 process-chain probe ( n=74 ) is exactly the process-chain goal type.
Cond.
R@1
R@3
R@5
R@10
MRR
nDCG@10
lat. (ms)
Full set ( n=138 )
Dense
0.472
0.787
0.870
0.928
0.902
0.868
202.9
Graph
0.132
0.292
0.394
0.617
0.352
0.397
15.6
Hybrid
0.472
0.810
0.885
0.945
0.893
0.875
208.4
Process-chain probe ( n=74 )
Dense
0.318
0.675
0.788
0.869
0.889
0.809
203.9
Table 3: RQ1 retrieval quality on the OData-Proc test split ( n=138 ). Hybrid is our process-aware method; Dense is FAISS-only retrieval; Graph is graph-only retrieval. The lower block shows the process-chain probe subset ( n=74 ); there, Hybrid beats Dense at Recall@3 with p=0.037 (paired test).
Cond.
Runs
Compl.
Calls/run
Invalid/run
Lat. (s)
ChunkRAG
243
0.103
0.89
0.107
5.3
Ours
243
0.387
2.45
0.103
11.5
Table 4: RQ3: mock-execution agent on the 81 multi-step test queries ( 3 runs per query, primary LLM). “Calls/run” is the mean number of tool calls per run, and “Invalid/run” the mean number of invalid calls per run ( GET metadatarequestsarenotcounted).Noruntimedout.Completiondifference+0.284,p<0.001(pairedtest,n=81$ ). Results on an earlier 26-query subset and with other LLMs are in Supplementary Sec. S5.
Cond.
POST runs
Inv. runs
Invalid
Codes
ChunkRAG
127
100/127
218
60
ChunkRAG+VH
128
59/128
120
0
Ours
101
3/101
3
0
Table 5: RQ4: code elicitation on the 56 -query probe ( 3 runs per query; 168 per condition; primary LLM). “POST runs”: runs with at least one POST. “Inv. runs”: POST runs with an invalid call. “Invalid”: all invalid calls (entity set, key, or code). “Codes”: invented codes. ChunkRAG+VH adds the value-help code lists as plain text to the retrieved chunks. No run timed out.
Retriever
R@3
R@5
R@10
Tokens
0.297
0.413
0.614
Hybrid (no BPMN)
0.307
0.392
0.573
Table 6: RQ5: generalization on SOCBench-D (OpenAPI, no BPMN layer), pooled over 11 GICS sectors ( n=110 queries). Hybrid vs. Tokens (paired tests): R@3 +0.010 , p=0.753 ; R@5 −0.021 , p=0.437 ; R@10 −0.041 , p=0.071 .
ChunkRAG
Ours
Retrieval tool
Dense top- k search over EntitySet summaries Pesl et al. (2026)
Dense seeds + bounded process-chain expansion
Retrieval returns
EntitySet, service, and property names
Full typed specification: required keys, navigation cardinality, value-help code lists, process-chain information
Execution
GET and POST through the same mock HTTP layer and validator
Table S1: Tool surface per condition, identical across RQ3 and RQ4.
Structural class
Source
Graph
Coverage
Exactness
EntitySets
132
132
1.000
1.000
EntityTypes
1,270
1,270
1.000
1.000
Properties
8,481
8,481
1.000
1.000
Navigation edges (incl. cardinality)
1,499
1,499
1.000
1.000
Activities (BPMN)
79
79
1.000
1.000
HAS_NEXT (BPMN)
65
65
1.000
1.000
Table S2: Graph-construction fidelity audit. Coverage: share of distinct source elements present in the graph. Exactness: share of graph elements whose attributes all match the source. The 79 activities are the 51 task activities plus 28 start and end events.
Query type
Golds
Seed
Exp.
Displ.
Miss
Single entity
35
33
0
1
1
Value-help dependent
1
0
0
0
1
Navigation chain
49
44
0
1
4
Multi-step workflow
13
6
4
2
1
Process-chain goal
206
120
28
12
46
All
304
203
32
16
53
Table S3: Gold-entity attribution for Hybrid at k=3 (counts over the 304 gold entities of the test split). Seed : in the dense top-3. Exp. : only in the Hybrid top-3, added or moved up by process expansion. Displ. : in the dense top-3 but pushed out by expansion. Miss : in neither. Seed + Exp. equals Hybrid Recall@3 and Seed + Displ. equals Dense Recall@3, micro-averaged over gold entities.
LLM
Cond.
Runs
Compl.
Calls/run
Invalid/run
Timeouts
Primary LLM; difference −0.103 , p=0.461 ( n=26 )
qwen3-coder:30b
ChunkRAG
78
0.295
2.12
0.128
0/78
qwen3-coder:30b
Ours
78
0.192
1.69
0.051
0/78
Sensitivity LLMs
gemma4:31b
ChunkRAG
26
0.192
1.00
0.000
0/26
gemma4:31b
Ours
26
0.115
0.65
0.000
0/26
Table S4: Agent results on the earlier 26-query subset. “Invalid/run” is the mean number of invalid calls per run ( GET $metadata requests are not counted). gemma4:31b runs once per query; the other LLMs run three times. The qwen3.6 rows were collected on CPU-only serving with an earlier version of the system and mainly measure serving instability.
Sector
Hybrid
Tokens
Δ
01 Energy
0.346
0.258
+0.088
02 Materials
0.533
0.432
+0.101
03 Industrials
0.364
0.307
+0.058
04 Consumer discretionary
0.502
0.477
+0.025
05 Consumer staples
0.397
0.394
+0.003
06 Health care
0.339
0.403
−0.064
Table S5: RQ5 per-sector Recall@5 on SOCBench-D (Hybrid vs. Tokens), 10 queries per GICS sector.
LLM-based agents offer new opportunities for automating business process execution beyond the limits of rule-based systems. However, general-purpose LLMs lack the organization-specific knowledge required for reliable execution, which is typically fragmented across human-oriented artifacts such as policies, process models, and standard operating procedures. While such knowledge can technically be encoded in individual prompts or agent-specific retrieval setups, this approach does not scale in enterprises, as it gives rise to knowledge silos and rule duplicates, and makes consistent updates and learning across agents difficult. We argue that this calls for an organizational memory for agentic business process execution: a shared, governed, and agent-consumable reference layer of evolving organization-specific procedural knowledge about how work should be executed. We derive requirements for such a memory, propose an architecture for its curation and consumption, and demonstrate its effectiveness in a proof-of-concept based on a procurement scenario.
Lukas Kirchdorfer, Adrian Rebmann, Christian Warmuth +3
Customer-service LLM agents must follow organizational policy when acting on a user's behalf. Compliance failures arise from either forbidden actions, such as granting an ineligible change, or omitted procedural requirements, such as identification or confirmation. Runtime safeguards can intervene on risky actions, but action-local checks do not guide an agent through a multi-step procedure. Workflow-following systems support prescribed process execution, but primarily target workflow completion rather than safeguarding agent behavior. PolicyGuide instead compiles each domain policy into a workflow graph and invokes a proactive verifier at user-turn boundaries. From persisted graph state, the verifier reconciles open requests and returns step-specific remediation along a policy-compliant path. Across the τ2-bench airline, retail, and telecom domains with a GPT-5.4 agent and verifier, PolicyGuide raises mean Pass4 from 0.42 to 0.62, with the largest gain on telecom (0.19 to 0.61), the most workflow-structured domain. The same workflows transfer to Claude Sonnet 4.6 and Gemini 2.5 Pro agents. Complementary evaluations find the lowest observed attack-success rate under adversarial users and the strongest procedural compliance in an author-designed workflow-level validation.
We present AgenticRAG, a practical agentic harness for retrieval and analysis over enterprise knowledge bases. Standard RAG pipelines place significant burden of grounding on the search stack, constraining the language model to a fixed candidate set chosen deep in the retrieval process. Our approach reduces this overdependence by layering a lightweight harness on top of existing enterprise search infrastructure, equipping a reasoning LLM with search, find, open, and summarize tools enabling the model to iteratively retrieve information, navigate within documents, and analyze evidence autonomously. On three open benchmarks we observe substantial gains: 49.6% recall@1 on BRIGHT (+21.8 pp over the best embedding baseline), 0.96 factuality on WixQA (+13% relative improvement), and 92% answer correctness on FinanceBench--within 2 pp of oracle access to true evidence. Ablation studies show that the most significant factor is the shift from single-shot retrieval to agentic tool use (5.9× improvement), while multi-query search and in-document navigation contribute to both quality and efficiency. We present various design choices in our agentic harness that were informed by pre-production deployments. Our results demonstrate its suitability for real-world enterprise production environments.