When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems
Authors: Shihao Weng, Longtian Wang, Yang Feng, Mingfei Cheng, Jia Liu, Chao Shen, He Ye, Federica Sarro, Xiaofei Xie
Organizations: Nanjing University, Nanjing, China · Xi’an Jiaotong University, Xi’an, China · Singapore Management University, Singapore · University College London, London, United Kingdom
Software quality has long rested on specifications that state what code must do, and on testing and verification that check that the code does it. Agentic systems change this foundation: they replace parts of their logic, such as choosing the next tool, judging whether an action is allowed, or writing code to run, with calls to a large language model (LLM). The program still relies on what this logic must guarantee, but the model's behavior is not fixed by code, varies across runs, and can be steered by its input. The guarantee must therefore come from the program around each LLM call site, as a contract of pre-conditions, prompt constraints, and post-conditions. We present an empirical study of these contracts in 496 LLM call sites of 40 popular open-source agentic systems (13.4 MLOC, four languages). Using a multi-agent pipeline validated by 17,428 human judgments, we trace where every model output flows, derive an expected contract for each call site without access to its code, and compare it with the contract that the code implements. We find that 48% of LLM outputs reach control flow or execution, and 73% of these never reach a person. Programs enforce 78% of their constraints in code, but in the prompt, 67% of the constraints are only requested from the model. Although 82% of call sites are checked before and after the call, 63% of the expected constraints are unmet and only 5.3% of call sites satisfy their whole contract; ten recurring patterns explain 99% of the gaps. 38 mechanisms behind these gaps trigger at runtime, and a small code fix that enforces the expected constraint stops all 38. Our results show that the presence of checks says little about their adequacy, and we derive implications for agent developers, framework maintainers, and researchers.
Figures & tables
Figure 1. A file read by Codewhale reaches a real process through three weak checks.
Figure 2. Overview of the study. The person icon marks steps with manual analysis or expert validation.
Table 1. Subject systems, grouped by task domain and sorted by GitHub stars.
Operation : what the model is asked to do
Group
Category
Definition
Boundary rule
Example
Open-ended
generate
New free-form content
not already in the input (vs. extract)
write a report or code
Grounded
compress
Lossy condensation of the input
drops information (vs. transform)
summarize a conversation
extract
Structured values from the input
copied, not created (vs. generate)
read fields from a page
transform
Meaning-preserving rewrite
keeps meaning (vs. compress)
reformulate a query
Bounded
select
Next action, tool, or resource
picks what to do next (vs. classify)
pick a tool and arguments
Table 2. Classification scheme for the outputs of LLM calls.
Figure 3. Multi-agent contract reconstruction for one responsibility unit.
ID
Name
Applies when
Exclusion rules : a gap is excluded if one rule applies
E1
Before the reply
the gap arises before a model reply exists, in the pre-condition or prompt phase
E2
Shared layer
the defect lies in an extraction, parsing, or retry layer that all outputs share
E3
Narrowed value
a default-on closed value set, validator, or sanitizer narrows the value to a safe range
E4
Downstream check
an independent check downstream stops the bad value before it has an effect
E5
Not live
reaching the defect needs a non-default configuration, or the code is dead
Table 3. Exclusion rules and threat-chain conditions of feasibility screening.
Step
By
Criterion
Review
Coverage
AC1 ( κ )
Calibration : R1, R2, and R3 refine the guidelines on six pilot systems, then freeze them before the main review
Call-site identification
real LLM invocation, not a wrapper or relay
R1 ∥ R2
all (1,226)
0.99 (0.98)
Output decomposition
output split
R1 ∥ R2
all (484)
0.94 (0.44)
operation label
R1 ∥ R2
all (870)
0.98 (0.98)
Consumer tracing
consumer location
R1 ∥ R2
sample (885)
0.91 (0.71)
Stage I
edge type
R1 ∥ R2
sample (885)
0.95 (0.94)
Table 4. Analysis and review protocol with inter-rater agreement.
Figure 4. Operations and consumers of the 917 typed outputs. (a) Outputs per operation. (b) Outputs per consumer type. (c) Consumer kinds per output. (d) Decision-relevant share per system.
Figure 5. A classify gate in STORM (abridged).
Role
Operation → consumer
Out.
Sites
Sys.
Example
Steering the program : the output decides which operations run, or is itself run
Action selection
select → control-flow
140
117
26
Codex: the chosen tool call is dispatched and run
Gating and judging
classify / score → control-flow
72
50
17
Gemini CLI: a policy verdict blocks a tool call
Free-text steering
generate → control-flow
139
126
20
Aider: files named in the reply join the chat
Code execution
generate → executed
66
66
9
Khoj: generated Python code runs in a sandbox
Feeding other components : the output becomes data, context, or a message to a person
Table 5. Recurring roles of LLM call sites.
Figure 6. Assurance of implemented constraints. (a) By phase, kind, call site. (b) Prompt-only share per system.
Mechanism
What it enforces
Constr.
Sys.
Example
Before the call : what the model receives and what it may do
Figure 7. Verdicts of expected constraints. (a) By phase and call site. (b) Required vs. implemented assurance.
Pattern
Expected →
Implemented
Gaps
Sites
Sys.
Example
Before the call : what the model receives and what it may do 420 gaps (25%)
Unchecked call inputs
inputs checked →
call made anyway
330
265
34
Continue: prompt not checked before call
Over-broad authority
least authority →
wider tool set
90
74
27
OpenManus: planner may also delete plans
In the request : how the model is asked 335 gaps (20%)
Underspecified prompt
task fully stated →
left unstated
242
168
32
CodeWhale: router not told request is data
Faulty request assembly
input intact, fenced →
altered or unfenced
93
88
27
PentAGI: raw tool output inlined unescaped
Table 7. Recurring gap patterns between expected and implemented contracts.
Figure 15
Step 1: Feasibility screening
Step 2: Gap attribution
Steps 3 to 6
RQ3
E1
E2
E3
E4
E5
E6
E7
No chain
Expl.
Direct
Partial
Shared
On chain
PoC
Trig.
Patched
Blocked
Gaps
1,697
716
30
165
138
84
132
137
5
290
48
66
176
114
–
–
68
32
Mech.
–
–
–
–
–
–
–
–
–
–
–
–
–
60
60
38
38
23
Table 8. Gaps and mechanisms through the six steps of Stage III.
Pattern
Gaps
Exploitable
Direct
Partial
Unchecked call inputs
330
7
3
0
Over-broad authority
90
15
7
3
Underspecified prompt
242
9
0
8
Faulty request assembly
93
12
1
11
Partial format check
249
72
6
5
Unchecked content
253
81
11
9
Table 9. Exploitable gaps by pattern.
System
Mechanism
Source
Hits
Causes of misses
Triggered in every run
AgentLaboratory
bare regex passes model text to exec()
attacker
5/5
JARVIS
picks a tool from an empty candidate pool
model error
5/5
gpt-researcher
chat renders report text as HTML
attacker
5/5
Triggered in some runs
skyvern
browser action policy only observes
attacker
4/5
cross-domain link flagged (1)
Table 10. Seven trigger tests against a real model, grouped by outcome.
Figure 10. Two mechanisms of Table 10 . (a) A render path reaches the browser with no sanitizer. (b) An unscoped opt-in reaches an internal address, with no injection needed.