Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation
Authors: Geert Trooskens, Aaron Karlsberg, Anmol Sharma, Lamara De Brouwer, Max Van Puyvelde, Matthew Young, John Thickstun, Gil Alterovitz, Walter A. De Brouwer
Organizations: XY.AI Labs, Palo Alto, CA · Stanford University School of Medicine, Stanford, CA · Cornell University, Ithaca, NY · Cornell University, Department of Computer Science, Ithaca, NY · Brigham and Women's Hospital / Harvard Medical School, Boston, MA · Brigham and Women’s Hospital / Harvard Medical School, Boston, MA
We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which workflows execute deterministically without further model invocation. This paradigm has antecedents in prior work on declarative pipeline optimization (DSPy) and hybrid neural-symbolic planning (LLM+P); our contribution is a systems-oriented study of its application to high-stakes enterprise workflows, with particular emphasis on healthcare settings where reliability and auditability are critical. By constraining generation to narrow business-logic functions embedded in validated templates, compiled AI trades runtime flexibility for predictability, auditability, cost efficiency, and reduced security exposure. We introduce (i) a system architecture for constrained LLM-based code generation, (ii) a four-stage generation-and-validation pipeline that converts probabilistic model output into production-ready code artifacts, and (iii) an evaluation framework measuring operational metrics including token amortization, determinism, reliability, security, and cost. We evaluate on two task types: function-calling (BFCL, n=400) and document intelligence (DocILE, n=5,680 invoices). On function-calling, compiled AI achieves 96% task completion with zero execution tokens, breaking even with runtime inference at approximately 17 transactions and reducing token consumption by 57x at 1,000 transactions. On document intelligence, our Code Factory variant matches Direct LLM on key field extraction (KILE: 80.0%) while achieving the highest line item recognition accuracy (LIR: 80.4%). Security evaluation across 135 test cases demonstrates 96.7% accuracy on prompt injection detection and 87.5% on static code safety analysis with zero false positives.
Figures & tables
Figure 1: Token consumption comparison across baselines (BFCL, n=400). Compiled AI incurs a one-time 9,600-token generation cost, then executes at zero marginal cost. Break-even vs. Direct LLM at n∗≈17 ; at 1,000 transactions, compiled AI reduces consumption by 57 × vs. Direct LLM and 84 × vs. AutoGen.
Figure 2: The code foundry architecture. Business intent (YAML) enters; validated Temporal activities emerge. Dashed red arrows show the regeneration loop on validation failure. The LLM runs once at generation time. Generated artifacts may include bounded LLM tool calls for steps requiring semantic understanding at runtime.
Method
GenTokens
Runtime/tx
Total@1K
BreakEven
CompRatio
Direct LLM
0
552
552,000
n/a
n/a
LangChain
0
740
740,000
n/a
0.75 ×
AutoGen
0
805
805,000
n/a
0.69 ×
Compiled AI
9,600
0
9,600
∼ 17 tx
∼ 57 ×
Table 1: Token consumption comparison (BFCL, n=400). Runtime tokens reported for the function-calling task; the Code Factory variant for document intelligence is reported in Table 3 .
Method
Inference
Infra
Total TCO
CostRatio
Direct LLM
$21,500
$500
$22,000
40 ×
LangChain
$28,900
$500
$29,400
53 ×
AutoGen
$31,400
$500
$31,900
57 ×
Compiled AI
$55
$500
$555
1 ×
Table 2: Total cost of ownership at 1M transactions/month (Claude Opus 4.5: 15/1Minput,75/1M output tokens).
Paradigm
Approach
KILE
LIR
Latency
LLM Calls
Compiled AI
Deterministic (Regex)
20.3%
59.7%
0.6 ms
None
Code Factory
80.0%
80.4%
2,695 ms †
Compiled
Runtime AI
Direct LLM
80.0%
74.5%
6,339 ms
Per-request
LangChain
80.0%
75.6%
6,207 ms
Per-request
AutoGen
77.8%
78.9%
13,742 ms
Per-request
† Median latency; mean 10,263 ms due to 10% retry outliers (max 183s). First compile: 57s.
Table 3: DocILE results. KILE: Key Information Line-level Extraction (exact-match on key fields, e.g. invoice number, date, total). LIR: Line Item Recognition (accuracy on structured line items). Code Factory makes compiled LLM calls; Deterministic uses pure regex.
Every LLM agent run re-derives its behavior token by token on a frontier model: brilliant, expensive, slow, and unbounded. We present Auto, a compiler that records live agent behavior, measures which parts are secretly deterministic, extracts them into verified programs or distilled specialists, and emits cognition binaries: WebAssembly artifacts whose manifests carry measured guarantees and whose declared capabilities are physically enforced by the sandbox. A tiered runtime executes compiled behavior behind conformally calibrated guards; guard trips deopt to the reference agent, and the captured trace recompiles back down, so nothing is figured out twice. We use "AGI compiler" in one narrow, testable sense: a system that autonomously converts novel experience into permanent, verified, near-free skill while measuring what it does not know. On AUTO-BENCH, a benchmark we introduce and pre-register, 87.1% of 560 recorded frontier-agent spans are witnessed-deterministic (three of the four censused task families measure 100.0%). On a 300-item stream with three scheduled distribution shifts, the closed loop compiles three artifact generations and drives marginal cost from 59 to 2 micro-dollars per item (6.4x end-to-end) at 96.9% parity on witnessed inputs with zero errors. The same stream also quantifies the failure modes: a loose guard silently mislabels 48.9% of compiled answers, and an unfaithful deopt reference causes the verification gate to refuse recompilation. Calibration and reference fidelity, not model capability, decide whether cheap stays correct. Code: https://github.com/RightNow-AI/auto
Enterprise compliance management requires rapid adaptation to evolving regulatory frameworks (e.g., DORA, AI RMF, FedRAMP) and tight remediation SLAs. Traditional static orchestrators often fail in hybrid cloud environments where event-driven assessments demand that automation code adapt to runtime context in seconds. This paper presents lessons learned from evaluating six large language models for AI-driven workflow generation in a production enterprise platform, benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration (2,784 runs total). Our initial pipeline used monolithic workflow generation, achieving 31.5-82.8% structural success rates (JSON schema validity and correct UI rendering), with most models struggling on complex JSON generation. We developed a redesigned piecewise pipeline that decomposes workflow construction into variable scaffolding, base block assembly, and nested block generation, raising structural success to 74.1-97.8% across all models. We analyze production tradeoffs including cost (USD 0.008-0.20 per workflow), latency (under 50s for interactive use), and model selection. Piecewise decomposition enables smaller models (e.g., mistral-small at 95.7% structural success and USD 0.01 per workflow) to reach production viability, removing dependency on expensive frontier models. While mistral-medium-2505 and gpt-oss-120b achieved the highest structural success (96.1% and 97.8%), mistral-medium-2505 carries a 19x cost premium versus mistral-small. Our deployment lessons highlight the need to separate structural validity from semantic correctness (logical fulfillment of user intent) and provide a solution for model-agnostic, scalable automation in cloud engineering.
Xavier Wrenn, Radoslav Raykov, Aleksandar Angelov +3
AI coding assistants are now used to generate production code in security-sensitive domains, yet the exploitability of their outputs remains unquantified. We address this gap with Broken by Default: a formal verification study of 3,500 code artifacts generated by seven widely-deployed LLMs across 500 security-critical prompts (five CWE categories, 100 prompts each). Each artifact is subjected to the Z3 SMT solver via the COBALT analysis pipeline, producing mathematical satisfiability witnesses rather than pattern-based heuristics. Across all models, 55.8% of artifacts contain at least one COBALT-identified vulnerability; of these, 1,055 are formally proven via Z3 satisfiability witnesses. GPT-4o leads at 62.4% (grade F); Gemini 2.5 Flash performs best at 48.4% (grade D). No model achieves a grade better than D. Six of seven representative findings are confirmed with runtime crashes under GCC AddressSanitizer. Three auxiliary experiments show: (1) explicit security instructions reduce the mean rate by only 4 points; (2) six industry tools combined miss 97.8% of Z3-proven findings; and (3) models identify their own vulnerable outputs 78.7% of the time in review mode yet generate them at 55.8% by default.