Agent Behavior as Code: Efficient and Robust LLM Agents with Programmatic Specifications

Peng Qi; Chunliang Lyu; Gang Li; Fabian Chan; Cheng Chang; Ignacio Cases; Will Lu

cs.AIOct 4, 2026

Agent Behavior as Code: Efficient and Robust LLM Agents with Programmatic Specifications

Authors: Peng Qi, Chunliang Lyu, Gang Li, Fabian Chan, Cheng Chang, Ignacio Cases, Will Lu

Organizations: Uniphore · Orby AI

Abstract

AI agents based on foundation models (FMs) have demonstrated strong capabilities to perform complex open-ended tasks. However, they face some common challenges in practice: (a) agent behavior can deviate drastically even for semantically similar tasks, leading to catastrophically propagated errors; (b) high cost and latency due to FM calls, repeated in full whenever a task recurs with different inputs; (c) FMs' limited context and instruction following capability confine how well agents manage the ever-growing execution context and follow complex plans. We introduce $\textbf{A}$ gent $\textbf{B}$ ehavior as $\textbf{C}$ ode $\textbf{Agent}$ (ABCAgent), which uses a symbolic program (e.g., Python code with potential neural functions) to fully specify the agent's behavior at runtime, with a powerful FM agent editing that program for flexibility. Behavior is thus specified without premature variable binding, and its execution is deterministic. We evaluate ABCAgent on six agent benchmarks, two of which we construct to test how well a derived program generalizes to variants of the task it was written for. ABCAgent matches a model-matched neural agent on GAIA and augmented GAIA, and surpasses it where robustness and long control flows matter: 98.3% against 97.3% on GSM-Symbolic ( $p = 0.001$ ), 71.9% against 47.4% $\mathrm{Pass}^4$ on the telecom domain of $τ^2$ -bench ( $p = 0.0001$ ), and more records written correctly at every loop length on our control-flow-augmented WorkArena benchmark. For more parametric task families, ABCAgent is also significantly superior in efficiency. Without authoring a new program, ABCAgent solves 92.6% of GSM-Symbolic instances and 20.1% of augmented GAIA variants, which yields $5.2\times$ lower latency and $7.0\times$ lower cost on GSM-Symbolic, 19% lower cost on augmented GAIA, and $9.5\times$ lower agent latency on $τ^2$ -telecom.

Figures & tables

Panel of Figure 1 — Figure 1: How ABCAgent improves over traditional FM agents. Left: Traditional FM agents (e.g., ReAct ( Yao et al., 2023 ) ) re-invoke the FM at every action step, making reruns costly when task inputs change. Center: ABCAgent separates design (task $\to$ program) from execution (program $\to$ result). The program can contain neural functions (e.g., papers() in this example) for flexibility. On any new input, only the program is re-run, eliminating redundant FM calls, especially orchestration calls. Right: Error reduction against cost efficiency on six benchmarks relative to the neural agent.

Figure 2: ABCAgent processes new tasks by authoring programs from scratch, or reusing them from previously solved similar tasks. In the latter case, ABCAgent attempts to substitute parameter values into the existing program (with a lightweight FM call for parsing only when necessary), before resorting to editing the program if substitution fails.

System	GAIA		WorkArena		GSM-Symbolic	$\tau^{2}$ -bench ( $\mathrm{Pass}^{4}$ )
System	base (165)	augmented (378)	L1 (31)	CF (42)	(5000)	telecom (114)	airline (50)
Reference	74.6	—	42.7	—	94.9	29.0	42.0
Neural agent	76.4	59.5	84.9	58.3	97.3	47.4	58.0
ABCAgent	72.7	57.1	62.4	64.3	98.3	71.9	60.0
$p$ -value	0.377	0.281	0.041	0.453	0.0011	0.0001	1.000
Cost efficiency	0.90 $\times$	1.24 $\times$	0.41 $\times$	1.02 $\times$	6.97 $\times$	12.93 $\times$	5.66 $\times$

Table 1: Task success rate (%) and efficiency of ABCAgent relative to model- and harness-matched neural agent on six agent benchmarks. Bold marks a statistically significant difference (

p<0.01

) and italics the stronger system otherwise; cost efficiency and speedup are the neural agent’s cost and time divided by those of ABCAgent on the tasks where both systems produced a result, highlighted when the deviation is

\geq 20\%

. The reference row is the strongest published result we are aware of, with — where none exists (see Appendix B for details).

	Augmented GAIA L1 (129)			GSM-Symbolic (1000)			WorkArena-CF (18)
Setting	SR	Free	Cost eff.	SR	Free	Cost eff.	SR	Records	Cost eff.
Neural agent	62.0	0.0%	1.00 $\times$	99.7	0.0%	1.00 $\times$	76.5	53.3	1.00 $\times$
ABCAgent	58.9	21.7%	1.27 $\times$	99.1	96.0%	11.91 $\times$	83.3 §	63.9	1.04 $\times$
$-$ reuse & rebind	62.8	0.0%	1.02 $\times$	99.4	0.0%	0.96 $\times$	77.8	63.3	0.80 $\times$
$-$ variant check	56.6	0.0%	1.30 $\times$	99.8	0.0%	0.97 $\times$	68.8	0.0 ‡	1.26 $\times$

Table 2: Ablation of ABCAgent ’s components. We use the L1 subset of Augmented GAIA and 20% of GSM-Symbolic for this experiment. Free is the share of tasks solved without authoring a program, and Records is the percentage of records written correctly on the loop tasks (

N=30

). § On WorkArena-CF , cheap rebinding never took effect, so this setting behaves as family reuse with the variant check alone. ‡ Without the variant check, the replayed programs commit records with wrong field values, including one program that ignores its per-record input entirely (details in Appendix H ).

Panel of Figure 3 — Figure 3: Cost efficiency of ABCAgent relative to the neural agent under three forms of amortization: (a) reusing a program across the tasks of a family, (b) reusing a loop body, and (c) evaluating a policy repeatedly. The open squares in (a) are ABCAgent on GSM-Symbolic without rebinding.

Transfer setting	pairs	Reuse	Control	$p$
GSM-Symbolic
Forward ( $j>i$ )	7 640	0.958	0.842	$\mathbf{<10^{-4}}$
Backward ( $j<i$ )	7 360	0.963	0.826	$\mathbf{<10^{-4}}$
Augmented GAIA
Forward ( $j>i$ )	235	0.332	0.302	0.410
Backward ( $j<i$ )	239	0.469	0.305	3.5e-5

Table 3: Fraction of task pairs within a family on which a program stored for one task (indexed by

i

) solves the other (indexed by

j

), for unseen (forward) and earlier (backward) tasks, with and without program reuse. On GSM-Symbolic, parameters are rebound with one FM call (Appendix B ); bold marks

p<0.01

and italics the stronger result otherwise.

System	Behavior specification	Adaptation signal	$\mathrm{Pass}^{1}$
Neural agent	prompt only	none	0.603
Neural agent $+$ traces	prompt $+$ prior transcripts	task transcripts	0.671
ABCAgent	symbolic program	task transcripts	0.789
$+$ reward at authoring	symbolic program	transcripts $+$ reward	0.943
Control : neural agent, transcript injection disabled			0.638

Table 4: Prompt-based neural agents and ABCAgent adapting in a continual learning setting on

\tau^{2}

-telecom. Revealing the reward at authoring is an oracle ceiling.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	GAIA				Augmented GAIA
System	L1 (53)	L2 (86)	L3 (26)	all (165)	L1 (129)	L2 (204)	L3 (45)	all (378)
Reference (HAL)	82.1	72.7	65.4	74.6	—	—	—	—
Neural agent	86.8	75.6	57.7	76.4	62.0	58.3	57.8	59.5
ABCAgent	86.8	72.1	46.2	72.7	58.9	57.4	51.1	57.1
McNemar $p$ -value	1.000	0.648	0.453	0.377	0.481	0.851	0.508	0.281

Appendix

Table 5: Task success rate (%) on GAIA and augmented GAIA by difficulty level, with the number of tasks in parentheses. The reference row is the strongest generalist agent on the HAL leaderboard ( Kapoor et al., 2025 ) ,

p

is the exact McNemar test, and italics mark the strongest system in each column; no level differs significantly.

Task Type	Task Names
Loop tasks	CreateMultipleChangeRequestTask
	CreateMultipleHardwareAssetTask
	CreateMultipleIncidentTask
	CreateMultipleProblemTask
	CreateMultipleUserTask
	DeleteMultipleUserTask

Appendix

Table 6: Complex Task types and corresponding task names

Panel of Figure 4 — Figure 4: Accuracy and token consumption on WorkArena-CF as a function of loop length $N$ , with the legend in (a) applying to both panels. In (a), circles give the fraction of records written correctly and squares the task success rate, with the non-loop tasks at $N\!=\!1$ ; in (b), the dotted lines are least-squares power-law fits, both sublinear in $N$ , labeled with $95\%$ confidence intervals on their exponents.

	telecom ( $n\!=\!114$ )		airline ( $n\!=\!50$ )
System	$\mathrm{Pass}^{1}$	$\mathrm{Pass}^{4}$	$\mathrm{Pass}^{1}$	$\mathrm{Pass}^{4}$
Reference ( Barres et al., 2025 )	0.59	0.29	0.59	0.42
Stock agent	0.6075	0.4737	0.6850	0.5800
ABCAgent	0.8158	0.7193	0.6850	0.6000
paired $\Delta$	$\mathbf{+0.208}$ (46/16/52)		$+0.000$ (8/8/34)
Wilcoxon $p$	0.0001		0.958

Appendix

Table 7: Performance on the two

\tau^{2}

-bench domains, with the benchmark’s own stock agent as the model-matched neural baseline. The paired row gives the mean per-task difference, ABCAgent minus stock, with win, loss and tie counts; the reference row is the strongest model evaluated by Barres et al. (2025) under the same stock agent.

Augmented GAIA
Transfer setting	pairs	Reuse	Control	$p$
All pairs	572	0.388	0.292	2.2e-5
Forward ( $j>i$ )	235	0.332	0.302	0.410
Backward ( $j<i$ )	239	0.469	0.305	3.5e-5
Backward, target known-passing	145	0.703	0.462	5.1e-5
Backward, rewritten & known-passing	103	0.718	0.388	1.2e-6

Appendix

Table 8: Transfer of a stored program to other tasks of the same family, to tasks it has not seen (forward) and to tasks it was authored on earlier (backward). Bold marks differences from the no-reuse control that are significant at

p<0.01

, by McNemar’s test on augmented GAIA and by a sign test over the

20

families on GSM-Symbolic, where a program whose parameters cannot be bound to the target counts as a failure. The frozen-parent rows bind a child task either with one FM call (FM-assisted) or with the parameter values from the child’s own program, and the last block binds every pair with one FM call, on GSM-Symbolic over

100

families with the full configuration’s programs, tested with the Wilcoxon test over families (Appendix B ), and on augmented GAIA from each family’s first child solved in both arms to its siblings.

Figure 5: Instances of a GSM-Symbolic family that a program frozen after seeing

k

of its

50

instances solves among the other

49

, split into instances it has already seen and instances it has not. Each bar’s total is shown above it, and the dashed line marks all

49

Figure 6: Programs authored by ABCAgent , taken verbatim from run artifacts. In (a), all instances of a GSM-Symbolic family share one derivation, so a sibling is solved by rebinding the highlighted constants; in (b), rebinding fails on an augmented GAIA family, because a new Boggle board requires a new search rather than a new constant.

Setting	SR	Free	$/task	Cost eff.
Neural agent	99.7	0.0%	0.1362	1.00 $\times$
ABCAgent , no components	99.8	0.0%	0.1408	0.97 $\times$
$+$ family reuse	100.0	0.0%	0.2138	0.64 $\times$
$+$ variant check	99.7	0.0%	0.2155	0.63 $\times$
$+$ cheap rebind	99.1	96.0%	0.0114	11.91 $\times$

Appendix

Table 9: Component ablation on GSM-Symbolic over 1000 tasks from 20 templates, adding components cumulatively. Cheap rebinding is the only component that reduces cost, and accuracy is near ceiling for every setting.

	executor loop off	executor loop on
variant check off	64.4	0.0 ‡
variant check on	63.3	63.9 ⋆
neural agent	53.3

Appendix

Table 10: Records written correctly (%) on WorkArena-CF at

N\!=\!30

, crossing the variant check with the executor loop, without family reuse. ⋆ is the configuration reported as ABCAgent ; ‡ without the variant check, the replayed programs commit records with wrong field values, including one program that ignores its per-record input entirely and submits the same values on every record.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 681 papers

LLM-as-Code: Agentic Programming for Agent Harness
Jun 14, 2026Junjia Qi, Zichuan Fu, Jingtong Gao +4AI Agent Reliability LLM Agent Orchestration
City University of Hong Kong · Tencent Jarvis Lab
Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents
Sep 22, 2026Laizhen Li, Jiarui Li, Juanjuan Zhao +4Agent Harness Optimization LLM Agent Workflow Optimization
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Shenzhen University of Advanced Technology +1
JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent Harnesses LLM Agents
University of Illinois Urbana-Champaign · Princeton University · Google

Jun 14, 2026J/K move · Enter open · S save

LLM-as-Code: Agentic Programming for Agent Harness

Junjia Qi, Zichuan Fu, Jingtong Gao, Wenlin Zhang+3

City University of Hong Kong · Tencent Jarvis Lab

AI Agent Reliability LLM Agent Orchestration Long-Horizon LLM Agents LLM Agent Reliability

Open paper AskPDF arXiv

Every major LLM agent framework gives the LLM the role of orchestrator; the model decides what to do next, when to call tools, and when to stop. We argue that token explosion, control-flow hallucination, and unreliable completion are not implementation bugs but architectural consequences of assigning the deterministic work of looping, branching, and sequencing to a probabilistic system. A better prompt or a stronger model cannot guarantee the reliability of the LLM agent. We therefore propose Agentic Programming, in which the program governs all control flow, and the LLM is itself part of it, an adaptive component we call LLM-as-Code and invoke only where a task calls for reasoning or generation. Within each call the model keeps full flexibility, but it cannot alter the program's execution path. With control in the program, the LLM's context is built from the execution history's call tree and forms a directed acyclic graph (DAG). Each call's context length is then determined by its call depth rather than by accumulation over steps. A case study of computer-use agents shows that the design is practical, not just a theoretical stance, substantially improving the stability of long visual operation sequences.

Abstract

LLM Agents

Explore similar work

LLM-as-Code: Agentic Programming for Agent Harness

Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs