AgentBetta: Verification-Driven Adaptive Configuration of an AI Nano-Agent through Selective Expansion and Verified Contraction

Md. Ashraful Babu

cs.AISep 20, 2026

AgentBetta: Verification-Driven Adaptive Configuration of an AI Nano-Agent through Selective Expansion and Verified Contraction

Authors: Md. Ashraful Babu

Organizations: Department of Physical Sciences · Explainable Artificial Intelligence and Optimization Research (XAiOR)

Abstract

Large language model agents are typically deployed with predefined configurations, although the required model capability, context, tools, permissions, memory, and computational resources can vary substantially across tasks. This study develops and evaluates AgentBetta, an adaptive AI Nano-Agent framework that represents these factors as an executable configuration and updates them through verification-driven diagnosis, selective expansion, and verification-based counterfactual contraction. The evaluation distinguishes controlled mechanism validation from external agent comparisons. On the AB-ConfigBench benchmark, AgentBetta achieved 91.38% verified success while reducing median context allocation from 64,000 to 8,000 context characters and median tool exposure from five tools to zero compared with the fully provisioned configuration. The configuration-deficiency diagnosis achieved a macro-F1 score of 0.819 with precision of 1.000 across the evaluated dimensions, and selective expansion avoided unnecessary changes to unrelated configuration dimensions. Post-success contraction preserved verification outcomes in 56.41% of evaluated one-dimension contraction probes, indicating that some successful configurations contained removable capability under the tested conditions. External evaluations indicate that adaptive configuration can improve the balance between verified task completion and capability exposure; however, the results vary across benchmarks and agent families. In particular, the cross-family replication did not reproduce the primary-backbone accuracy ordering, and specialized systems remained advantageous for certain task domains. These results support interpreting AgentBetta as a configuration-adaptation mechanism that regulates capability allocation and inference expenditure rather than as a universal replacement for specialized agent architectures.

Figures & tables

Method	Primary adaptive object	Runtime feedback or adaptation	Principal distinction from the present study
ARC [ 8 ]	Workflow, tools, budget, prompts	Learns query-wise configurations from correctness and cost	Does not make post-failure configuration-dimension diagnosis and verified post-success contraction the central runtime loop.
E3 [ 13 ]	Execution scope	Verification failure triggers progressive scope expansion	Establishes lean execution and recovery, but primarily varies execution footprint rather than a general heterogeneous configuration.
AOrchestra [ 9 ]	Instruction, context, tools, model	Creates specialized sub-agents during execution and learns orchestration from trajectories	Establishes runtime model/context/tool specialization; permissions, resource bounds, and configuration contraction are outside its principal abstraction.
AgentFlow [ 11 ]	Planning and tool-use policy	Verifier signals and final outcomes support in-flow policy learning	Verification-driven learning is central, whereas the full executable configuration is not the primary optimization variable.
CHILL-Harness [ 19 ]	Harness workflow interventions	Counterfactual effect estimates authorize advantageous workflow changes	Establishes counterfactual harness control rather than explicit necessity testing over model, context, tools, permissions, memory, and resources.
AgentSlimming [ 7 ]	Multi-agent workflow nodes and node models	Candidate pruning/replacement is re-evaluated against an acceptance threshold	Establishes validated structural contraction, mainly at workflow/task level rather than as per-instance bidirectional configuration adaptation.

Table 1: Positioning of AgentBetta relative to closely related adaptive-agent methods. The entries describe each method’s primary optimization scope rather than every auxiliary capability of the corresponding implementation.

Figure 1: Overall AgentBetta system architecture. The adaptive configuration engine maintains the executable configuration while provider and tool registries expose only policy-admissible capabilities. Runtime validators determine whether execution is acceptable, and run records preserve verification outcomes, adaptation history, and empirical configuration evidence.

Symbol	Dimension	Operational interpretation
$M$	Model capability	Provider/model identity, capability tier, reasoning or modality support, and inference parameters that are under policy control.
$C$	Context	Model-visible task information, retrieved evidence, history, and admissible context budget.
$T$	Tools	Explicit set of callable tools or functions exposed to the agent for the current task.
$P$	Permissions	Authorized operations and resource scopes; permission expansion can request additional authority but cannot grant it autonomously.
$\mathrm{Mem}$	Memory	Working or persistent memory available to the task, together with retention and retrieval policy.
$R$	Resources	Token, compute, monetary, or other quantitative resource budgets not represented elsewhere.

Table 2: Operational dimensions of the AgentBetta configuration.

Figure 2: Verification-driven adaptive control flow in AgentBetta. A task is characterized and assigned an initial configuration before execution and runtime verification. Unsuccessful or indeterminate verification triggers configuration-deficiency diagnosis; only policy-admissible diagnosed dimensions may be expanded. Verified configurations may undergo controlled counterfactual contraction in replayable settings, after which accepted evidence is recorded for subsequent history-informed initialization.

Method	Verified success (%)	Median context	Median tools	Median adaptations
AgentBetta Full	91.38	8,000	0	1.0
Fixed Full	93.10	64,000	5	0.0
Wholesale Escalation	93.10	64,000	5	1.0
Fixed Lean	6.90	8,000	0	0.0
AgentBetta $-$ Diagnosis	15.52	8,000	0	0.0
AgentBetta $-$ Contraction	91.38	8,000	0	1.0

Table 3: Held-out AB-ConfigBench performance under the deterministic configuration-aware provider. The experiment isolates controller behavior and is not a general LLM comparison.

Dimension	Support	Precision	Recall	F1
Model	218	1.000	0.716	0.834
Context	161	1.000	0.752	0.858
Tools	322	1.000	0.354	0.523
Permissions	187	1.000	0.610	0.758
Memory	142	1.000	0.739	0.850
Resources	154	1.000	0.818	0.900

Table 4: Per-dimension configuration-deficiency diagnosis in the controlled campaign.

Figure 3: Distribution of true and predicted configuration deficiencies in the controlled campaign. The diagonal concentration is consistent with the high precision reported in Table 4 ; residual error is dominated by missed rather than spurious deficiencies.

Method	Initially failed	Recovery (%)	Mean adaptations per initially failed task	Median dimensions changed	Unrelated-change rate	Added context
AgentBetta Full	482	89.00	0.814	1	0.000	0
Scope-only Expansion	192	89.06	0.947	1	0.000	0
Wholesale Escalation	192	91.67	0.909	8	0.733	56,000

Table 5: Recovery after initial failure under selective and wholesale adaptation.

Figure 4: Median context, tool, and permission exposure for AgentBetta, scope-only expansion, and wholesale escalation. Wholesale escalation increases dimensions unrelated to the diagnosed deficiency, whereas the selective methods retain the lean exposure profile for dimensions that do not require expansion.

Dimension	Probes attempted	Probes passed	Acceptance (%)
Model	1,566	1,389	88.70
Context	357	234	65.55
Tools	156	78	50.00
Memory	15	0	0.00
Resources	103	0	0.00
Execution limits	103	0	0.00

Table 6: Counterfactual contraction outcomes by configuration dimension.

Figure 5: Acceptance rate of verification-gated counterfactual contraction by configuration dimension. A contraction is accepted only when re-execution under the reduced configuration remains verified.

Comparator	Benchmark	AgentBetta (%)	Comparator (%)	Difference (pp)	95% CI (pp)	Adj. $p$	Principal resource effect
ReAct	BFCL	87.35	83.94	+3.40	$[+0.09,+6.89]$	0.0103	tokens +16.9%; calls +20.2%; cost +18.1%
Reflexion	GAIA	27.19	31.58	$-4.39$	$[-15.35,+5.70]$	0.4531	tokens $-41.0$ %; calls $-31.1$ %; cost $-48.9$ %
AutoGen	BFCL subset	88.02	31.77	+56.25	$[+45.3,+66.7]$	$5.8\times 10^{-11}$	tokens $-66.4$ %; calls $-53.0$ %; cost $-70.5$ %
CrewAI	BFCL subset	88.02	13.54	+74.48	$[+65.1,+83.3]$	$8.9\times 10^{-15}$	tokens $-77.0$ %; calls $-67.7$ %; cost $-81.2$ %
SWE-agent	SWE-bench pilot	6.25	75.00	$-68.75$	$[-93.8,-37.5]$	0.0625	tokens $-79.4$ %; calls $-35.2$ %; cost $-76.4$ %
Fixed Full	Cross-family BFCL	31.25	31.25	0.00	$[0.0,0.0]$	–	context $-75.0$ %; tokens +134.9%

Table 7: Summary of external comparisons. Resource effects are AgentBetta relative to the comparator and are computed from task-level repetition aggregates. “Adj.

p

” denotes the Holm-adjusted significance level within the predeclared confirmatory family.

Method	Scoreable runs	Excluded runs	Tasks complete 3/3	Task-level success (%)	Pooled success (%)	Mean tokens/task	Mean calls/task	Mean cost/task (USD)
AgentBetta	550	23	178/191	87.35	88.00	1,075	1.20	0.000262
ReAct Matched	573	0	191/191	83.94	83.94	920	1.00	0.000222

Table 8: Repeated BFCL comparison with matched ReAct. Success and resource values are task-level repetition aggregates; pooled run-level success is included only as a descriptive quantity.

Figure 6: Task-level verified success for AgentBetta and matched ReAct on the repeated BFCL panel. AgentBetta’s incomplete repetition coverage is reported explicitly rather than imputed.

Figure 7: BFCL success and task-level inference expenditure for AgentBetta and matched ReAct. AgentBetta attains higher verified success in this panel but uses more tokens and model calls.

Figure 8: Verified success and inference overhead for AgentBetta and the evaluated AutoGen and CrewAI configurations on the complete 64-task BFCL framework subset.

Method	Scoreable runs	Excluded runs	Tasks complete 3/3	Task-level success (%)	Pooled success (%)	Mean tokens/task	Mean calls/task	Mean cost/task (USD)
AgentBetta	115	5	37/40	27.19	26.96	9,364	4.47	0.003189
Reflexion	65	55	11/40	31.58	38.46	15,874	6.48	0.006243

Table 9: GAIA comparison between AgentBetta and Reflexion. The repeated panel remains incomplete because frozen-provider recovery was unavailable; the result is therefore interpreted as exploratory.

Figure 9: Task-level GAIA success and resource expenditure for AgentBetta and Reflexion. The figure is descriptive because the repeated panel is incomplete, particularly for Reflexion.

Figure 10: Observed success and execution consumption in the eight-instance SWE-bench pilot. The specialized SWE-agent configuration achieved substantially higher task resolution, while AgentBetta consumed fewer inference resources.

Figure 11: Cross-family BFCL replication on the 16-task stratified subset. All four controllers reached the same verified success, so the main-backbone accuracy ordering did not replicate.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 661 papers

Oct 1, 2026

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

Agent evaluations increasingly go beyond a single success rate, reporting metrics such as cost, consistency, and robustness. Yet they typically treat the agent itself as fixed. In practice, an agent is a configurable system: users decide what to tell it, how long to let it run, and which model to use, and each of these choices can change how well and how consistently it performs. We study these choices on a new benchmark of four scientific tasks, where a coding agent must find and correctly operate a published specialist model. We investigate five parts of the agent's configuration: task information, reasoning, self-verification, time budget, and backbone model. We find substantial run-to-run variability, with approximately 54% of the outcome variance coming from repeating the same configuration rather than changing it. Across configurations, the information provided to the agent has the largest effect, exceeding both time budget and model size, while also reducing cost and improving calibration. Configuration choices also interact: additional time helps only when the agent has sufficient information or a capable enough model to use it. Finally, a trajectory-based taxonomy of agent behavior reveals that prompting an agent to verify its answer has little effect on its verification behavior, whereas providing a dedicated verification tool changes that behavior substantially. These results suggest that agents should be evaluated as configurable systems themselves, and that some desired behaviors are more effectively implemented in the system than requested through prompting. We release the benchmark and more than 18,000 agent trajectories.

Luis Wiedmann, Leander Girrbach, Cordelia Schmid +1

Technical University of Munich, MCML · Helmholtz Munich · Inria, École normale supérieure, CNRS, PSL Research University

AI Agent Reliability AI Agent Evaluation AI Agent Benchmarks Automated Agent Evaluation

Ask PDF

Jul 23, 2026

GuardianAgentBench: Where Agents Fail and How to Guard Them

As large language model agents increasingly operate autonomously with access to tools and external environments, ensuring their safe and reliable behavior becomes critical. We present GuardianAgentBench (GABench), a benchmark of 580 scenarios across six domains evaluated on three production-ready frameworks: LangChain, LlamaIndex, and Vectara. The benchmark incorporates rigorous multi-stage validation and five adversarial attack modes. Experiments with six state-of-the-art models reveal that even the strongest configuration achieves only 74.8% overall accuracy and expose two distinct failure regimes: stronger models under-call required tools, while weaker models mis-select and over-call tools. Performance degrades monotonically with both tool-set size and sequential turn depth, with long-horizon planning proving the steeper bottleneck. Our guardrail implementation consistently outperforms system-prompt-based defenses across all models, recovering 19.9% of failures at a false positive rate of just 0.5%. These results demonstrate that execution-time structural intervention improves safety without disrupting correct agent behavior.

Vishal Ishwar Naik, Chenyu Xu, Donna Dong +5

Vectara, Inc., Palo Alto, CA, USA · Iowa State University, Ames, IA, USA

AI Agent Reliability LLM Guardrails AI Agent Safety Runtime Enforcement for AI Agents AI Agent Safety Benchmarks Tool-Using Language Agents+1

Ask PDF

Apr 3, 2026

NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration

As agentic network management gains popularity, there is a critical need for evaluation frameworks that transcend static, one-shot testing. To address this, we introduce NetAgentBench, a dynamic benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization guaranteeing determinism, correctness, and bounded execution. This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors. Our empirical evaluation of four state-of-the-art LLM agents through diverse network configuration tasks reveals stark deficiencies: while agents can solve basic tasks, they suffer severe exploration meltdowns and coherence collapse during expert-level configurations. Ultimately, NetAgentBench demonstrates that systematically evaluating multi-turn behavioral stability is an indispensable step toward realizing trustworthy, fully autonomous networks.

Ahmed Twabi, Yepeng Ding, Tohru Kondo

Informatics & Data Science, Hiroshima University, Hiroshima, Japan

Finite-State Machine LLM Agent Evaluation AI Agent Benchmarks AI Agent Reliability

Ask PDF

Abstract

LLM Agents

Explore similar work

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

GuardianAgentBench: Where Agents Fail and How to Guard Them

NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration