BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

Mesut Toruk

cs.AIAug 27, 2026

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

Authors: Mesut Toruk

Organizations: Istanbul, Turkiye

Abstract

Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,057 deterministic, committed test tasks. Every task is verifier-checkable gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model's own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy-tool-call adherence, URL hallucination and source-match, and per-model token cost and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released.

Figures & tables

Agent	Category	Tools	$n$	Skill probed
ar_math	arithmetic	calc	154	multi-step computation
ar_multistep	arithmetic	lookup, calc	152	constant lookup + compute
sql_analytics	structured	list_tables, schema, run_sql	157	SQL analytics (joins, group-by)
sec_sqli	security	scan_sql	154	SQL-injection judgment
sec_context_poison	security	scan_context	154	prompt-injection judgment
sec_guard	security	security_scan	151	LLM security-risk gating

Table 1: The

13

agents of the BekchiAI-Benchmark, their category, committed test-set size

n

, the tools they call, and the skill each isolates.

Family	Tools	Backing
arithmetic	calc , lookup	evaluator + constant table
SQL	list_tables , schema , run_sql	in-memory SQLite (3 tables)
security	scan_sql , scan_context , security_scan	signature scanners (imperfect)
web / URL	websearch , fetch	offline index of real pages
business	find_customer , list_invoices , get_payment , fx_rate , account_discount	customer / finance store
failover	cache_get , mirror_get , authdb_get	tiered read with faults

Table 2: The shared tool library the agents call.

Family	Category	Qwen3.7-Max	gemma-4-31B	gemma4:26b	gpt-oss-120b
ar_math	arithmetic	0.90	0.86	0.91	0.69
ar_multistep	arithmetic	0.97	0.97	0.85	0.65
sql_analytics	structured	0.87	0.81	0.77	0.24
sec_sqli	security	0.99	0.99	0.95	0.94
sec_guard	security	0.99	0.96	0.93	0.85
sec_context_poison	security	0.92	0.91	0.84	0.82

Table 3: Test-set accuracy (success rate on

[0,1]

) per family across four models, computed from the committed test sets in data/eval (

2{,}057

tasks per model). Best per family in bold. The wide per-family spread—not an aggregate—is the benchmark’s ranking signal.

Category	$n$	Qwen3.7-Max	gemma-4-31B	gemma4:26b	gpt-oss-120b
arithmetic	306	0.93	0.91	0.88	0.67
structured	157	0.87	0.81	0.77	0.24
security	459	0.97	0.95	0.91	0.87
url-grounding	305	0.63	0.65	0.49	0.83
planning	332	0.97	0.90	0.96	0.57
orchestration	196	0.86	0.95	0.91	0.94

Table 4: The same accuracies collapsed to the seven task categories (each cell is the task-weighted mean over that category’s families;

n

is its total test tasks). This is the level the radar in Figure 1 plots; best per category in bold.

Figure 1: Zero-shot accuracy by task category for the four models—one axis per category (the task-weighted mean over its families of Table 4 ; radius

=

success rate on

[0,1]

). The polygons expose each model’s profile : gpt-oss-120b (red) caves in on the structured (

0.24

), planning (

0.57

), and tool-policy (

0.62

) categories yet reaches farthest on url-grounding (

0.83

), while the other three trace a broadly similar high envelope—a shape an overall score cannot convey.

Model	LLM calls	input (M)	output (M)	total (M)
Qwen3.7-Max	6,965	3.86	1.96	5.83
gemma-4-31B	7,715	8.32	1.80	10.12
gemma4:26b	7,258	4.61	0.82	5.43
gpt-oss-120b	7,158	6.56	2.25	8.81

Table 5: Operational cost of the same runs, measured by the platform: LLM calls and token usage per model over the full

2{,}057

-task suite (input + output tokens in millions). Read alongside Table 3 , this shows capability and cost from one instrumented stream: gemma-4-31B and gpt-oss-120b spend the most tokens without leading on accuracy, while gemma4:26b is the leanest.

Model behaviour	Platform signal (event)	Reveals
task correctness	outcome (success/mismatch)	accuracy, per family
model call	llm_call (input/output tokens, latency)	token usage and speed
tool use	tool_call , plan-vs-actual	tool competence, adherence
URL arguments	url_access (grounded/hallucinated)	URL-hallucination, source-match

Table 6: How each benchmark behavior maps onto BekchiAI-Platform signals. One instrumented run yields all four.

LLM Agent Skill Evaluation

See all 25 papers

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley
The Amazing Agent Race: Strong Tool Users, Weak Navigators
Apr 11, 2026Zae Myung Kim, Dongseok Lee, Jaehyung Kim +2LLM Agent Evaluation AI Agent Benchmarks
University of Minnesota Twin Cities · Yonsei University · Grammarly
AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering
Sep 28, 2026Chanhee Park, Jeongho Yoon, Sungbin Han +2Multi-Hop QA Tool-Augmented Language Model Agents
Korea University · Sookmyung Women’s University

Jul 6, 2026J/K move · Enter open · S save

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability Failure Attribution for LLM Agents

Open paper AskPDF arXiv

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Abstract

LLM Agent Skill Evaluation

Explore similar work

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

The Amazing Agent Race: Strong Tool Users, Weak Navigators

AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering