BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

Mesut Toruk

cs.AIAug 27, 2026

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

Authors: Mesut Toruk

Organizations: Istanbul, Turkiye

Abstract

Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,057 deterministic, committed test tasks. Every task is verifier-checkable gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model's own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy-tool-call adherence, URL hallucination and source-match, and per-model token cost and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released.

Figures & tables

Agent	Category	Tools	$n$	Skill probed
ar_math	arithmetic	calc	154	multi-step computation
ar_multistep	arithmetic	lookup, calc	152	constant lookup + compute
sql_analytics	structured	list_tables, schema, run_sql	157	SQL analytics (joins, group-by)
sec_sqli	security	scan_sql	154	SQL-injection judgment
sec_context_poison	security	scan_context	154	prompt-injection judgment
sec_guard	security	security_scan	151	LLM security-risk gating

Table 1: The

13

agents of the BekchiAI-Benchmark, their category, committed test-set size

n

, the tools they call, and the skill each isolates.

Family	Tools	Backing
arithmetic	calc , lookup	evaluator + constant table
SQL	list_tables , schema , run_sql	in-memory SQLite (3 tables)
security	scan_sql , scan_context , security_scan	signature scanners (imperfect)
web / URL	websearch , fetch	offline index of real pages
business	find_customer , list_invoices , get_payment , fx_rate , account_discount	customer / finance store
failover	cache_get , mirror_get , authdb_get	tiered read with faults

Table 2: The shared tool library the agents call.

Family	Category	Qwen3.7-Max	gemma-4-31B	gemma4:26b	gpt-oss-120b
ar_math	arithmetic	0.90	0.86	0.91	0.69
ar_multistep	arithmetic	0.97	0.97	0.85	0.65
sql_analytics	structured	0.87	0.81	0.77	0.24
sec_sqli	security	0.99	0.99	0.95	0.94
sec_guard	security	0.99	0.96	0.93	0.85
sec_context_poison	security	0.92	0.91	0.84	0.82

Table 3: Test-set accuracy (success rate on

[0,1]

) per family across four models, computed from the committed test sets in data/eval (

2{,}057

tasks per model). Best per family in bold. The wide per-family spread—not an aggregate—is the benchmark’s ranking signal.

Category	$n$	Qwen3.7-Max	gemma-4-31B	gemma4:26b	gpt-oss-120b
arithmetic	306	0.93	0.91	0.88	0.67
structured	157	0.87	0.81	0.77	0.24
security	459	0.97	0.95	0.91	0.87
url-grounding	305	0.63	0.65	0.49	0.83
planning	332	0.97	0.90	0.96	0.57
orchestration	196	0.86	0.95	0.91	0.94

Table 4: The same accuracies collapsed to the seven task categories (each cell is the task-weighted mean over that category’s families;

n

is its total test tasks). This is the level the radar in Figure 1 plots; best per category in bold.

Figure 1: Zero-shot accuracy by task category for the four models—one axis per category (the task-weighted mean over its families of Table 4 ; radius

=

success rate on

[0,1]

). The polygons expose each model’s profile : gpt-oss-120b (red) caves in on the structured (

0.24

), planning (

0.57

), and tool-policy (

0.62

) categories yet reaches farthest on url-grounding (

0.83

), while the other three trace a broadly similar high envelope—a shape an overall score cannot convey.

Model	LLM calls	input (M)	output (M)	total (M)
Qwen3.7-Max	6,965	3.86	1.96	5.83
gemma-4-31B	7,715	8.32	1.80	10.12
gemma4:26b	7,258	4.61	0.82	5.43
gpt-oss-120b	7,158	6.56	2.25	8.81

Table 5: Operational cost of the same runs, measured by the platform: LLM calls and token usage per model over the full

2{,}057

-task suite (input + output tokens in millions). Read alongside Table 3 , this shows capability and cost from one instrumented stream: gemma-4-31B and gpt-oss-120b spend the most tokens without leading on accuracy, while gemma4:26b is the leanest.

Model behaviour	Platform signal (event)	Reveals
task correctness	outcome (success/mismatch)	accuracy, per family
model call	llm_call (input/output tokens, latency)	token usage and speed
tool use	tool_call , plan-vs-actual	tool competence, adherence
URL arguments	url_access (grounded/hallucinated)	URL-hallucination, source-match

Table 6: How each benchmark behavior maps onto BekchiAI-Platform signals. One instrumented run yields all four.

LLM Agent Skill Evaluation

See all 25 papers

Jul 6, 2026

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability Failure Attribution for LLM Agents

Ask PDF

Apr 11, 2026

The Amazing Agent Race: Strong Tool Users, Weak Navigators

Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of 2 to 5 steps. We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph (DAG) puzzles (or "legs") with fork-merge tool chains. We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs). Agents must navigate Wikipedia, execute multi-step tool chains, and aggregate results into a verifiable answer. Legs are procedurally generated from Wikipedia seeds across four difficulty levels with live-API validation. Three complementary metrics (finish-line accuracy, pit-stop visit rate, and roadblock completion rate) separately diagnose navigation, tool-use, and arithmetic failures. Evaluating three agent frameworks on 1,400 legs, the best achieves only 37.2% accuracy. Navigation errors dominate (27 to 52% of trials) while tool-use errors remain below 17%, and agent architecture matters as much as model scale (Claude Code matches Codex CLI at 37% with 6x fewer tokens). The compositional structure of AAR reveals that agents fail not at calling tools but at navigating to the right pages, a blind spot invisible to linear benchmarks. The project page can be accessed at: https://minnesotanlp.github.io/the-amazing-agent-race

Zae Myung Kim, Dongseok Lee, Jaehyung Kim +2

University of Minnesota Twin Cities · Yonsei University · Grammarly

LLM Agent Evaluation AI Agent Benchmarks Tool-Using Agents LLM Agents

Ask PDF

Sep 28, 2026

AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering

Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources. Due to this complexity, agentic task failures arise from various sources, and pinpointing these failure causes is essential to diagnose and improve agentic systems. Existing benchmarks, however, tend to focus on a single leaderboard score, leaving the underlying failure modes opaque. To fill this gap, we introduce AgentHop, a diagnostic benchmark of 1,011 multiple-choice questions paired with a controlled seven-tool sandbox under fixed token, turn, and tool-call constraints. AgentHop reveals model vulnerabilities by dissecting a single accuracy score along four axes of agent operation: retrieval, synthesis, tool-call, and resource management. Across 19 models, we find that behavior clusters by model family, with tool-call signatures revealing distinct family fingerprints: GPT models commit early, Anthropic and GLM checkpoints verify before committing, DeepSeek and Kimi over-search, and Gemini-3 Pro stays balanced. Decomposed axes further expose within-family structure: Claude Opus 4.6 and Sonnet 4.6 land within one accuracy point yet diverge on retrieval-versus-synthesis emphasis, with Opus retrieving more and Sonnet synthesizing better. We release the full benchmark set and the harness to support diagnostic agent benchmarking.

Chanhee Park, Jeongho Yoon, Sungbin Han +2

Korea University · Sookmyung Women’s University

Multi-Hop QA Tool-Augmented Language Model Agents LLM Agent Evaluation Scientific QA AI Agent Benchmarks Tool-Using Language Agents

Ask PDF

Abstract

LLM Agent Skill Evaluation

Explore similar work

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

The Amazing Agent Race: Strong Tool Users, Weak Navigators

AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering