AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

Shreeyans Arora; Utsav Avaiya; Zera Lyngkhoi; Hem Gosalia; Ram Mohan Rao Kadiyala; Vinay Kumar Sankarapu; Pratinav Seth

cs.CLOct 15, 2026

AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

Authors: Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth

Organizations: Lexsi Labs

Abstract

Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows. We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics. Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.

Figures & tables

Panel of Figure 1 — Figure 1: Evaluation flow through the shared result object, including replay, harness, agent, and fingerprint paths. Solid, dotted, and dashed arrows denote standard execution, replay, and harness execution, respectively.

Family	Representative metrics	What it measures	#
Built-in	ExactMatch , QuasiExactMatch , Acc , AccNorm	Exact / normalized correctness against a target	4
Code / deterministic	Contains , Regex , IsJson , F1Score , Levenshtein , Equals , StartsWith , EndsWith , WordCount	Structural or exact string/JSON correctness	9
Generation quality	Bleu , RogueL (registry name rouge_l ), ChrF [ 32 ] , WordErrorRate , Perplexity , BertScore	N-gram overlap, fluency, LM confidence	6
Embedding similarity	CosineSimilarity , TokenOverlap , BM25Similarity	Semantic and lexical similarity	3
Retrieval-augmented generation (RAG)	LexicalGroundedness , ContextOverlap , ContextCoverage , AnswerOverlap	Groundedness and retrieval quality (lexical-overlap heuristic)	4
Hallucination	FactualConsistency	Groundedness of generated claims against source text	1

Table 3: Built-in metric families, representative metrics, and registry counts (registered implementations, not independently validated metrics).

Panel of Figure 2 — Figure 2: Episode-level agent evaluation separates verified outcomes from trace diagnostics. Dashed arrows mark caller-supplied artifact links.

Retriever	Hit@3	P@3	R@3	MRR	nDCG@3	QEM	F1
BM25	0.833	0.500	0.750	0.750	0.704	0.333	0.370
TF-IDF	0.833	0.389	0.583	0.750	0.602	0.333	0.333

Table 11: Retriever and answer metrics for HotpotQA (

N\!=\!6

k\!=\!3

, one shared generator).

Evidence	Granularity	Ids given	Result
AlignTune regression report	aggregate	artifact name, format	joined (artifact)
AlignTune audit report	aggregate	checkpoint digest	joined (caller)
CircuitKIT faithfulness report	aggregate	model digest	joined (caller)
SafeTune audit event	per event	episode id, call id	joined (caller)
SafeTune audit event	per event	decision only	refused: describes, does not identify
CuratorKIT manifest	aggregate	row id	refused: aggregate cannot pin one row

Table 12: Evidence-linking outcomes for a sample AgentTune episode (constructed placeholder artifacts).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark Auditing AI Agent Evaluation
Anthropic Fellows Program · Anthropic
BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
Aug 27, 2026Mesut TorukLLM Agent Skill Evaluation LLM Agent Evaluation
Istanbul, Turkiye
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
Aug 27, 2026Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian +1LLM-as-a-Judge LLM Agent Evaluation
ServiceNow AI Hyderabad, India

Feb 26, 2026J/K move · Enter open · S save

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

Abhay Sheshadri, Aidan Ewart, Elias Kempf, Kai Fronsdal+7

Anthropic Fellows Program · Anthropic

Benchmark Auditing AI Agent Evaluation Language Model Auditing

Open paper AskPDF arXiv

We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.

Abstract

LLM Evaluation

Explore similar work

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling