From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning

Seungdong Yoa; Sanghyu Yoon; Suhee Yoon; Dongmin Kim; Ye Seul Sim; Junhyun Lee; Woohyung Lim

cs.CLFeb 27, 2026

From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning

Authors: Seungdong Yoa, Sanghyu Yoon, Suhee Yoon, Dongmin Kim, Ye Seul Sim, Junhyun Lee, Woohyung Lim

Organizations: LG AI Research · Division of Computer Engineering, Hankuk University of Foreign Studies

Abstract

The evaluation of large language models (LLMs) has predominantly relied on static datasets, which offer limited scalability and fail to capture the evolving reasoning capabilities of recent models. To overcome these limitations, we propose an agent-centric benchmarking paradigm that moves beyond static datasets by introducing a dynamic protocol in which autonomous agents iteratively generate, validate, and solve problems. Within this protocol, a teacher agent generates candidate problems, an orchestrator agent rigorously verifies their validity and guards against adversarial attacks, and a student agent attempts to solve the validated problems. An invalid problem is revised by the teacher agent until it passes validation. If the student correctly solves the problem, the orchestrator prompts the teacher to generate more challenging variants. Consequently, the benchmark scales in difficulty automatically as more capable agents are substituted into any role, enabling progressive evaluation of large language models without manually curated datasets. Adopting text anomaly detection as our primary evaluation format, which demands cross-sentence logical inference and resists pattern-matching shortcuts, we demonstrate that this protocol systematically exposes corner-case reasoning errors that conventional benchmarks fail to reveal. We further advocate evaluating systems along several complementary axes including cross-model pairwise performance and progress between the initial and orchestrator-finalized problems. By shifting the focus from fixed datasets to dynamic protocols, our approach offers a sustainable direction for evaluating ever-evolving language models and introduces a research agenda centered on the co-evolution of agent-centric benchmarks.

Figures & tables

Figure 1: Comparison of text anomaly samples. Left : Existing benchmarks include obvious anomalies (e.g., complete off-topic from sports news to economy news) that are clear but too trivial. Right : ATAD examples introduce subtle shifts within context (e.g., benefits to ethics in healthcare AI), preserving clarity while presenting reasoning-intensive challenges. Our collaborative agents resolve the clarity-difficulty trade-off through iterative task refinement.

Figure 2: Illustration of the overall ATAD protocol. Three agents iteratively interact to generate progressively challenging benchmarks designed to uncover subtle reasoning weaknesses in LLMs.

Figure 3: Examples of the seven task types of text anomalies. With the exception of T2, each task requires identifying a guaranteed anomaly within the sample (e.g., by selecting a sentence or choice), rather than performing a simple binary classification.

Evaluation Model	T1	T2	T3	T4	T5	T6	T7	Avg.
GPT-3.5-Turbo OpenAI (2024a)	59.00	16.00	66.75	48.50	55.75	51.75	81.50	54.18
GPT-4o-mini OpenAI (2024c)	57.25	17.00	62.50	54.00	52.50	58.75	83.00	55.00
GPT-4o OpenAI (2024b)	62.00	21.25	68.25	53.25	49.25	56.75	81.00	55.96
GPT-o4-mini OpenAI (2024d)	63.25	30.25	68.50	53.00	47.25	57.25	80.00	57.07
Gemini-1.5-Flash Reid et al. (2024)	6.00	11.25	62.00	48.75	17.50	10.75	21.00	25.32
Gemini-2.0-Flash-Lite Google DeepMind (2024)	64.00	10.75	63.50	52.25	62.75	62.00	86.25	57.36

Table 1: Overall Performance of LLMs on our Text Anomaly Detection Benchmark. Average accuracy of each LLM, across the four datasets generated by four agent families (GPT, Gemini, Claude, LLaMA), is shown for each anomaly type (T1-T7) and overall.

Evaluation Model	GPT-4o		Gemini-2.0-Flash		Claude-3.5-Sonnet		LLaMA-3.3-70B
Evaluation Model	Base	Final	Base	Final	Base	Final	Base	Final
GPT-3.5-turbo	91.00	67.71	80.00	42.00	83.71	61.43	86.00	45.57
GPT-4o-mini	93.00	68.29	80.43	42.71	84.14	57.86	87.43	51.14
GPT-4o	94.29	72.43	83.29	44.71	87.29	62.71	89.29	44.00
GPT-o4-mini	91.86	72.43	83.57	47.14	87.29	61.86	87.71	46.86
Gemini-1.5-Flash	50.57	30.29	40.14	17.00	40.43	28.29	41.43	25.71

Table 2: Comparison of the LLMs’ performance on the initial (base) datasets, consisting of the base problems, and the final versions of the benchmark datasets. Each column represents a different dataset, generated by GPT-4o, Claude-3.5-Sonnet, Gemini-2.0-Flash, and LLaMA-3.3-70B, respectively. The observed performance drop from base to final problems highlights the effectiveness of ATAD in exposing the weaknesses of LLM reasoning.

Evaluation Model	Performance (%)		Problem Quality
	w/o Orch.	w/ Orch.	Validity (1–5)		Coherence (1–5)		Fairness (1–5)		Approval Rate (%)
	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.
GPT-4o	68.29	72.43	4.30	4.85	3.71	4.74	3.20	4.65	38.14	87.14
Gemini-2.0-Flash	65.00	71.86	5.00	5.00	4.97	5.00	4.93	4.94	99.00	100.00
Claude-3.5-Sonnet	65.00	72.86	4.61	4.92	4.11	4.69	3.41	4.42	55.57	90.43
LLaMA-3.3-70B	65.71	72.43	4.66	4.87	4.37	4.76	4.34	4.80	66.00	88.29

Table 3: Comparison of LLMs’ Performance and Problem Quality on the benchmark generated by GPT-4o agents. Problem quality is evaluated by each model acting as a reviewer, comparing benchmarks generated with and without the use of an Orchestrator.

Evaluation Model	GPT-4o
Evaluation Model	Base	Final
GPT-o3-mini	93.71	72.14
GPT-o4-mini	91.86	72.43
GPT-4o	94.29	72.43
GPT-4o-mini	93.00	68.29

Table 4: Simulated future scenario with GPT-o3/o4-mini (future) vs. GPT-4o/4o-mini (current), showing sustained relative evaluation.

Figure 4: Consistency in Benchmark Generation.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Performance of evaluation models on benchmarks generated by GPT-4o, Gemini-2.0-Flash, Claude-3.5-Sonnet, and LLaMA-3.3-70B.

Task ID	Task Name	Input	Output
T1	Sentence Context Anomaly	5–6 sentence paragraph	Index of off-topic sentence
T2	Paragraph Order Consistency	5-sentence paragraph	Boolean (True/False)
T3	Blank-based Choice Anomaly	Sentence with blank + 5 choices	Index of most inappropriate choice
T4	Bridge Sentence Evaluation	Two paragraphs + 5 bridge candidates	Index of incoherent bridge
T5	Referential Ambiguity	5-sentence paragraph	Index of ambiguous sentence
T6	Logical Contradiction	5-sentence paragraph	Index of logically inconsistent sentence

Appendix

Table 6: Input/output structure for each text anomaly detection task.

Task ID	Reasoning Type	Challenge Factors	Topics (Domains)
T1	Contextual reasoning	Minor topic shift, semantic deviation	Philosophy, society, psychology
T2	Discourse coherence	Sentence reordering	Science, economics, politics
T3	Lexical + pragmatic reasoning	Lexical fit, collocation	Literature, psychology, philosophy
T4	Logical bridging + topic shift detection	Weak logical connection, abrupt topic shift	Economics, society, policy
T5	Coreference resolution	Ambiguous pronouns, unclear referents	Psychology, literature, philosophy
T6	Causal and contradiction reasoning	Contradictory claims, causal reversal	Science, economics, politics

Appendix

Table 7: Reasoning types, challenge factors, and domain topics per anomaly detection task.

Figure 5: Refined after rejection. The left side shows a rejected T1 (Sentence Context Anomaly) problem where the anomaly was conceptually weak and difficult to identify. The Orchestrator’s feedback noted the lack of semantic inconsistency and suggested stronger topic divergence. The revised version (right) introduces a scientifically framed yet incorrect statement about Camus’s influences, resulting in a clearer and more pedagogically effective anomaly. This highlights the Orchestrator’s role in guiding high-difficulty problem construction.

Figure 6: A structurally valid T3 problem rejected by the Orchestrator due to an unclear anomaly. Despite adhering to the task format, the anomaly is too ambiguous—leading two out of three LLMs to answer incorrectly.

Generator Family	BiasIndex	Highest-Accuracy Model
GPT	-0.53	Claude-3.5-Sonnet
Gemini	-0.24	Claude-3.5-Sonnet
Claude	2.00	LLaMA-3.3-70B
LLaMA	-0.57	Gemini-2.0-Flash

Appendix

Table 8: BiasIndex across generator model families.

Generator (Teacher / Student / Orchestrator)	Easy	Hard	Extreme	Impossible
GPT-4o family (4o / 4o / 4o)	83.94	83.33	80.71	70.94
Gemini-2.0-Flash family (2.0F / 2.0F / 2.0F)	81.67	76.98	74.44	60.32
Cross-family (Claude-4 / Gemini-2-Flash / GPT-4o)	80.54	74.31	70.26	67.90

Appendix

Table 9: Average accuracy (%) across evaluation models as a function of difficulty tier. Each row corresponds to a benchmark generated under a different agent configuration. Difficulty tiers emerge along a single generation trajectory (Easy

\rightarrow

Hard

\rightarrow

Extreme

\rightarrow

Impossible).

LLMs	Same-family	Cross-family
gpt-3.5-turbo	54.18	51.71
gpt-4o-mini	55.00	53.86
gpt-4o	55.96	57.57
gemini-2.0-flash-lite	57.36	56.43
gemini-2.0-flash	58.93	54.57
claude-3-haiku	53.54	52.57

Appendix

Table 10: Comparison between same-family and cross-family agent configurations.

Criterion	DyVal	DARG	Benchmark Self-Evolving	ATAD (Ours)
Domain	Math, logic	Graph augmentation	Benchmark refresh	Text anomaly reasoning
Generation Architecture	Generator + Filter	Generator + Critic	Generator + Selector	Teacher $\rightarrow$ Orchestrator $\rightarrow$ Student
Role Separation	$\times$ Single loop	$\times$ Mostly 2-agent	$\times$ No 3-agent	✓ 3-agent separation
Difficulty Scaling	Static	No Student-conditioned loop	No difficulty conditioning	✓ Student-conditioned escalation
Validation Mechanism	Basic filtering	Code verifier	Quality scoring	✓ Strict Orchestrator validation

Appendix

Table 11: Conceptual comparison between ATAD and prior dynamic benchmark frameworks.

Language Model Reasoning Evaluation

See all 52 papers

Jun 4, 2026

Benchmark Everything Everywhere All at Once

Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.

Shiyun Xiong, Dongming Wu, Peiwen Sun +5

MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen +3

LLM Evaluation Benchmark Design Synthetic Benchmark Generation

Ask PDF

Jul 30, 2026

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.

Philipp D. Siedler, Jordan Sassoon

Aleph Alpha Research, Heidelberg, Germany.

LLM Evaluation Benchmark Auditing Benchmark Construction LLM Auditing

Ask PDF

May 7, 2026

Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

Evaluating large language models (LLMs) today rests on fixed benchmarks that apply the same set of items to any model, producing ceiling and floor effects that mask capability gaps. We argue that the most informative evaluation signal lies at the boundary, where the per-prompt pass probability is near

0.5

under random-sampling decoding strategy, and propose Dynamic Boundary Evaluation (DBE), which actively locates each model's boundary and places it on a globally comparable difficulty scale. DBE delivers three artifacts: (i) a calibrated item bank covering safety, capability, and truthfulness, with per-item difficulty labels validated across

9

reference LLMs; (ii) Skill-Guided Boundary Search (SGBS), a search algorithm that finds boundary items for a given target LLM using only API-level query access; and (iii) an evaluation protocol that places a new LLM on a unified ability scale and grows the evaluation set adaptively when the target falls outside the bank's coverage. We instantiate DBE on four categories spanning safety (harmful request refusal, over-refusal), capability (constrained instruction following), and truthfulness (multi-turn sycophancy resistance). The resulting evaluation covers a broader model spectrum without saturation while remaining compatible with existing datasets.

Haoxiang Wang, Da Yu, Huishuai Zhang

Peking University · Google Research · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence

LLM Evaluation Language Model Generation Evaluation Language Model Safety Evaluation

Ask PDF

Abstract

Language Model Reasoning Evaluation

Explore similar work

Benchmark Everything Everywhere All at Once

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models