From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning

Seungdong Yoa; Sanghyu Yoon; Suhee Yoon; Dongmin Kim; Ye Seul Sim; Junhyun Lee; Woohyung Lim

cs.CLFeb 27, 2026

From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning

Authors: Seungdong Yoa, Sanghyu Yoon, Suhee Yoon, Dongmin Kim, Ye Seul Sim, Junhyun Lee, Woohyung Lim

Organizations: LG AI Research · Division of Computer Engineering, Hankuk University of Foreign Studies

Abstract

The evaluation of large language models (LLMs) has predominantly relied on static datasets, which offer limited scalability and fail to capture the evolving reasoning capabilities of recent models. To overcome these limitations, we propose an agent-centric benchmarking paradigm that moves beyond static datasets by introducing a dynamic protocol in which autonomous agents iteratively generate, validate, and solve problems. Within this protocol, a teacher agent generates candidate problems, an orchestrator agent rigorously verifies their validity and guards against adversarial attacks, and a student agent attempts to solve the validated problems. An invalid problem is revised by the teacher agent until it passes validation. If the student correctly solves the problem, the orchestrator prompts the teacher to generate more challenging variants. Consequently, the benchmark scales in difficulty automatically as more capable agents are substituted into any role, enabling progressive evaluation of large language models without manually curated datasets. Adopting text anomaly detection as our primary evaluation format, which demands cross-sentence logical inference and resists pattern-matching shortcuts, we demonstrate that this protocol systematically exposes corner-case reasoning errors that conventional benchmarks fail to reveal. We further advocate evaluating systems along several complementary axes including cross-model pairwise performance and progress between the initial and orchestrator-finalized problems. By shifting the focus from fixed datasets to dynamic protocols, our approach offers a sustainable direction for evaluating ever-evolving language models and introduces a research agenda centered on the co-evolution of agent-centric benchmarks.

Figures & tables

Figure 1: Comparison of text anomaly samples. Left : Existing benchmarks include obvious anomalies (e.g., complete off-topic from sports news to economy news) that are clear but too trivial. Right : ATAD examples introduce subtle shifts within context (e.g., benefits to ethics in healthcare AI), preserving clarity while presenting reasoning-intensive challenges. Our collaborative agents resolve the clarity-difficulty trade-off through iterative task refinement.

Figure 2: Illustration of the overall ATAD protocol. Three agents iteratively interact to generate progressively challenging benchmarks designed to uncover subtle reasoning weaknesses in LLMs.

Figure 3: Examples of the seven task types of text anomalies. With the exception of T2, each task requires identifying a guaranteed anomaly within the sample (e.g., by selecting a sentence or choice), rather than performing a simple binary classification.

Evaluation Model	T1	T2	T3	T4	T5	T6	T7	Avg.
GPT-3.5-Turbo OpenAI (2024a)	59.00	16.00	66.75	48.50	55.75	51.75	81.50	54.18
GPT-4o-mini OpenAI (2024c)	57.25	17.00	62.50	54.00	52.50	58.75	83.00	55.00
GPT-4o OpenAI (2024b)	62.00	21.25	68.25	53.25	49.25	56.75	81.00	55.96
GPT-o4-mini OpenAI (2024d)	63.25	30.25	68.50	53.00	47.25	57.25	80.00	57.07
Gemini-1.5-Flash Reid et al. (2024)	6.00	11.25	62.00	48.75	17.50	10.75	21.00	25.32
Gemini-2.0-Flash-Lite Google DeepMind (2024)	64.00	10.75	63.50	52.25	62.75	62.00	86.25	57.36

Table 1: Overall Performance of LLMs on our Text Anomaly Detection Benchmark. Average accuracy of each LLM, across the four datasets generated by four agent families (GPT, Gemini, Claude, LLaMA), is shown for each anomaly type (T1-T7) and overall.

Evaluation Model	GPT-4o		Gemini-2.0-Flash		Claude-3.5-Sonnet		LLaMA-3.3-70B
Evaluation Model	Base	Final	Base	Final	Base	Final	Base	Final
GPT-3.5-turbo	91.00	67.71	80.00	42.00	83.71	61.43	86.00	45.57
GPT-4o-mini	93.00	68.29	80.43	42.71	84.14	57.86	87.43	51.14
GPT-4o	94.29	72.43	83.29	44.71	87.29	62.71	89.29	44.00
GPT-o4-mini	91.86	72.43	83.57	47.14	87.29	61.86	87.71	46.86
Gemini-1.5-Flash	50.57	30.29	40.14	17.00	40.43	28.29	41.43	25.71

Table 2: Comparison of the LLMs’ performance on the initial (base) datasets, consisting of the base problems, and the final versions of the benchmark datasets. Each column represents a different dataset, generated by GPT-4o, Claude-3.5-Sonnet, Gemini-2.0-Flash, and LLaMA-3.3-70B, respectively. The observed performance drop from base to final problems highlights the effectiveness of ATAD in exposing the weaknesses of LLM reasoning.

Evaluation Model	Performance (%)		Problem Quality
	w/o Orch.	w/ Orch.	Validity (1–5)		Coherence (1–5)		Fairness (1–5)		Approval Rate (%)
	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.	w/o Orch.	w/ Orch.
GPT-4o	68.29	72.43	4.30	4.85	3.71	4.74	3.20	4.65	38.14	87.14
Gemini-2.0-Flash	65.00	71.86	5.00	5.00	4.97	5.00	4.93	4.94	99.00	100.00
Claude-3.5-Sonnet	65.00	72.86	4.61	4.92	4.11	4.69	3.41	4.42	55.57	90.43
LLaMA-3.3-70B	65.71	72.43	4.66	4.87	4.37	4.76	4.34	4.80	66.00	88.29

Table 3: Comparison of LLMs’ Performance and Problem Quality on the benchmark generated by GPT-4o agents. Problem quality is evaluated by each model acting as a reviewer, comparing benchmarks generated with and without the use of an Orchestrator.

Evaluation Model	GPT-4o
Evaluation Model	Base	Final
GPT-o3-mini	93.71	72.14
GPT-o4-mini	91.86	72.43
GPT-4o	94.29	72.43
GPT-4o-mini	93.00	68.29

Table 4: Simulated future scenario with GPT-o3/o4-mini (future) vs. GPT-4o/4o-mini (current), showing sustained relative evaluation.

Figure 4: Consistency in Benchmark Generation.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Performance of evaluation models on benchmarks generated by GPT-4o, Gemini-2.0-Flash, Claude-3.5-Sonnet, and LLaMA-3.3-70B.

Task ID	Task Name	Input	Output
T1	Sentence Context Anomaly	5–6 sentence paragraph	Index of off-topic sentence
T2	Paragraph Order Consistency	5-sentence paragraph	Boolean (True/False)
T3	Blank-based Choice Anomaly	Sentence with blank + 5 choices	Index of most inappropriate choice
T4	Bridge Sentence Evaluation	Two paragraphs + 5 bridge candidates	Index of incoherent bridge
T5	Referential Ambiguity	5-sentence paragraph	Index of ambiguous sentence
T6	Logical Contradiction	5-sentence paragraph	Index of logically inconsistent sentence

Appendix

Table 6: Input/output structure for each text anomaly detection task.

Task ID	Reasoning Type	Challenge Factors	Topics (Domains)
T1	Contextual reasoning	Minor topic shift, semantic deviation	Philosophy, society, psychology
T2	Discourse coherence	Sentence reordering	Science, economics, politics
T3	Lexical + pragmatic reasoning	Lexical fit, collocation	Literature, psychology, philosophy
T4	Logical bridging + topic shift detection	Weak logical connection, abrupt topic shift	Economics, society, policy
T5	Coreference resolution	Ambiguous pronouns, unclear referents	Psychology, literature, philosophy
T6	Causal and contradiction reasoning	Contradictory claims, causal reversal	Science, economics, politics

Appendix

Table 7: Reasoning types, challenge factors, and domain topics per anomaly detection task.

Figure 5: Refined after rejection. The left side shows a rejected T1 (Sentence Context Anomaly) problem where the anomaly was conceptually weak and difficult to identify. The Orchestrator’s feedback noted the lack of semantic inconsistency and suggested stronger topic divergence. The revised version (right) introduces a scientifically framed yet incorrect statement about Camus’s influences, resulting in a clearer and more pedagogically effective anomaly. This highlights the Orchestrator’s role in guiding high-difficulty problem construction.

Figure 6: A structurally valid T3 problem rejected by the Orchestrator due to an unclear anomaly. Despite adhering to the task format, the anomaly is too ambiguous—leading two out of three LLMs to answer incorrectly.

Generator Family	BiasIndex	Highest-Accuracy Model
GPT	-0.53	Claude-3.5-Sonnet
Gemini	-0.24	Claude-3.5-Sonnet
Claude	2.00	LLaMA-3.3-70B
LLaMA	-0.57	Gemini-2.0-Flash

Appendix

Table 8: BiasIndex across generator model families.

Generator (Teacher / Student / Orchestrator)	Easy	Hard	Extreme	Impossible
GPT-4o family (4o / 4o / 4o)	83.94	83.33	80.71	70.94
Gemini-2.0-Flash family (2.0F / 2.0F / 2.0F)	81.67	76.98	74.44	60.32
Cross-family (Claude-4 / Gemini-2-Flash / GPT-4o)	80.54	74.31	70.26	67.90

Appendix

Table 9: Average accuracy (%) across evaluation models as a function of difficulty tier. Each row corresponds to a benchmark generated under a different agent configuration. Difficulty tiers emerge along a single generation trajectory (Easy

\rightarrow

Hard

\rightarrow

Extreme

\rightarrow

Impossible).

LLMs	Same-family	Cross-family
gpt-3.5-turbo	54.18	51.71
gpt-4o-mini	55.00	53.86
gpt-4o	55.96	57.57
gemini-2.0-flash-lite	57.36	56.43
gemini-2.0-flash	58.93	54.57
claude-3-haiku	53.54	52.57

Appendix

Table 10: Comparison between same-family and cross-family agent configurations.

Criterion	DyVal	DARG	Benchmark Self-Evolving	ATAD (Ours)
Domain	Math, logic	Graph augmentation	Benchmark refresh	Text anomaly reasoning
Generation Architecture	Generator + Filter	Generator + Critic	Generator + Selector	Teacher $\rightarrow$ Orchestrator $\rightarrow$ Student
Role Separation	$\times$ Single loop	$\times$ Mostly 2-agent	$\times$ No 3-agent	✓ 3-agent separation
Difficulty Scaling	Static	No Student-conditioned loop	No difficulty conditioning	✓ Student-conditioned escalation
Validation Mechanism	Basic filtering	Code verifier	Quality scoring	✓ Strict Orchestrator validation

Appendix

Table 11: Conceptual comparison between ATAD and prior dynamic benchmark frameworks.

Language Model Reasoning Evaluation

See all 36 papers

Benchmark Everything Everywhere All at Once
Jun 4, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5LLM Evaluation Benchmark Design
MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen +3
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM Evaluation Benchmark Auditing
Aleph Alpha Research, Heidelberg, Germany.
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
May 7, 2026Haoxiang Wang, Da Yu, Huishuai ZhangLLM Evaluation Language Model Generation Evaluation
Peking University · Google Research · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence

Jun 4, 2026J/K move · Enter open · S save

Benchmark Everything Everywhere All at Once

Shiyun Xiong, Dongming Wu, Peiwen Sun, Yuang Ai+4

MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen+3

LLM Evaluation Benchmark Design Synthetic Benchmark Generation

Open paper AskPDF arXiv

Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Benchmark Everything Everywhere All at Once

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models