RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis

Andrew Zhuoer Feng; Cunxiang Wang; Yu Luo; Bosi Wen; Yidong Wang; Lin Fan; Yilin Zhou; Zikang Wang

cs.CLFeb 28, 2026

RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis

Authors: Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo, Bosi Wen, Yidong Wang, Lin Fan, Yilin Zhou, Zikang Wang,

Organizations: Tsinghua University · Peking University

Abstract

Large Language Models have evolved from single-round generators into long-horizon agents, capable of complex text synthesis scenarios. However, current evaluation frameworks lack the ability to assess the actual synthesis operations, such as outlining, drafting, and editing. Consequently, they fail to evaluate the actual and detailed capabilities of LLMs. To bridge this gap, we introduce RAVEL, an agentic framework that enables the LLM testers to autonomously plan and execute typical synthesis operations, including outlining, drafting, reviewing, and refining. Complementing this framework, we present C3EBench, a comprehensive benchmark comprising 1,258 samples derived from professional human writings. We utilize a "reverse-engineering" pipeline to isolate specific capabilities across four tasks: Cloze, Edit, Expand, and End-to-End. Through our analysis of 14 LLMs, we uncover that most LLMs struggle with tasks that demand contextual understanding under limited or under-specified instructions. By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability rather than raw generative capacity. Furthermore, we find that a strong reasoner can guide a weaker generator to yield higher-quality results, whereas the inverse does not hold. Our code and data are available at this link: https://github.com/ZhuoerFeng/RAVEL-Reasoning-Agents-Text-Eval.

Figures & tables

Figure 1: An Illustration of the evaluation mode gap.

Figure 2: RAVEL : an evaluation framework that integrates the testing LLM in agentic text synthesis. After receiving the writing instruction, the testing LLM will autonomously reason the synthesis actions to take, and carry out the actions to update its state.

Figure 3: Overview of the C3EBench task formulations.

Task	Lang.	Instruction			Input		Reference		Total
Task	Lang.	Avg.	Max.	Min.	Avg.	Max.	Avg.	Max.	Total
Cloze	CN	44.40	60	33	2037.44	4795	422.04	1458	200
Cloze	EN	19.00	19	19	951.89	5179	408.55	2412	150
Expand	CN	109.94	183	64	359.57	750	1779.80	3894	200
Expand	EN	246.72	387	141	239.23	342	1546.07	4863	99
Edit	CN	178.00	178	178	2434.84	4338	982.32	2159	200

Table 1: Statistics of the Chinese (CN) and English (EN) datasets. Lengths are measured in tokens/characters.

Model	C3EBench Tasks (Score $\uparrow$ )				Agentic Dynamics					Execution Efficiency
Model	Cloze	Expand	Edit	E2E	$\mathcal{S}$ %	$\eta_{\text{traj}}$ $\%\downarrow$	$\rho_{\text{ref}}$ %	$\delta_{\text{gain}}$	Judge	$\text{ERR}_{\pi_{\theta}}\%\downarrow$	$\|\mathcal{O}\|$	$\|\mathcal{T}\|$
Proprietary LLMs
GPT-5.2-2025-11-20 ( Singh et al., 2025 )	4.53	7.89	7.50	7.37	64.3	2.35	17.9	0.00	6.64	0.95	15.5	30.2
Gemini-3 Pro ( Gemini Team, Google, 2025 )	4.44	7.79	7.18	7.51	95.1	2.32	2.40	27.96	7.05	0.00	7.1	16.2
Claude-4.5 Sonnet ( Anthropic, 2025 )	4.44	7.79	7.42	7.52	68.1	2.95	100.5	19.55	6.68	0.00	10.3	28.0
Grok-4 ( xAI, 2025 )	4.42	7.76	7.12	7.39	69.0	2.50	38.2	3.52	6.31	0.03	7.4	18.4

Table 2: Main benchmarking results on the C3EBench and RAVEL . Metrics are grouped into Task Capabilities , Agentic Dynamics , and Overall Efficiency . Bold indicates top performance per category.

Figure 4: Analysis of Paragraph Lifecycles across LLMs. We plot the distribution of the initial drafting step ( Start Step ) versus the terminal completion step ( End Step ) for all generated sections. The difference of the medium (

\Delta_{\text{med}}

) is captioned in each figure. Some LLMs (GPT-5.2) exhibit a sequential drafting pattern with staggered start times. In contrast, efficiency-oriented LLMs ( Gemini, DeepSeek) demonstrate a “batch” approach , initializing all content early (

t\approx 0

) followed by global refinement.

Figure 5: Sensitivity analysis of reasoning vs. generation capabilities within RAVEL .

Setting	Cloze	Expand	Edit	End2End
Current	88.0	88.0	90.0	92.0
$-$ Rubric	76.0	78.0	74.0	72.0
$-$ Traits	78.0	82.0	78.0	76.0
$-$ Reference	72.0	64.0	64.0	56.0

Table 3: Ablation on LLM-as-a-judge setting.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Writing genre distribution and Taxonomy.

Figure 7: Correlations between the data input-instruction and reference.

	CN Writer	PW4ES	SeptES	ZJPub	Officials
1	0	153	20	0	2
2	12	351	134	3	15
3	1137	13188	10261	272	8705
4	4468	72908	3216	722	6957
5	19	861	73	86	204

Appendix

Table 4: Filter score from the coarse rubric scoring system implemented with Claude-4-5-sonnet.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis
Sep 29, 2026Yuan Li, Hanyun Jiang, Guowei Tian +2LLM Evaluation LLM-Based Program Synthesis
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · National University of Singapore
Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows
Jun 12, 2026Shikun Liu, Mufei Li, Dongqi Fu +5Multi-Agent LLM Systems KV-Cache Management
1Georgia Institute of Technology · 2Meta
LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
Mar 6, 2026Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2RAG Evaluation LLM Evaluation
neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

Sep 29, 2026J/K move · Enter open · S save

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

Yuan Li, Hanyun Jiang, Guowei Tian, Chengpeng Wang+1

The State Key Laboratory of Blockchain and Data Security, Zhejiang University · National University of Singapore

LLM Evaluation LLM-Based Program Synthesis Implicit Reasoning in Language Models

Open paper AskPDF arXiv

Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require users to state their intent as input-output examples. Large language models (LLMs) suggest a more natural route, text-to-Datalog synthesis from a natural-language question, yet how well they do so has not been systematically evaluated. We present DatalogBench, a benchmark of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts. Synthesized programs are graded by execution on held-out inputs against an oracle validated by mutation analysis. Across six LLMs and four prompting configurations, exact match peaks at 68.4%, and relation descriptions or an input-output example have only modest, model-dependent effects. Under direct prompting, most failures occur at compile time, typically because a model invents auxiliary predicates that it never declares or types consistently. Two coding agents reach up to 83.8% and eliminate nearly all such failures, leaving mostly semantic errors concentrated in recursive tasks. DatalogBench thus identifies recursive reasoning and decomposition as open challenges for current LLMs and agents, and offers a reliable, execution-grounded measure of both.

Abstract

Language Model Generation Evaluation

Explore similar work

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation