A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs

Yifan Song

cs.CLAug 13, 2026

A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs

Authors: Yifan Song

Organizations: Taiyuan Institute of Technology

Abstract

Long-form generation exposes fundamental limitations of large language models. Even 70B-parameter models exhibit length collapse at 16k-token outputs, and multi-chapter stories frequently trigger the attribute drift characteristic of the lost-in-the-middle'' effect. The outline-first, write-later'' paradigm has gained wide adoption, yet existing research evaluates the final writing rather than the outline itself, conflating two evaluation objects that should be decoupled. We construct a unified head-to-head benchmark covering 7 representative long-form generation frameworks across 3 generation granularities -- single-chapter, multi-chapter, and whole-book -- and propose an anchor-based LLM-as-a-judge protocol that directly assesses outlines against the source text on a 5-point anchored scale. Across 21 framework-granularity cells, no single framework dominates; performance depends on the match between a framework's intrinsic output form and the target granularity. SuperWriter ranks first in the length-constrained single-chapter mode, but this advantage degrades in whole-book mode. The outline-side ranking correlates only moderately with the writing-side ranking, supporting the outline--writing decoupling principle. Compute constraints limit the writing-side evaluation to a subset of cases; follow-up experiments will expand the sample size and add cross-model evaluators to enable stronger statistical inference.

Figures & tables

Subset	Language	Samples	Chapters/Sample
Chinese Novels	Chinese	100	10
English Novels	English	100	10
Other Genres	Mixed	100	1
Total	—	300	—

Table 1: Subset composition and sample counts.

Framework	#LLM calls	Outline Form	Key Design
naive	1	—	No intermediate steps
cot	1	—	Single call with embedded CoT instruction
selfrefine	2	—	Fixed 2 rounds of self-feedback
CogWriter	2	Structured JSON	Two-stage block-level planning (constraint set $T$ not on execution path)
Dome	$\geq 1$	Five-act paragraphs	Campbell five-act + summary / legacy dual mode
StoryWriter	3–5	Event quintuples	Event generation + planning + reordering (writing agent not implemented)

Table 2: Comparison of the comparative frameworks: number of LLM calls and outline format.

#	Metric	Source	Cost	Definition
M1	Word Count	Character count	0 LLM calls	Word / character count
M2	$S_{l}$	StoryWriter + LongBench-Write	0 LLM calls	Length-compliance score
M3	Ent-2	Dome	0 LLM calls	2-gram word entropy
M4	Win Rate	SuperWriter	LLM $\times 2$ directions	Pairwise win rate
M5	HANNA 6 dims	StoryWriter	LLM	RE / CH / EM / SU / CR / CX
M6	WriteJudge 6 dims	SuperWriter	LLM	Rel / Coher / Clar / Creat / Spec / Tone

Table 3: The seven evaluation metrics used in the writing-side evaluation.

Framework	Key Deviation
CogWriter (A-1) / SuperWriter (A-2)	CogWriter’s constraint set is disabled at the configuration layer; SuperWriter’s refinement phase is disabled by default.
selfrefine / others	Fixed 2 rounds without specified review dimensions; CogWriter’s examples are removed, SuperWriter’s iteration count is reduced to 1, StoryWriter’s pseudo-multi-round cap is set to 10.

Table 4: Representative reproducibility divergences across frameworks.

Framework	A1 Cov.	A2 Faith.	A3 Cons.	A4 Rel.	B1 Gran.	B2 Hier.	C1 Exec.	Overall
naive	2.70	3.42	3.39	3.37	2.33	2.25	2.35	2.76
cot	3.14	3.43	3.68	3.62	2.52	2.67	2.49	2.95
selfrefine	3.02	3.55	3.79	3.75	2.38	2.23	2.85	3.03
CogWriter	3.60	4.09	4.26	3.90	3.44	2.86	3.52	3.65
StoryWriter	3.73	4.13	4.24	4.03	3.80	2.97	4.10	3.55
SuperWriter	4.78	4.43	4.71	4.91	4.74	3.46	4.92	4.36

Table 5: 7 frameworks on outline-side primary metrics (three-mode per-cell mean,

n=3

–

6

per mode). 1–5 integer scale; the overall score is the arithmetic mean of 8 dimensions. Bold indicates the best in the column. Per-mode data is in Appendix Appendix B Per-mode Per-dimension Mean Tables (B.1–B.4) .

Framework	single	multichap	wholebook
naive	2,419 $\pm$ 795	2,038 $\pm$ 531	10,630 $\pm$ 3,155
cot	2,833 $\pm$ 816	2,704 $\pm$ 1,011	16,379 $\pm$ 3,897
selfrefine	3,262 $\pm$ 903	2,021 $\pm$ 201	11,103 $\pm$ 4,244
CogWriter	3,346 $\pm$ 1,154	2,754 $\pm$ 764	12,449 $\pm$ 3,322
Dome	3,665 $\pm$ 262	2,182 $\pm$ 900	19,896 $\pm$ 5,396
StoryWriter	4,142 $\pm$ 1,679	3,140 $\pm$ 727	28,307 $\pm$ 10,844

Table 6: Output length (in characters) of the 7 frameworks by mode. Each cell reports mean

\pm

standard deviation; arranged in ascending order by the single column. Sorting of wholebook / multichap may differ from single.

Framework	HANNA RE	HANNA CH	HANNA EM	HANNA SU	HANNA CR	HANNA CX	HANNA Overall	WJ Overall
SuperWriter	4.9 $\pm$ 1.1	6.8 $\pm$ 1.2	7.3 $\pm$ 0.7	5.9 $\pm$ 1.7	7.3 $\pm$ 1.2	6.8 $\pm$ 1.5	6.50 $\pm$ 0.91	6.61 $\pm$ 1.00
naive	4.8 $\pm$ 1.6	6.7 $\pm$ 1.6	6.3 $\pm$ 2.2	5.4 $\pm$ 1.5	7.1 $\pm$ 1.5	6.1 $\pm$ 2.0	6.07 $\pm$ 1.59	6.70 $\pm$ 1.37
cot	3.9 $\pm$ 1.5	6.3 $\pm$ 1.6	6.8 $\pm$ 1.7	6.0 $\pm$ 1.9	6.7 $\pm$ 2.1	5.9 $\pm$ 2.1	5.93 $\pm$ 1.66	6.17 $\pm$ 1.32
StoryWriter	4.2 $\pm$ 1.6	6.8 $\pm$ 1.4	6.1 $\pm$ 2.0	5.3 $\pm$ 1.9	6.6 $\pm$ 1.9	5.4 $\pm$ 2.4	5.74 $\pm$ 1.63	6.19 $\pm$ 1.70
CogWriter	4.1 $\pm$ 1.4	6.2 $\pm$ 1.9	6.1 $\pm$ 1.4	5.3 $\pm$ 1.3	6.7 $\pm$ 1.9	5.7 $\pm$ 2.4	5.69 $\pm$ 1.46	6.00 $\pm$ 1.80
Dome	4.6 $\pm$ 1.4	6.4 $\pm$ 1.9	5.9 $\pm$ 2.3	5.1 $\pm$ 1.5	6.2 $\pm$ 2.0	5.2 $\pm$ 2.4	5.57 $\pm$ 1.77	6.08 $\pm$ 1.78

Table 7: Quality of the 7 frameworks on HANNA 6 dimensions and WriteJudge overall on the writing side. Each cell reports the mean

\pm

standard deviation across 3 modes

\times

3 samples (

n=9

). Bold indicates the best in each column. Per-mode means are in Appendix Appendix B Per-mode Per-dimension Mean Tables (B.1–B.4) , Table 10 .

	HANNA			WriteJudge
Framework	single	multichap	wholebook	single	multichap	wholebook
SuperWriter	6.83 $\pm$ 0.76	6.78 $\pm$ 1.23	5.89 $\pm$ 0.63	6.94 $\pm$ 0.63	6.33 $\pm$ 1.45	6.56 $\pm$ 1.08
StoryWriter	6.22 $\pm$ 1.23	5.50 $\pm$ 2.36	5.50 $\pm$ 1.74	6.61 $\pm$ 1.46	5.44 $\pm$ 2.36	6.50 $\pm$ 1.64
CogWriter	6.11 $\pm$ 1.46	5.72 $\pm$ 1.67	5.22 $\pm$ 1.73	5.78 $\pm$ 2.08	6.22 $\pm$ 2.22	6.00 $\pm$ 1.89
Dome	5.94 $\pm$ 1.42	5.00 $\pm$ 2.17	5.78 $\pm$ 2.26	6.58 $\pm$ 2.47	5.50 $\pm$ 2.08	6.33 $\pm$ 1.69
selfrefine	5.78 $\pm$ 2.56	5.72 $\pm$ 2.00	5.00 $\pm$ 2.33	6.56 $\pm$ 2.15	6.11 $\pm$ 2.37	5.50 $\pm$ 2.18

Table 8: Comparison of writing-side quality across modes on HANNA and WriteJudge overall for the 7 frameworks. Mean

\pm

standard deviation (

n=3

per cell). Arranged in descending order by the HANNA single column. SuperWriter ranks first on HANNA single, and second on WJ single (

0.23

behind naive , a directional pattern).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Framework	Mode	A1 Cov.	A2 Faith.	A3 Cons.	A4 Rel.	Overall	D1/M1
naive	single	7.2 $\pm$ 0.8	6.5 $\pm$ 1.1	6.8 $\pm$ 1.0	7.1 $\pm$ 0.9	6.7 $\pm$ 1.0	—
naive	multichap ∗	6.8 $\pm$ 0.9	6.2 $\pm$ 1.3	6.5 $\pm$ 1.2	6.9 $\pm$ 1.0	6.4 $\pm$ 1.1	M1=5.9 $\pm$ 1.4
naive	wholebook ∗	6.5 $\pm$ 1.0	6.0 $\pm$ 1.4	6.3 $\pm$ 1.3	6.7 $\pm$ 1.1	6.2 $\pm$ 1.2	D1=5.7 $\pm$ 1.5
cot	single	7.0 $\pm$ 0.9	6.3 $\pm$ 1.2	6.6 $\pm$ 1.1	6.9 $\pm$ 1.0	6.5 $\pm$ 1.1	—
cot	multichap ∗	6.7 $\pm$ 1.0	6.0 $\pm$ 1.4	6.4 $\pm$ 1.3	6.7 $\pm$ 1.1	6.3 $\pm$ 1.2	M1=5.8 $\pm$ 1.5
cot	wholebook ∗	6.4 $\pm$ 1.1	5.9 $\pm$ 1.5	6.2 $\pm$ 1.4	6.5 $\pm$ 1.2	6.1 $\pm$ 1.3	D1=5.6 $\pm$ 1.6

Appendix

Table 9: Outline-side primary-metric means (

n\approx 89

; A1–A4 + overall_quality + D1/M1; 1–10 scale). Cells marked with ∗ are affected by known defects in Section 3.1 — the English-side data of these cells may be degraded.

Framework	HANNA RE	HANNA CH	HANNA EM	HANNA SU	HANNA CR	HANNA CX	HANNA Overall	WJ Overall
SuperWriter	4.9	6.8	7.3	5.9	7.3	6.8	6.50	6.61
naive	4.8	6.7	6.3	5.4	7.1	6.1	6.07	6.70
cot	3.9	6.3	6.8	6.0	6.7	5.9	5.93	6.17
StoryWriter	4.2	6.8	6.1	5.3	6.6	5.4	5.74	6.19
CogWriter	4.1	6.2	6.1	5.3	6.7	5.7	5.69	6.00
Dome	4.6	6.4	5.9	5.1	6.2	5.2	5.57	6.08

Appendix

Table 10: Writing-side track means (same

n=6

subset; HANNA 6 dimensions + WriteJudge overall; 1–10 scale).

Framework	single	multichap	wholebook
naive	6	1	2
cot	2	1	2
selfrefine	1	1	2
CogWriter	1	2	3
Dome	1	2	1
StoryWriter	1	2	2

Appendix

Table 11: Dual-list ranking difference (HANNA overall vs. WriteJudge overall;

n\approx 89

). “Dual-list agreement” is operationally defined as ranking difference

\leq 1

; of the 21 cells, 12 cells show dual-list agreement, 9 cells show disagreement.

Dimension Pair	$\rho$	Interpretation
A2_faithfulness (outline) vs. HANNA RE (writing)	$0.45$	Weak-moderate positive correlation
A3_consistency (outline) vs. HANNA CH (writing)	$0.52$	Weak-moderate positive correlation
A4_relevance (outline) vs. WJ relevance (writing)	$0.58$	Weak-moderate positive correlation
overall_quality (outline) vs. WJ overall (writing)	$0.43$	Weak-moderate positive correlation
(EM, A2_faithfulness)	$-0.32$	Weak negative correlation (trade-off)
(SU, A2_faithfulness)	$-0.28$	Weak negative correlation (trade-off)

Appendix

Table 12: Outline vs. Writing Spearman

\rho

(

n\approx 89

; primary-evidence overall). All

\rho

interpretations should be treated with caution — self-preference bias is not mitigated; expansion fidelity is uncontrolled as an intermediate variable.

Sample ID	Language	Genre	Length Tier	Notes
novel_ch_idx15	Chinese	Novel	Medium ( $<5000$ chars)	Most stable output under single mode
novel_ch_idx41	Chinese	Novel	Long context ( $\geq 5000$ chars)	Long-context scenario
novel_en_idx4	English	Novel	Medium ( $<5000$ chars)	Representative English novel
novel_en_idx55	English	Novel	Medium ( $<5000$ chars)	Representative English novel
other_idx32	English	Other (academic / business)	Medium ( $<5000$ chars)	Cross-genre
other_idx76	Chinese	Other (academic / business)	Medium ( $<5000$ chars)	Cross-genre

Appendix

Table 13: Writing 6-sample properties.

Long-Form Document Generation

Long-form document generation studies methods for producing coherent, factual, well-structured books, reports, and other extended texts with AI.

See all 32 papers

Jun 8, 2026

IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking

Generating coherent and controllable long-form content remains a persistent challenge for Large Language Models (LLMs). While reasoning-enhanced models have demonstrated success in logic-intensive domains, our evaluation reveals that they suffer from a severe length collapse in open-ended writing, where performance degrades sharply as target lengths exceed 2,000 words. We attribute this failure to the limitation of static hierarchical planning, which struggles to provide dynamic guidance over extended contexts. To bridge this gap, we introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework. Unlike external agentic workflows, IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process, enabling continuous strategy adaptation and global alignment without additional assistance. Based on this framework, we construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B. Experiments demonstrate that IS-Writer-8B achieves state-of-the-art performance on challenging long-form benchmarks (e.g., +3.08 vs. DeepSeek-V3.2 on LongBench-Write), exhibiting robust length compliance and coherence competitive with significantly larger proprietary models.

Zechen Sun, Yuyang Sun, Zecheng Tang +6

Institute of Computer Science and Technology, Soochow University · 2Information Research Center of Military Science, PLA Academy of Military Science

Open-Ended Generation Long-Form Document Generation CoT Reasoning LLM Training Text Generation

Ask PDF

Jun 1, 2026

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

As large language models (LLMs) are increasingly used for long-form generation, reliably evaluating long-form outputs has become a critical challenge. LLM-as-a-judge offers a scalable alternative to human evaluation, yet its reliability in long-form output evaluation remains underexamined: existing meta-evaluation benchmarks focus mainly on short-form outputs. Compared with short-form evaluation, long-form evaluation is not merely a matter of output length; it often requires judges to make more complex document-level assessments of overall organization, task-relevant coverage and depth, cross-section consistency, and scenario-specific quality criteria. In this work, we introduce LongJudgeBench, a comprehensive benchmark for evaluating LLM judges on long-form outputs across diverse real-world scenarios and judging protocols. We systematically evaluate a broad range of LLM judges, covering multiple base models and judging settings. Our results reveal a substantial reliability gap: current LLM judges remain unstable across scenarios, and rubrics or references are helpful but not always sufficient. We hope LongJudgeBench will support future research on more robust, context-aware, and human-aligned LLM-as-a-judge methods. Our code is available at https://github.com/cjj826/LongJudgeBench.

Junjie Chen, Yuxi Dong, Haitao Li +5

Department of Computer Science and Technology, Tsinghua University · University of Science and Technology Beijing

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation

Ask PDF

Sep 28, 2026

Scaling Long-Form Story Generation via Narrative State Tracking

LLMs have demonstrated strong capabilities in creative writing. However, scaling them to full-length novels remains challenging, as maintaining narrative consistency becomes increasingly difficult. Existing story-generation methods typically focus on stories of up to about ten thousand words, leaving their ability to scale to full-length novels underexplored. In this work, we introduce Narrative State Tracking Agent (NstAgent), a training-free agentic framework that allows LLMs to track a structured narrative state including characters, past events and future requirements. We extend an existing benchmark to compare narrative consistency across lengths, and use it together with a writing-quality benchmark to systematically evaluate stories ranging from 10K to 100K words. We show that NstAgent achieves better narrative consistency and writing quality as stories grow longer, and neither of them degrades noticeably as length increases, suggesting that it provides an effective approach to scaling story generation toward full-length novels.

Zhennan Wan, Jianfei Chen

Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University

Long-Form Document Generation LLM Agents Text Generation Narrative Generation

Ask PDF

Abstract

Long-Form Document Generation

Explore similar work

IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

Scaling Long-Form Story Generation via Narrative State Tracking