PlotTwist: A Creative Plot Generation Framework with Small Language Models

Abhinav Thorat; Ravi Kolla; Jyotin Goel; Madhav Kataria; Niranjan Pedanekar

cs.CLMar 17, 2026

PlotTwist: A Creative Plot Generation Framework with Small Language Models

Authors: Abhinav Thorat, Ravi Kolla, Jyotin Goel, Madhav Kataria, Niranjan Pedanekar

Organizations: Sony Research India

Abstract

Creative plot generation presents a fundamental challenge for language models: transforming a concise premise into a coherent narrative that sustains global coherence, character development, pacing, tone consistency, and emotional progression. Although recent Large Language Models (LLMs) demonstrate strong fluency on general-purpose tasks, they require preference alignment to perform well on domain-specific tasks such as creative plot generation. However, conducting such alignment at the scale of frontier LLMs is computationally prohibitive, significantly limiting accessibility and practical deployment. To address this, we present PlotTwist, a structured framework that enables Small Language Models (SLMs) with $\leq$ 3B active parameters to generate high-quality, premise-conditioned plots competitive with frontier systems of vastly greater parameter scale. Our approach decomposes generation into three specialized components: (1) an Aspect Rating Reward Model, trained via a novel Positive-Negative prompting strategy; (2) a Mixture-of-Experts (MoE) plot generator aligned via Direct Preference Optimization (DPO); and (3) an Agentic Evaluation module using a cross-family jury for unbiased, independent post-hoc assessment. Extensive experiments demonstrate that PlotTwist consistently outperforms all baselines, including frontier models, across multiple Narrative Quality Dimensions (NQDs), achieving higher win rates against every baseline except the strongest, with which it remains competitive. Further validation confirms strong sensitivity to narrative quality, as the framework reliably distinguishes plots derived from critically acclaimed versus widely panned screenplays. Together, these results establish structured, preference-based alignment as a resource-efficient approach to high-quality creative plot generation. Project page: https://abhinavthorat.github.io/plottwist/

Figures & tables

Figure 2: The Cross-Family Jury - Agentic Evaluation Framework

Model	Character Development	Tone Consistency	Pacing	Narrative Coherence	Emotional Turning Points	Overall
Qwen-2.5-14B	$7.41\pm 0.59$	$7.53\pm 0.43$	$7.52\pm 0.31$	$7.69\pm 0.48$	$7.95\pm 0.39$	$7.62\pm 0.33$
Claude Sonnet 4	$\mathbf{8.30\pm 0.41}$	$8.23\pm 0.35$	$8.09\pm 0.24$	$8.49\pm 0.38$	$\mathbf{8.63\pm 0.24}$	$8.35\pm 0.24$
Gemini 2.0 Flash	$8.01\pm 0.41$	$7.98\pm 0.32$	$7.91\pm 0.19$	$8.27\pm 0.32$	$8.40\pm 0.29$	$8.11\pm 0.24$
GPT-4.1	$7.88\pm 0.52$	$8.14\pm 0.29$	$7.99\pm 0.19$	$8.37\pm 0.34$	$8.41\pm 0.28$	$8.16\pm 0.24$
Llama-3.3-70B	$7.11\pm 0.69$	$7.39\pm 0.55$	$6.94\pm 0.56$	$6.89\pm 0.75$	$7.56\pm 0.53$	$7.18\pm 0.48$
DeepSeek-R1-14B	$7.29\pm 0.66$	$7.67\pm 0.42$	$7.53\pm 0.31$	$7.61\pm 0.48$	$7.89\pm 0.41$	$7.60\pm 0.35$

Table 1: PlotTwist Performance Evaluation. Results are reported as mean

\pm

standard deviation over 160 test premises using the Cross-Family Jury Agentic Evaluation framework. The best and second-best results in each column are shown in bold and underlined, respectively.

Baseline	Mean	WR	$95\%$ CI	$p_{\text{Holm}}$
Claude Sonnet 4	8.35	$\mathbf{0.563}$	[0.488, 0.638]	$0.13$
Agents’ Room	8.25	$\mathbf{0.734}$	[0.663, 0.800]	$4.4\times 10^{-9}$
GPT-4.1	8.16	$\mathbf{0.906}$	[0.863, 0.950]	$2.0\times 10^{-27}$
Gemini 2.0 Flash	8.11	$\mathbf{0.938}$	[0.900, 0.975]	$1.3\times 10^{-32}$
Qwen-3-32B	7.90	$\mathbf{0.987}$	[0.969, 1.000]	$2.1\times 10^{-43}$
Qwen-2.5-14B	7.62	$\mathbf{0.950}$	[0.913, 0.981]	$6.4\times 10^{-35}$

Table 2: Per-premise paired win rates of PlotTwist against each baseline under the Cross-Family Jury framework (ties counted as

0.5

), with premise-level bootstrap

95\%

CIs and Holm-corrected exact sign tests. Mean: the baseline’s overall mean from Table 1 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Aspect 1: Narrative Coherence

Positive Prompt (

r^{+}_{a}

)

You are a professional movie critic whose only output must be a single JSON object with exactly one integer field (0–10):

Narrative Coherence: Field Definition (Positive Focus):

Narrative clarity, logical plot progression, coherent world-building, strong cause–effect relationships, and well-integrated subplots.

Strict output rules:

1. Output only a valid JSON object. 2. Include only Narrative_Coherence . 3. Integer value from 0 to 10. 4. Score generously.

Appendix

Table 5

Figure 3: Dataset curation for performing DPO

System Prompt 1: Narrative Coherence Evaluation

Task Overview

Evaluate a movie plot’s narrative structure and logical consistency using a 10-criteria framework. Assign precise numerical scores reflecting coherence quality.

Evaluation Methodology

Each criterion is scored from 0–1 (increments of 0.1 allowed). Scores are summed for a total out of 10.

Scoring Criteria

Plot Structure and Logic (4 points)

Appendix

Table 7

System Prompt 3: Character Development Evaluation

Task Overview

Evaluate protagonist character development using a 10-criteria framework assessing motivation, arc progression, and narrative function.

Scoring Criteria

Core Character Elements (4 points)

1. Motivation Clarity 2. Behavioral Consistency Clear goals and desires. Actions align with personality. 3. Character Arc 4. Psychological Depth Believable transformation. Emotional and psychological complexity.

Character Foundation (3 points)

Appendix

Table 8

System Prompt 5: Tone Consistency Evaluation

Task Overview

Evaluate tonal coherence using a 10-criteria framework assessing atmosphere, stylistic unity, and emotional continuity.

Scoring Criteria

1. Initial Atmosphere Establishment 2. Scene-to-Scene Consistency 3. Tonal Relief Integration 4. Earned Tone Shifts 5. Dialogue Style Consistency 6. Visual Reinforcement 7. Stakes Alignment 8. Comedy/Drama Balance 9. Ending Consistency 10. Motif and Symbol Unity

Output Format

1. Initial Atmosphere Establishment: X.X 2. Scene-to-Scene Consistency: X.X 3. Tonal Relief Integration: X.X 4. Earned Tone Shifts: X.X 5. Dialogue Style Consistency: X.X 6. Visual Reinforcement: X.X 7. Stakes Alignment: X.X 8. Comedy/Drama Balance: X.X 9. Ending Consistency: X.X 10. Motif and Symbol Unity: X.X

Appendix

Table 9

System Prompt 6: Grounded Output Contract (Independent Scoring)

Applied in

Round independent scoring. Appended verbatim to each of the five rubrics above, so every juror returns one structured, evidence-grounded record per NQD.

Instruction (verbatim)

Return ONLY a single JSON object and nothing else, with the exact shape below. List every rubric criterion in order. Every evidence_quote must be copied verbatim from the plot text, not paraphrased. Do not add any text outside the JSON object.

Output schema

Appendix

Table 10

System Prompt 7: Deliberation Instruction (Juror Revision)

Applied in

Both deliberation rounds, on cells the independent panel contests (a spread of at least two points between juror scores). Appended to the same rubric; each juror sees its own prior assessment and the anonymized, order-shuffled arguments and cited spans of the other jurors, with peer scores hidden to prevent vote-matching.

Instruction (verbatim)

You already assessed this plot on the dimension defined by the rubric above. Below the plot you are shown (a) your own prior assessment and (b) the anonymized arguments of the other independent jurors, with the verbatim spans they cited. Reconsider your assessment in light of their EVIDENCE. Change your score only if a peer cites stronger grounded evidence than you did; do NOT change it to match a majority or because a peer sounds confident. If your original judgement still holds, keep your score and say why. Stay blind to anything outside the plot text (there is no premise, genre label, or reference answer).

Output schema

Appendix

Table 11

System Prompt 8: Consolidation Instruction (Judge LLM)

Applied in

Final consolidation by the held-out Judge LLM (Llama-3.3-70B) after the two deliberation rounds, once per contested NQD. The judge reads the plot, the anonymized independent assessments, and both rounds of revisions before issuing one consolidated score.

Instruction (verbatim)

You are the presiding judge. Below the plot you are shown the anonymized, grounded assessments of several independent jurors on the dimension defined by the rubric above, each with the verbatim spans it cited. Produce ONE final score for this dimension by WEIGHING the arguments on how well each is grounded in evidence actually present in the plot, not by averaging the jurors and not by counting how many agree. Discount any claim whose cited span does not support it or is absent from the plot. Name the main point of disagreement and how the evidence resolves it. Stay blind to anything outside the plot text.

Output schema

Appendix

Table 12

Plot Generation with PlotTwist

Task Description

The following example demonstrates long-form creative movie plot generation under the PlotTwist setting. The task emphasizes narrative coherence, concealed truths, emotional stakes, and morally consequential decision-making, serving as a qualitative illustration of model behavior.

Premise

Generate a movie plot that follows a dedicated detective as he investigates a murder that implicates his beloved, forcing him to navigate a web of concealed truths, blackmail, and the desperate lengths one will go to protect a dark secret.

PlotTwist Generated Plot

Detective Aris Thorne, a man whose sharp eyes and even sharper instincts have solved countless cases but whose heart remains scarred by the unsolved murder of his wife Elara five years prior, finds his world shattered when the body of City Councilman Silas Croft—a powerful figure known for his ruthless political maneuvering and rumored ties to the city’s underbelly—is discovered in his opulent penthouse with a single, precise stab wound to the heart. The evidence points damningly toward Lena Voss, Aris’s former lover and now a devoted social worker fighting for the city’s forgotten children. Witnesses report a heated argument between Lena and Croft hours before his death, and a fragmented security recording captures her leaving the building, her face pale with fear. Aris, torn between duty and lingering love, cannot believe she is capable of murder—yet the physical evidence is overwhelming. A rare antique switchblade, identical to one Aris gifted Lena years earlier, is found clutched in Croft’s hand. Even more troubling is a cryptic encrypted file on Croft’s laptop, labeled Project Chimera , implicating Lena in a scheme involving missing children. As Aris investigates further, the case reveals a labyrinth of concealed truths. Croft was not merely corrupt; he was the linchpin of an extensive blackmail network exploiting vulnerable families. The encrypted files reveal that Croft had been threatening Lena with exposure—not only of her past, but of evidence tied directly to Elara’s death. Aris had long believed Elara’s death was a tragic accident and had quietly suppressed details to protect her reputation. The blackmail was never solely about Lena—it was about Aris himself. Lena ultimately murdered Croft in desperation to stop him from exposing the truth, protect the children he was trafficking, and prevent Aris’s buried past from destroying them both. Aris finds himself ensnared between the city’s elite, an ambitious District Attorney eager to frame Lena, Croft’s remaining associates intent on silencing her, and his own conscience, which recognizes that Lena’s act was born of sacrifice rather than malice. Aris uncovers the final truth: Elara had been investigating Croft’s operations as a journalist and was killed in a staged car accident when she came too close. In a climactic confrontation at the rain-lashed docks, Aris dismantles the remnants of Croft’s network, but the true reckoning is internal. Choosing love over career, he confesses to the murder to protect Lena, sacrificing his badge and freedom. As rain washes over the city, Aris walks away from everything he once was, the weight of concealed truths finally lifted. What remains is the memory of Lena—not as a suspect, but as the woman he loved—and the understanding that truth and love, when intertwined, can demand a devastating cost.

Appendix

Table 13

PlotTwist Core Components

Qwen-3-30B-A3B (MoE) — PlotTwist Plot Generator

MoE backbone with 3B active parameters ; preference-aligned via Direct Preference Optimization (DPO) to generate high-quality premise-conditioned plots. Only 3B parameters are active per token despite a total count of 30B, classifying it as an SLM under the paper’s definition.

Qwen-3-32B (4-bit) — PlotTwist Aspect Rating Reward Model

Fine-tuned via regression-aware SFT using a weighted combination of cross-entropy loss and Huber loss to predict continuous aspect-level narrative quality scores across all five NQDs.

Cross-Family Jury — PlotTwist Agentic Evaluation Module

Independent post-hoc evaluation module operating separately from the training pipeline. A panel of five open-weight Juror LLMs drawn from five model families disjoint from the Qwen family of the generator and reward model: GPT-OSS-120B , Gemma-4-31B , Mistral-Medium-3.5-128B , Nemotron-3-Super-120B-A12B , and GLM-4.6V . Each juror independently scores all five NQDs using ten-criterion, weakness-focused rubrics with verbatim evidence grounding. Score disagreements of at least two points are resolved through two rounds of structured deliberation , consolidated by a held-out Judge LLM , Llama-3.3-70B , drawn from a sixth model family disjoint from both the panel and the generator.

Appendix

Table 14

Stratum	NQD	Orig.	Gen.	$\Delta$	95% CI	$d_{z}$	$P(\mathrm{gen}>\mathrm{orig})$	Welch $p$
Excellent	Narrative coherence	8.24	8.64	$+0.40$	$[0.13,\,0.72]$	0.41	0.70	0.012
( $n=40$ )	Emotional turning points	8.37	8.69	$+0.32$	$[0.02,\,0.72]$	0.28	0.55	0.088
	Character development	7.72	8.22	$+0.50$	$[0.20,\,0.88]$	0.44	0.75	0.021
	Pacing	8.18	8.16	$-0.02$	$[-0.21,\,0.19]$	$-0.04$	0.38	0.828
	Tone consistency	8.18	8.51	$+0.33$	$[0.17,\,0.50]$	0.62	0.75	$4.47\times 10^{-4}$
	Overall	8.14	8.45	$\mathbf{+0.31}$	$\mathbf{[0.09,\,0.57]}$	0.38	0.65	0.023

Appendix

Table 12: Per-stratum jury statistics for original and PlotTwist-generated plots.

\Delta

denotes the mean paired difference between generated and original plots, with bootstrap 95% confidence intervals. The statistic

d_{z}

is the paired Cohen’s effect size, and

P(\mathrm{gen}>\mathrm{orig})

is the proportion of films for which the generated plot receives a higher score than its paired original.

Juror	Family	Precision
GPT-OSS-120B	OpenAI (open-weight)	MXFP4
Gemma-4-31B	Google	BF16
Mistral-Medium-3.5-128B	Mistral	Q6_K
Nemotron-3-Super-120B-A12B	NVIDIA	FP8
GLM-4.6V	Zhipu	FP8

Appendix

Table 13: Jury composition. The five jurors span five model families, all disjoint from the Qwen family of the plot generator and of the single-judge development evaluator.

NQD	Krippendorff’s $\alpha$	Gwet’s AC2
Narrative coherence	0.504	0.889
Emotional turning points	0.280	0.827
Character development	0.405	0.863
Pacing	0.476	0.932
Tone consistency	0.366	0.911
Pooled	0.413	0.887

Appendix

Table 14: Inter-juror agreement per NQD (interval-level Krippendorff’s

\alpha

; ordinal-weighted Gwet’s AC2). The pooled AC2

95\%

CI is

[0.880,0.893]

Model	Execution	Usage in PlotTwist
PlotTwist Core Components — Local (4 $\times$ L40S)
Qwen-3-30B-A3B (MoE)	Local	DPO fine-tuning and inference for PlotTwist Plot Generator
Qwen-3-32B (4-bit)	Local	SFT training of Aspect Rating Reward Model
Qwen-3-32B (16-bit)	Local	Inference for Agentic Evaluation module
Ensemble Rating Models — Local (4 $\times$ L40S)
Qwen-2.5-7B	Local	Positive–negative aspect rating ensemble

Appendix

Table 18

Narrative Generation

See all 32 papers

StoryAlign: Evaluating and Training Reward Models for Story Generation
May 6, 2026Haotian Xia, Hao Peng, Yunjia Qi +4Reward Modeling Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University
Towards Human-Level Book-Writing Capability
May 16, 2026Jan Zierstek, Matteo Batelic, Maya Medjad +1Long-Form Document Generation LLM Fine-Tuning
Pageshift Entertainment
Planning Beyond Text: Graph-based Reasoning for Complex Narrative Generation
Apr 23, 2026Hanwen Gu, Chao Guo, Junle Wang +2LLM Planning Narrative Generation
Institute of Automation, Chinese Academy of Sciences · Tencent Turing Lab

May 6, 2026J/K move · Enter open · S save

StoryAlign: Evaluating and Training Reward Models for Story Generation

Haotian Xia, Hao Peng, Yunjia Qi, Xiaozhi Wang+3

Department of Computer Science and Technology, Tsinghua University

Reward Modeling Language Model Generation Evaluation Human Preference Modeling Text Generation Reward Model Evaluation Narrative Generation

Open paper AskPDF arXiv

Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly advanced text generation, stories generated by LLMs still diverge from human-authored works regarding complex narrative structure and human-aligned preferences. A key reason is the absence of effective modeling of human story preferences, which are inherently subjective and under-explored. In this work, we systematically evaluate the modeling of human story preferences and introduce StoryRMB, the first benchmark for assessing reward models on story preferences. StoryRMB contains

1,133

high-quality, human-verified instances, each consisting of a prompt, one chosen story, and three rejected stories. We find existing reward models struggle to select human-preferred stories, with the best model achieving only

66.3\%

accuracy. To address this limitation, we construct roughly

100,000

high-quality story preference pairs across diverse domains and develop StoryReward, an advanced reward model for story preference trained on this dataset. StoryReward achieves state-of-the-art (SoTA) performance on StoryRMB, outperforming much larger models. We also adopt StoryReward in downstream test-time scaling applications for best-of-n (BoN) story selection and find that it generally chooses stories better aligned with human preferences. We will release our dataset, model, and code to facilitate future research. Related code and data are available at https://github.com/THU-KEG/StoryReward.

Abstract

Narrative Generation

Explore similar work

StoryAlign: Evaluating and Training Reward Models for Story Generation

Towards Human-Level Book-Writing Capability

Planning Beyond Text: Graph-based Reasoning for Complex Narrative Generation