cs.CLJan 23, 2026
SaveRetell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Retelling
Organizations: University of New South Wales, Sydney, Australia
Abstract
Counterfactual storytelling exposes LLM shortcomings in causal reasoning, where few-shot prompting and SFT fail to produce logical and rational narratives. This case study introduces Retell, Reward, Repeat (RRR), a narrative theory-informed RL pipeline that teaches causal story structure. We extend an existing counterfactual stories dataset with human-annotated stages of narrative equilibrium, synthesising Structuralist Narratology with scalar narrativity. An aligned LLM-as-judge automates this framework to output reward signals, teaching the policy LLM to generate texts with causal narrative structure. Evaluations show that RRR-trained 7/8B LLMs outperform few-shot and SFT baselines in logic, rationality, and completeness, also aligning with a blind human preference subset. RRR offers a linguistically grounded, cost-effective post-training mechanism for LLM storytelling using a small, query-only dataset, showing the value of integrating linguistic theories into contemporary NLP.
Figures & tables
Figure 1: Todorov (1971) ’s narrative theory applied to Qin et al. (2019) ’s counterfactual storytelling task. Narrativity measures the presence of narrative structure.
Figure 2: The Retell Reward Repeat (RRR) post-training pipeline uses d-RLAIF ( Lee et al., 2024 ) with a narrative theory-informed LLM-as-judge to generate the reward signal for GRPO ( Shao et al., 2024 ) .
| Narrativity ( Equilibrium → Disruption → Recognition → Attempt → New Equilibrium ) |
| A: (4) Evan was ready to buy himself a bicycle. He went to the store but forgot his money. Then he looked around some more, trying to find something else he could afford or a way to get home. |
| B: (4) Evan was ready to buy himself a bicycle. He went to the store but forgot his money. Then he looked around some more, trying to find something else he could afford or a way to get home. |
| C: (3) Evan was ready to buy himself a bicycle. He went to the store but forgot his money. Then he looked around some more, trying to find something else he could afford or a way to get home. |
| Is the story logical? |
| A: (3) I interpret ‘but forgot his money’ as Evan forgetting a large sum of money intended for buying a bicycle. Evan may still have the usual money he carries every day, and is able to logically find something else he could afford. |
| B: (3) The edited text is logical. Forgetting his money and then looking for an alternative or a way home is internally consistent. |
Table 1: An example of 3 human annotators (A/B/C) applying our framework. More examples are in Appendix J .
| Annotator | Logical | Rational | Complete N | Narrativity |
| A | 0.31 | 0.53 | 0.46 | 0.43 |
| B | 0.24 | 0.37 | 0.38 | 0.53 |
| C | 0.17 | 0.46 | 0.37 | 0.49 |
| Average | 0.24 | 0.46 | 0.40 | 0.48 |
Table 2: Correlation (Spearman’s ) of each annotator vs the median of the other two ( ). All .
| (query) Original Story + Counterfactual | |
| Premise | Kimmy was on a trip to Paris. |
| Initial | Her plane had just landed and she and her classmates her [sic] excited. |
| Original Ending | They couldn’t wait to start seeing museums and churches! They spent lots of time shopping and seeing historical sites. Kimmy was very excited. |
| Counterfactual | Kimmy’s plane crashed over the ocean and killed everyone on board. |
Table 3: Example test outputs from models based on Llama-3.1-8B-Instruct. Ratings are given by LLM-as-judge (Gemini-3-Flash): L=Logical, R=Rational, C=Complete N , M=min LRC . Additional examples in Appendix K .
| Model | Logical | Rational | Complete N | Narrativity |
| Selene | 0.31 | 0.23 | 0.39 | |
| Prometheus | 0.24 | 0.29 | 0.33 | 0.32 |
| Gemini | 0.34 | 0.39 | 0.52 | 0.50 |
Table 4: Correlation (Spearman’s ) of LLM-as-Judge vs the median of 3 human annotators ( ). All except Selene Logical ( ).
Figure 3: Mean and standard deviation of the reward signal R N generated by Selene-1-mini during d-RLAIF.
| Model | Method | Logical | Rational | Complete N | min LRC | Narrativity | BLEU-4 | ROUGE-L |
| Ground Truths | Crowd Workers | 2.884 | 2.927 | 2.933 | 2.811 | 4.718 | - | - |
| Llama-3.1-8B-Inst. | BASE | 2.735 | 2.385 | 2.528 | 2.080 | 4.347 | 0.340 | 0.502 |
| few-shot | 2.768 | 2.615 | 2.505 | 2.213 | 4.303 | 0.306 | 0.473 | |
| SFT | 2.320 | 2.501 | 2.934 | 2.260 | 4.692 | 0.810 | 0.835 | |
| RRR O -Prometheus | 2.801 | 2.825 | 2.594 | 2.400 | 4.534 | 0.179 | 0.386 | |
| RRR N -Selene | 2.785 | 2.860 | 2.740 | 2.554 | 4.733 | 0.087 | 0.299 |
Table 5: Evaluation of post-trained LLMs using the test split (n=1871) from the TimeTravel dataset using Gemini-3-Flash. For each base LLM, the highest score in each column is bolded. Significance testing in Appendix I .
| Llama (SFT) | Llama (RRR N ) | |
| Narrativity | 4.820 | 4.580 |
| 2.220 | 2.480 | |
| Win-Rate | 28% | 72% |
Table 6: Human preference test on a subset of 100.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Rating | A (%) | B (%) | C (%) |
| Logical | Agree | |||
| Neutral | ||||
| Disagree | ||||
| Rational | Agree | |||
| Neutral | ||||
| Disagree |
Table 7: Distribution: 3 human annotators (A, B, C), percentages shown (200 per metric per annotator). Agree+ is Strongly Agree; Disagree- is Strongly Disagree.
| Criteria | AC2 | % Agree | |
| Logical | 0.7648 [0.71, 0.82] | 0.7967 | 0.1797 [0.0567, 0.2968] |
| Rational | 0.7082 [0.65, 0.76] | 0.7792 | 0.4068 [0.3024, 0.5035] |
| Complete N | 0.5611 [0.50, 0.62] | 0.6750 | 0.3180 [0.2322, 0.4129] |
| min LRC | 0.5544 [0.49, 0.61] | 0.6767 | 0.2981 [0.2116, 0.3851] |
| Narrativity | 0.7415 [0.69, 0.79] | 0.8660 | 0.3764 [0.2690, 0.4796] |
Table 8: Inter-annotator agreement measured by Gwet’s AC2, % Agreement, and Quadratic Weighted . Values in brackets represent the 95% Confidence Interval (CI).
| Annotator | Logical | Rational | Complete N | Overall | min LRC | Narrativity |
| Humans Only | 0.76/0.18 | 0.71/0.41 | 0.56/0.32 | 0.55/0.30 | 0.74/0.38 | |
| w/ Selene-1-mini | 0.80/0.09 | 0.71/0.27 | 0.58/0.25 | 0.49/0.20 | 0.54/0.20 | 0.73/0.31 |
| w/ M-Prometheus | 0.81/0.12 | 0.71/0.30 | 0.56/0.25 | 0.47/0.21 | 0.48/0.23 | 0.73/0.28 |
| w/ Gemini-3-Flash | 0.78/0.22 | 0.71/0.36 | 0.59/0.36 | 0.53/0.29 | 0.55/0.31 | 0.77/0.40 |
Table 9: Inter-annotator agreement, reported by AC2/ . “w/” indicates when an LLM-evaluator is added.
| Author | Logical | Rational | Complete N | min LRC | Narrativity |
| human | 2.98 / 2.54 / 2.96 | 2.72 / 2.66 / 2.52 | 2.82 / 2.72 / 2.28 | 2.58 / 2.30 / 1.96 | 4.70 / 4.62 / 3.72 |
| Olmo | 2.80 / 2.50 / 2.96 | 2.52 / 2.66 / 2.48 | 1.92 / 2.04 / 2.00 | 1.70 / 1.84 / 1.72 | 3.84 / 4.00 / 3.36 |
| Qwen | 2.64 / 2.40 / 2.92 | 2.36 / 2.52 / 2.36 | 2.50 / 2.54 / 2.12 | 2.12 / 2.12 / 1.68 | 4.22 / 4.32 / 3.50 |
| Llama | 2.70 / 2.36 / 2.80 | 2.02 / 2.16 / 2.04 | 1.54 / 1.92 / 1.52 | 1.24 / 1.60 / 1.24 | 3.50 / 3.68 / 2.84 |
Table 10: Human and LLM author comparison across evaluation dimensions with 3 human annotators (A / B / C).
| Model | Logical | Rational | Complete N | Overall | min LRC | Narrativity |
| human | 2.80 / 2.94 / 3.00 | 2.92 / 2.94 / 2.96 | 2.96 / 2.38 / 2.82 | 2.52 / 2.46 / 2.62 | 2.72 / 2.38 / 2.80 | 4.90 / 3.62 / 5.00 |
| Qwen | 2.60 / 3.00 / 2.96 | 2.80 / 3.00 / 3.00 | 2.36 / 2.54 / 2.86 | 2.40 / 2.74 / 2.76 | 2.04 / 2.54 / 2.82 | 4.78 / 3.42 / 4.88 |
| Olmo | 2.72 / 3.00 / 3.00 | 2.80 / 2.98 / 3.00 | 1.88 / 2.14 / 2.72 | 2.20 / 2.40 / 2.76 | 1.76 / 2.14 / 2.72 | 4.38 / 3.26 / 4.76 |
| Llama | 2.68 / 2.92 / 2.94 | 2.24 / 2.72 / 2.52 | 1.52 / 1.90 / 2.22 | 1.60 / 2.10 / 2.28 | 1.36 / 1.86 / 2.12 | 3.82 / 2.76 / 4.18 |
Table 11: Model comparison across evaluation dimensions with LLM judges (Gemini / Selene / Prometheus).
| Author | Logical | Rational | Complete N | min LRC | Narrativity | BLEU-4 | ROUGE-L |
| Llama-3.1-8B-Instruct | |||||||
| BASE | 2.735 | 2.385 | 2.528 | 2.080 | 4.347 | 0.340 | 0.502 |
| few-shot | 2.768 | 2.615 | 2.505 | 2.213 | 4.303 | 0.306 | 0.473 |
| SFT | 2.320 | 2.501 | 2.934 | 2.260 | 4.692 | 0.810 | 0.835 |
| GRPO ROUGE-L | 2.007 | 2.212 | 2.938 | 1.952 | 4.628 | 0.855 | 0.863 |
| RRR O -Prometheus | 2.801 | 2.825 | 2.594 | 2.400 | 4.534 | 0.179 | 0.386 |
Table 12: Evaluation of post-trained LLMs using the test split (n=1871) from TimeTravel dataset using Gemini-3-Flash. Additional models trained on Llama-3.1.
| Model | Method | Logical | Rational | Complete N | min LRC | Narrativity |
| Llama-3.1-8B-Inst. | BASE | 2.735* | 2.385* | 2.528* | 2.080* | 4.347* |
| few-shot | 2.768 | 2.615* | 2.505* | 2.213* | 4.303* | |
| SFT | 2.320* | 2.501* | 2.934 * | 2.260* | 4.692 | |
| RRR O -Prometheus | 2.801 | 2.825* | 2.594* | 2.400* | 4.534* | |
| RRR N -Selene | 2.785 Base | 2.860 Base | 2.740 Base | 2.554 Base | 4.733 Base |
Table 13: Evaluation of Llama-3.1-8B variants using the test split ( ) from the TimeTravel dataset. Values marked with * indicate a statistically significant difference ( via Wilcoxon signed-rank test) compared to the RRR N -Selene baseline. Bolded values represent the best performance for each metric.
| Model | Method | Logical | Rational | Complete N | min LRC | Narrativity |
| Qwen-3-8B | BASE | 2.429* | 2.132* | 2.759* | 1.940* | 4.551* |
| few-shot | 2.543* | 2.219* | 2.686 | 2.006* | 4.559* | |
| SFT | 2.402* | 2.532* | 2.925 * | 2.317* | 4.693* | |
| RRR O -Prometheus | 2.717* | 2.336* | 2.566* | 2.024* | 4.356* | |
| RRR N -Selene | 2.781 Base | 2.890 Base | 2.705 Base | 2.516 Base | 4.894 Base |
Table 14: Evaluation of Qwen-3-8B variants using the test split ( ) from the TimeTravel dataset. Values marked with * indicate a statistically significant difference ( via Wilcoxon signed-rank test) compared to the RRR N -Selene baseline. Bolded values represent the best performance for each metric within the model family.
| Model | Method | Logical | Rational | Complete N | min LRC | Narrativity |
| Olmo-3-7B-Inst. | BASE | 2.762 | 2.812* | 2.400* | 2.222* | 4.328* |
| few-shot | 2.745* | 2.837* | 2.377* | 2.208* | 4.415* | |
| SFT | 2.295* | 2.441* | 2.910 * | 2.211* | 4.647* | |
| RRR O -Prometheus | 2.770 | 2.857* | 2.491* | 2.325* | 4.460* | |
| RRR N -Selene | 2.793 Base | 2.893 Base | 2.595 Base | 2.434 Base | 4.899 Base |
Table 15: Evaluation of Olmo-3-7B variants using the test split ( ) from the TimeTravel dataset. Values marked with * indicate a statistically significant difference ( via Wilcoxon signed-rank test) compared to the RRR N -Selene baseline. Bolded values represent the best performance for each metric within the model family.
| Narrativity ( Equilibrium → Disruption → Recognition → Attempt → New Equilibrium ) |
| A: (4) My nephew had a bad back one day. He took some ibuprofen and was fine. The pills were never even opened, and his back felt better by the next day. |
| B: (4) My nephew had a bad back one day. He took some ibuprofen and was fine. The pills were never even opened, and his back felt better by the next day. |
| C: (4) My nephew had a bad back one day. He took some ibuprofen was fine. The pills were never even opened, and his back felt better by the next day. |
| Logical |
| A: (1) It’s illogical and inconceivable that the nephew took ibuprofen yet the pills were never opened. |
| B: (3) The edited text is logically coherent. Taking ibuprofen and leaving the prescription pills unopened is fully consistent. |
Table 16: An example of valid annotator disagreement when tagging and evaluating stories.
| Narrativity ( Equilibrium → Disruption → Recognition → Attempt → New Equilibrium ) |
| A: (5) I started using coupons. I realized it was too much work, with no real savings. I still collected coupons from the Sunday paper, emails and snail mail, but I didn’t put much effort into researching stores that double the value of coupons. In the end, I didn’t clip, tear, or cut any coupons, and I didn’t end up saving any money last week. |
| B: (5) I wanted to see if i could save some money so I started using coupons. I realized it was too much work, with no real savings. I still collected coupons from the Sunday paper, emails and snail mail, but I didn’t put much effort into researching stores that double the value of coupons. In the end, I didn’t clip, tear, or cut any coupons, and I didn’t end up saving any money last week. |
| C: (4) I wanted to see if i could save some money so I started using coupons. I realized it was too much work, with no real savings. I still collected coupons from the Sunday paper, emails and snail mail, but I didn’t put much effort into researching stores that double the value of coupons. In the end, I didn’t clip, tear, or cut any coupons, and I didn’t end up saving any money last week. |
| Logical |
| A: (1) It is illogical that the narrator would keep collecting coupons after realising that it was too much work with no real savings. |
| B: (3) The edited text is logically coherent. The speaker tries coupons, recognises the effort outweighs the savings, reduces that effort, and ends without saving money. |
Table 17: An example of valid annotator disagreement when tagging and evaluating stories.
| Narrativity ( Equilibrium → Disruption → Recognition → Attempt → New Equilibrium ) |
| A: (4) Evan was ready to buy himself a bicycle. He went to the store but forgot his money. Then he looked around some more, trying to find something else he could afford or a way to get home. |
| B: (4) Evan was ready to buy himself a bicycle. He went to the store but forgot his money. Then he looked around some more, trying to find something else he could afford or a way to get home. |
| C: (3) Evan was ready to buy himself a bicycle. He went to the store but forgot his money. Then he looked around some more, trying to find something else he could afford or a way to get home. |
| Logical |
| A: (3) I interpret ‘but forgot his money’ as Evan forgetting a large sum of money intended for buying a bicycle. Evan may still have the usual money he carries every day, and is able to logically find something else he could afford. |
| B: (3) The edited text is logical. Forgetting his money and then looking for an alternative or a way home is internally consistent. |
Table 18: An example of valid annotator disagreement when tagging and evaluating stories.
| Narrativity ( Equilibrium → Disruption → Recognition → Attempt → New Equilibrium ) |
| A: (1) I had to go to Boston for a work trip. It was a boring trip filled with conferences and no fun. We attended several conferences and had no memorable experiences, but I learned a lot for my career that I will carry with me. |
| B: (4) I had to go to Boston for a work trip. It was a boring trip filled with conferences and no fun. We attended several conferences and had no memorable experiences, but I learned a lot for my career that I will carry with me. |
| C: (4) I had to go to Boston for a work trip It was a boring trip filled with conferences and no fun. We attended several conferences and had no memorable experiences, but I learned a lot for my career that I will carry with me |
| Logical |
| A: (3) The scenario is conceivable and breaks no logical rules. “boring/unmemorable” and “learn a lot” are not mutually exclusive. |
| B: (1) The edited text contains a weak internal contradiction: the trip is described as having “no memorable experiences,” yet it also results in a significant and lasting professional benefit. Because this shift is not sufficiently explained, the narrative’s internal logic is weakened. |
Table 19: An example of valid annotator disagreement when tagging and evaluating stories.
| Narrativity ( Equilibrium → Disruption → Recognition → Attempt → New Equilibrium ) |
| A: (5) In the last mile of a marathon only two runners were in contention. One runner got cramps on the ground and tripped the other. They both rested on the ground. They rested for a few second [sic] before they drop out. |
| B: (5) In the last mile of a marathon only two runners were in contention. They both got cramps and had to drop out of the race. One runner got cramps on the ground and tripped the other. They both rested on the ground. They rested for a few second [sic] before they drop out. |
| C: (3) In the last mile of a marathon only two runners were in contention. They both got cramps and had to drop out of the race. One runner got cramps on the ground and tripped the other. They both rested on the ground. They rested for a few second [sic] before they drop out. |
| Logical |
| A: (3) Conceivable. The events are not narrated in chronological order. “They both got cramps and had to drop out” is a summary of the explanations that follow. |
| B: (1) The edited text is not logical. The claim that one runner “got cramps on the ground and tripped the other” is internally unclear and weakens the event sequence. |
Table 20: An example of valid annotator disagreement when tagging and evaluating stories.
| Source | Text | L | R | C | M | N |
| Original Story | ||||||
| Premise | Max and May were at a tavern. | |||||
| Initial | They saw the dartboard and challenged each other to a game. | |||||
| Original Ending | Max played very well. But May thru even better than him. May won the game. | |||||
| Counterfactual Retelling | ||||||
| Premise | Max and May were at a tavern. | |||||
Table 21: Example outputs from the test set. Ratings are given by LLM-as-judge (Gemini-3-Flash): L=Logical, R=Rational, C=Complete N , M=min LRC , N=Narrativity.
| Source | Text | L | R | C | M | N |
| Original Story | ||||||
| Premise | Kimmy was on a trip to Paris. | |||||
| Initial | Her plane had just landed and she and her classmates her [sic] excited. | |||||
| Original Ending | They couldn’t wait to start seeing museums and churches! They spent lots of time shopping and seeing historical sites. Kimmy was very excited. | |||||
| Counterfactual Retelling | ||||||
| Premise | Kimmy was on a trip to Paris. | |||||
Table 22: Example outputs from the test set. Ratings are given by LLM-as-judge (Gemini-3-Flash): L=Logical, R=Rational, C=Complete N , M=min LRC , N=Narrativity.
| Source | Text | L | R | C | M | N |
| Original Story | ||||||
| Premise | Amy’s class went on a field trip to see A Mid Summer Night’s Dream. | |||||
| Initial | They arrived at the theater and found their seats. | |||||
| Original Ending | Amy was in awe of the actors’ discipline and talent. After the play she was able to speak with the cast. Amy was thrilled. | |||||
| Counterfactual Retelling | ||||||
| Premise | Amy’s class went on a field trip to see A Mid Summer Night’s Dream. | |||||
Table 23: Example outputs from the test set. Ratings are given by LLM-as-judge (Gemini-3-Flash): L=Logical, R=Rational, C=Complete N , M=min LRC , N=Narrativity.