cs.CLNov 2, 2026

Distilling Literary Sentiment from LLM Annotations into an Ordinal BERT Model

Authors: Artem Suslov, Takehiro Hashimoto

Organizations: Hokkai-Gakuen University, Sapporo, Japan · Chuo University, Tokyo, Japan

Abstract

The scarcity of human-annotated literary corpora limits the development of domain-specific sentiment analysis models. This study investigates whether LLM-generated annotations can supervise smaller models for literary sentiment analysis. We fine-tune five BERT models using ordinal regression on 10,889 literary sentences annotated by Gemma 4 31B on a ten-point sentiment scale. The models are evaluated against 1,923 human-annotated sentences from Hemingway's "The Old Man and the Sea", alongside the teacher model and conventional baselines. BERT-large achieves a Kendall's ττ of 0.589, approaching the teacher's 0.604. Although a constant-neutral predictor achieves the lowest overall mean absolute error (MAE), BERT-base outperforms the teacher on non-neutral sentences, achieving an MAE of 0.726 versus 1.116. These findings demonstrate the potential of LLM-generated supervision for developing smaller literary sentiment models, while highlighting the limitations of aggregate error metrics on predominantly neutral datasets.

Explore similar work

CardsList
  1. Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Model

    Jun 28, 2026Sercan Karakaş, Yusuf ŞimşekSupervised Fine-TuningLLM Evaluation

  2. NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis

    Oct 13, 2026Gaurab Baral, Mimansak Nepal, Grishma BasnetSentiment AnalysisLLM Evaluation

  3. The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs

    Oct 25, 2026Michael Foster, Barbara McGillivray, Gard B. Jenset +7LLM EvaluationLanguage Model Post-Training