As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's α) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.
Figures & tables
Naturalness
→
Completeness Clarity
Quality
→
Interpretability Conciseness
Informativeness
→
Accuracy Relevance
Figure 1: Hierarchical structure of the proposed evaluation rubric.
Krippendorff’s α
Our Method
0.5774
Full finetuning without augmentation
0.4304
Early stopping without augmentation
0.4380
LoRA dropout ( p=0.1 )
0.4067
Zero-shot GPT-4o
0.1964
Zero-shot GPT-5-mini-2025-08-07
0.2462
Table 1: Annotator agreement results on SPS dataset.
Accuracy
Macro-F1
Our Method
0.8163
0.6380
Full SFT without Augmentation
0.7819
0.4967
Early Stopping without Augmentation
0.7917
0.5685
LoRA Dropout ( p=0.1 )
0.7937
0.4998
Zero-shot GPT-4o
0.4741
0.3732
Zero-shot GPT-5-mini-2025-08-07
0.4732
0.3875
Table 2: Classification performance of proposed approach and baselines on the GoEmotions dataset. Classification performance is reported in place of inter-annotator agreement as the dataset labels are taken as ground truth.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Quantization
4-bit
LoRA Rank ( r )
16
Precision
bfloat16
Epochs
5
Batch Size
32
Gradient Accumulation
1
Appendix
Table 3: Hyperparameter settings for model fine-tuning.
Figure 2: Plot of Training and Validation loss without using Data Augmentation
Figure 3: Plot of Training and Validation loss using Proposed Data Augmentation
Figure 4: Plot of Training and Validation loss using LoRA Dropout