Large language models (LLMs) are increasingly used for text annotation, but their outputs are often reduced to a single categorical label before downstream analysis. This hardening step discards the model's relative support across categories, which may contain useful information when category boundaries are ambiguous. We study whether retaining the full probability vector improves downstream regression when LLM annotations are combined with a limited amount of human annotation. Using matched soft and hard representations from three LLMs across three social-science text datasets, we compare direct regression and human-assisted correction. Soft inputs are not uniformly better when used directly as outcomes, but after human correction, using the probability vector consistently improves regression efficiency across models and datasets. Analyses of repeated human judgments and controlled simulations further show that these gains arise when probability variation beyond the hard category carries useful information about the human-coded outcome. Our results identify output representation as an important design choice in LLM-assisted measurement and support preserving probability outputs for downstream statistical analysis. Code and reproducibility materials are available at https://anonymous.4open.science/r/soft-label-LLM-D201/
Figures & tables
Dataset
Corpus
Outcome / downstream contrasts
Primary human budget
BABE
3,575
Opinion; left/right vs. center (6 slopes)
500 judgments (100 × 5)
MHS
2,974
Hate speech; Twitter/YouTube vs. Reddit (6 slopes)
300 comments
CCC
4,271
Event type; year + region contrasts (24 slopes)
450 labels
Table 1: Empirical datasets, downstream contrasts, and primary human-label budgets.
Figure 1: Human vote alignment within fixed LLM hard predictions in BABE. Within each predicted category, the x-axis bins pmax and the y-axis gives the mean fraction of five human annotators selecting that category. Open markers denote bins with fewer than 20 texts.
Dataset
Model
Soft
Hard
BABE
Luna
0.157
0.294
Sonnet 5
0.130
0.259
Qwen3-32B
0.162
0.303
MHS
Luna
0.268
0.293
Sonnet 5
0.178
0.232
Qwen3-32B
0.280
0.332
Table 2: Mean Jensen–Shannon divergence between LLM representations and empirical human vote distributions. Lower values indicate closer alignment with the observed vote distribution.
Model
Uncorrected
PPI-style
DSL-style
Human
BABE
Luna
14.70 /19.57
9.63 /11.70
10.73 /11.18
12.35
Sonnet 5
14.08/ 13.16
9.15 /11.01
10.42 /10.82
Qwen3-32B
16.80/ 12.27
9.82 /12.40
11.30 /11.51
MHS
Luna
4.62 /4.86
5.78 /5.99
4.81 /4.86
5.23
Sonnet 5
5.16 /5.51
4.84 /5.67
4.66 /4.81
Qwen3-32B
3.84/ 3.26
5.62 /6.25
4.78 /4.81
Table 3: Aggregate coefficient RMSE (percentage points) at the primary budgets. Paired entries are soft/hard; Human is the same-budget human-only estimator. Bold marks the lower RMSE. Corrected and human-only results use 2,000 paired repetitions.
Coverage
Width
Dataset
Model
S
H
S
H
BABE
Luna
93.62
93.61
40.92
42.70
Sonnet 5
93.66
93.86
39.72
41.30
Qwen3-32B
93.39
93.33
43.29
44.00
MHS
Luna
95.30
95.11
18.59
18.72
Sonnet 5
95.22
95.23
17.98
18.51
Table 4: Mean pointwise coverage (%) and interval width (percentage points) at the primary budgets. S and H denote linear Soft+DSL and DSL. All results use 2,000 paired repetitions.
Figure 2: Annotation savings for Soft+DSL relative to DSL at matched aggregate coefficient RMSE. Positive values favor soft inputs; error bars are 95% Monte Carlo intervals. See Appendix D.4 for construction details.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Confusion matrices on BABE, with human plurality labels in rows and model predictions in columns. F: factual; M: mixed; W: writer opinion; Acc.: overall classification accuracy; Pred.: predicted category; n : number of texts in each cell; Row %: percentage within each human-category row.
Figure 4: Confusion matrices comparing LLM hard labels with human-reference labels for the same 2,974 MHS comments with valid predictions from all three models. Human labels are the highest-vote categories, with ties resolved uniformly at random once and fixed across models. LLM hard labels are the categories with the highest predicted probabilities. Rows correspond to human labels and columns to LLM predictions. N, U, and Y denote no hate speech, unclear, and hate speech, respectively; the corresponding human-label counts are 2,033, 187, and 754. Each cell reports its count ( n ) above its row-normalized percentage. Blue shading represents row percentages on a shared 0–100% scale. Acc. denotes overall classification accuracy.
Figure 5: Confusion matrices comparing LLM hard labels with human-reference labels for the same 4,271 CCC events with valid predictions from all three models. Human labels are the dataset’s original event categories. LLM hard labels are the categories with the highest predicted probabilities. Rows correspond to human labels and columns to LLM predictions. P, R, D, and M denote protest, rally, demonstration, and march, respectively; the corresponding human-label counts are 2,700, 1,010, 97, and 464. Each cell reports its count ( n ) above its row-normalized percentage. Blue shading represents row percentages on a shared 0–100% scale. Acc. denotes overall classification accuracy.
Figure 6: Human inter-annotator reliability by LLM probability concentration in BABE. Panels show Luna, Sonnet 5, and Qwen3-32B. Each point reports nominal Krippendorff’s α computed from human annotations within a probability-concentration group. Probability concentration is measured by 1−H(Qi)/logK , with higher values indicating more concentrated LLM predictions. Error bars indicate 95% confidence intervals; the dashed line marks α=0 .
Figure 7: Human inter-annotator reliability by LLM probability concentration in MHS. Panels show Luna, Sonnet 5, and Qwen3-32B. Each point reports nominal Krippendorff’s α computed from human annotations within a probability-concentration group. Probability concentration is measured by 1−H(Qi)/logK , with higher values indicating more concentrated LLM predictions. Error bars indicate 95% confidence intervals; the dashed line marks α=0 .
Votes
Reference agreement
Scenario
Uncorrected
PPI-style
DSL-style
Human only
1
60%
Random
2.337 / 5.269
3.904 / 4.758
3.739 / 3.749
3.750
1
60%
Covariate-dependent
16.544 / 28.638
3.744 / 4.554
3.695 / 3.726
3.750
1
80%
Random
0.699 / 14.649
3.688 / 4.472
3.688 / 3.716
3.750
1
80%
Covariate-dependent
7.414 / 18.551
3.664 / 4.386
3.665 / 3.698
3.750
1
80%
Class-information-only
1.094 / 14.649
3.860 / 4.472
3.716 / 3.716
3.750
5
60%
Random
8.812 / 2.508
8.370 / 10.566
8.208 / 8.291
8.236
Appendix
Table 5: Aggregate coefficient RMSE at 500 human judgments, in percentage points, pooled over six non-intercept slopes and 2,000 paired repetitions of 2,000 simulated observations each. Paired entries report soft / hard; DSL-style denotes linear Soft+DSL / DSL. One-vote and five-vote plurality outcomes yield 500 and 100 labeled observations, respectively, and define different population targets. Reference agreement is the calibration target for agreement between argmaxQ and argmaxP , not accuracy against realized human labels. The class-information-only rows use the hard labels from the 80% random-error scenario and replace Q by a deterministic soft recoding of H . Uncorrected estimates do not depend on the annotation budget.
Dataset
Model
Max. abs. bias (S/H)
Improved slopes
BABE
Luna
0.490 / 0.643
6/6
Sonnet 5
0.364 / 0.536
6/6
Qwen3-32B
0.514 / 0.423
6/6
MHS
Luna
0.175 / 0.166
4/6
Sonnet 5
0.078 / 0.089
6/6
Qwen3-32B
0.104 / 0.097
5/6
Appendix
Table 6: Additional coefficient diagnostics for linear Soft+DSL (S) and DSL (H) at the primary human-label budgets. Maximum absolute bias is the largest absolute repeat-mean coefficient bias, in percentage points. The final column counts coefficients with lower RMSE under Soft+DSL, based on unrounded results. All comparisons use 2,000 paired repetitions.
Luna
Sonnet 5
Qwen3-32B
nS
RMSE S/H
Saving [95% MC]
RMSE S/H
Saving [95% MC]
RMSE S/H
Saving [95% MC]
BABE: budget in human judgments; five judgments per sentence
250
17.3095/17.9502
8.65[6.55,10.55]
16.7222/17.3160
8.37[6.28,10.23]
18.0496/18.2089
2.32[0.39,4.05]
500
10.7247/11.1813
11.95[10.34,13.33]
10.4226/10.8168
10.81[8.87,12.71]
11.3027/11.5136
5.71[4.23,7.19]
1,250
5.9302/6.1347
–†
5.7561/5.9393
–†
6.2065/6.2934
–†
MHS: budget in annotated comments
Appendix
Table 7: Complete budget results for linear Soft+DSL and DSL. Each model block reports same-budget aggregate coefficient RMSE (soft / hard, in percentage points) and estimated annotation savings with 95% Monte Carlo intervals (in percent). Bold budgets mark the points displayed in Figure 2 .
Model
Correction
RMSE
Max. abs. bias
Mean coverage
Mean width
Luna
PPI-style
9.593 / 11.373
0.778 / 0.894
94.83 / 95.00
37.46 / 45.50
Linear DSL-style
10.651 / 11.016
0.898 / 0.648
93.33 / 93.58
41.22 / 43.05
Forest DSL-style
9.298 / 9.843
0.751 / 0.717
94.08 / 93.75
35.34 / 37.78
Sonnet 5
PPI-style
8.988 / 10.990
0.543 / 0.777
94.92 / 94.33
35.35 / 42.72
Linear DSL-style
10.411 / 10.802
0.698 / 0.878
93.25 / 93.42
40.06 / 41.72
Forest DSL-style
9.017 / 9.554
0.597 / 0.583
94.08 / 94.75
34.73 / 37.04
Appendix
Table 8: Matched BABE auxiliary-model comparison at 500 human judgments using 200 paired repetitions. Each paired entry is soft/hard. RMSE, maximum absolute bias, and mean width are in percentage points; mean coverage is in percent. Linear and forest DSL-style correction denote Soft+DSL/DSL with the corresponding auxiliary learner.
Figure 8: Concentration of the three models’ probability outputs on the same 2,974 MHS comments. Panel A shows the empirical cumulative distribution of pmax,i=maxkQik . Panel B shows the percentage of comments with pmax,i at least 0.90, 0.95, or 0.99.
Model
Original Soft+DSL
DSL
Log-ratio Soft+DSL
Reduction (%)
Luna
4.814
4.859
4.613
4.17
Sonnet 5
4.660
4.808
4.734
-1.58
Qwen3-32B
4.783
4.811
4.813
-0.61
Appendix
Table 9: Aggregate coefficient RMSE across six MHS platform slopes on the common 2,974-comment corpus at 300 labeled comments, in percentage points. Original-probability and log-ratio Soft+DSL use the same linear auxiliary regression and differ only in input representation. The final column reports the relative RMSE reduction from original-probability to log-ratio Soft+DSL, computed from unrounded results.
Model
Max. abs. bias
Mean coverage
Mean width
Luna
0.114
95.13
17.91
Sonnet 5
0.107
94.99
18.16
Qwen3-32B
0.116
95.00
18.51
Appendix
Table 10: Log-ratio Soft+DSL interval diagnostics on the common 2,974-comment MHS corpus at 300 labeled comments, using 2,000 paired repetitions. Maximum absolute bias and mean width are in percentage points; mean coverage is in percent.
Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. We then study retrieval-augmented annotation. Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. On the two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74--93% of the improvement, indicating that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.
Viraj Mishra, Shubham Gupta, Dhruv Gupta +2
Department of Computer Science and Engineering, Pennsylvania State University · Department of Statistics, Pennsylvania State University
Large language models (LLMs) are increasingly used in annotation tasks with varying success. While several studies have demonstrated the opportunities and shortcomings of LLMs for annotation, a critical question remains unanswered: is it possible to predict how well they will perform on a new dataset? To answer this and related questions, we construct two task families which induce 461 text classification datasets. We find that it is possible to predict LLM performance - to a certain degree - and that contrary to other work which highlights the usefulness of LLM verbalizations of self-confidence, internal log-probabilities are the most predictive signal.
Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.