High-quality labeled data is critical for training reliable machine learning and deep learning models, yet manual annotation remains costly and error-prone. Programmatic labeling addresses this challenge by using label functions (LFs), i.e., heuristic rules that automatically generate weak labels for training datasets. However, existing automated LF generation methods either rely on large language models (LLMs) to synthesize surface-level heuristics or employ model-based synthesis over hand-crafted primitives. These approaches often result in limited coverage and unreliable label quality. In this paper, we introduce EXPONA, an automated framework for programmatic labeling that formulates LF generation as a principled process balancing diversity and reliability. EXPONA systematically explores multi-level LFs, spanning surface, structural, and semantic perspectives. EXPONA further applies reliability-aware mechanisms to suppress noisy or redundant heuristics while preserving complementary signals. To evaluate EXPONA, we conducted extensive experiments on eleven classification datasets across diverse domains. Experimental results show that EXPONA consistently outperformed state-of-the-art automated LF generation methods. Specifically, EXPONA achieved nearly complete label coverage (up to 98.9%), improved weak label quality by up to 87%, and yielded downstream performance gains of up to 46% in weighted F1. These results indicate that EXPONA's combination of multi-level LF exploration and reliability-aware filtering enabled more consistent label quality and downstream performance across diverse tasks by balancing coverage and precision in the generated LF set.
Figures & tables
Figure 1: Expona : Approach Overview
Figure 2: Prompt templates of Expona for surface label function exploration.
Dataset
#classes
∣D∣
∣Dl∣
∣Dtest∣
YouTube
2
1,586
18
250
SMS
2
4,135
77
517
IMDb
2
20,000
375
2,500
Yelp
2
30,400
570
3,800
Clickbait
2
25,600
480
3,200
AGNews
4
96,000
1800
12,000
Table 1: Dataset overview
YouTube
SMS
IMDb
Yelp
Clickbait
Agnews
Finance
Chemprot
Massive
#LFs
Snuba
25
3
25
25
25
–
–
–
–
Alchemist
15
15
15
15
15
15
15
15
15
DataSculpt
84
167
90
78
171
216
185
181
131
Expona
60
60
60
60
60
60
58
46
60
Coverage
Snuba
1.000
1.000
0.605
0.650
0.185
–
–
–
–
Alchemist
1.000
0.999
0.982
0.928
0.635
0.996
1.000
1.000
0.935
Table 2: Labeling performance comparison on binary- (YouTube, SMS, IMDb, Yelp, and Clickbait) and multi-class (the others) datasets.
Pubmed
PaperAbs
#LFs
Alchemist
210
90
DataSculpt
6241
2679
Expona
772
332
Coverage
Alchemist
1.000
0.986
DataSculpt
0.963
0.972
Expona
1.000
1.000
Table 3: Labeling performance comparison on multi-label datasets.
Figure 3: Per-class proportion of instances (orange bars) as well as per-class coverage and F1-Score of Expona for dataset ChemProt .
LF Config.
Coverage
Label. Quality
E2E Perf.
IMDb
Surface
0.953
0.761
0.739
Structural
0.998
0.801
0.740
Semantic
0.413
0.345
0.698
Full config. ( Expona )
1.000
0.827
0.748
Finance
Surface
1.000
0.615
0.576
Structural
0.956
0.511
0.514
Table 4: Expona ’s performance under controlled LF-category configurations, where only the specified LF category (or combination of categories) is enabled during LF exploration and inter-type filtering is disabled.
LLM
Coverage
Label. Quality
E2E Perf.
GPT-5
1.000
0.828
0.756
GPT-4.1
1.000
0.827
0.748
Gemini-2.5-Flash
1.000
0.814
0.734
Qwen-2.5-7B
0.999
0.788
0.732
Gemma3-4B
1.000
0.796
0.717
Table 5: Effect of the choice of LLM used to generate surface LFs on the performance of Expona . Dataset: IMDb .
Classifier
Coverage
Label. Quality
E2E Perf.
LR
0.990
0.859
0.850
DT
0.978
0.820
0.847
SVM
1.000
0.865
0.853
KNN
0.983
0.820
0.845
MLP
1.000
0.857
0.852
Table 6: Performance of Expona on Yelp when different classifiers are used to generate Structural LFs.
Pretrained Model
Coverage
Label. Quality
E2E Perf.
bert-base
0.976
0.746
0.598
bge-base-en-v1.5
1.000
0.833
0.636
bart-base
0.992
0.731
0.605
multilingual-e5-large
0.995
0.817
0.621
Table 7: Performance of Expona on Massive when different embedding models are used to generate Semantic LFs.
Regime
Coverage
Label. Quality
E2E Perf.
YouTube
Non abst.
1.000
0.803
0.782
Abstain
1.000
0.848
0.870
Agnews
Non abst.
1.000
0.817
0.796
Abstain
1.000
0.855
0.827
Finance
Non abst.
1.000
0.599
0.596
Abstain
1.000
0.623
0.617
Table 8: Performance of Expona with and without abstain allowance in generating LFs.
Figure 4: Impact of weighting factor β when allowing abstain on Expona ’s performance. Dataset: Massive .
Figure 5: Expona ’s run time and E2E performance as a function of filtering parameter α . Dataset: Massive .
Figure 6: Coverage and performance of Expona as a function of the number of LFs per category Kc . Dataset: Massive .
Dataset
Label Model
Label. Quality
E2E Perf.
IMDb
MultiVote
0.832
0.756
W. MultiVote
0.832
0.757
DawidSkene
0.826
0.748
FlyingSquid
0.830
0.755
Snorkel
0.829
0.752
Finance
MultiVote
0.595
0.561
Table 9: Influence of label models on the overall performance of Expona .
Figure 7: Coverage and performance of Expona as a function of the proportion of labeled instances ( ∣Dl∣/(∣D∣+∣Dl∣) ). Dataset: Chemprot .
Crowdsourced labeling provides valuable labeled data for domains across natural language processing, computer vision, and video. Label aggregation aims to infer latent true labels from noisy and biased annotations, with the key lying in annotator reliability estimation. Despite promising progress, existing approaches struggle with one real-world bottleneck: most individual annotators label only a small subset of tasks, making accurate annotator estimation highly intractable. In this paper, we focus on the considerably more challenging multi-class label aggregation and propose AHEAD (cross-Annotator learning and High-confidEnce Annotator-guideD label aggregation), a cross-annotator learning framework that advances annotator reliability estimation by leveraging the population-level data. Specifically, AHEAD first learns high-dimensional cross-annotator contexts via a graph neural network, deriving multi-view, complementary annotator embeddings by aggregating individual-level annotator features with contextual information. These embeddings are then decoded into interpretable annotator-specific confusion matrices to fit the observed labels. We formulate a composite objective incorporating high-confidence annotators to alleviate the unsupervised training issues faced by prior models. Experiments on 10 real-world datasets spanning NLP, CV, Video, and Audio show that AHEAD substantially improves label accuracy, increasing average accuracy from 68.75% to 73.23%, with gains of up to 14.9% in the best case. Meanwhile, scalability experiments on the largest dataset further demonstrate the overall superiority of our method.
Ju Chen, Sijia Xu, Jun Feng +2
Hohai University, China · University of New South Wales, Australia · Southeast University, China +1
Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a fixed annotation budget. Unlike prior approaches relying on model confidence or uncertainty estimates, QUORUM leverages feature-based signals to estimate instance difficulty and supports multiple annotations per instance, combining them through agreement-based rewards to improve reliability. We evaluate QUORUM across diverse closed- and open-ended annotation tasks in English and multilingual settings, and QUORUM improves annotation quality by up to 34.4% while reducing costs by 8.8% over competing methods. Code can be found at https://github.com/amazon-science/QUORUM.
Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu +2
Amazon · Sapienza University of Rome · Université Côte d’Azur, CNRS, Inria, I3S
The explosive growth of short videos has presented significant challenges to content governance. Traditional manual labeling is inefficient, slow to iterate, and costly. Therefore, we explore and develop a multi-stage autopilot labeling framework aimed at progressively increasing the level of automation in short video content annotation. To move beyond the heavy reliance on human expert labeling at the initial L0 stage, we first introduce a discriminative MLLM labeler at the L1 stage, which is fine-tuned with supervised data to directly predict labels for short videos. Advancing to the L2 stage, we shift to a domain-adaptive MLLM pretrained on short video data and use it for zero-shot labeling, enabling flexible generalization without task-specific supervision. In the L3 stage, we propose an agentic annotation workflow, and equip the agent with several key tools, which empower the agent to handle complex, unfamiliar, or vaguely defined cases, significantly reducing the need for human intervention. We conduct extensive experiments across various content issues, and results demonstrate that each stage of the autopilot labeling framework progressively improves labeling accuracy, and has significantly reduced manual labor costs in real-world applications by up to 85%.