Annotation Budget

Annotation budget, the limited resources available for labeling data in machine learning, is a critical constraint impacting model performance across various tasks, including named entity recognition, question answering, and image segmentation. Research focuses on optimizing annotation strategies, such as active learning and data selection algorithms (including those leveraging pre-trained models), to maximize model accuracy with minimal labeling effort. These efforts aim to improve the efficiency and cost-effectiveness of training machine learning models, particularly in resource-constrained settings or when dealing with complex, subjective tasks where annotation is expensive or requires expert knowledge. The resulting advancements have significant implications for practical applications by reducing the financial and time burdens associated with data annotation.

Papers

October 26, 2024

Annotation Efficiency: Identifying Hard Samples via Blocked Sparse Linear Bandits
Adit Jain, Soumyabrata Pal, Sunav Choudhary, Ramasuri Narayanam, Vikram Krishnamurthy
Linear Bandit Hard Sample Annotation Budget Block Sparsity Sparse Linear Bandit

June 19, 2024

In-Context Learning on a Budget: A Case Study in Named Entity Recognition
Uri Berger, Tal Baumel, Gabriel Stanovsky
Case Study Entity Recognition Different Context Local Authority Budget Annotation Budget Large Scale Annotated

February 21, 2024

Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation
Preni Golazizian, Alireza S. Ziabari, Ali Omrani, Morteza Dehghani
Annotation Strategy Annotation Budget Subjective Task

January 17, 2024

Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints
Kunpeng Guo, Dennis Diefenbach, Antoine Gourru, Christophe Gravier
Pre Trained Language Model Fine Tuning Strategy Annotation Budget Domain Specific Question Answering QA Model

November 16, 2023

Large Language Models for Propaganda Span Annotation
Maram Hasanain, Fatema Ahmad, Firoj Alam
Annotation Budget Propaganda Technique Propaganda Content Identification

November 4, 2023

Perturbation-based Active Learning for Question Answering
Fan Luo, Mihai Surdeanu
Active Learning Question Answering Acquisition Function Training Strategy Annotation Budget

September 30, 2023

Deep Active Learning with Noisy Oracle in Object Detection
Marius Schubert, Tobias Riedlinger, Karsten Kahl, Matthias Rottmann
Deep Active Learning Noisy Annotation Object Detection Performance Annotation Budget

September 29, 2023

Towards Free Data Selection with General-Purpose Models
Yichen Xie, Mingyu Ding, Masayoshi Tomizuka, Wei Zhan
Active Learning Data Selection Annotation Budget General Purpose Model Batch Selection Free Data

September 12, 2023

Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection
Sophia Althammer, Guido Zuccon, Sebastian Hofstätter, Suzan Verberne, Allan Hanbury
Pre Trained Language Model Active Learning Pretrained Language Model Data Annotation Neural Ranker Annotation Budget Random Selection

June 6, 2023

Efficient Anomaly Detection with Budget Annotation Using Semi-Supervised Residual Transformer
Hanxi Li, Jingqi Wu, Hao Chen, Mingwen Wang, Chunhua Shen
Anomaly Detection Industrial Anomaly Detection Annotation Budget Efficient Anomaly Detection Patch Matching Residual Transformer Supervised Anomaly Detection AD Dataset

May 24, 2023

Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions
London Lowmanstone, Ruyuan Wan, Risako Owan, Jaehyung Kim, Dongyeop Kang
Imputation Algorithm Imputation Method Initial Study Model Prediction Data Annotation Annotation Budget Personalized Prediction Dynamic Distribution

March 25, 2023

Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm
Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, Wei Zhan
Semantic Segmentation Vision Task Unsupervised Pre Training Finetune Paradigm Annotation Budget Active Finetuning

March 21, 2023

Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns
Javier Gamazo Tejero, Martin S. Zinkernagel, Sebastian Wolf, Raphael Sznitman, Pablo Márquez Neila
Segmentation Model Segmentation Dataset Weak Annotation Annotation Budget Adaptive Strategy

March 3, 2023

Unified Perception: Efficient Depth-Aware Video Panoptic Segmentation with Minimal Annotation Costs
Kurt Stolle, Gijs Dubbelman
Panoptic Segmentation Scene Understanding Annotation Budget Object Embeddings Video Panoptic Segmentation

October 13, 2022

Reducing Annotation Effort by Identifying and Labeling Contextually Diverse Classes for Semantic Segmentation Under Domain Shift
Sharat Agarwal, Saket Anand, Chetan Arora
Semantic Segmentation Domain Adaptation Active Learning Domain Shift Annotation Budget Active Domain Adaptation Augmentation Approach Annotation Effort

October 11, 2022

Analyzing Text Representations under Tight Annotation Budgets: Measuring Structural Alignment
César González-Gutiérrez, Audi Primadhanty, Francesco Cazzaro, Ariadna Quattoni
Text Representation Latent Structure Efficient Representation Annotation Budget

September 30, 2022

Did You Get What You Paid For? Rethinking Annotation Cost of Deep Learning Based Computer Aided Detection in Chest Radiographs
Tae Soo Kim, Geonwoon Jang, Sanghyup Lee, Thijs Kooi
Deep Learning Chest Radiograph Data Annotation Efficient Annotation Annotation Budget High Quality Annotation Annotation Cost

September 25, 2022

Partial annotations for the segmentation of large structures with low annotation cost
Bella Specktor Fadida, Daphna Link Sourani, Liat Ben Sira Elka Miller, Dafna Ben Bashat, Leo Joskowicz
Segmentation Performance Selective Annotation Annotation Budget

July 31, 2022

Deep Active Learning with Budget Annotation
Kinyua Gikunda
Active Learning Pre Trained Unlabeled Data Deep Active Learning Annotation Budget Uncertainty Metric

April 18, 2022

Active Learning with Weak Supervision for Gaussian Processes
Amanda Olmin, Jakob Lindqvist, Lennart Svensson, Fredrik Lindsten
Active Learning Gaussian Process Supervised Learning Weak Supervision Active Learning Algorithm Annotation Budget Annotation Error Rate

Annotation Budget

Papers

Annotation Efficiency: Identifying Hard Samples via Blocked Sparse Linear Bandits

In-Context Learning on a Budget: A Case Study in Named Entity Recognition

Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation

Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints

Large Language Models for Propaganda Span Annotation

Perturbation-based Active Learning for Question Answering

Deep Active Learning with Noisy Oracle in Object Detection

Towards Free Data Selection with General-Purpose Models

Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

Efficient Anomaly Detection with Budget Annotation Using Semi-Supervised Residual Transformer

Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions

Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm

Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns

Unified Perception: Efficient Depth-Aware Video Panoptic Segmentation with Minimal Annotation Costs

Reducing Annotation Effort by Identifying and Labeling Contextually Diverse Classes for Semantic Segmentation Under Domain Shift

Analyzing Text Representations under Tight Annotation Budgets: Measuring Structural Alignment

Did You Get What You Paid For? Rethinking Annotation Cost of Deep Learning Based Computer Aided Detection in Chest Radiographs

Partial annotations for the segmentation of large structures with low annotation cost

Deep Active Learning with Budget Annotation

Active Learning with Weak Supervision for Gaussian Processes