Human Annotations

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 111

All topics

Optimal Design for Active Preference Learning with Biased LLM Judges
Sep 30, 2026Zhongman Du, Huiming Zhang, Haodong Zhu +1Preference Learning Preference Alignment Learning
Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University
Curating Synthetic Data for Task-Specific Visual Perception
Sep 29, 2026Saptarshi Neil Sinha, Paul Julius Kühn, Michael WeinmannSynthetic Data Indoor Scene Generation
Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands
VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics
Sep 29, 2026Bo Lv, Mao Zheng, Zheng Li +3Multimodal Benchmarks Unpaired Image-To-Image Translation
Foundation Model Department, Tencent
Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks
Sep 28, 2026Manya Singh, Arjun PakrashiAmbiguity Multi-Label Classification
School of Computer Science, University College Dublin, Ireland
Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows
Sep 23, 2026Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang +4Human Annotations Agentic Workflow Design
University of Washington, USA · Georgia Institute of Technology, USA · Google DeepMind, USA
Learning to Defer with Guidance on Real World Medical Data
Sep 22, 2026Emma Sun, Joshua Strong, Alison NobleHuman Annotations Multi-Stage Training
Department of Engineering Science, University of Oxford, UK
Decomposing Error and Style in Automated Clinical Coding
Sep 21, 2026Han-Chin Shing, Jack Moriarty, Ryan Ware +6International Classification Of Disease Coding Code Quality
Amazon
Canonical Procedural Actions: An Auditable Annotation Protocol for Tool-Use Agent Traces
Sep 21, 2026Songqi Li, Dongqing Li, Zheqiao ChengTraces Tool Invocation
Cambridge Language Sciences, Raised Faculty Building, Sidgwick Avenue, Cambridge CB3 9DA, United Kingdom · Department of Statistics, University of Oxford, 24–29 St Giles’, Oxford OX1 3LB, United Kingdom · Zhejiang University, China
Measuring Annotation Efficiency for Handwritten Devanagari Recognition: Sample-Complexity Curves for Four Pretraining Regimes
Sep 15, 2026Manglesh Kumar Pandey, Sumit Kumar BanshalHandwritten Text Recognition Pretraining
Alliance School of Liberal Arts and Sciences, Alliance University Bengaluru, India · Department of Computer Science and Engineering, Alliance University Bengaluru, India
OpenEnded: An Open-Response Speech Corpus for Speaking Proficiency Assessment with Human Annotations and ALM Supervision
Sep 14, 2026Yu-Wen Chen, Eric Zhou, Evelyn Ding +3Proficiency Text Corpora
Department of Computer Science, Columbia University, USA
TEAR: Table Extraction with Attribute Recommendation from Texts via Large Language Models
Sep 14, 2026Tong Li, Shuye Ding, Jiachuan Wang +4Information Extraction Text Analysis and Detection
Hong Kong University of Science and Technology Hong Kong SAR, China · Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · South China Normal University Guangzhou, China
What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts
Sep 14, 2026Mohamad Al Mdfaa, Nursultan Askarbekuly, Ahmed Helaly +2Transcript Human Annotations
LemoniLab FZCO (lemonilab.com) · Innopolis University (innopolis.ru)
A Multi-Vehicle Dataset with Camera, LiDAR, and Radar Sensors and Scanned 3D Models for Custom Auto-Annotation using RTK-GNSS
Sep 14, 2026Philipp Berthold, Bianca Forkel, Mirko MaehlischAutonomous Driving Radar
Institute for Autonomous Driving of the University of the Bundeswehr Munich, Neubiberg, Germany
Single-Query Person-Centric Bimanual Hand-Object Interaction Detection
Sep 14, 2026Jonghyun Kim, Junho Roh, Yubin Yoon +5Hand-Object Interaction Detection Human Annotations
AI Lab, CTO Division, LG Electronics, South Korea
OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
Sep 11, 2026Jianjiang Yang, Peihang Li, Shanqing Xu +3Multimodal Large Language Models Cross-Modal
The University of Manchester · The University of Hong Kong · Huazhong University of Science and Technology +2
SignMatch: Matching Dictionary Signs to Continuous Sign Language Video
Sep 1, 2026Ryan Wong, Youngjoon Jang, Liliane Momeni +2Sign Language Translation Sign
Visual Geometry Group Department of Engineering Science University of Oxford UK · LIGM, École des Ponts, IP Paris, Univ Gustave Eiffel, CNRS France
Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics
Sep 1, 2026Maksim Evdokimov, Matvey Ivanov, Dmitrii Tsiupin +3Optical Character Recognition Data-Curation
T-Tech
On the Human and Computer Alignment of Attribute-Based Music Matches
Sep 1, 2026Roser Batlle-Roca, Woosung Choi, Joan Serrà +5Music Structure Analysis Generative Artificial Intelligence
Music Technology Group, Universitat Pompeu Fabra · Sony AI · Joint Research Centre, European Commission +1
Human-Anchored Factuality Evaluation with Strategic Annotation
Aug 31, 2026Yu Wang, Craig Erickson, Kevin SmallHuman Annotators Facts
Amazon AGI
Expert-like Bone Ultrasound Segmentation through Expert-in-the-loop Mask-conditioned Progressive Learning
Aug 31, 2026Arash Tavangar, Larissa K. Chiu, Hamidreza Khodashenas +2Ultrasound U-Net
Department of Surgery, McGill University, Montreal, QC, Canada · Research Institute of the McGill University Health Centre, Montreal, QC, Canada · Division of Orthopedic Surgery, Department of Surgery, McGill University,2026 Montreal, QC, Canada
Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
Aug 31, 2026Alessio Galatolo, Meriem BeloucifSequential Preference Optimization Preference Alignment Learning
Uppsala University
Lot Machine: Multimodal Lot Extraction from Auction Catalogs
Aug 31, 2026Mathias Zinnen, Alisha Mund, Sabine Lang +3Catalogs Extraction
Pattern Recognition Lab, FAU Erlangen · Department of Digital Humanties and Social Studies, FAU Erlangen
WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition
Aug 21, 2026Abigail G. Grassick, Jerome Tze-Hou Hsu, Ethan Lin +10Df-Wild Dataset Fish
Cornell University, 14850 Ithaca, NY · University of Colorado Boulder, 80309 Boulder, CO · HHMI Janelia Research Campus, 20147, Ashburn VA
Attune: A Self-Annotation Tool for Understanding Robot Operator Attention Profiles
Aug 12, 2026Puqi Zhou, Sungsoo Ray Hong, David PorfirioRobot Systems Multi-Robot Systems
Department of Computer Science George Mason University Fairfax, VA, USA · Department of Information Sciences and Technology George Mason University Fairfax, VA, USA
Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication
Aug 12, 2026Xiaobin Shen, Chloe Y. H. Huang, Jonathan Elmer +1Clinical Prediction Predictive Uncertainty
Heinz College of Information Systems and Public Policy, Carnegie Mellon University · Department of Emergency Medicine, University of Pittsburgh
When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation
Aug 12, 2026Jinhyung Bae, Dain Kil, Seongmin Oh +1Machine Translation Quality Human Annotations
Hankuk University of Foreign Studies
CIDER: A Dataset of Contextual Disclosure Boundaries for Privacy Preference Alignment
Aug 10, 2026Bingcan Guo, Eryue Xu, Jijie Zhou +2Privacy Disclosures
University of Washington · UIUC · Northeastern University
TradeVerse: A Longitudinal Benchmark of Political Negotiation in International Trade
Aug 6, 2026Debodeep Banerjee, Amitangshu DasguptaDialogue Benchmarks Negotiation
DI, University of Pisa · RazorPay · DISI, University of Trento
Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search
Aug 3, 2026Han Wang, Alex Whitworth, Pak Ming Cheung +5Relevance Human Annotations
Pinterest · San Francisco, CA, USA
Automatic Annotation of Ancient Greek Vowel Length
Aug 3, 2026Albin Thörn Cleland, Eric CullhedGreek Consonants
Centre for Languages and Literature, Lund University, Sweden · Department of Linguistics and Philology, Uppsala University, Sweden
HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems
Aug 1, 2026Pritom Saha Akash, Phanideep Gampa, Chao Shen +2Relevance Human Annotations
Amazon
Challenges in annotations by humans and LLMs: A case study of evaluative language
Jul 30, 2026Mirela Imamovic, Aenne Cecilia Kristine Knierim, Khushi Pitroda +1Large Language Model Annotations Human Annotations
University of Hildesheim, Institute of Translation Studies and Specialised Communication, Germany · University of Paris 8 Vincennes-Saint Denis, TransCrit, France · University of Hildesheim, Institute of Information Science and Language Technology, Germany
Automated Cardiac Adipose Tissue Segmentation in Computed Tomography: A Literature Review
Jul 18, 2026Andreas W. Aspe, Jonas Jalili Pedersen, Andreas Ohrt Johansen +4Echocardiography Multi Organ Computed Tomography Segmentation
DTU Compute, Technical University of Denmark, Kongens Lyngby, Denmark · Heart Centre, Rigshospitalet, Copenhagen, Denmark, Denmark · Novo Nordisk A/S, Søborg, Denmark
Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement
Jul 12, 2026Mahammed Kamruzzaman, Shrabon Kumar Das, Gene Louis KimLarge Language Model Annotations Demographics
Bellini College of AI, Cybersecurity and Computing · University of South Florida
Why Domain Matters: Domain-Aware Benchmarking of Underwater Object Detection and Annotation Quality
Jul 12, 2026Melanie Wille, Dimity Miller, Tobias Fischer +1Underwater Object Detection Unsupervised Detection
QUT Centre for Robotics, Queensland University of Technology, Brisbane, Australia
VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness
Jul 9, 2026Iulia-Maria Udrea, Alexandra Diaconu, Bogdan AlexeHigh Confidence Pseudo-Labels Video Dataset
Department of Computer Science, University of Bucharest
GKDT: General Keypoint Detection Transformer
Jul 1, 2026Changsheng Lu, Yuxin Chen, Haokun Gui +5Keypoint Detection Open-Vocabulary Object Detection
Hong Kong University of Science and Technology, Hong Kong, China · Australian National University, Canberra, Australia · Tencent Inc., China +1
RareDxR1: Autonomous Medical Reasoning for Rare Disease Diagnosis Beyond Human Annotation
Jun 30, 2026Deyang Jiang, Haoran Wu, Ziyi Wang +4Rare Disease Clinical Reasoning Training
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Peking Union Medical College Hospital, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China
A Point Cloud Transformer for Remote Monitoring and Automated Assessment of Physical Rehabilitation Exercises
Jun 29, 2026Kazi Rafat, Md. Ismail Hossain, M M Lutfe Elahi +4Rehabilitation Point Clouds
Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh · Apurba Technologies Limited, Dhaka, Bangladesh
HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
Jun 29, 2026Xinrui Ruan, Zhenyu Zhao, Waverly Wei +4Model Evaluation Human Annotations
Division of Biostatistics, University of California, Berkeley · Roblox Corporation · Department of Data Sciences and Operations, University of Southern California +2
AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search
Jun 24, 2026Md Omar Faruk Rokon, Shasvat Desai, Hong Yao +1Relevance Human Annotations
Walmart Global Tech, Sunnyvale, CA, USA
PrivacyAlign: Contextual Privacy Alignment for LLM Agents
Jun 19, 2026Manveer Singh Tamber, Abhay Puri, Marc-Etienne Brunet +3Large Language Model Agents Privacy
University of Waterloo · †Work done during internship at ServiceNow AI Research · 2ServiceNow AI Research +2
$μ$ Match: Foundation Models for Semi-supervised Learning and Domain Adaptation in EM
Jun 19, 2026Marei Freitag, Olesia Korchevaia, Luca Freckmann +2Recent Vision Foundation Models Wireless Foundation Models
Life and Medical Sciences Institute (LIMES), University of Bonn, Germany · Institute of Computer Science, Georg-August-University Göttingen, Germany
Speeding up the annotation process in semantic segmentation industrial applications
Jun 18, 2026Marta Fernandez-Moreno, Margarita Guerrero, Rosalia Rementeria +2Semantic Segmentation Human Annotations
Department of Computer Science and Artificial Intelligence, Andalusian Research Institute in Data Science and Computational Intelligence, DaSCI, University of Granada, 18071, Granada, Spain · Ingénica STS, Av. Pablo Iglesias 38, Bj, 33205, Gijón, Spain · ArcelorMittal Global R&D, SLab—Steel Labs, Calle Marineros 4, 33490, Avilés, Spain +1
AnnotateAnything: Automatic Annotation of 3D Assets for Robot Manipulation
Jun 16, 2026Haoran Lu, Mutian Shen, Shuyang Yu +9Robotic Data Acquisition 3D Assets
Northwestern University · Peking University
IMPACTeen: Intentions, Manipulation, Persuasion, Annotations, and Consequences in Teen Communication Dataset
Jun 15, 2026Aleksander Szczęsny, Wiktoria Mieleszczenko-Kowszewicz, Maciej Markiewicz +5Social Influence Textual Dataset
Wrocław University of Science and Technology, Wrocław, Poland
LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation
Jun 14, 2026Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1Llm-As-A-Judge Large Language Model Judges
Department of Computer Science, Graduate School of Engineering, Chubu University, Kasugai, Aichi 487-8501, Japan · Mitsubishi Heavy Industries, Ltd., Research & Innovation Center, Heat Transfer Research Department, Takasago, Hyogo 676-8686, Japan
LLM-Assisted Stance Detection in Scientific Discourse: A Test Case in Bayesian Cognitive Science
Jun 14, 2026Eyup Engin Kucuk, Tarik Kelestemur, Ömer Dağlar TanrikuluQualitative Research Discourse
University of New Hampshire · Department of Psychology · Independent Researcher
Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability
Jun 12, 2026Alyssa Unell, Natalie Dullerud, Naomi Boneh +4Large Language Model Judges Judges
Department of Computer Science Stanford University · Department of Medicine Stanford University
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5Llm-As-A-Judge Large Language Model Alignment
Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University +2
Profy: Interpretable Visualization of Expertise-Dependent Motor Skills Toward Supporting Piano Practice
Jun 9, 2026Kazuki Kawamura, Fujiki Nakamura, Hayato Nishioka +3Instrument Proficiency
The University of Tokyo · Sony Computer Science Laboratories
Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency
Jun 7, 2026Md Abdullah Al Forhad, Weishi ShiNoisy Labels Human Annotations
Active Machine Learning Lab · Learning · Language · Vision (LLaVi) Lab · Department of Computer Science and Engineering +1
Cross-Source Reasoning-based Correction for Author Name Disambiguation
Jun 7, 2026Fanjin Zhang, Yunhe Pang, Bo Chen +4Authorship Disambiguation
School of Information Engineering Research Center of Database and Business Intelligence Key Laboratory of Data Engineering and Knowledge Engineering Renmin University of China · School of Computer Science and Engineering Sun Yat-Sen University · University of Oslo Department of Informatics +1
Sycophantic Praise: Evaluating Excessive Praise in Language Models
Jun 5, 2026Daniel Vennemeyer, Phan Anh Duong, Meryl Ye +2Sycophancy Large Language Model Alignment
University of Cincinnati · Carnegie Mellon University · Texas A&M University
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
Jun 3, 2026Abhishek DivekarHuman Annotations Reliable
Amazon
Towards Characterizing Scientific Image Utility and Upgradability
Jun 2, 2026WenZhe Li, Qihang Yan, Liang Chen +6Scientific Figure Scientific Discovery
Shanghai Research Institute for Intelligent Autonomous Systems, TongJi University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling
Jun 1, 2026Andrea Brunello, Cristian Curaba, Luca Geatti +3Legal Natural Language Processing Benchmarks Natural Language Inference
University of Udine, Italy
Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
Jun 1, 2026Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli +10Human Annotations Annotations
NLLG Lab University of Technology Nuremberg · Interdisciplinary Transformation University, Austria
Knowledge-Intensive Video Generation
May 31, 2026Chenxu Wang, Mingda ChenInteractive Video Generation Human Annotations
Fudan University · Shanghai Jiao Tong University
Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety Alignment
May 29, 2026Arkadiy Saakyan, Charvi Rastogi, Lora AroyoCultural Awareness Multilingual Safety
Columbia University, New York, NY, USA. · Google DeepMind, New York, NY, USA.

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Optimal Design for Active Preference Learning with Biased LLM Judges

Zhongman Du, Huiming Zhang, Haodong Zhu, Baochang Zhang

Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University

Preference Learning Preference Alignment Learning Pool-Based Active Learning Reinforcement Learning From Human Feedback Large Language Model Judges Optimal Design+2

Open paper AskPDF arXiv

Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.

Momentum

Related topics

Latest papers 111