Human Annotators

Ask about this topic

Momentum

8 papers in the last four weeks, up 14% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 71

All topics
CardsList

  1. Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

    Sep 24, 2026Fardeen Sadab, Adib SakhawatMultilingual BenchmarkHuman Annotators

    Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh

  2. onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

    Sep 21, 2026Lei Yang, Mengyin Liu, Jia Wang +9Large Language Model AnnotationsLarge Language Model Alignment

    StepFun · Xiamen University

  3. Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs

    Sep 15, 2026Toqeer Ehsan, Nico Penttilä, Richard Schmidt +2Vision-Language Foundation ModelsHallucination Detection

    Reliable Intelligence Team, Physical AI, VTT Technical Research Centre of Finland

  4. Through the Eyes of the Beholder: Biometric and Demographic Conditioning for Multimodal Sexism Detection

    Sep 14, 2026Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truică +1Multimodal Deep LearningHuman Annotators

    National University of Science and Technology POLITEHNICA Bucharest, Splaiul Independenţei 313, Bucureşti 060042, Romania · Academy of Romanian Scientists, Ilfov 3, Bucharest, 050044, Romania

  5. Option-Aware Retrieval and Task-Specific VLM Adaptation for Medical VQA

    Sep 14, 2026Tristan Kirscher, Niklas C. Koser, Soren PirkMedical Visual Question AnsweringMultiple-Choice Questions

    ICube, Institut Strauss · ICube Laboratory, CNRS UMR 7357, University of Strasbourg, Strasbourg, France · CLCC Institut Strauss, Strasbourg, France +2

  6. How broad is that claim? Mapping Generalisation in NLP Research

    Sep 13, 2026Chenxin Diao, Nataliya Stepanova, Emily AllawayScientific DiscoveryGeneralisation

    School of Informatics, University of Edinburgh

  7. Human Agreement and Return Association Are Not Interchangeable Criteria

    Sep 10, 2026AS Aravinthakshan, Laven Srivastava, Harsh NandwaniFinancial Sentiment AnalysisSentiment

    Manipal Institute of Technology · Perssonify

  8. Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

    Sep 7, 2026Raphael Lorenzo-Louis, Bertrand Luvison, Serena IvaldiHuman-Robot InteractionIntuition

    Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France

  9. Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection

    Sep 3, 2026Joe Cecil, Marjorie FreedmanCitation Hallucination DetectionHuman Annotators

    Information Sciences Institute, University of Southern California

  10. Does task decomposition improve automatic NLG evaluation?

    Sep 1, 2026Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1Llm-As-A-JudgeNatural Language Datasets

    Amazon, Barcelona, Spain · Amazon, Berlin, Germany

  11. Human-Anchored Factuality Evaluation with Strategic Annotation

    Aug 31, 2026Yu Wang, Craig Erickson, Kevin SmallHuman AnnotatorsFacts

    Amazon AGI

  12. LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurts

    Aug 31, 2026Daniela Occhipinti, Andrea Piergentili, Marco GueriniDemographicsLarge Language Model Bias

    Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento, Italy · Almawave Labs, Via Di Casal Boccone, 188/190, Rome

  13. Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

    Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8Scientific IdeationAssessment

    1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1

  14. Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment

    Aug 21, 2026Emma Granqvist, Rocío Mercado, Samuel GenhedenLlm-As-A-JudgeDrug Discovery

    Molecular AI, Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden · Department of Computer Science and Engineering, Section for Data Science and AI, Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden · Science for Life Laboratory (SciLifeLab), Gothenburg, Sweden

  15. Toward a Theory of Value in AI Alignment

    Aug 10, 2026Andrew Smart, Shazeda Ahmed, Jackie Kay +3Artificial Intelligence AlignmentHuman Values

    1Google Research · 2UCLA · 3Google Deep Mind +2

  16. Conformal Coverage Guarantees for Any Video Temporal Grounder

    Aug 7, 2026Aseel Mohamed, Rasul Khanbayov, Erchin Serpedin +1Temporal GroundingSpatial Coherence

    Department of Electrical and Computer Engineering, Texas A&M University at Qatar, Doha, Qatar · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar · Department of Computer and Electrical Engineering, Texas A&M University, College Station, TX, USA

  17. ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation

    Aug 5, 2026Xiaoyan Gu, Yifang Wang, Wenqing Zheng +6Artistic OwnershipArt

    School of Computer and Computing Science, Hangzhou City University Hangzhou, Zhejiang, China

  18. VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs

    Aug 4, 2026Andrei Chetvergov, Alexander Evseev, Timofei Sivoraksha +4Valence-Arousal EstimationSentiment

  19. Oh Deer, How Should I Handle This? Seasonal Priors for Selective Wildlife Annotation and Classification

    Aug 3, 2026Hugo Markoff, Christoph Praschl, Anton Hjalte Jørgensen +5Animal Re-IdentificationSpecies

    Aalborg University, Department of Chemistry and Bioscience, Aalborg, Denmark · University of Applied Sciences Upper Austria, Hagenberg, Austria · Massachusetts Institute of Technology, Cambridge, MA, USA

  20. SERUM: State Extraction and Refinement for User Modeling

    Jul 31, 2026Andy J. Phu, James Mooney, Karin de Langis +2User ModelingEgocentric Video

    Minnesota NLP Lab · University of Minnesota · Minneapolis, MN 55455, USA

  21. DODA: A Database of Datasets for Aesthetics Research

    Jul 30, 2026Lisa Koßmann, Ralf Bartho, Christoph Redies +1AestheticsImage Quality Assessment

    Laboratory of Experimental Psychology, Department of Brain and Cognition, University of Leuven (KU Leuven), Leuven, Belgium · Leuven AI Institute, University of Leuven (KU Leuven), Leuven, Belgium · Experimental Aesthetics Group, Institute for Anatomy I, University Hospital Jena, Germany

  22. FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery

    Jul 28, 2026Reihaneh Rostami, Brian GoodwinPool-Based Active LearningActive Learning

    RAIC Labs · RAIC Labs USA

  23. A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection

    Jul 27, 2026Gabriel Singer, Samuel Gruffaz, Olivier Vo Van +2Imbalanced ClassificationMinority

    SNCF, Paris, France · Université Paris-Saclay, ENS Paris-Saclay, Centre Borelli, France · 2Tampere University, Finland, Biostatistics group

  24. The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

    Jul 20, 2026Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann +4Similarity and MetricsRecent Vision-Language Models

    1Carnegie Mellon University · 2Adobe Research · 3UC Berkeley

  25. AI_LectureNote: A Retrospective Pilot Study of a Post-ASR Workflow for English-Script Rendering and Semantic Drift in Korean-English Medical Lectures

    Jul 19, 2026Kyeongeon Lee, Donghoon Chang, Seungryeol Baek +2Automatic Speech RecognitionHuman Annotators

    Sungkyunkwan University, Republic of Korea

  26. Validity of LLMs as data annotators: AMALIA on authority

    Jul 9, 2026Manuel PitaHuman AnnotatorsValidation

    Artificial Intelligence, Social Interaction and Complexity Laboratory · CICANT, Universidade Lusófona

  27. Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

    Jul 8, 2026Quentin BrabantEvaluation MetricsHievi-Rag

    Orange Research, Lannion, France

  28. Who's Behind It? Annotating and Extracting Conspiratorial Actors from German Telegram Posts

    Jul 6, 2026Helena Mihaljević, Jolanda Beer, Mareike Lisker +1Human AnnotatorsArtificial Intelligence Detection

    HTW Berlin, Germany · Goethe University, Frankfurt, Germany

  29. Multi-Agentic System Leveraging Open-Source LLMs to Mitigate Disinformation Threats

    Jun 29, 2026Sebastian Kula, Martin TamajkaMisinformation DetectionMulti-Agent Large Language Model Systems

    Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · West Pomeranian University of Technology in Szczecin, Szczecin, Poland

  30. Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

    Jun 26, 2026Manuel PitaCorrectnessInter-Annotator Agreement

    Artificial Intelligence, Social Interaction and Complexity Laboratory CICANT, Universidade Lusófona 376, Campo Grande, 1700-097 Lisbon, Portugal

  31. Interpretable Probabilistic Medical Image Segmentation via Gaussian Process with Explicit Modelling of Annotation Bias and Variability

    Jun 22, 2026Qi Li, Yuliang Huang, Shaheer U. Saeed +7Semi-Supervised Medical Image SegmentationInter-Annotator Agreement

    UCL Hawkes Institute, Department of Medical Physics and Biomedical Engineering, University College London, London, U.K. · Centre for Bioengineering, Digital Environment Research Institute, School of Engineering and Materials Science, Queen Mary University of London, London, U.K. · Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, U.K. +4

  32. Counsel: A Meta-Evaluation Dataset for Agentic Tasks

    Jun 19, 2026Sashank Pisupati, Henry Broomfield, Eujeong Choi +5Agentic BenchmarksLlm-As-A-Judge

    1Atla AI · 2Cohere AI · 3Mistral AI +1

  33. Findings of the MAGMaR 2026 Shared Task

    Jun 10, 2026Alexander Martin, Dengjia Zhang, Joel Brogan +7Multimodal Retrieval Augmented GenerationMultimodal Retrieval

    Johns Hopkins University · OpenAI · University of Maryland, Baltimore County +4

  34. Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails

    Jun 4, 2026Marco Antonio Stranisci, A Pranav, Rossana Damiano +2Large Language Model BiasGuardrail

    University of Turin, Italy · Trustworthy AI Lab, University of Hamburg, Germany · IT University of Copenhagen, Denmark

  35. SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs

    Jun 3, 2026Jingyao Wu, Ashley Wang, Keane Ong +2Automatic LabellingLarge Language Model Alignment

    MIT Media Lab, Massachusetts Institute of Technology · National University of Singapore

  36. SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformation

    Jun 2, 2026Junxiao Yang, Minghao Zhang, Xiaoce Wang +4CredibilityHuman Annotators

    The Conversational AI (CoAI) group, DCST, Tsinghua University

  37. The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction

    Jun 1, 2026Mirko Lai, Alessandra Urbinati, Simona Frenda +2Large Language Model AnnotationsHuman Annotators

    Heriot-Watt University, Edinburgh, Scotland · aequa-tech, Torino, Italy · Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA +2

  38. Machine Learning for Coding Retail Product Names to Consumer-Price Categories: A Rule-plus-Bag-of-Words Pipeline with Reliability-Weighted Human-in-the-Loop Labeling

    Jun 1, 2026Vladimir BeskorovainyiConsumptionHuman Annotators

    Besk Tech · Moscow Institute of Physics and Technology (MIPT)

  39. EvoPool: Evolutionary Programmatic Annotation for Label-Efficient Specialized Supervision

    Jun 1, 2026Tianyi Xu, Yaolun Zhang, Xuan Ouyang +1Human AnnotatorsMulti-Agent Evolution

    Oregon State University · University of Wisconsin–Madison

  40. Digging Up Citations: FOSSIL, a Dataset and Workflow for Reference Extraction in Law and the Humanities

    May 31, 2026Luca Foppiano, Christian BoulangerCitationsHuman Annotators

    ScienciaLAB, Portugal · Max Planck Institute for Legal History and Legal Theory, Germany

  41. The Representation-Rationalizability Tradeoff in Reward Learning

    May 29, 2026Jing Dong, Yaoliang Yu, Pascal PourpartReinforcement Learning From Human FeedbackReward Functions

    Vector Institute · University of Waterloo & Vector Institute

  42. Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization

    May 27, 2026Beiduo Chen, Pingjun Hong, Ziyun Zhang +3Human AnnotatorsProcess-Level Supervision

    U MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Æ Munich Center for Machine Learning (MCML), Munich, Germany · [ University of Vienna, Austria +1

  43. Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation

    May 27, 2026Niantong Li, Guangzheng Hu, Weixu Qiao +35CreativityFidelity

    Alibaba Inc.

  44. Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation

    May 22, 2026Konstantin Berlin, Adam SwandaContent ModerationAutomatic Labelling

    Cisco AI Defense

  45. AraHopeCorpus: Annotation Guidelines and Dataset for Hope Speech in Arabic Social Media Crisis Discourse

    May 22, 2026Esra'a Sharqawi, Wajdi ZaghouaniArabicHate Speech Detection

    Hamad Bin Khalifa University, Northwestern University in Qatar Doha, Qatar

  46. Same Target, Different Basins: Hard vs. Soft Labels for Annotator Distributions

    May 20, 2026Mirerfan Gheibi, Gashin GhazizadehLabel PropagationHuman Annotators

    Independent Researcher.

  47. Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech

    May 17, 2026Kaavya Chaparala, Thomas Thebaud, Jesús Villalba López +3TranscriptSummarization

    Johns Hopkins University 3400 N. Charles St., Baltimore, MD 21218

  48. Voice "Cloning" is Style Transfer

    May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Cross-Lingual Voice CloningVoice Conversion

    Cornell University · TogetherAI · Stanford University

  49. Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models

    May 15, 2026Cheng Zhang, Yuer Liu, Zhiyu Zhou +2Emotion RecognitionEmotion

    Department of Computer Science and Technology, Jilin University, Changchun, China · Department of Computer Science, National Taiwan University, Taipei, Taiwan

  50. Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks

    May 11, 2026Tadesse Destaw Belay, Ibrahim Said Ahmad, Idris Abdulmumin +6Human AnnotatorsDisagreement

    1Instituto Politécnico Nacional · University of Wisconsin–Stevens Point · University of Pretoria +3

  51. Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation

    May 7, 2026Maximilian Maurer, Maximilian Linde, Gabriella LapesaHuman AnnotatorsLinguistics

    1GESIS - Leibniz Institute for the Social Sciences · 2Heinrich-Heine University Düsseldorf

  52. Understanding Annotator Safety Policy with Interpretability

    May 6, 2026Alex Oesterling, Donghao Ren, Yannick Assogba +4Human AnnotatorsArtificial Intelligence Safety

    Harvard University, Cambridge, MA, USA · Apple, Seattle, WA, USA · Apple, Cambridge, MA, USA +1

  53. Leveraging Vision-Language Models as Weak Annotators in Active Learning

    May 1, 2026Phuong Ngoc Nguyen, Kaito Shiku, Ryoma Bise +2Pool-Based Active LearningActive Learning

    Kyushu University, Japan.

  54. Three Models of RLHF Annotation: Extension, Evidence, and Authority

    Apr 28, 2026Steve CoyneReinforcement Learning From Human FeedbackHuman Annotators

    University of Toronto, Canada · University of Toronto, Toronto, Canada

  55. Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment?

    Apr 28, 2026António Branco, João Silva, Nuno Marques +10Cultural AwarenessLarge Language Model Alignment

    †University of Lisbon · ‡University of Beira Interior

  56. SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning

    Apr 23, 2026Hans Ole Hatzel, Ekaterina Artemova, Haimo Paul Stiemer +2NarrativesSimilarity and Metrics

    University of Hamburg1 · Toloka AI2 · German UDS3 +1

  57. Fine-Grained Perspectives: Modeling Explanations with Annotator-Specific Rationales

    Apr 23, 2026Olufunke O. Sarumi, Charles Welch, Daniel BraunHuman AnnotatorsExplainable AI Methods

    Marburg University, Marburg, Germany · McMaster University, Hamilton, Ontario, Canada

  58. Modeling Human Perspectives with Socio-Demographic Representations

    Apr 20, 2026Leixin Zhang, Cagri ColtekinHuman AnnotatorsDemographics

    University of Tübingen

  59. From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?

    Apr 20, 2026Hasan Amin, Harry Yizhou Tian, Xiaoni Duan +3Human AnnotatorsBiases

    Purdue University · Washington University in St. Louis

  60. REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations

    Apr 19, 2026Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh +1Large Language Model AnnotationsLarge Language Model Fine-Tuning

    Department of Electrical and Computer Engineering UC Santa Barbara Santa Barbara, CA 93106, USA