cs.CLNov 15, 2026

Ambiguous Examples Help, But Full Data Wins: Revisiting Dataset Cartography for Extractive Adversarial QA

Authors: Paul K. Mandal

Organizations: Neurint LLC, Baton Rouge, LA · U.S. Army Cyber Corps, U.S. Army Reserve, Grand Prairie, TX · Northwestern State University of Louisiana, Natchitoches, LA

Abstract

Dataset cartography argues that training dynamics can reveal regions of a dataset that matter differently for learning: easy-to-learn examples aid optimization, hard-to-learn examples often expose annotation problems, and ambiguous examples contribute strongly to out-of-distribution generalization. A striking implication of the original work is that a model trained on only 33% ambiguous examples can outperform full-data training on several OOD evaluations, including a QNLI reformulation of SQuAD evaluated on Adversarial SQuAD. We revisit this claim in a stricter setting: open-ended extractive question answering, where models must predict answer spans rather than classify question-sentence pairs. For each encoder family, we compute clean-source SQuAD training dynamics, train on full, random, easy-ranked, ambiguous-ranked, and hard-ranked subsets, and evaluate robustness on adversarial-only AddSent and AddOneSent rows. Ambiguous-ranked 33% subsets beat random 33% subsets on exact-match accuracy in all 14 model-evaluation cells and on F1 in 12/14 cells. However, full-data training beats ambiguous-ranked training in 12/14 exact-match cells and 12/14 F1 cells, with mean full-minus-ambiguous gains of 3.40 EM and 4.10 F1. The most thoroughly replicated ELECTRA-small panel is also the main exception to the ambiguous-over-random F1 gains. These results show that clean-source cartography remains useful for reduced-data selection, but its OOD advantage does not carry over as a general replacement for full-data training in extractive adversarial QA.

Figures & tables

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

    Feb 27, 2026Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray ChoudhuryLLM EvaluationLLM-as-a-Judge

  2. Intrinsic Sequence-Likelihood Confidence in Retrieval-Dominated Extractive QA: Two Pre-Specified Negatives, and What They Do and Do Not Attribute

    Sep 17, 2026Gunwoo Lee, Changmin Sung, Sang-Hwan Gwak +3LLM EvaluationSelective Prediction

  3. OpenThoughts-Agent: Data Recipes for Agentic Models

    Jun 23, 2026Negin Raoof, Richard Zhuang, Marianna Nezhurina +47Training Data CurationLLM Agent Training