Ambiguous Examples Help, But Full Data Wins: Revisiting Dataset Cartography for Extractive Adversarial QA
Organizations: Neurint LLC, Baton Rouge, LA · U.S. Army Cyber Corps, U.S. Army Reserve, Grand Prairie, TX · Northwestern State University of Louisiana, Natchitoches, LA
Abstract
Dataset cartography argues that training dynamics can reveal regions of a dataset that matter differently for learning: easy-to-learn examples aid optimization, hard-to-learn examples often expose annotation problems, and ambiguous examples contribute strongly to out-of-distribution generalization. A striking implication of the original work is that a model trained on only 33% ambiguous examples can outperform full-data training on several OOD evaluations, including a QNLI reformulation of SQuAD evaluated on Adversarial SQuAD. We revisit this claim in a stricter setting: open-ended extractive question answering, where models must predict answer spans rather than classify question-sentence pairs. For each encoder family, we compute clean-source SQuAD training dynamics, train on full, random, easy-ranked, ambiguous-ranked, and hard-ranked subsets, and evaluate robustness on adversarial-only AddSent and AddOneSent rows. Ambiguous-ranked 33% subsets beat random 33% subsets on exact-match accuracy in all 14 model-evaluation cells and on F1 in 12/14 cells. However, full-data training beats ambiguous-ranked training in 12/14 exact-match cells and 12/14 F1 cells, with mean full-minus-ambiguous gains of 3.40 EM and 4.10 F1. The most thoroughly replicated ELECTRA-small panel is also the main exception to the ambiguous-over-random F1 gains. These results show that clean-source cartography remains useful for reduced-data selection, but its OOD advantage does not carry over as a general replacement for full-data training in extractive adversarial QA.
Figures & tables
| Study / task formulation | Evaluation | Full | Random 33% | Ambiguous 33% | Amb.-Full |
|---|---|---|---|---|---|
| Swayamdipta et al. QNLI (sentence classification) | Adv. SQuAD-derived acc. | 81.7 | 78.3 | 83.9 | +2.2 |
| This work, extractive QA | AddSent EM | 55.64 | 49.05 | 51.90 | -3.74 |
| This work, extractive QA | AddOneSent EM | 55.22 | 48.85 | 51.93 | -3.29 |
| This work, extractive QA | AddSent F1 | 61.73 | 55.35 | 56.87 | -4.86 |
| This work, extractive QA | AddOneSent F1 | 61.74 | 55.16 | 56.94 | -4.80 |
| AddSent EM | AddOneSent EM | |||||
|---|---|---|---|---|---|---|
| Model | Full | Random | Ambiguous | Full | Random | Ambiguous |
| ELECTRA-small | 42.60 1.23 | 34.08 1.06 | 34.22 0.85 | 42.57 1.16 | 33.81 1.16 | 34.43 1.13 |
| BERT-base | 51.00 0.23 | 44.30 1.28 | 47.42 1.35 | 50.49 0.60 | 43.91 1.34 | 47.73 2.04 |
| ELECTRA-base | 62.57 0.78 | 54.85 1.60 | 59.45 1.39 | 62.35 0.32 | 54.36 1.92 | 59.81 1.20 |
| RoBERTa-base | 61.25 0.52 | 57.76 1.45 | 59.90 0.79 | 62.13 0.13 | 57.81 1.31 | 59.76 0.57 |
| DistilBERT | 46.08 1.14 | 38.34 1.41 | 40.08 1.22 | 45.70 1.48 | 37.78 1.27 | 40.24 1.27 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Logged value | Notes |
|---|---|---|
| Python | 3.11.15 | Runtime recorded in canonical environment logs. |
| Nix | 2.34.7 | Used to enter the system shell for local and GPU-server execution. |
| uv | 0.9.30 | Used to sync the Python environment from the checked-in project definition. |
| PyTorch | 2.3.1+cu121 | CUDA 12.1 wheel used for GPU runs. |
| Transformers | 4.41.2 | Hugging Face implementation used for encoder QA fine-tuning. |
| Datasets / Evaluate / Accelerate | 2.19.2 / 0.4.2 / 0.30.1 | Used for data handling, metrics support, and Trainer execution. |
| Setting | Value | Notes |
|---|---|---|
| Task | Extractive QA | Models predict start and end positions over SQuAD-style contexts. |
| Training epochs | 3 | Used for full, random, easy-ranked, ambiguous-ranked, and hard-ranked conditions. |
| Maximum sequence length | 384 | QA tokenization length for training and evaluation. |
| Learning rate | Fixed across panels; no evaluation-set tuning. | |
| Warmup ratio | 0.0 | Fixed across panels. |
| Weight decay | 0.0 | Fixed across panels. |
| Model | Random draws | Full/selected train runs | Random train runs | Total train runs | Eval cmds |
|---|---|---|---|---|---|
| ELECTRA-small | 10 | 12 | 30 | 42 | 126 |
| BERT-base | 10 | 12 | 30 | 42 | 126 |
| ELECTRA-base | 5 | 12 | 15 | 27 | 81 |
| RoBERTa-base | 5 | 12 | 15 | 27 | 81 |
| DistilBERT | 5 | 12 | 15 | 27 | 81 |
| ALBERT-base | 5 | 12 | 15 | 27 | 81 |
| Model family | Checkpoint used | Approximate parameter count |
|---|---|---|
| ELECTRA-small | google/electra-small-discriminator | 14M |
| BERT-base | bert-base-uncased | 110M |
| ELECTRA-base | google/electra-base-discriminator | 110M |
| RoBERTa-base | roberta-base | 125M |
| DistilBERT | distilbert-base-uncased | 67M |
| ALBERT-base | albert-base-v2 | 11M–12M |
| Model | GPU | Train cmds | Eval cmds | Train h | Eval h | Logged cmd h |
|---|---|---|---|---|---|---|
| ELECTRA-small | 1 RTX 4090 | 42 | 126 | 2.76 | 2.27 | 5.04 |
| BERT-base | 1 RTX 4090 | 42 | 126 | 5.18 | 2.97 | 8.16 |
| ELECTRA-base | 1 RTX 4090 | 27 | 81 | 4.80 | 1.76 | 6.57 |
| RoBERTa-base | 1 RTX 4090 | 27 | 81 | 4.74 | 1.78 | 6.52 |
| DistilBERT | 1 RTX 4090 | 27 | 81 | 2.70 | 1.57 | 4.28 |
| ALBERT-base | 1 RTX 4090 | 27 | 81 | 6.49 | 1.80 | 8.30 |
| Artifact | Source / citation | License or terms | Use in this work |
|---|---|---|---|
| SQuAD v1.1 | Rajpurkar et al. (2016) | CC BY-SA 4.0 | Clean-source training data and source examples for cartography. The grouped training map contains 87,599 examples. The benchmark is English reading comprehension over Wikipedia passages. |
| AddSent/AddOneSent | Jia and Liang (2017) | SQuAD-derived challenge data; original adversarial-squad code released under MIT | Robustness evaluation. We report adversarial-only rows and paired original/adversarial failure rates rather than blending clean and adversarial rows. |
| Pretrained encoders | Model papers cited in Section 3.3 | Model-card licenses: Apache-2.0 for BERT, ELECTRA, DistilBERT, and ALBERT; MIT for RoBERTa and DeBERTa-v3 | Initializations for extractive QA fine-tuning. We do not release modified model weights in the reproducibility package. |
| Experiment package | This work | Research artifact | Contains run configurations, environment definitions, command logs, metrics, subset files, raw source-run dynamics, and figure/table-generation scripts. Intended for research reproduction and audit, not deployment. |
| Model | Mean conf. | Mean var. | Mean corr. | |||
|---|---|---|---|---|---|---|
| ELECTRA-small | 0.367 | 0.121 | 0.531 | 30,735 | 17,371 | 39,493 |
| BERT-base | 0.510 | 0.157 | 0.639 | 38,793 | 19,234 | 29,572 |
| ELECTRA-base | 0.552 | 0.134 | 0.675 | 44,601 | 16,502 | 26,496 |
| RoBERTa-base | 0.537 | 0.128 | 0.664 | 44,024 | 15,873 | 27,702 |
| DistilBERT | 0.450 | 0.152 | 0.590 | 34,026 | 19,566 | 34,007 |
| ALBERT-base | 0.538 | 0.121 | 0.666 | 45,107 | 14,956 | 27,536 |
| Model | Easy ( 100%) | Amb. ( 50%, 100%) | Hard ( 50%) | Easy Amb. | Easy Hard | Amb. Hard |
|---|---|---|---|---|---|---|
| ELECTRA-small | 85.3 | 42.8 | 97.1 | 44.1 | 0.0 | 1.7 |
| BERT-base | 96.9 | 49.5 | 88.9 | 15.7 | 0.0 | 18.5 |
| ELECTRA-base | 99.8 | 44.2 | 84.2 | 9.6 | 0.0 | 26.6 |
| RoBERTa-base | 99.4 | 42.4 | 86.5 | 12.3 | 0.0 | 23.9 |
| DistilBERT | 90.3 | 49.3 | 93.5 | 27.6 | 0.0 | 8.0 |
| ALBERT-base | 99.6 | 40.5 | 86.0 | 12.5 | 0.0 | 24.8 |
| Eval set | All rows | Original | Adversarial |
|---|---|---|---|
| AddSent | 3560 | 1000 | 2560 |
| AddOneSent | 1787 | 1000 | 787 |
| Model | Map examples | 10K sample | Global subset size | Easy shown | Amb./Hard shown |
|---|---|---|---|---|---|
| ELECTRA-small | 87,599 | 10,000 | 29,170 | 3,280 | 3,355/3,348 |
| BERT-base | 87,599 | 10,000 | 29,170 | 3,309 | 3,304/3,338 |
| ELECTRA-base | 87,599 | 10,000 | 29,170 | 3,248 | 3,365/3,379 |
| RoBERTa-base | 87,599 | 10,000 | 29,170 | 3,332 | 3,284/3,349 |
| DistilBERT | 87,599 | 10,000 | 29,170 | 3,307 | 3,308/3,380 |
| ALBERT-base | 87,599 | 10,000 | 29,170 | 3,322 | 3,329/3,331 |
| Model | Full | Random | Easy | Ambiguous | Hard | |
|---|---|---|---|---|---|---|
| ELECTRA-small | 42.60 1.23 | 34.08 1.06 | 29.78 0.80 | 34.22 0.85 | 29.70 0.74 | +0.13 |
| BERT-base | 51.00 0.23 | 44.30 1.28 | 38.53 1.00 | 47.42 1.35 | 38.37 0.62 | +3.12 |
| ELECTRA-base | 62.57 0.78 | 54.85 1.60 | 49.45 1.90 | 59.45 1.39 | 48.49 0.09 | +4.61 |
| RoBERTa-base | 61.25 0.52 | 57.76 1.45 | 50.61 0.99 | 59.90 0.79 | 49.27 1.25 | +2.14 |
| DistilBERT | 46.08 1.14 | 38.34 1.41 | 32.30 0.57 | 40.08 1.22 | 35.26 1.26 | +1.74 |
| ALBERT-base | 55.40 1.04 | 49.77 1.50 | 41.84 1.40 | 52.90 2.06 | 43.53 1.73 | +3.14 |
| Model | Full | Random | Easy | Ambiguous | Hard | |
|---|---|---|---|---|---|---|
| ELECTRA-small | 42.57 1.16 | 33.81 1.16 | 29.56 1.18 | 34.43 1.13 | 28.29 0.95 | +0.62 |
| BERT-base | 50.49 0.60 | 43.91 1.34 | 39.35 1.51 | 47.73 2.04 | 35.96 0.77 | +3.82 |
| ELECTRA-base | 62.35 0.32 | 54.36 1.92 | 50.28 2.17 | 59.81 1.20 | 47.27 0.55 | +5.45 |
| RoBERTa-base | 62.13 0.13 | 57.81 1.31 | 51.29 0.60 | 59.76 0.57 | 48.84 1.41 | +1.95 |
| DistilBERT | 45.70 1.48 | 37.78 1.27 | 32.70 0.73 | 40.24 1.27 | 33.55 1.44 | +2.46 |
| ALBERT-base | 55.65 0.92 | 49.99 1.45 | 41.00 1.73 | 53.66 1.73 | 43.03 1.80 | +3.68 |
| Model | Full | Random | Easy | Ambiguous | Hard | |
|---|---|---|---|---|---|---|
| ELECTRA-small | 49.37 1.41 | 41.08 1.09 | 34.90 0.86 | 39.82 0.82 | 40.36 0.80 | -1.26 |
| BERT-base | 57.66 0.39 | 51.27 1.32 | 43.70 1.06 | 53.45 1.41 | 49.98 1.04 | +2.18 |
| ELECTRA-base | 68.37 1.21 | 61.25 1.70 | 53.78 1.80 | 65.27 1.53 | 60.15 0.35 | +4.02 |
| RoBERTa-base | 67.67 0.26 | 64.14 1.26 | 54.95 1.02 | 65.58 1.02 | 60.21 1.57 | +1.44 |
| DistilBERT | 53.15 1.17 | 45.61 1.33 | 37.64 0.90 | 46.31 1.27 | 47.02 1.39 | +0.71 |
| ALBERT-base | 61.69 0.78 | 56.03 1.50 | 46.56 1.48 | 58.05 1.98 | 53.94 1.45 | +2.02 |
| Model | Full | Random | Easy | Ambiguous | Hard | |
|---|---|---|---|---|---|---|
| ELECTRA-small | 49.10 1.39 | 40.92 1.13 | 34.55 1.07 | 39.89 1.01 | 39.43 0.91 | -1.04 |
| BERT-base | 57.10 0.99 | 50.47 1.40 | 44.19 1.66 | 53.50 2.14 | 47.47 1.15 | +3.03 |
| ELECTRA-base | 68.45 0.71 | 60.57 2.18 | 54.22 2.19 | 65.47 1.53 | 59.07 0.91 | +4.90 |
| RoBERTa-base | 68.23 0.09 | 63.95 1.30 | 55.42 0.36 | 65.33 0.74 | 59.82 1.53 | +1.38 |
| DistilBERT | 52.50 1.54 | 44.80 1.23 | 38.24 1.20 | 46.37 1.58 | 45.29 1.60 | +1.56 |
| ALBERT-base | 62.00 0.90 | 55.99 1.48 | 45.62 1.81 | 58.25 1.77 | 53.53 1.79 | +2.26 |
| AddSent failure rate | AddOneSent failure rate | |||||
|---|---|---|---|---|---|---|
| Model | Rand | Amb. | Rand | Amb. | ||
| ELECTRA-small | 68.0 | 67.9 | -0.1 | 53.3 | 54.0 | +0.7 |
| BERT-base | 59.8 | 57.6 | -2.3 | 43.5 | 40.5 | -3.0 |
| ELECTRA-base | 49.6 | 44.9 | -4.8 | 36.3 | 31.6 | -4.7 |
| RoBERTa-base | 43.9 | 40.5 | -3.3 | 31.7 | 30.4 | -1.3 |
| DistilBERT | 64.7 | 64.2 | -0.5 | 47.4 | 46.2 | -1.2 |
| SQuAD-dev EM | SQuAD-dev F1 | |||||
|---|---|---|---|---|---|---|
| Model | Full | Random | Ambiguous | Full | Random | Ambiguous |
| ELECTRA-small | 77.48 0.18 | 71.33 0.39 | 72.41 0.33 | 85.47 0.15 | 80.75 0.38 | 80.67 0.30 |
| BERT-base | 80.65 0.29 | 76.27 0.20 | 78.76 0.16 | 88.34 0.17 | 85.07 0.19 | 86.39 0.12 |
| ELECTRA-base | 85.29 0.07 | 82.77 0.37 | 84.51 0.19 | 91.95 0.08 | 90.07 0.24 | 91.17 0.11 |
| RoBERTa-base | 85.39 0.15 | 83.47 0.25 | 85.20 0.10 | 91.91 0.18 | 90.49 0.16 | 91.41 0.13 |
| DistilBERT | 77.54 0.12 | 71.77 0.38 | 73.92 0.31 | 85.77 0.08 | 81.34 0.32 | 82.25 0.13 |
| Run | Article | Question | Gold answer(s) | Original prediction | Adversarial prediction |
|---|---|---|---|---|---|
| ELECTRA-small full | Ctenophora | What does the Beroe eat? | other ctenophores; ctenophores | other ctenophores | hamsters |
| BERT ambiguous | Warsaw | Over how many species of trees can be found in the Saxon Garden? | 100; over 100 | 100 | 600 |
| ELECTRA-base ambiguous | Nikola Tesla | What was Galileo Ferraris? | physicist; Italian physicist | Italian physicist | Rosetta Lamborghinis |
| RoBERTa ambiguous | Normans | What kind of bureaucracy did the Normans institute? | meritocratic | meritocratic | deep |
| DeBERTa ambiguous | Teacher | What objective would be labeled as practical? | skill | skill | hamster |
| Condition | AddSent EM | AddSent F1 | AddOneSent EM | AddOneSent F1 |
|---|---|---|---|---|
| Full | 42.60 1.23 | 49.37 1.41 | 42.57 1.16 | 49.10 1.39 |
| Rand | 34.08 1.06 | 41.08 1.09 | 33.81 1.16 | 40.92 1.13 |
| Easy | 29.78 0.80 | 34.90 0.86 | 29.56 1.18 | 34.55 1.07 |
| Amb. | 34.22 0.85 | 39.82 0.82 | 34.43 1.13 | 39.89 1.01 |
| Hard | 29.70 0.74 | 40.36 0.80 | 28.29 0.95 | 39.43 0.91 |
| Condition | AddSent EM | AddSent F1 | AddOneSent EM | AddOneSent F1 |
|---|---|---|---|---|
| Full | 51.00 0.23 | 57.66 0.39 | 50.49 0.60 | 57.10 0.99 |
| Rand | 44.30 1.28 | 51.27 1.32 | 43.91 1.34 | 50.47 1.40 |
| Easy | 38.53 1.00 | 43.70 1.06 | 39.35 1.51 | 44.19 1.66 |
| Amb. | 47.42 1.35 | 53.45 1.41 | 47.73 2.04 | 53.50 2.14 |
| Hard | 38.37 0.62 | 49.98 1.04 | 35.96 0.77 | 47.47 1.15 |
| Condition | AddSent EM | AddSent F1 | AddOneSent EM | AddOneSent F1 |
|---|---|---|---|---|
| Full | 62.57 0.78 | 68.37 1.21 | 62.35 0.32 | 68.45 0.71 |
| Rand | 54.85 1.60 | 61.25 1.70 | 54.36 1.92 | 60.57 2.18 |
| Easy | 49.45 1.90 | 53.78 1.80 | 50.28 2.17 | 54.22 2.19 |
| Amb. | 59.45 1.39 | 65.27 1.53 | 59.81 1.20 | 65.47 1.53 |
| Hard | 48.49 0.09 | 60.15 0.35 | 47.27 0.55 | 59.07 0.91 |
| Condition | AddSent EM | AddSent F1 | AddOneSent EM | AddOneSent F1 |
|---|---|---|---|---|
| Full | 61.25 0.52 | 67.67 0.26 | 62.13 0.13 | 68.23 0.09 |
| Rand | 57.76 1.45 | 64.14 1.26 | 57.81 1.31 | 63.95 1.30 |
| Easy | 50.61 0.99 | 54.95 1.02 | 51.29 0.60 | 55.42 0.36 |
| Amb. | 59.90 0.79 | 65.58 1.02 | 59.76 0.57 | 65.33 0.74 |
| Hard | 49.27 1.25 | 60.21 1.57 | 48.84 1.41 | 59.82 1.53 |
| Condition | AddSent EM | AddSent F1 | AddOneSent EM | AddOneSent F1 |
|---|---|---|---|---|
| Full | 46.08 1.14 | 53.15 1.17 | 45.70 1.48 | 52.50 1.54 |
| Rand | 38.34 1.41 | 45.61 1.33 | 37.78 1.27 | 44.80 1.23 |
| Easy | 32.30 0.57 | 37.64 0.90 | 32.70 0.73 | 38.24 1.20 |
| Amb. | 40.08 1.22 | 46.31 1.27 | 40.24 1.27 | 46.37 1.58 |
| Hard | 35.26 1.26 | 47.02 1.39 | 33.55 1.44 | 45.29 1.60 |
| Condition | AddSent EM | AddSent F1 | AddOneSent EM | AddOneSent F1 |
|---|---|---|---|---|
| Full | 55.40 1.04 | 61.69 0.78 | 55.65 0.92 | 62.00 0.90 |
| Rand | 49.77 1.50 | 56.03 1.50 | 49.99 1.45 | 55.99 1.48 |
| Easy | 41.84 1.40 | 46.56 1.48 | 41.00 1.73 | 45.62 1.81 |
| Amb. | 52.90 2.06 | 58.05 1.98 | 53.66 1.73 | 58.25 1.77 |
| Hard | 43.53 1.73 | 53.94 1.45 | 43.03 1.80 | 53.53 1.79 |
| Condition | AddSent EM | AddSent F1 | AddOneSent EM | AddOneSent F1 |
|---|---|---|---|---|
| Full | 74.67 0.90 | 80.03 1.14 | 75.01 0.73 | 80.11 0.80 |
| Rand | 71.84 0.94 | 78.15 0.83 | 71.83 1.05 | 77.84 1.09 |
| Easy | 60.51 2.19 | 64.85 2.24 | 61.33 2.27 | 65.60 2.36 |
| Amb. | 74.99 1.11 | 80.33 1.29 | 75.35 1.44 | 80.35 1.65 |
| Hard | 59.99 0.26 | 73.35 0.29 | 58.20 0.77 | 72.11 1.04 |
Training Data Selection
Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.
See all 196 papers