NER-D: A cross-domain benchmark for German NER
Organizations: Computational Linguistics, Department of Linguistics, Bielefeld University · Center for Cognitive Interaction Technology, Faculty of Technology, Bielefeld University
Abstract
We present NER-D, a cross-domain benchmark for German NER comprising 13 datasets spanning 9 domains, including General/Politics, Wikipedia, Legal, Medical, Historical, Financial, Agriculture and Archaeology. We examine the performance of both proprietary LLMs and state-of-the-art small open-source models using normal prompting and further examine the performance gains achievable via in-context learning and LoRA fine-tuning. Our results show that without any sort of optimization a proprietary model (GPT-5.4 mini) outperforms the smaller open-source models on 6 out of 13 datasets, but LoRA fine-tuning enables the smaller models to substantially outperform the proprietary baseline. We further analyze the influence of the domain and the tagset used in each task on the performance achieved in the different evaluation settings. Our benchmark will be available as a resource for the development of future German NER methods on both general and specialized domains.\footnote{We make our code, data, and models publicly available upon acceptance of this paper.
Figures & tables
| Dataset | Domain | Train/Val/Test | Labels |
|---|---|---|---|
| Europarl-NER | General (politics) | 639 / 80 / 80 | 4: PER, ORG, LOC, MISC |
| SwissNER | General (Swiss) | - / - / 172 | 13: Canton, City, Company, Country, Event, Lake, Landmark, Mountain, Organization, Person, Region, River, Village |
| NoSta-D | Wikipedia | 24000 / 2200 / 5100 | 12: PER, PERpart, PERderiv, LOC, LOCpart, LOCderiv, ORG, ORGpart, ORGderiv, OTH, OTHpart, OTHderiv |
| WikiBiography | Wikipedia | 19046 / 2337 / 2162 | 5: LOC, ORG, PERSON, TEMP, UNDEF_NE |
| German LER | Legal | 22754 / 2732 / 2893 | 16: PER, RR, AN, LD, ST, STR, ORG, UN, INN, GRT, MRK, GS, VO, EUN, VS, VT |
| GerPS | Legal | 10718 / 595 / 596 | 10: Mitwirkender, Handlungsgrundlage, Ergebnisempfänger, Hauptakteur, Aktion, Dokument, Signalwort, Frist, Bedingung, Datenfeld |
| Ministral 8B | Qwen 3.5 9B | Gemma 4 12B | GPT-5.4 mini | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot |
| FAIRagro | 0.492 | 0.523 | 0.731 | 0.582 | 0.692 | 0.857 | 0.484 | 0.542 | 0.769 | 0.441 |
| Archaeo NER | 0.412 | 0.450 | 0.679 | 0.471 | 0.487 | 0.681 | 0.516 | 0.534 | 0.671 | 0.476 |
| CO-Fun | 0.652 | 0.698 | 0.869 | 0.639 | 0.692 | 0.844 | 0.619 | 0.711 | 0.829 | 0.614 |
| WikiBiography | 0.434 | 0.428 | 0.856 | 0.555 | 0.590 | 0.840 | 0.539 | 0.618 | 0.837 | 0.584 |
| Europarl-ner | 0.328 | 0.387 | 0.688 | 0.460 | 0.540 | 0.754 | 0.648 | 0.708 | 0.761 | 0.603 |
| Ministral 8B | Qwen 3.5 9B | Gemma 4 12B | GPT-5.4 mini | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot |
| FAIRagro | 0.677 | 0.677 | 0.808 | 0.727 | 0.731 | 0.857 | 0.710 | 0.712 | 0.846 | 0.610 |
| Archaeo NER | 0.576 | 0.604 | 0.788 | 0.607 | 0.627 | 0.784 | 0.649 | 0.669 | 0.760 | 0.617 |
| CO-Fun | 0.840 | 0.856 | 0.909 | 0.780 | 0.818 | 0.889 | 0.793 | 0.866 | 0.870 | 0.768 |
| WikiBiography | 0.571 | 0.571 | 0.873 | 0.635 | 0.677 | 0.857 | 0.600 | 0.680 | 0.855 | 0.646 |
| Europarl-ner | 0.463 | 0.495 | 0.766 | 0.633 | 0.657 | 0.783 | 0.745 | 0.764 | 0.851 | 0.679 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Ministral 8B | Qwen 3.5 9B | Gemma 4 12B | GPT-5.4 mini | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot |
| FAIRagro | 0.517 | 0.456 | 0.650 | 0.597 | 0.637 | 0.717 | 0.462 | 0.506 | 0.627 | 0.487 |
| Archaeo NER | 0.367 | 0.398 | 0.623 | 0.450 | 0.447 | 0.642 | 0.489 | 0.521 | 0.636 | 0.466 |
| CO-Fun | 0.685 | 0.724 | 0.875 | 0.666 | 0.715 | 0.853 | 0.661 | 0.739 | 0.836 | 0.649 |
| WikiBiography | 0.359 | 0.357 | 0.730 | 0.433 | 0.450 | 0.633 | 0.442 | 0.477 | 0.679 | 0.457 |
| Europarl-ner | 0.356 | 0.403 | 0.719 | 0.412 | 0.496 | 0.779 | 0.698 | 0.720 | 0.788 | 0.633 |
| Ministral 8B | Qwen 3.5 9B | Gemma 4 12B | GPT-5.4 mini | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot | 5-shot | LoRA | 0-shot |
| FAIRagro | 0.723 | 0.629 | 0.723 | 0.737 | 0.687 | 0.717 | 0.748 | 0.730 | 0.752 | 0.660 |
| Archaeo NER | 0.504 | 0.526 | 0.711 | 0.560 | 0.560 | 0.723 | 0.588 | 0.625 | 0.705 | 0.578 |
| CO-Fun | 0.844 | 0.860 | 0.909 | 0.783 | 0.821 | 0.891 | 0.805 | 0.868 | 0.871 | 0.774 |
| WikiBiography | 0.472 | 0.472 | 0.748 | 0.505 | 0.526 | 0.652 | 0.499 | 0.534 | 0.698 | 0.515 |
| Europarl-ner | 0.526 | 0.545 | 0.799 | 0.616 | 0.628 | 0.800 | 0.792 | 0.800 | 0.878 | 0.716 |
Named Entity Recognition
Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.
See all 114 papers