An Extreme Multi-label Text Classification (XMTC) Library Dataset: What if we took "Use of Practical AI in Digital Libraries" seriously?

Jennifer D'Souza; Sameer Sadruddin; Maximilian Kähler; Andrea Salfinger; Luca Zaccagna; Francesca Incitti; Lauro Snidaro; Osma Suominen

cs.CLMar 11, 2026

An Extreme Multi-label Text Classification (XMTC) Library Dataset: What if we took "Use of Practical AI in Digital Libraries" seriously?

Authors: Jennifer D'Souza, Sameer Sadruddin, Maximilian Kähler, Andrea Salfinger, Luca Zaccagna, Francesca Incitti, Lauro Snidaro, Osma Suominen

Organizations: TIB Leibniz Information Centre for Science and Technology, Germany · Deutsche Nationalbibliothek, Germany · University of Udine, Italy · National Library of Finland, Finland

Abstract

Subject indexing is vital for discovery but hard to sustain at scale and across languages. We release a large bilingual (English/German) corpus of catalog records annotated with the Integrated Authority File (GND), plus a machine-actionable GND taxonomy. The resource enables ontology-aware multi-label classification, mapping text to authority terms, and agent-assisted cataloging with reproducible, authority-grounded evaluation. We provide a brief statistical profile and qualitative error analyses of three systems. We invite the community to assess not only accuracy but usefulness and transparency, toward authority-anchored AI co-pilots that amplify catalogers' work.

Figures & tables

Type/Split	en	de	Total
Article	1,738 (1.27)	10 (0.01)	1,748 (1.28)
train	1,103 (0.81)	7 (0.01)	1,110 (0.81)
dev	212 (0.16)	2 (0.00)	214 (0.16)
test	423 (0.31)	1 (0.00)	424 (0.31)
Book	44,877 (32.86)	55,704 (40.79)	100,581 (73.65)
train	30,545 (22.37)	35,899 (26.29)	66,444 (48.65)

Table 1: Record counts and relative shares (%) by split, type, and language for the updated dataset. Blue: type-level shares; Cyan: split-level (Total column); Gray: totals. Cells

\geq

15% are shaded.

Record (title) (Linked)	Lang	Type	Domains	Subjects
Chapter 86: Explaining the Comparative Statics in Step-Level Public Good Games	en	Article	Economics	Experiment; Wirtschaftswissenschaften; Wirtschaftsforschung; Methodologie; Experimentelle Wirtschaftsforschung
Das Beil von Wandsbek: Roman; 1938–1943	de	Book	Literature Studies	Nationalsozialismus
Window on Freedom: Race, Civil Rights, and Foreign Affairs, 1945–1988	en	Book	Social Sciences; History	Außenpolitik; Rassendiskriminierung; Menschenrecht; Schwarze
Sicherung des Familieneinflusses in Familienunternehmen: Symposium … (6./7. Okt. 2016)	de	Conf.	Law	Einfluss; Familienbetrieb; Familie
Charge Carrier Recombination and Open Circuit Voltage in Organic Solar Cells: From Bilayer Model Systems to Hybrid Multi-junctions	en	Thesis	Electrical engineering; Physics	Organische Solarzelle; Reihenschaltung; Mehrschichtsystem; Fluor; Donator (Chemie); Dotierung; Rekombination

Table 2: Representative examples of multi-domain, multi-subject annotations from the library records dataset. Only the title metadata are shown for compactness; links point to the full JSON-LD records.

Metric	Train–Dev	Train–Test	Dev–Test
Jaccard	0.3687	0.3608	0.3668
Tanimoto	0.2064	0.2530	0.3771

Table 3: Jaccard and Tanimoto scores showing subject overlap across dataset splits.

Model	View	N( $\geq$ 1)	N( $\geq$ 2)	Mean	Max
Gemma-300m	name	$\sim$ 19.8K	$\sim$ 4.0K	0.15	83
	context	$\sim$ 15.7K	$\sim$ 3.6K	0.13	75
E5-small	name	$\sim$ 195.7K	$\sim$ 188.1K	38.77	1.49K
	context	$\sim$ 165.8K	$\sim$ 147.0K	24.62	800
Jina-v2-DE	name	$\sim$ 15.0K	$\sim$ 2.8K	0.11	36
	context	$\sim$ 14.5K	$\sim$ 3.7K	0.13	53

Table 4: Semantic-similarity graph at cosine

\geq 0.90

(topK

=50

N=203{,}763

subjects. N(

\geq

1): count of subjects that have at least one similar neighbor; N(

\geq

2): subjects with at least two similar neighbors (stronger polysemy signal).

Team Name	nDCG@k
	k=5	k=10	k=15	k=20
System 1	0.3639	0.3977	0.4143	0.4247
System 2	0.4919	0.4880	0.4879	0.4879
System 3	0.6020	0.6391	0.6560	0.6652

Table 5: nDCG@k scores at different ranked cutoffs for three systems on our subject indexing dataset.

Figure 2: nDCG@5 scores by the five record types.

Figure 3: nDCG@5 scores by the two languages.

Panel of Figure 4 — Figure 4: nDCG@k scores, where k=5,10,15,20, for qualitative evaluation on 10 records per five domains. For this exercise, subject predictions were manually labeled Y and I by subject specialists at the library.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Error Category	Absolute Count
FNM	79
FNC	41
False Negatives (total)	120

Appendix

Table 6: Distribution of false negatives for 250 subject suggestions from System 1 (5 predicted labels per test set document) on 50 test-set documents with 161 gold-standard subjects.

Prediction	R	Rank	Sim.	Eval.	Error Category
Elektronische Gegenmaßnahme [electronic countermeasure] gnd:1082384615	A	1	0.687	N	FPAR
Störsender [jammer] gnd:4273774-6	A	2	0.687	Y	– (TP, Y)
Radar [radar] gnd:4176765-2	A	3	0.687	Y	– (Y)
Sonar [sonar] gnd:4181785-0	A	4	0.652	N	FPAR
Signalverarbeitung [signal processing] gnd:4054947-1	A	5	0.652	Y	– (Y)
AWACS [airborne warning and control system] gnd:4309079-5	O	22	0.406		OROR

Appendix

Table 7: System 1’s predictions for the test set record with ID 3A1007389885 , “Recent advancements in airborne radar signal processing: emerging research and opportunities” , dissecting which reasoning branch ( R ) identified each predicted label – analogical (A) or ontological (O) reasoning. The first five results are the top-5 predictions returned by the fusion system after merging the results from both A and O branches; the last row shows an actual ground-truth subject excluded from the retrieved list due to being outranked. Rank denotes the ranking of each subject within each reasoning branch (analogical vs. ontological), Sim. shows the cosine similarity to the query document in embedding space determining the ranking, and Eval. lists the human expert’s judgment of the predicted label.

LLM-suggested term	Mapped GND term	Error Category
Signalverarbeitung [Signal processing]	Signalverarbeitung gnd:4054947-1 [Signal processing]	FP, but relevant
Jamming [Jamming]	Störsender gnd:4273774-6 [jamming transmitter]	TP
Spoofing [Spoofing]	betrügen gnd:4554780-4 [deceive]	MEU
Luftfahrtradar [aviation radar]	Wetterradar gnd:4270420-0 [Weather radar]	MEU

Appendix

Table 8: System 2 output for the record-ID 3A1007389885 “Recent advancements in airborne radar signal processing : emerging research and opportunities” with error category annotation.

Gold-standard term	GND identifier
Störsender [jamming transmitter]	gnd:4273774-6
AWACS [airborne warning and control system]	gnd:4309079-5
Raum-Zeit-Signalverarbeitung [space-time signal processing]	gnd:4834654-8
Bordradar [on board radar]	gnd:4456131-3
Zielerkennung [target recognition]	gnd:4190792-9
Täuschung (Militär) [deception (military)]	gnd:4184333-2

Appendix

Table 9: Gold-standard terms for the record-ID 3A1007389885 .

Error Category	Absolute Count
FNM	68
FNC	28
False Negatives (total)	96

Appendix

Table 10: Distribution of false negatives for 176 subject suggestions from System 2 on 50 test-set documents with 162 gold-standard subjects.

Prediction	GND identifier	Eval.	Error Cat.
Kreditrisiko [credit risk]	gnd:4114309-7	Y	FP, but relevant
Kreditmarkt [credit market]	gnd:4073788-3	Y	FP, but relevant
Zinsstrukturtheorie [term structure theory]	gnd:4117720-4	Y	FP, but relevant
Modellierung [modelling]	gnd:4170297-9	I	FP, technically correct
Semimartingal [semimartingale]	gnd:4180967-1	Y	TP

Appendix

Table 11: System 3 top-5 output for the record-ID 3A168734406X “Dynamic term structure modeling beyond the paradigm of absolute continuity” with expert evaluations and error category annotation.

Gold-standard subject	GND identifier	Freq.	Error Cat.
Arbitrage [arbitrage]	gnd:4002820-3	0	FNM
Semimartingal [semimartingale]	gnd:4180967-1	4	TP
Ausfallrisiko [default risk]	gnd:4205942-2	4	FNM5
Zinsstruktur [term structure]	gnd:4067855-6	1	FNC
HJM-Modell [HJM model]	gnd:4642940-2	1	FNM5

Appendix

Table 12: Gold-standard subjects for the record-ID 3A168734406X with train-set frequencies and error category annotation.

Figure 5: Train-frequency distribution (binned) for the System 3 predictions, split into true positives and three false-negative subtypes.

Library and Information Science

Library and information science studies how information is organized, preserved, discovered, and used across libraries, archives, and digital systems.

See all 19 papers

Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature
Jul 16, 2026Maximilian Kähler, Katja Konermann, Lisa Kluge +1Long-Tailed Classification Multi-Label Classification
Deutsche Nationalbibliothek Leipzig/ Frankfurt am Main, Germany
A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing
May 5, 2026Eric H. C. ChowLibrary and Information Science LLM Agents
School of Humanities, The University of Hong Kong
Automatic Thematic Indexing of Large Literary Corpora: A Machine Learning Approach to Voltaire's Complete Works
Jul 10, 2026Miguel Arana-Catania, Gillian Pink, Glenn RoeComputational Literary Studies Multi-Label Classification
University of Oxford, UK

Jul 16, 2026J/K move · Enter open · S save

Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature

Maximilian Kähler, Katja Konermann, Lisa Kluge, Markus Schumacher

Deutsche Nationalbibliothek Leipzig/ Frankfurt am Main, Germany

Long-Tailed Classification Multi-Label Classification Library and Information Science Text Generation

Open paper AskPDF arXiv

With a large controlled vocabulary as the label set, the task of automated subject indexing in a library can be understood as a multi-label classification task. If the set of subject terms is large, the problem fits the Extreme Multi-Label Classification (XMLC) objective. In this study, we apply a selection of specialised supervised XMLC methods to the test case of subject indexing contemporary German scientific literature, collected at the German National Library (DNB). We contrast these results by including a classical lexical matching baseline and three of our own recently developed LLM-based methods into the benchmark. Algorithms are evaluated and compared in several metrics. This includes binary relevance comparisons with previously indexed material, as well as graded relevance ratings by professional subject librarians. A challenge for all methods is to reliably make suggestions from the long tail of the subject vocabulary. We find that supervised XMLC algorithms relying on transformer-based dense features give best results in terms of overall binary relevance metrics. However, focusing on graded relevance and performance in the long tail of our subject vocabulary, the LLM-based generative methods give better results, making them a promising alternative for future productive use.

Abstract

Library and Information Science

Explore similar work

Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature

A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing

Automatic Thematic Indexing of Large Literary Corpora: A Machine Learning Approach to Voltaire's Complete Works