Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining

Mahmoud Amiri; Jamile Mohammad Jafari; Sara Mostafapour; Thomas Bocklitz

cs.DBApr 14, 2026

Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining

Authors: Mahmoud Amiri, Jamile Mohammad Jafari, Sara Mostafapour, Thomas Bocklitz

Organizations: Leibniz Institute of Photonic Technology, Member of Leibniz Health Technologies, Member of the Leibniz Centre for Photonics in Infection Research (LPI), Albert-Einstein-Strasse 9, 07745 Jena, Germany. · Institute of Physical Chemistry (IPC) and Abbe Center of Photonics (ACP), Friedrich Schiller University Jena, Member of the Leibniz Centre for Photonics in Infection Research (LPI), Helmholtzweg 4, 07743 Jena, Germany

Abstract

We present Lit2Vec, a reproducible workflow for constructing and validating a chemistry corpus from the Semantic Scholar Open Research Corpus using conservative, metadata-based license screening. Using this workflow, we assembled an internal study corpus of 582,683 chemistry-specific full-text research articles with structured full text, token-aware paragraph chunks, paragraph-level embeddings generated with the intfloat/e5-large-v2 model, and record-level metadata including abstracts and licensing information. To support downstream retrieval and text-mining use cases, an eligible subset of the corpus was additionally enriched with machine-generated brief summaries and multi-label subfield annotations spanning 18 chemistry domains. Licensing was screened using metadata from Unpaywall, OpenAlex, and Crossref, and the resulting corpus was technically validated for schema compliance, embedding reproducibility, text quality, and metadata completeness. The primary contribution of this work is a reproducible workflow for corpus construction and validation, together with its associated schema and reproducibility resources. The released materials include the code, reconstruction workflow, schema, metadata/provenance artifacts, and validation outputs needed to reproduce the corpus from pinned public upstream resources. Public redistribution of source-derived text and broad text-derived representations is outside the scope of the general release. Researchers can reproduce the workflow by using the released pipeline with publicly available upstream datasets and metadata services.

Figures & tables

Abbr.	Meaning	Abbr.	Meaning
ANN	Approximate nearest neighbor	MAG	Microsoft Academic Graph
BART	Bidirectional and Auto-Regressive Transformers	MLP	Multilayer perceptron
BERTScore	Bidirectional Encoder Representations from Transformers Score	MMR	Maximal Marginal Relevance
CC	Creative Commons	NLP	Natural language processing
CC0	Creative Commons Zero	OA	Open access
CC-BY	Creative Commons Attribution	RAG	Retrieval-augmented generation

Table 1

Figure 1: Overview of the Lit2Vec workflow. The primary pipeline constructs a legally screened chemistry corpus from S2ORC through domain filtering, text normalization, chunking, embedding, and license screening. Optional enrichment modules are applied to records with eligible abstracts to generate TL;DR summaries and subfield annotations.

Step	Key Processing	Records Retained	% of Reference Stage
1. Data Acquisition	Download JSONL dump	200,000,000	—
2. Domain Filtering	Restrict to fields_of_study = Chemistry	18,621,044	9.3% of S1
3. Abstract Alignment	Join with abstract corpus via corpus_id	8,660,979	46.5% of S2
4. Full-Text Alignment	Join with full-text corpus via corpus_id	1,074,638	5.8% of S2
5. Markdown Structuring	Parse annotations; normalize section headers	966,946	89.9% of S4
6. Chunking	Token-aware text segmentation	963,697	99.7% of S5

Table 2: Overview of the Lit2Vec pipeline, showing inputs, key processing steps, retained records, and retention rates relative to explicitly defined reference stages. Steps 9 and 10 (optional enrichment) were applied only to the subset of records with eligible abstracts.

Panel of Figure 2 — Figure 2: (a) Overlap between available full-text and abstract records for chemistry-labeled S2ORC papers. Most records contain only abstracts (43.0%), while 4.9% contain both and 1.0% contain only full text. (b) Coverage of license metadata sources (Unpaywall, OpenAlex, Crossref) for the chemistry full-text subset (N = 1,074,638). Overall, 89.9% of papers have license information from at least one source, with 49.5% covered by all three.

Validation Area	Key Checks	Summary of Results
Schema & Structure	JSON schema compliance, field typing, embedding format	79% pass (460,397); 21% did not satisfy full enrichment/schema completeness criteria (122,286) — mainly due to missing abstracts or abstract-dependent fields omitted for abstracts below the embedding eligibility threshold
Metadata	Completeness of title, authors, venue, year, DOI, license	73% pass (423,519); 27% warn (159,164) — most warnings from missing venue, or publication date.
Subfield Labels	Vocabulary match, valid confidence scores	79% pass (460,397); 21% did not satisfy full enrichment/schema completeness criteria (122,286) — primarily attributable to missing or short upstream abstracts under the abstract-embedding eligibility threshold.
Text Quality	Length limits, Unicode validity, abstract–full-text alignment	68% pass (398,064); 32% warn (184,619) — $\sim$ 72% strong alignment; most records free of quality flags.
Chunking	Token size limits, Unicode integrity, paragraph–embedding mapping	61% pass (354,064); 39% warn (228,619) — warnings only for naturally short final chunks.
Embeddings	Presence, size, reproducibility	100% pass (582,683) — perfect cosine similarity agreement.

Table 3: Technical validation summary showing pass, warning, and non-complete rates (relative to the stated criteria) for each validation area.

Figure 3: Schema validation results for 582,683 records in the chemistry full-text subset. Left: Pass/not-fully-complete distribution under schema/enrichment criteria, with non-complete cases broken down by the number of errors per record. Center: Most frequent non-completeness drivers are (i) short abstract embedding size (82,711 occurrences), (ii) missing abstract-embedding key (39,575), and (iii) missing predicted subfield annotation (39,575). Right: Normalized co-occurrence matrix showing two tight clusters: empty abstracts always co-occur with short abstract embedding size, while short abstracts ( $<\!1000$ characters) always co-occur with both missing abstract-embedding keys and missing predicted subfield annotations. These non-complete cases are primarily attributable to missing or short upstream abstracts under the abstract-length policy and are not indicative of embedding instability or licensing conflicts.

Figure 4: Metadata validation results for 582,683 records in the chemistry full-text subset. Left: Distribution of validation outcomes by category, showing that 72.7% of records pass all checks. The most common warnings are missing venue information (15.5%), missing open-access license metadata (10.4%), and missing publication date (1.0%). Less frequent issues include malformed author fields (0.3%), missing authors (0.2%), and short titles ( $<5$ characters, rare). Right: Histogram of the number of warnings per record, with most affected records containing only a single warning.

Figure 5: Distribution of the number of predicted disciplinary subfield labels per document for records that passed schema validation. Most documents are assigned two subfields, followed by those with zero or one label, and a smaller portion with three labels. This reflects the multi-label nature of the classifier, which allows documents to be categorized into multiple overlapping chemistry subfields.

Figure 6: Character length distributions for abstracts ( top ) and full texts ( bottom ) in the chemistry full-text subset, shown on a log scale. For abstracts, the median length is 1,296 characters, with 83,877 flagged as too short ( $<100$ characters) and 82,711 having zero characters. For full texts, the median length is 29,077 characters, with no zero-length cases. Dashed vertical lines indicate quartiles and mean values.

Figure 7: Distribution of ROUGE-1 recall alignment scores between abstracts and full texts in the chemistry full-text subset, grouped into bins of width 0.05. Most documents have high alignment, with the largest peak in the 0.95–1.0 range. Bars are color-coded by text validation flags, such as short abstracts, corrupted characters, low whitespace ratio, low American Standard Code for Information Interchange (ASCII) ratio, low sentence count, or missing/invalid abstract embeddings. The shaded area on the left indicates records with ROUGE-1 recall scores below 0.5.

Figure 8: Paragraph-level chunk validation for the chemistry full-text subset. First row: Validation status showing that 61% of documents pass without issues, while 39% contain short chunks. Second row: All affected documents contain exactly one short chunk. Third row: All short chunks occur in the final chunk of the document. Forth row: Distribution of mean chunk token length per document, with a median of 167 tokens and most values falling between 162 (Q1) and 171 (Q3).

Figure 9: Embedding reproducibility analysis for the chemistry full-text subset. Left: Distribution of mean cosine similarity between regenerated embeddings and their stored counterparts, with values tightly clustered around 1.0, indicating high reproducibility. Right: Distribution of maximum cosine drift (largest per-token deviation), showing that even the largest observed differences are on the order of $10^{-7}$ , confirming numerical stability across regeneration.

Figure 10: License validation results for 582,683 chemistry full-text records. Left: Sources from which license information was resolved, showing that most records (62.6%) were resolved by agreement across all three metadata sources (Crossref, Unpaywall, OpenAlex), followed by Crossref+Unpaywall (24.6%), Unpaywall+OpenAlex (12.2%), and Crossref+OpenAlex (0.6%). Right: Distribution of resolved license types, with CC-BY dominating (87.5%), followed by CC-BY-NC (10.7%), CC-BY-NC-SA (0.9%), CC-BY-SA (0.5%), and public domain (0.4%).

Figure 11: Relationship between abstract length and machine-generated summary evaluation scores for records with available summaries. Lines show binned mean scores with 95% confidence intervals for BERTScore_F1, ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum. The histogram (grey bars) indicates the number of abstracts per length bin, while the red gradient shows the proportion of abstracts exceeding the model’s input limit ( $\sim$ 1024 tokens $\approx$ 3500–4500 characters, dashed vertical line). Scores generally peak at shorter lengths and decline as abstracts become longer, particularly beyond the model input limit.

Figure 12: Temporal trends in SERS and Raman spectroscopy usage across 582,683 full-text chemistry papers in Lit2Vec. (Top) Document counts and smoothed shares for gold and silver nanoparticle use in SERS. (Bottom) Document counts and normalized shares for 785 nm and 532 nm excitation in Raman experiments. Trend analysis uses paragraph-level filtering and metadata-driven grouping, without fine-tuned models.

Role	Mahmoud A.	Thomas B.	Sara M.	Jamile M.J.
Conceptualization	X
Methodology	X	X
Software	X
Validation	X		X	X
Formal Analysis	X
Investigation	X

Table 4: Author contributions according to the CRediT taxonomy.

Corpus	Description	Limitations
CHEMDNER [ 16 ]	84K mentions in 10K PubMed abstracts; strong NER benchmark.	Abstracts only; limited license; no embeddings.
CDR [ 18 ]	1.5K abstracts with chemical–disease relations; expert-annotated.	Small; abstracts only; no embeddings.
NLM-Chem [ 19 ]	Full-text NER in PMC; good chemical coverage.	NER-only; no embeddings or structure.
ChEMU 2020 [ 20 ]	1.5K patent segments with NER/reaction events.	Small; patent-focused; no embeddings.
ChemRxivQuest [ 21 ]	970 QA pairs from 155 preprints in 17 subfields.	Small; QA-only; no full-text.
ChemNLP [ 22 ]	1M abstracts + 100K PubChem records for NLP.	No full-text; mixed licensing; no embeddings.

Table 5: Comparison of chemistry and general scientific corpora by scope and features.

Model	R-1	R-2	R-L	ROUGE-Lsum	BERT P	BERT R	BERT F1	Sec.
lit2vec_tldr_finetuned	56.38	31.00	44.28	45.64	90.81	92.14	91.46	$255.16$
distilbart_cnn_12_6_zeroshot	$37.67$	$15.57$	$26.68$	$30.54$	$87.14$	$88.02$	$87.57$	$318.94$
bart_large_cnn	$37.00$	$15.02$	$25.90$	$30.01$	$87.12$	$87.80$	$87.44$	$402.62$
lexrank	$35.65$	$14.78$	$24.73$	$28.57$	$85.74$	$88.79$	$87.22$	$15.05$
lead3	$33.74$	$13.79$	$23.53$	$27.44$	$85.45$	$88.06$	$86.71$	13.01
scibert_extractive	$33.74$	$13.79$	$23.53$	$27.44$	$85.45$	$88.06$	$86.71$	$24.07$

Table 6: Test-set performance across abstractive and extractive summarization models. ROUGE and BERTScore metrics are reported as percentages (higher is better). Sec. indicates total evaluation runtime on a single NVIDIA GeForce RTX 3090 under the default generation settings used by the training script.

Figure 13: Analysis of predicted research subfields from document-level classification. Top row: (left) counts of all predicted subfields across documents and (right) counts of only the top-confidence subfield per document. Bottom row: distribution of prediction scores for each subfield when it appears as (left) the top-predicted category and (right) the second-predicted category (if applicable). Subfield names are abbreviated (legend) for clarity.

Label	Precision	Recall	F1-score	Support
Catalysis	0.83	0.78	0.80	197
Organic Chemistry	0.85	0.60	0.70	245
Polymer Chemistry	0.80	0.65	0.72	120
Inorganic Chemistry	0.84	0.62	0.71	203
Materials Science	0.82	0.78	0.80	917
Analytical Chemistry	0.87	0.60	0.71	633

Table 7: Per-label classification report on the test set (threshold

\tau=0.5

). Macro average is the unweighted mean across labels; micro average aggregates decisions across labels; weighted average weights by label support; samples average is the average across instances.

Figure 14: Label support in the training set (bars, left axis) versus F1-score on the test set (red line, right axis) for each subfield. Frequent subfields such as Biochemistry and Medicinal Chemistry achieve high F1-scores, whereas underrepresented subfields including Chemical Engineering and Supramolecular Chemistry remain challenging.

Field	Type	Description
schema_version	string	Schema version (e.g., ”1.0”).
corpus_id	integer	Primary identifier (S2ORC CorpusId) for traceability and file naming.
metadata	object	Bibliographic metadata from S2ORC including title, authors, venue, year, URLs, and external IDs; see Table 7.8 .
abstract	string	Original abstract text from S2ORC.
fulltext	string	Reconstructed full text (plain text).
paragraphs	array[string]	Paragraph-level text units used for retrieval (aligned with embeddings by index).

Table 8: Top-level fields for each JSON record. Complete nested field dictionaries and a machine-readable schema are provided in the Supplementary Information.

Validation Rule	Description	Flag (on failure)
Top-level Required Fields	Must include: schema_version, corpus_id, metadata, abstract, paragraphs, embeddings, tldr, abstract_embedding, predicted_subfield.	missing_¡field¿ — Raised if any required field is missing (e.g., missing_abstract).
Additional Properties	Extra fields not defined in the schema are disallowed at all levels (top-level and nested).	additional_property_¡field¿ — Raised when unexpected field is present (e.g., additional_property_validation_info).
Type Enforcement	Fields must conform to expected types: strings, integers, booleans, arrays, or objects.	type_mismatch_¡field¿ — Type does not match schema definition (e.g., string provided where object expected).
Fixed-Length Arrays	embeddings and abstract_embedding must be numeric arrays of exactly length 1024.	too_short_¡field¿ or too_long_¡field¿ — Array length is incorrect (e.g., too_short_abstract_embedding).
Paragraph Key Pattern	Keys in paragraphs and embeddings must match regex ^\d+P\d+$ .	pattern_violation_¡field¿ — Raised when key format is invalid (e.g., 1Para2 ).
Field Format Constraints	Fields like metadata.url must be valid URIs; publicationdate must be ISO 8601 date.	invalid_format_¡field¿ — Raised for malformed dates or URLs (e.g., invalid_format_metadata_url).

Table 9: Schema Validation Rules and Diagnostic Flags for Lit2Vec-Style Records

Example Failing Record	Example Passing Record
"schema_validation": { "status": "fail", "summary": "1 schema validation error(s) found.", "metrics": { "validation_errors": [ { "message": "[] is too short", "field": "abstract_embedding", "schema_path": "properties/abstract_embedding/minItems", "validator": "minItems" } ], "flags": { "too_short_abstract_embedding": true } }, "checked_at": "2025-07-30T23:41:30.136585" }	"schema_validation": { "status": "pass", "summary": "Document conforms to the JSON schema.", "metrics": { "validation_errors": [], "flags": {} }, "checked_at": "2025-07-30T12:08:22.875185" }

Table 10: Example schema validation outputs: failing (left) and passing (right).

Validation Rule	Description	Flag
Missing Required Fields	Required metadata fields are missing.	missing:<field>
Type Mismatch	Field type does not match expected type (e.g., string vs int).	type:<field>:<found_type>to<expected_type>
Empty Fields	Field is present but empty (e.g., empty string, list, or dict).	empty:<field>
Title Too Short	Title has fewer than 5 characters (possible placeholder).	title_short
Malformed Author Entries	Author missing name or ID, or not a valid dict.	authors_malformed
Publication Year Mismatch	Parsed publicationdate year disagrees with year field.	year_vs_date

Table 11: Metadata Validation Rules and Diagnostic Flags

Warning Record

Passing Record

"metadata_validation": { "status": "warn", "summary": "Metadata field & content validation", "metrics": { "error_count": 0, "warning_count": 1, "errors": [], "warnings": [ "empty:venue" ], "diagnostics": { "missing_fields": [], "wrong_type_fields": [], "empty_fields": [ "venue" ], "malformed_authors": [], "title_short": false, "year_mismatch": false, "future_date": false, "date_issue": null, "no_chemistry_field": false, "oa_missing_licenses": [] } }, "checked_at": "2025-07-30T12:10:51.411572" }

"metadata_validation": { "status": "pass", "summary": "Metadata field & content validation", "metrics": { "error_count": 0, "warning_count": 0, "errors": [], "warnings": [], "diagnostics": { "missing_fields": [], "wrong_type_fields": [], "empty_fields": [], "malformed_authors": [], "title_short": false, "year_mismatch": false, "future_date": false, "date_issue": null, "no_chemistry_field": false, "oa_missing_licenses": [] } } }

Table 12: Example metadata validation outputs: warning (left) and passing (right).

Validation Rule	Description	Flag (on failure)
Raw Field Extraction	License fields are extracted from raw or nested metadata under Crossref, Unpaywall, and OpenAlex.	missing_license_¡source¿ — Raised when no usable license is found in source.
Normalization to Canonical ID	Raw values are mapped to canonical forms using direct mappings or regex (e.g., cc-by, cc0, public-domain).	unknown_license — Raised if license is unrecognized or ambiguous.
Source Agreement Logic	If two or more sources provide the same informative license, that license is accepted as resolved.	license_conflict — Raised when two or more informative sources disagree.
Three-Way Conflict Detection	If all three sources provide different informative licenses, a multi-way conflict is recorded.	license_conflict — Set to true and conflict expression recorded (e.g., conflict:cc-by_vs_closed_vs_cc0).
Resolved License Recording	The final resolved license is stored explicitly in resolved_license.	Not a flag — Used for downstream compliance filters.
Open License Whitelist	Licenses must be from an accepted set (e.g., cc0, cc-by, cc-by-sa).	fail — If resolved license is missing, restrictive, or not from allowed list.

Table 13: License Validation Rules and Conflict Flags

Failing Record	Passing Record
"license_validation": { "status": "fail", "summary": "Checks licence consistency and open-access compliance.", "metrics": { "resolved_license": "conflict:closed_vs_cc-by-nc", "license_source": "unpaywall+openalex", "license_conflict": true, "input_licenses": { "crossref": "missing", "unpaywall": "closed", "openalex": "cc-by-nc" } }, "checked_at": "2025-07-30T23:41:30Z" }	"license_validation": { "status": "pass", "summary": "Checks licence consistency and open-access compliance.", "metrics": { "resolved_license": "cc-by", "license_source": "crossref+unpaywall", "license_conflict": false, "input_licenses": { "crossref": "cc-by", "unpaywall": "cc-by", "openalex": "missing" } }, "checked_at": "2025-07-30T12:08:22Z" }

Table 14: Example license validation outputs: failing (left) and passing (right).

Validation Rule	Description	Flag (on failure)
Abstract Character Length	Abstract must be at least 100 characters.	abstract_too_short
Full Text Character Length	Full text must be at least 1000 characters.	fulltext_too_short
Abstract Sentence Count	Must contain at least 2 sentence-ending punctuation marks.	abstract_low_sentence_count
Full Text Sentence Count	Must contain at least 50 sentence-ending punctuation marks.	fulltext_low_sentence_count
Heading Markers in Full Text	Detects Markdown or numbered headings (e.g., # Introduction).	fulltext_missing_heading_markers
Corrupted Characters	Unicode replacement char (U+FFFD) present in text.	abstract_has_corrupted_chars, fulltext_has_corrupted_chars

Table 15: Text Validation Rules and Diagnostic Flags

Warning Record

(Placeholder for Passing Record)

"text_validation": { "status": "warn", "flags": { "abstract_too_short": false, "fulltext_too_short": false, "abstract_has_corrupted_chars": false, "fulltext_has_corrupted_chars": false, "abstract_low_whitespace_ratio": false, "fulltext_low_whitespace_ratio": false, "abstract_low_ascii_ratio": false, "fulltext_low_ascii_ratio": false, "abstract_low_sentence_count": false, "fulltext_low_sentence_count": false, "fulltext_missing_heading_markers": false, "language_mismatch_or_low_confidence": false, "low_rouge1_overlap": true, "abstract_embedding_norm_off": false, "abstract_embedding_missing_or_invalid": false }, "metrics": { "abstract_chars": 1344, "fulltext_chars": 15561, "abstract_bad_chars": { "valid": 1145, "corrupted": 0, "control": 0, "formatting": 0, "unassigned": 0, "whitespace": 199 }, "fulltext_bad_chars": { "valid": 13300, "corrupted": 0, "control": 0, "formatting": 0, "unassigned": 0, "whitespace": 2261 }, "abstract_ws_ratio": 0.8519345238095238, "fulltext_ws_ratio": 0.8547008547008547, "abstract_ascii_ratio": 0.8050595238095238, "fulltext_ascii_ratio": 0.80573227941649, "abstract_sentence_count": 5, "fulltext_sentence_count": 104, "fulltext_heading_markers": 9, "language": "en", "language_score": 0.9999956877319424, "rouge1_recall": 0.45145631067961167, "abstract_embedding_norm": 1.0 }, "examples": { "abstract_problem_chars": [], "fulltext_problem_chars": [] }, "checked_at": "2025-07-22T06:38:13.266722+00:00" }

Table 16: Example text validation output: warning (left).

Validation Rule	Description	Flag (on failure)
Empty Paragraphs	Paragraph value is missing, empty, or not a string	empty_chunks — Count of non-usable paragraph entries
Token Length Too Short	Paragraph has fewer than 100 tokens	chunks_too_short — Count of under-length paragraphs
Token Length Too Long	Paragraph has more than 300 tokens	chunks_too_long — Count of over-length paragraphs
Corrupted Characters	Paragraph contains replacement characters (�)	char_counts.corrupted — Aggregated count per paragraph
Control or Formatting Characters	Paragraph includes invisible control (Cc), formatting (Cf), or unassigned (Cn) code points	char_counts.control, formatting, unassigned — Diagnostic only
Missing Embedding	No vector found for paragraph key	missing_embeddings — Count of missing embeddings

Table 17: Chunk-Level Validation Rules and Diagnostic Flags

Warning Record

"chunk_validation": { "status": "warn", "summary": "0 chunk(s) over max tokens; 1 under min tokens.", "metrics": { "paragraph_count": 36, "token_length_distribution": { "Q1": 147.75, "Q2": 165.5, "Q3": 183.25, "min": 56, "max": 239, "mean": 166.38888888888889 }, "chunks_too_long": 0, "chunks_too_short": 1, "empty_chunks": 0, "missing_embeddings": 0, "invalid_embedding_vectors": 0, "max_token_limit": 300, "min_token_limit": 100 }, "paragraphs": { "9991603P0": { "token_length": 201, "char_counts": { "valid": 864, "corrupted": 0, "control": 0, "formatting": 0, "unassigned": 0 }, "problem_characters": [] }, "9991603P1": { "token_length": 232, "char_counts": { "valid": 1068, "corrupted": 0, "control": 0, "formatting": 0, "unassigned": 0 }, "problem_characters": [] }, ⋮ "9991603P35": { "token_length": 56, "char_counts": { "valid": 330, "corrupted": 0, "control": 0, "formatting": 0, "unassigned": 0 }, "problem_characters": [] } }, "checked_at": "2025-07-22T06:37:27.593943+00:00" }

Table 18: Example chunk validation output: warning status with one chunk under the minimum token limit.

Validation Rule	Description	Flag (on failure)
Field Type Check	paragraphs and embeddings must be dictionaries.	invalid_structure — Raised when fields are not of type dict.
Paragraph–Embedding Alignment	Each paragraph must have a corresponding embedding and vice versa.	missing_embedding, extra_embedding — Raised for mismatched IDs.
Vector Length	Each embedding must have exactly 1024 float elements.	invalid_shape_embedding — Raised for incorrect dimensionality.
Finite Float Values	All elements in embedding must be finite 32-bit floats.	nonfinite_values_embedding — Raised if NaNs or Infs are present.
Unit Norm (Approximate)	Embedding must be approximately unit-normalized within a 0.05 tolerance.	unnormalized_embedding — Raised if norm deviates from 1.0 beyond tolerance.
Cosine Similarity Threshold	Randomly sampled paragraph–embedding pairs must match re-encoded vectors (cosine similarity ≥ threshold).	cosine_mismatch — Raised if any sampled vector falls below similarity threshold.

Table 19: Embedding Validation Rules and Diagnostic Flags for Lit2Vec-Style Records

Failing Record	Passing Record
"embedding_validation": { "status": "fail", "summary": "Embedding validation complete", "metrics": { "paragraphs": 5, "embeddings": 5, "alignment": { "missing": [], "extra": [], "bad": ["p3"] }, "determinism": { "mean_cos": 0.821, "min_cos": 0.765, "max_delta": 0.235, "threshold": 0.01, "passed": false } }, "checked_at": "2025-07-30T22:05:49.882404" }	"embedding_validation": { "status": "pass", "summary": "Embedding validation complete", "metrics": { "paragraphs": 6, "embeddings": 6, "alignment": { "missing": [], "extra": [], "bad": [] }, "determinism": { "mean_cos": 0.998, "min_cos": 0.996, "max_delta": 0.004, "threshold": 0.01, "passed": true } }, "checked_at": "2025-07-30T22:08:41.208336" }

Table 20: Example embedding validation outputs: failing (left) and passing (right).

Validation Rule	Description	Flag (on failure)
Field Presence	The predicted_subfield field must exist in the record.	missing_predicted_subfield — Field is entirely absent.
Type Constraint	The field must be a dictionary.	not_a_dict — Field is not a dict (e.g., list or string given).
Empty Dictionary Check	Empty predictions are optionally disallowed.	empty_predictions — No predicted subfields found.
Vocabulary Match	Each subfield label must match a known entry in the controlled vocabulary.	contains_invalid_labels — Unknown label(s) detected.
Score Validity	Each score must be a float in the range [0.0, 1.0].	contains_invalid_scores — Value is non-float or out-of-bounds.
Validation Outcome	Determines status: ”pass”, ”warn”, or ”fail” based on aggregated rule checks.	Not a flag — Derived from combination of failure conditions.

Table 21: Validation Rules and Diagnostic Flags for Predicted Subfield Assignments

Failing Record	Warning Record	Passing Record
"predicted_subfield_validation": { "status": "fail", "flags": { "missing_predicted_subfield": true, "not_a_dict": false, "contains_invalid_labels": false, "contains_invalid_scores": false, "empty_predictions": false }, "metrics": { "label_count": 0, "invalid_labels": [], "invalid_scores": [] }, "checked_at": "2025-07-30T23:41:30.136585Z" }	"predicted_subfield_validation": { "status": "warn", "flags": { "missing_predicted_subfield": false, "not_a_dict": false, "contains_invalid_labels": true, "contains_invalid_scores": true, "empty_predictions": false }, "metrics": { "label_count": 2, "invalid_labels": [ "Quantum Wizardry" ], "invalid_scores": [ {"Catalysis": 1.2} ] }, "checked_at": "2025-07-30T23:41:31.004728Z" }	"predicted_subfield_validation": { "status": "pass", "flags": { "missing_predicted_subfield": false, "not_a_dict": false, "contains_invalid_labels": false, "contains_invalid_scores": false, "empty_predictions": false }, "metrics": { "label_count": 3, "invalid_labels": [], "invalid_scores": [] }, "checked_at": "2025-07-30T23:41:32.288271Z" }

Table 22: Example predicted subfield validation outputs: failing (left), warning (center), and passing (right).

Validation Rule	Description	Flag (on failure)
Non-empty Abstract and TLDR	Both abstract and tldr must be non-empty strings.	empty_abstract, empty_tldr
Minimum ROUGE-Lsum Threshold	ROUGE-Lsum must exceed a configurable minimum threshold (default: 10.0).	low_rouge_lsum
Minimum BERTScore-F1 Threshold	BERTScore_F1 must exceed a configurable minimum threshold (default: 30.0).	low_bertscore_f1
Optional Composite Score	Composite score may be defined as the average of ROUGE-Lsum and BERTScore-F1. Records with low composite score may be flagged.	low_summary_quality
Evaluation Fallback	If online evaluation tools are unavailable, fall back to native libraries without compromising score fidelity.	Not a flag — automatically handled by backend logic.
Evaluation Output Fields	Evaluation must produce all 7 scores: ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum, BERTScore Precision, Recall, and F1.	missing_score_¡metric¿

Table 23: Summary Evaluation Rules and Diagnostic Flags

Failing Record	Passing Record
"summary_validation": { "scores": { "ROUGE-1": 4.12, "ROUGE-2": 1.34, "ROUGE-L": 3.56, "ROUGE-Lsum": 5.89, "BERTScore_Precision": 22.74, "BERTScore_Recall": 24.65, "BERTScore_F1": 23.44 }, "flags": { "low_rouge_lsum": true, "low_bertscore_f1": true }, "summary_score": 14.66, "evaluated_at": "2025-07-30T13:22:18.543672" }	"summary_validation": { "scores": { "ROUGE-1": 48.21, "ROUGE-2": 31.04, "ROUGE-L": 43.89, "ROUGE-Lsum": 46.75, "BERTScore_Precision": 88.93, "BERTScore_Recall": 90.14, "BERTScore_F1": 89.52 }, "flags": {}, "summary_score": 68.13, "evaluated_at": "2025-07-30T13:25:14.781129" }

Table 24: Example summary validation outputs: failing (left) and passing (right).

Validation Rule	Description	Flag (on failure)
ID Agreement Check	The filename_id, corpus_id, and metadata.corpusid must all be present and equal.	corpusid_mismatch — Raised if values disagree.
Missing Metadata ID	metadata.corpusid must be defined if other IDs are present.	missing_metadata_id — Raised if missing.
DOI Normalization	DOIs from metadata, unpaywall, and openalex are normalized to a canonical form.	Not a flag — Used internally for comparison.
DOI Agreement Check	DOIs from all available sources must match after normalization.	doi_mismatch — Raised if two or more DOIs differ.
DOI Coverage Check	At least one DOI must be available across all sources.	doi_missing_sources — Raised if all DOI sources are missing.
Missing Embeddings	All paragraph IDs must have corresponding embedding IDs.	missing_embeddings — Raised if any paragraph is unrepresented.

Table 25: Consistency Validation Rules and Diagnostic Flags

Failing Record	Warning Record	Passing Record
"consistency_validation": { "status": "fail", "flags": { "missing_metadata_id": false, "corpusid_mismatch": true, "doi_mismatch": false, "missing_embeddings": false, "orphan_embeddings": false, "doi_missing_sources": false }, "metrics": { "corpus_ids": { "filename_id": 123456, "corpus_id": 123456, "metadata_id": 789012 }, "doi_set": [], "para_count": 10, "embed_count": 10, "missing_para_ids": [], "extra_embed_ids": [], "doi_sources_present": { "metadata_doi": false, "unpaywall_doi": false, "openalex_doi": false } }, "checked_at": "2025-07-30T19:08:15.015264" }	"consistency_validation": { "status": "warn", "flags": { "missing_metadata_id": false, "corpusid_mismatch": false, "doi_mismatch": true, "missing_embeddings": true, "orphan_embeddings": false, "doi_missing_sources": false }, "metrics": { "corpus_ids": { "filename_id": 101010, "corpus_id": 101010, "metadata_id": 101010 }, "doi_set": [ "10.1000/example", "10.1000/other" ], "para_count": 5, "embed_count": 3, "missing_para_ids": [ "1P3", "1P4" ], "extra_embed_ids": [], "doi_sources_present": { "metadata_doi": true, "unpaywall_doi": true, "openalex_doi": false } }, "checked_at": "2025-07-30T19:35:42.554110" }	"consistency_validation": { "status": "pass", "flags": { "missing_metadata_id": false, "corpusid_mismatch": false, "doi_mismatch": false, "missing_embeddings": false, "orphan_embeddings": false, "doi_missing_sources": false }, "metrics": { "corpus_ids": { "filename_id": 202020, "corpus_id": 202020, "metadata_id": 202020 }, "doi_set": [ "10.1000/example" ], "para_count": 8, "embed_count": 8, "missing_para_ids": [], "extra_embed_ids": [], "doi_sources_present": { "metadata_doi": true, "unpaywall_doi": true, "openalex_doi": true } }, "checked_at": "2025-07-30T20:05:08.123456" }

Table 26: Example consistency validation outputs: failing (left), warning (center), and passing (right).

Training Data Curation

Training data curation improves the quality, diversity, organization, and reliability of datasets used to train machine-learning models.

See all 155 papers

ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings
Aug 3, 2025Ali Shiraee Kasmaee, Mohammad Khodadad, Mahdi Astaraki +4Text Embedding Evaluation Information Retrieval
Department of Computational Science and Engineering, McMaster University, Canada · BASF Canada Inc., Canada · BASF Corporation, USA
ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery
Apr 10, 2026Shahar Levy, Eliya Habba, Reshef Mintz +3LLM-Assisted Annotation Document Information Extraction
School of Computer Science and Engineering, The Hebrew University of Jerusalem · Faculty of Law, The Hebrew University of Jerusalem · Allen Institute for AI
RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature
Oct 23, 2026Jiaxing Sun, Junyuan Gao, Ruikang Zhang +16Multimodal Dataset Document Information Extraction
Shanghai Artificial Intelligence Laboratory · Peking University · South China Normal University +7

Aug 3, 2025J/K move · Enter open · S save

ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings

Ali Shiraee Kasmaee, Mohammad Khodadad, Mahdi Astaraki, Mohammad Arshi Saloot+3

Department of Computational Science and Engineering, McMaster University, Canada · BASF Canada Inc., Canada · BASF Corporation, USA

Text Embedding Evaluation Information Retrieval Text Embedding Models

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) systems in chemistry heavily depend on accurate and relevant retrieval of chemical literature. However, general-purpose text embedding models frequently fail to adequately represent complex chemical terminologies, resulting in suboptimal retrieval quality. Existing embedding models for chemistry are outdated, and none is tailored to chemical literature retrieval, leaving a substantial performance gap. To address this challenge, we introduce ChEmbed, the first purpose-built family of domain-adapted text embedding models engineered for chemical literature retrieval. These models are fine-tuned via contrastive learning on a dataset comprising chemistry-specific text from the PubChem, Semantic Scholar, and ChemRxiv corpora. To create effective training data, we employ large language models to synthetically generate queries, resulting in approximately 1.7 million high-quality query-passage pairs. Additionally, we augment the tokenizer by adding 900 chemically specialized tokens to previously unused slots, which reduces the fragmentation of chemical entities, such as IUPAC names. ChEmbed also maintains an 8192-token context length, enabling retrieval of longer passages than many open-source embedding models allow. Evaluated on our newly introduced ChemRxiv Retrieval benchmark, ChEmbed outperforms state-of-the-art general embedding models, raising MRR@10 from 0.781 to 0.882 (+10.1 pp). It also substantially outperforms domain-specific embedding models such as Chemical-BERT, improving MRR@10 from 0.096 to 0.882. A role-based retrieval analysis using PubChem descriptions and ChEBI annotations shows that the improvement extends to chemical-role queries. ChEmbed represents a practical, lightweight, and reproducible embedding solution that effectively improves chemical literature retrieval.

Abstract

Training Data Curation

Explore similar work

ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings

ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery

RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature