LLM Information Extraction

LLM: Large Language Model

Overview

This field studies how to turn text, tables, images, and long documents into structured data that can support search, analysis, and decision-making. Common tasks include named-entity and relation extraction, event and attribute extraction, clinical report parsing, and building knowledge graphs from scientific or legal literature. Approaches range from prompting and retrieval-augmented examples to fine-tuning, schema-constrained decoding, agent workflows, and verification against source evidence. Models can produce well-formed outputs, but accurate and complete values remain difficult in long, noisy, multimodal, or specialized documents, so evaluation, provenance, and human review remain central.

Latest papers

247 papers16 papers in the last four weeks, down 52% on the four weeks before. 0.1% of all new papers.

Jan 25, 2026

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass decoder-based LLMs with in-context learning techniques; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP by 25.50 to 45.32 F1 points due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.
Jan 23, 2026

Strategies for Span Labeling with Large Language Models

Large language models (LLMs) are increasingly used for text analysis tasks, such as named entity recognition or error detection. Unlike encoder-based models, however, generative architectures lack an explicit mechanism to refer to specific parts of their input. This leads to a variety of ad-hoc prompting strategies for span labeling, often with inconsistent results. In this paper, we categorize these strategies into three families: tagging the input text, indexing numerical positions of spans, and matching span content. To address the limitations of content matching, we introduce LogitMatch, a new constrained decoding method that forces the model's output to align with valid input spans. We evaluate all methods across four diverse tasks. We find that while tagging remains a robust baseline, LogitMatch improves upon competitive matching-based methods by eliminating span matching issues and outperforms other strategies in some setups.
Nov 9, 2025

Rep2Text: Decoding Full Text from a Single LLM Token Representation

Large language models represent text as hidden numerical patterns, but what information these representations actually contain remains a mystery. Can we recover the original text from just one of them? In this paper, we introduce Rep2Text, a method designed to answer this question by reconstructing text from a single hidden representation. We find that these representations preserve much more information than we might expect. For short texts of about 16 words, Rep2Text can recover roughly half of the original words while preserving their overall meaning. As the text gets longer, recovering the exact wording becomes harder, but the main ideas can still be recovered. These findings offer a new way to understand what language models store internally and highlight potential privacy risks, as hidden representations may reveal parts of the original input.
Oct 28, 2025

LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature

Wide access to advanced experimental methods in materials science has given rise to an abundance of procedural knowledge, which is scattered across decades of scientific literature and recorded in unstructured formats that are challenging to analyze systematically. In this work, we present LeMat-Synth Parser, a modular, open-source, and multi-modal extraction toolbox that utilizes large language models (LLMs) and vision language models (VLMs) to automatically structure synthesis protocols and performance metrics extracted from both text and figures of publications. Applying LeMat-Synth Parser to 81K open-access publications, we curate LeMat-Synth, an extensive dataset of 58K synthesis procedures and to our knowledge the largest and most diverse structured inorganic materials synthesis dataset to date, covering 35 synthesis methods and 16 material classes based on a domain-specific ontology. We validate extraction quality against annotations by domain experts and a scalable LLM-as-a-judge framework, and benchmark a suite of models to identify optimal configurations and characterize cross-model biases. To demonstrate the extensibility of LeMat-Synth Parser, we apply it to two distinct domains. First, we link synthesis protocols and catalyst identity to thermocatalytic performance across a corpus of ammonia-decomposition publications. Second, we cross-validate text- and figure-reported critical transition temperatures across 1,384 superconductivity papers, then use the validated pipeline to recover the critical transition temperature for every composition in a sample series. We release LeMat-Synth Parser and the LeMat-Synth dataset openly on GitHub and Hugging Face
Oct 8, 2025

Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5

LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we provide a framework and the results of a multi-dimensional analysis of GPTKB v1.5 (Hu et al., 2025a), a recursively elicited Knowledge Base (KB) of 100 million facts (or beliefs) of a frontier LLM, namely, GPT-4.1. Given the scale of the elicited facts, we provide a multi-dimensional approach to qualitatively and quantitatively analyze these facts as opposed to the mainstream fact completion benchmarks, which are prone to availability bias. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities in neuro-symbolic AI concerning extraction, consolidation and verification of factual LLM knowledge.
Jul 8, 2025

GPTKB v1.5: A Massive Knowledge Base for Exploring Factual LLM Knowledge

Language models are powerful artifacts, yet their factual knowledge is still poorly understood, and inaccessible to ad-hoc browsing and scalable statistical analysis. This demonstration introduces GPTKB v1.5, a densely interlinked 100-million-triple knowledge base (KB) built for $14,000 from GPT-4.1, using the GPTKB methodology for massive-recursive LLM knowledge materialization. This demo focuses on three use cases: (1) link-traversal-based LLM knowledge exploration, (2) SPARQL-based structured LLM knowledge querying, (3) comparative exploration of the strengths and weaknesses of LLM knowledge. Massive-recursive LLM knowledge materialization is a groundbreaking opportunity both for the systematic analysis of LLM knowledge, as well as for automated KB construction.
Apr 28, 2025

Towards Large Language Models for Lunar Mission Planning and In Situ Resource Utilization

A key factor for lunar mission planning is the ability to assess the local availability of raw materials. However, many potentially relevant measurements are scattered across a variety of scientific publications. In this paper we consider the viability of obtaining lunar composition data by leveraging LLMs to rapidly process a corpus of scientific publications. While leveraging LLMs to obtain knowledge from scientific documents is not new, this particular application presents interesting challenges due to the heterogeneity of lunar samples and the nuances involved in their characterization. Accuracy and uncertainty quantification are particularly crucial since many materials properties can be sensitive to small variations in composition. Our findings indicate that off-the-shelf LLMs are generally effective at extracting data from tables commonly found in these documents. However, there remains opportunity to further refine the data we extract in this initial approach; in particular, to capture fine-grained mineralogy information and to improve performance on more subtle/complex pieces of information.