Latent entity extraction (LEE) tackles the challenge of identifying implicit, contextually inferred entities within free text-an area where traditional entity extraction methods fall short. In this paper, we introduce LentEx, a novel framework for latent entity extraction that leverages synthetic data generation and instruction fine-tuning to optimize smaller, efficient large language models (LLMs). Latent entities, which are often abstract and thematic, are crucial for applications such as retrieval-augmented generation (RAG), customer persona analysis, and knowledge graph enrichment. LentEx addresses the scarcity of labeled datasets by employing a template-based approach to generate diverse, contextually rich synthetic data, ensuring high variability and alignment with real-world distributions. To our knowledge, LentEx is the first to systematically approach LEE through the lens of LLMs. LentEx demonstrates significant performance improvements across multiple tasks, notably surpassing state-of-the-art models on the MTEB Clustering Benchmark. Furthermore, our methodology enables robust generalization to unseen domains, making LentEx highly applicable in real-world NLP tasks, including RAG and clustering, thereby establishing a new paradigm for latent entity understanding and extraction in natural language processing.
Figures & tables
Fig. 1: Overview of the LentEx workflow in three steps: (1) Template-based domain generation using Claude-3 with few-shot examples, (2) Synthetic input-output instance generation with similarity filtering, and (3) Instruction fine-tuning using Mistral-7B-Instruct-v0.2
Fig. 2: Diversity of synthetic data. Inner circle represents the 100 synthetic domains in synthetic data. Outer circle represents the domains clustered based on their similarity
Fig. 3: Synthetic data distributions: Top figure shows the distribution of length of input tokens across instances, middle figure shows the distribution of range of count of domains and the corresponding number of unique entities, while the bottom figure shows the size of dataset at different values of Rouge-L
Task
Prompt
Domain Generation
Generate a synthetic data dictionary containing 10 example entries. For each entry, the input should describe a broad category of text, and the output should specify a general category of latent entities. Latent entities are those that can be inferred from a text, even though they are not explicitly mentioned in the text. Ensure that the examples span a diverse range of topics, including but not limited to entertainment, education, technology, and health. The output categories should be broad, general, in 1-3 words, like ’movie genre’ or ’product type’, indicative of the kind of inference a reader can make from the given text type. Example: seed_task_dict = { ”sports commentary”: ”sport_name”, ”Google place review”: ”place category”, ”book summary”: ”book genre”, ”legal case_summary”: ”legal case type” }
Input-output Instance Generation
Generate a list of 10 detailed instances, each with their corresponding type or category. For each instance, provide comprehensive content that is representative of its category, ensuring clarity, relevance, and authenticity. The instances should cover a diverse range of subjects, showcasing unique and specific characteristics of each category. Format the output as follows: Each instance should be encapsulated in a dictionary with two keys: ”text_type” and ”latent_entity_type”. The ”text_type” key should include the full content of the instance, detailed and structured appropriately for its type. The ”latent_entity_type” key should specify the category or type that the instance belongs to. Structure each dictionary entry as a separate line in a JSONL format, where each line represents a single, complete dictionary corresponding to one instance and its category. Encapsulate the output in tag ¡rows¿.
Paraphrased Instructions for Training and Inference
1. From the given {text_type}, determine the {latent_entity_type}. 2. Based on the provided {text_type}, identify the {latent_entity_type}. 3. From the specified {text_type}, ascertain the {latent_entity_type}. 4. Using the given {text_type}, pinpoint the {latent_entity_type}. 5. Examine the {text_type} provided and determine the {latent_entity_type}.
TABLE I: Prompts for multiple stages. First row prompt is used in step 1, second row in step 2 of LentEx workflow. Third row shows the paraphrased prompts used in step 3 for instruction fine-tuning the model
Model
Mean Semantic Similarity %
Mistral-7B-Instruct-v0.2
61.17
Claude-3-Haiku
66.48
LentEx (Ours)
77.79
TABLE II: Experiment results from automated evaluation of model predictions on latent entity extraction task
Inference without entity options (Clustering Task)
Model
Average
arxiv-*
biorxiv-*
medrxiv-*
reddit-*
stackexchange-*
twenty-newsgroups
p2p
s2s
p2p
s2s
p2p
s2s
p2p
p2p
gte-Qwen2-7B-instruct
56.92
56.46
51.74
50.09
46.65
46.23
44.13
73.55
74.13
79.86
49.41
53.91
bge-en-icl
57.89
54.44
49.33
53.05
48.38
45.86
44.33
72.33
72.72
81.32
46.05
68.98
NV-Embed-v2
58.46
55.80
51.26
54.09
49.60
46.09
44.86
71.10
74.94
82.10
48.36
64.82
Mistral-7B-Instruct-v0.2
39.74
40.18
37.30
36.94
35.16
40.74
39.53
48.10
36.58
45.81
35.41
41.44
TABLE III: Clustering performance (v-measure scores) across different domains. p2p and s2s denote paragraph-to-paragraph and sentence-to-sentence tasks respectively. Best scores are in bold, second-best are underlined
Model
COLIEE
BioASQ
Precision
Recall
F-1
Precision
Recall
F-1
RAG (Baseline)
0.248
0.167
0.199
0.612
0.441
0.513
RAG + Mistral
0.644
0.536
0.585
0.681
0.496
0.574
RAG + Claude
0.715
0.663
0.688
0.704
0.528
0.603
RAG + Pre-trained NER
0.652
0.513
0.574
0.657
0.513
0.576
RAG + Topic Modeling
0.667
0.591
0.627
0.649
0.502
0.566
TABLE IV: Retrieval Metrics for COLIEE and BioASQ Datasets. CAPTAIN results are only available for COLIEE as it is a legal domain-specific system
Fig. 4: LLM evaluation of model responses over 1000 data points. Claude-3-Sonnet model was prompted to rate model responses as A, B, or C based on the SOP shown. Evaluations for different models were conducted independently, allowing for identical ratings across models.
Fig. 5: Effect of dataset size on model performance. Using synthetic outputs as reference labels, semantic similarity is measured between model predictions and labels. The y-axis represents the mean semantic similarity on test dataset.
Quality Review
Yes %
Is synthetic input text aligned with the text type?
98
Is latent entity label correct?
91.5
TABLE V: Manual data quality evaluation of input texts and labels from synthetic data, on a stratified sample of 2000 examples
Entity linking is a key component of many downstream NLP systems, yet existing approaches are often tied to the specific target knowledge bases and domains, limiting their real world application. In this paper, we extend LELA, a modular and domain-agnostic LLM-based entity disambiguation method, into a practical Python library that integrates zero-shot Named Entity Recognition (NER) -thereby providing a complete end-toend pipeline for entity-linking in real-world usage. We provide experimental results validating LELA's performance and robustness across diverse entity linking settings. In our demo, users can play with the system on their own input texts.
Samy Haffoudhi, Nikola Dobričić, Fabian Suchanek +1
IP Paris, LTCI, DIG · T´el´ecom Paris, Institut Polytechnique de Paris · IP Paris +1
Extracting structured data from unstructured text using large language models (LLMs) becomes challenging when target schemas are large and complex. In such cases, including the full schema in the prompt increases cost and latency, risks lost-in-the-middle performance degradation, and can exceed context length limits. We propose SchemaRAG, a retrieval-augmented generation (RAG) framework that dynamically prunes the output schema space for schema-conditioned information extraction tasks by leveraging schema metadata and few-shot examples when available. We evaluate SchemaRAG on real-world healthcare and e-commerce datasets. Our results show that SchemaRAG can achieve up to an 8.8% increase in micro-F1, a 47% reduction in latency, and a 48% reduction in token costs, demonstrating its practicality for large-schema extraction.
Retrieval-Augmented Generation (RAG) has become a standard approach for enhancing large language models (LLMs) with external knowledge, mitigating hallucinations, and improving factuality. However, existing systems rely on generating natural language queries at each hop and maintaining a strict architectural separation between retriever and generator, preventing them from leveraging the full representational capacity of the LLM. We propose LAnR (Latent Abstraction for RAG), a unified framework in which a single LLM jointly performs encoding, retrieval, and generation entirely within its own latent space. Rather than generating textual queries, LAnR produces dense retrieval vectors from the hidden states of a designated [PRED] token and uses them to match against encoded document representations from the same model. Furthermore, LAnR adaptively decides when sufficient evidence has been retrieved using a lightweight MLP control head over those same hidden states, eliminating both the separate retriever and explicit token-level stopping reasoning. This design is motivated by our empirical observation that answer token entropy reliably signals retrieval sufficiency. Extensive experiments on six QA benchmarks spanning single-hop and multi-hop settings demonstrate that LAnR outperforms existing RAG methods, while achieving improved inference efficiency through reduced number of retrieval calls and tighter model integration.