Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER

Ahmed Ewais; Ahmed Hashish; Amr Ali

cs.CLApr 6, 2026

Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER

Authors: Ahmed Ewais, Ahmed Hashish, Amr Ali

Organizations: WitnessAI

Abstract

Large language models encode extensive world knowledge valuable for zero-shot named entity recognition. However, their causal attention mechanism, where tokens attend only to preceding context, prevents effective token classification when disambiguation requires future context. Existing approaches use LLMs generatively, prompting them to list entities or produce structured outputs, but suffer from slow autoregressive decoding, hallucinated entities, and formatting errors. We propose Just Pass Twice (JPT), a simple yet effective method that enables causal LLMs to perform discriminative token classification with full bidirectional context. Our key insight is that concatenating the input to itself lets each token in the second pass attend to the complete sentence, requiring no architectural modifications. We combine these representations with definition-guided entity embeddings for flexible zero-shot generalization. Our approach achieves state-of-the-art results on zero-shot NER benchmarks, surpassing the previous best method by +7.9 F1 on average across CrossNER and MIT benchmarks, being over 20x faster than comparable generative methods. Interactive demo and access to the model are available at https://witness-ai-jpt-ner.hf.space/

Figures & tables

Figure 1: Just Pass Twice (JPT) enables bidirectional token classification in causal LLMs. (Top) Standard causal masking restricts the target token “Paris” from attending to future context like “album” (red barrier), leading to ambiguity. (Bottom) JPT duplicates the input sequence. In the second pass, the target token (green box) attends backwards to the entire original sequence (green arrows), resolving the entity type without architectural modifications.

Figure 2: Architecture of the proposed model. (Top) Entity definitions are encoded via a text encoder (dimension $d_{\text{enc}}$ ) and projected into the shared space $\mathbb{R}^{d_{p}}$ using the Entity Projection MLP, yielding entity embeddings $\mathbf{p}_{\text{per}},\mathbf{p}_{\text{loc}},\ldots$ (Bottom) The Causal LLM processes the duplicated input sequence. Hidden states from the second pass ( $\mathbf{h}_{5},\ldots,\mathbf{h}_{7}$ ) are projected to token embeddings $\mathbf{t}_{1},\ldots,\mathbf{t}_{n}\in\mathbb{R}^{d_{p}}$ and scored against entity embeddings via a bilinear classifier.

Generative Baselines
Model	AI	Lit.	Music	Politics	Science	Movie	Rest.	Avg
UniNER-7B	62.9	64.9	70.6	66.9	70.8	61.2	35.2	61.8
GoLLIE	59.1	62.7	67.8	57.2	55.5	63.0	43.4	58.4
InstructUIE	49.0	47.2	53.2	48.2	49.3	63.0	21.0	47.8
SaM (MoE)	60.9	66.9	73.5	74.4	62.6	72.1	52.9	66.2
Discriminative Baselines

Table 1: Zero-shot F1 scores on CrossNER and MIT benchmarks. JPT-8B achieves the best overall average, improving by +7.9 F1 over the strongest baseline (SaM). Results for baselines are taken from prior work ( Ding et al., 2025 ) .

Dataset	UniNER-7B	GLiNER-L	JPT-4B
ACE05	36.9	27.3	44.6
AnatEM	25.1	33.3	37.2
bc2gm	46.2	47.9	54.8
bc4chemd	47.9	43.1	53.3
bc5cdr	68.0	66.4	70.4
Broad Twitter	67.9	61.2	71.2

Table 2: Extended zero-shot F1 results across 20 NER benchmarks spanning biomedical, social media, and multilingual domains. Results for UniNER-7B and GLiNER-L are reported from ( Zaratiana et al., 2024 ) .

Generative Methods
Method	Cost( $)$ \downarrow$	Time (sec) $\downarrow$	F1 $\uparrow$
UniNER-7B	2.77	1970.2	61.8
GNER	8.21	5831.5	75.8
GPT-5 †	0.49	579.6	67.1
Discriminative Methods
GLiNER-L	0.05	33.3	60.9

Table 3: Efficiency comparison on CrossNER-Politics. Cost: compute time

\times

$5.07/hour (local) or API pricing (GPT). † GPT results vary with prompting strategy.

Configuration	Avg Micro F1
Input Duplication
Single Pass	55.7
Entity Definitions
No Definitions	58.3
Prompt-only definitions	63.3
Embedding-only definitions	65.2

Table 4: Ablation studies on JPT-4B. Avg. micro-F1 across CrossNER and MIT benchmarks. Input duplication provides +15.2 F1; dual-channel definitions provide +12.6 F1 over no definitions.

Figure 3: Attention weights averaged across all transformer layers from second-pass tokens (rows) to first-pass tokens (columns). The model attends to corresponding positions (diagonal) and semantically relevant context, suggesting effective bidirectional information flow.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Base LLM (Frozen)
Component	JPT-4B	JPT-8B
Model	Qwen3-4B	Qwen3-8B
Hidden dim ( $d_{\text{llm}}$ )	2560	4096
Layers / Attention heads	36 / 32	36 / 32
LoRA Adapters (Trained)
Rank ( $r$ ) / Alpha ( $\alpha$ )	32 / 64	128 / 256

Appendix

Table 5: Detailed architecture configuration for JPT models. Only a small fraction of backbone parameters are trained via LoRA adapters and lightweight projection heads.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$5\times 10^{-5}$
LR scheduler	Cosine with warmup
Warmup ratio	10% of total steps
Effective batch size	8
Gradient accumulation	2

Appendix

Table 6: Training hyperparameters used for both JPT-4B and JPT-8B.

Statistic	Value
Total sentences	17,489
Total tokens	3,391,899
Total entity mentions	374,705
Unique entity types	5,009
Avg. sentence length	194.9 tokens
Entity/non-entity ratio	1:1.5

Appendix

Table 7: Training dataset statistics.

Figure 4: Distribution of entity type frequencies in training data. The long-tail distribution (71% of types have

\leq

10 mentions, median=4) encourages the model to leverage definition semantics rather than memorizing patterns.

Example 1: Sports Domain
“the men’s foil competition ${}_{\textsc{[Competition]}}$ in fencing ${}_{\textsc{[Sport]}}$ at the 2016 olympic games ${}_{\textsc{[Event]}}$ in rio de janeiro ${}_{\textsc{[City]}}$ was held on 7 august ${}_{\textsc{[Date]}}$ at the carioca arena 3 ${}_{\textsc{[Venue]}}$ . the medals were presented by paul tergat ${}_{\textsc{[Person]}}$ , ioc ${}_{\textsc{[Org]}}$ member, kenya ${}_{\textsc{[Country]}}$ ”
Example 2: Natural Disaster Domain
“the 2014 mae lao earthquake ${}_{\textsc{[Earthquake]}}$ occurred at 18:08:43 ${}_{\textsc{[Time]}}$ indochina time ${}_{\textsc{[TimeZone]}}$ on may 5 ${}_{\textsc{[Date]}}$ . the epicenter was located 9 km ${}_{\textsc{[Distance]}}$ south of mae lao district ${}_{\textsc{[District]}}$ , 27 km ${}_{\textsc{[Distance]}}$ southwest of chiang rai ${}_{\textsc{[City]}}$ , thailand ${}_{\textsc{[Country]}}$ ”
Example 3: Biographical Domain
“ helga mühlberg-ulze ${}_{\textsc{[Athlete]}}$ is an east german ${}_{\textsc{[Nationality]}}$ sprint canoeist ${}_{\textsc{[SportDisc]}}$ who competed in the early to mid 1960s ${}_{\textsc{[TimePeriod]}}$ . she won gold ${}_{\textsc{[Medal]}}$ ( k-2 500m ${}_{\textsc{[RaceCat]}}$ : 1966 ${}_{\textsc{[Year]}}$ ) and two bronzes ${}_{\textsc{[Medal]}}$ ”

Appendix

Figure 5: Training examples showing fine-grained entity types across diverse domains. Entity spans are underlined in blue with type labels in subscript. The dataset contains 5,009 unique entity types including domain-specific categories like Earthquake , TimeZone , and RaceCategory .

Type	Generic Definition	Precise Definition	$\Delta$ F1
Location	“A geographical place”	“Any word indicating WHERE: explicit places (NYC, downtown), relative indicators (nearby, around, close by), directional phrases (east, south side). Tag the location word itself.”	+32.6
Price	“A monetary value”	“Explicit monetary amounts ($50, 100 dollars) AND qualitative price indicators (cheap, expensive, budget-friendly, overpriced, pricey).”	+34.9
Amenity	“An available service”	“A feature, facility, or service offered by a restaurant. Includes: physical features (bar, parking), services (delivery, reservations, takeout), atmosphere descriptors (romantic, casual, family-friendly).”	+14.29

Appendix

Table 8: Generic vs. precise entity definitions and their impact. F1 computed per-type on MIT Restaurant. Precise definitions with boundary cases and examples dramatically improve recognition.

Model	Params	Token F1
Qwen3-1.7B	1.7B	92.4
Qwen3-4B	4B	93.9
Qwen3-8B	8B	94.1
Qwen3-14B	14B	94.5

Appendix

Table 9: Impact of base LLM size on token-level F1 (private evaluation set). Larger models yield consistent improvements, with diminishing returns beyond 8B parameters.

LoRA Rank	Token F1
$r=0$ (frozen)	82.0
$r=16$	93.5
$r=32$ (default)	93.9
$r=64$	94.2
$r=128$	94.5
$r=256$	94.7

Appendix

Table 10: Impact of LoRA rank on token-level F1 using JPT-4B (private evaluation set). Adaptation is essential (

r=0

drops 12 points), but returns diminish beyond

r=32

Model	AI	Lit.	Music	Politics	Science	Avg.
Qwen3-4B	49.9	47.9	74.7	56.2	58.1	57.4
JPT-4B	68.3	73.7	84.1	76.4	69.5	74.4

Appendix

Table 11: Comparison between JPT-4B and its causal backbone, Qwen3-4B , on the CrossNER benchmark. JPT-4B consistently outperforms the backbone across all domains.

Entity at start of sentence	GT Type	Single-input prediction	Why future context matters
Type confusion at sentence start
"Autoencoders are trained to minimise…"	algorithm	product	Need “trained to minimise” to determine that it refers to an algorithm
"François Bayrou (…) is a French centrist politician…"	politician	person	Need “politician” later in the sentence to resolve the fine-grained type
Completely missed at sentence start
"The Color Purple is a 1982 epistolary novel…"	book	(missed)	Need “novel” to recognize it as a book title
"Young’s work on squid giant axons…"	scientist	(missed)	Need scientific context later in the sentence to identify the mention as a scientist

Appendix

Table 12: Representative errors made by the single-input JPT variant. When entity mentions appear early in the sentence, the absence of future context can lead to type confusion or missed predictions.

Panel of Figure 7 — Figure 7: Additional attention visualizations. Rows: second-pass tokens. Columns: first-pass tokens. The diagonal pattern (self-attention across passes) combined with off-diagonal attention to context tokens demonstrates effective bidirectional information flow.

Dataset	Entity Type	Gold Span	Predicted Span
CrossNER-Science	ORGANIZATION	Virgo interferometer	Virgo interferometer collaboration
CrossNER-Science	ACADEMIC JOURNAL	the Springer Series Complexity	of the Springer Series Complexity
CrossNER-Politics	POLITICAL PARTY	Social Credit Party of Canada	national Social Credit Party of
CrossNER-Music	BAND	Collective Consciousness Society	The Collective Consciousness
CrossNER-Literature	LITERARY GENRE	Germany Romanticism	Romanticism era
MIT-Movie	TITLE	the reflecting skin	reflecting skin movie

Appendix

Table 13: Representative boundary detection errors across CrossNER and MIT. JPT often predicts the correct type and core mention but may over-extend or truncate spans when entities appear with modifiers, appositions, or colloquial phrasing.

Figure 8: Entity type confusion matrix aggregated across CrossNER and MIT. Most confusions occur between semantically adjacent types (e.g., PER vs. ORG , LOC vs. COUNTRY , and POL_PARTY vs. POLITICIAN ), suggesting that errors often stem from fine-grained boundary cases and overlapping semantic definitions rather than arbitrary label flips.

Error Type	Dataset	Text (excerpt)	Gold	Pred.	Analysis
Type Confusion	CrossNER-AI	“NIST also differs from Bilingual evaluation understudy in its calculation…”	metrics	conference	Acronym ambiguity
Type Confusion	CrossNER-Politics	“The provinces ceded to Augustus for that ten-year period…”	person	politician	Role vs. entity
Type Confusion	CrossNER-Music	“…on The Clash ’s London Calling …”	album	location	Title–location ambiguity
Missed Entity	CrossNER-AI	“Rethink Robotics introduced Baxter as an industrial robot …”	product	(none)	Descriptive form
Missed Entity	CrossNER-Politics	“growth in the Melanesian countries of Solomon Islands…”	location	(none)	Adjectival reference
Over-prediction	CrossNER-AI	“A frame language is a technology used for knowledge representation…”	(none)	prog. lang.	Over-generalization

Appendix

Table 14: Representative JPT-4B error instances with context excerpts, gold labels, predictions, and diagnostics.

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 106 papers

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models
Aug 1, 2026Xuankang Zhang, Jiangming LiuNamed Entity Recognition Zero-Shot Learning
Yunnan University
SAM-NER: Semantic Archetype Mediation for Zero-Shot Named Entity Recognition
May 5, 2026Ruichu Cai, Juntao Gan, Miao Mai +2Named Entity Recognition Cross-Domain Transfer Learning
School of Computer Science, Guangdong University of Technology · Peng Cheng Laboratory · Nanfang Media Group(Nanfang Daily) +1
DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition
Apr 17, 2026Siun Kim, Hyung-Jin YoonNamed Entity Recognition LLM-Assisted Annotation
Seltasquare Seoul, Korea · Biomedical Research Institute, Seoul National University Hospital · Seoul National University College of Medicine Seoul, Korea

Aug 1, 2026J/K move · Enter open · S save

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models

Xuankang Zhang, Jiangming Liu

Yunnan University

Named Entity Recognition Zero-Shot Learning LLM Prompting LLM Information Extraction

Open paper AskPDF arXiv

Recent advancements of zero-shot Named Entity Recognition (NER) establish strong baselines by formulating sequence labeling into question answering where Large Language Models (LLMs) can be naturally adopted. However, existing LLM-based zero-shot NER methods suffer from the limitations of prompt and demonstration engineering. To address these issues with minimal human interventions, we introduce DE-NER, a dialogue elicitation framework which elicits the chatting ability of LLMs to fully extract the knowledge encoded in LLMs. Our experiments demonstrate that the proposed method outperform the competitive baselines in zero-shot settings across multiple benchmarks, with an average improvement of 3.75% F1 points. Codes are released in https://github.com/kkkenshi/DE-NER.

Abstract

Named Entity Recognition

Explore similar work

DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models

SAM-NER: Semantic Archetype Mediation for Zero-Shot Named Entity Recognition

DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition