Verb-ICL: Rethinking In-Context Learning for Structured Prediction

Fan Bai; Hengshuo Miao; Sanjit S Batra; Hamid Reza Hassanzadeh; Ardavan Saeedi; Mark Dredze

cs.CLOct 3, 2026

Verb-ICL: Rethinking In-Context Learning for Structured Prediction

Authors: Fan Bai, Hengshuo Miao, Sanjit S Batra, Hamid Reza Hassanzadeh, Ardavan Saeedi, Mark Dredze

Organizations: Johns Hopkins University · Optum

Abstract

Structured prediction tasks pose unique challenges for in-context learning (ICL): their compositional outputs require modeling fine-grained, token-level patterns that sentence-level approaches fail to capture, and their task-specific annotation conventions are human-defined artifacts that cannot be acquired through pretraining alone. We propose Verb-ICL, a selective annotation framework for ICL-based structured prediction that addresses both challenges. Verb-ICL first selects representative examples using a token-level coverage strategy that captures local semantic patterns critical for structured prediction, then generates actionable error feedback that codifies task-specific annotation guidelines and incorporates this feedback into ICL demonstrations. We evaluate Verb-ICL on six structured prediction datasets spanning information extraction and semantic parsing. Experiments with recent LLMs show that Verb-ICL consistently outperforms strong selective annotation baselines under low-resource settings and continues to provide gains as the annotation budget increases. Extended analyses demonstrate that the generated feedback is predominantly useful across a four-category quality taxonomy, generalizes as task-level guidance beyond instance-specific corrections, and improves performance regardless of the underlying selection strategy.

Figures & tables

Figure 1: Overview of Verb-ICL . In Step 1 , we use a coverage-driven strategy to select $M$ examples that best cover the unlabeled pool $\mathcal{U}$ . In Step 2 , we collect model errors from annotated examples, and prompt the LLM to generate textual feedback that is incorporated as “Notes” in future demonstrations to guide subsequent predictions.

$\|\mathcal{L}\|$	Method	NCBI	TwNER7	PHEE	GeoQ	ATIS	MTOP	Avg.
16	Random	61.2 4.4	52.3 1.4	56.6 2.7	23.0 3.8	27.3 0.8	11.9 2.1	38.7
	K-means	63.8 0.6	54.1 1.0	58.5 1.5	28.0 3.6	30.7 2.0	17.2 0.8	42.1
	IDDS ( 2022 )	65.0 0.3	55.4 0.3	57.7 0.0	27.7 0.2	35.6 0.3	15.3 0.2	42.8
	Vote- $k$ ( 2023 )	58.0 2.8	51.9 2.2	52.8 2.6	25.6 0.9	27.0 3.2	11.8 0.6	37.9
	+ SET-BSR	59.4 3.0	52.9 1.6	54.6 1.9	29.3 2.6	27.7 4.1	14.7 2.0	39.8
	LDCAL ( 2024 )	56.3 0.3	51.9 0.1	51.9 0.1	16.3 0.2	16.0 0.0	9.1 0.1	33.6

Table 1: Comparison of Verb-ICL and seven baselines across six datasets using GPT-4o-mini , evaluated under annotation budgets of 16 and 64 examples. Results are averaged over three random seeds, with standard deviations reported as subscripts. For vote- k , IDEAL and CoverICL, we also report results using their original sentence embedding similarity-based retrievers. Verb-ICL outperforms baselines in both budget settings.

Figure 2: Performance scaling with increasing annotation budgets from 16 to 512 examples on two representative datasets: NCBI (left) and ATIS (right). We compare Verb-ICL with best-performing baselines: Random, K-Means, IDDS, vote- k , and CoverICL. Verb-ICL outperforms baselines at every budget level on both datasets.

	Feedback Design				Accuracy
Method	Source	Grounding	Notes	Position	NCBI	ATIS
(a) Verb-ICL w/o EF	–	–	–	–	69.6 0.1	48.7 0.5
(b1) LEAP-style	zero-shot	gold	all	global	68.0 0.9	50.4 0.8
(b2) –	LOO	gold	all	global	69.9 0.2	49.5 1.5
(c) –	LOO	gold	retrieved	global	67.7 0.4	52.1 0.5
(d) Verb-ICL	LOO	gold	retrieved	inline	71.3 0.7	52.9 1.4

Table 2: Error feedback design comparison on NCBI (mention-level F1) and ATIS (exact match), using GPT-4o-mini at a 64-example budget, averaged over three seeds. All rows use Verb-ICL ’s coverage-driven selection, so only the feedback design differs; EF denotes error feedback, and LOO means leave-one-out. Each adjacent pair isolates one dimension: error source (b1 vs. b2), feedback selection (b2 vs. c), positioning (c vs. d), and grounding (d vs. e).

Dataset	$n$	Concrete	General	Misdiag.	Over-gen./Vague
NCBI	38	5 (13%)	28 (74%)	4 (11%)	1 (3%)
TweetNER7	179	20 (11%)	151 (84%)	0	8 (4%)
PHEE	240	1 (0%)	225 (94%)	14 (6%)	0
ATIS	63	19 (30%)	43 (68%)	–	1 (2%)
GeoQuery	66	48 (73%)	14 (21%)	–	4 (6%)
MTOP	81	65 (80%)	16 (20%)	–	0

Table 3: Distribution of feedback quality categories across all six datasets (64-shot). Misdiagnosis is verified for information extraction tasks only; “–” indicates not applicable for semantic parsing tasks. Over-generalized and vague categories are merged for compactness.

	ATIS			MTOP
Method	Total	Struct.	Value	Total	Struct.	Value
Random	608	491	117	737	633	104
K-Means	547	427 (–13.0%)	120	700	577 (–8.8%)	123
Verb-ICL w/o EF	520	424 (–13.6%)	96	661	521 (–17.7%)	140
Verb-ICL	487	400 (–18.5%)	87	643	492 (–22.3%)	151

Table 4: Error composition on ATIS and MTOP ( GPT-4o-mini , 64-example budget, seed 42). Incorrect predictions are split into structural and value violations; parentheses give the relative reduction in structural violations against Random. Both components reduce structural violations; value counts rise because a value violation is counted only when the structure is correct.

		Error Feedback
		4o-mini	4o	4.1
Dataset	Cov. Sel.	(10/01/23)	(10/01/23)	(06/01/24)
NCBI	69.4 0.0	71.1 0.3	71.3 0.6	71.3 0.6
ATIS	48.7 0.5	49.7 0.4	51.2 0.1	53.0 1.1
PHEE	61.1 0.1	61.3 0.3	62.0 0.3	62.1 0.1

Table 5: Left: LLM ablation for error feedback generation under the 64-example setting using GPT-4o-mini for ICL inference. Stronger models yield larger gains on complex tasks; gains on PHEE (released 2024) from pre-2024-cutoff models alleviate contamination concerns. Right: LLM cost comparison on NCBI (64-example). Verb-ICL costs less by generating feedback only for coverage-selected examples.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

		# Instances
Task Type	Dataset	(train / dev / test)
Info. Extraction	NCBI ( 2014 )	5.4k / 0.9k / 0.9k
	TweetNER7 ( 2022 )	7.1k / 0.8k / 0.6k
	PHEE ( 2024 )	2.9k / 1.0k / 1.0k
Semantic Parsing	GeoQuery ( 1996 )	0.7k / - / 0.2k
	ATIS ( 1990 )	4.0k / - / 1.0k

Appendix

Table 6: Statistics of the six evaluated datasets.

$\|\mathcal{L}\|$	Method	NCBI	TwNER7	PHEE	GeoQ	ATIS	MTOP	Avg.
16	Random	50.9 6.2	42.7 0.3	56.8 2.5	19.3 1.1	28.6 1.2	9.5 2.1	34.6
	K-means	52.0 1.8	44.0 2.7	57.9 0.7	26.5 0.7	31.3 2.5	14.4 0.7	37.7
	IDDS ( 2022 )	45.0 0.0	42.2 0.3	58.1 0.0	24.5 0.2	38.9 0.0	12.4 0.0	36.9
	Vote- $k$ ( 2023 )	29.0 0.0	35.6 0.0	56.2 0.0	19.3 0.0	29.9 0.0	11.3 0.0	30.2
	LDCAL ( 2024 )	34.2 0.2	40.3 0.0	55.8 0.0	15.0 0.0	17.1 0.0	9.2 0.2	28.6
	CoverICL ( 2024 )	48.8 3.7	38.3 2.1	58.0 0.5	28.2 0.5	27.1 2.7	10.8 1.0	35.2

Appendix

Table 7: Comparison of Verb-ICL and seven baselines across six datasets using Qwen2.5-7B , evaluated under two annotation budgets. Verb-ICL outperforms baselines, demonstrating its robustness across different LLMs.

		Information Extraction			Semantic Parsing
LLM	Method	NCBI	TwNER7	PHEE	GeoQ	ATIS	MTOP	Avg.
Qwen2.5-7B	Sen. Emb.	66.7	45.0	61.7	63.2	67.7	61.1	60.9
	BSR	68.7	47.2	62.7	66.4	72.7	64.3	63.6
	SET-BSR	69.2	48.1	63.6	68.2	75.8	64.7	65.0
gpt-4o-mini	Sen. Emb.	71.5	57.9	63.0	70.7	68.2	67.8	66.5
	BSR	73.9	60.6	64.2	73.2	74.4	72.0	69.8

Appendix

Table 8: Comparison of demonstration retrieval methods using the full training set. SET-BSR consistently outperforms sentence embedding-based retrievers across all settings.

		gpt-4o-mini			Qwen2.5-7B
Task	Dataset	$\|\mathcal{L}\|=16$	Full	$\Delta$	$\|\mathcal{L}\|=16$	Full	$\Delta$
Information Extraction	NCBI ( 2014 )	61.2 4.4	77.7	+16.5	50.9 6.2	69.2	+18.3
	TweetNER7 ( 2022 )	52.3 1.4	60.8	+8.5	42.7 0.3	48.1	+5.4
	PHEE ( 2024 )	56.6 2.7	64.7	+8.1	56.8 2.5	63.6	+6.8
Semantic Parsing	GeoQ ( 1996 )	23.0 3.8	75.0	+52.0	19.3 1.1	68.2	+48.9
	ATIS ( 1990 )	27.3 0.8	75.5	+48.2	28.6 1.2	75.8	+47.2

Appendix

Table 9: Comparison between randomly selecting 16 demonstrations and using the full training set with SET-BSR on six structured prediction datasets. Scaling from 16 to the full training set yields substantial improvements, confirming that these tasks benefit from additional annotations and more careful selection strategies.

Dataset	# token/sent.	Results
NCBI	5	68.9 0.5
	10	70.7 0.2
	15	68.6 0.2
	full	69.6 0.1
GeoQuery	5	51.6 0.5
	10	55.1 0.5

Appendix

Table 10: Verb-ICL coverage-driven selection results (64-example budget) with different column sizes for

\mathbf{C}_{i}

. We can see that reducing the column size to 10 does not impact the performance of coverage-driven selection significantly.

Method	16-shot	64-shot	256-shot
Label Reweight	2.5	8.3	47.8
Verb-ICL	30.3	44.4	52.1

Appendix

Table 11: Fine-tuning BERT-base performance with data selected by Verb-ICL vs. Label Reweight ( Luo et al., 2023 ) on CoNLL03. Across all three annotation budgets (16, 64, and 256 examples), data selected by Verb-ICL consistently leads to higher fine-tuning performance.

$\|\mathcal{L}\|$	Model	NCBI	PHEE	GeoQ	Average
16	DeBERTa-Large-MNLI	66.2 0.2	59.1 0.1	35.5 0.4	53.6
	DeBERTa-Base-MNLI	65.4 0.2	57.4 0.2	34.9 0.6	52.6
	BERT-base-uncased	64.3 0.2	56.7 0.2	39.2 0.4	53.4
	all-mpnet-base-v2	63.2 0.2	57.3 0.2	34.0 0.2	51.5
64	DeBERTa-Large-MNLI	69.7 0.1	61.1 0.1	53.1 0.3	61.3
	DeBERTa-Base-MNLI	67.8 0.2	61.1 0.2	54.3 0.3	61.1

Appendix

Table 12: Comparison of four embedding models for ICL using GPT-4o-mini . DeBERTa-Large-MNLI demonstrates great performance in most settings.

	NCBI		ATIS		GeoQuery		TweetNER7		PHEE		MTOP
	16	64	16	64	16	64	16	64	16	64	16	64
#Total feedback	8	26	15	44	12	43	16	60	14	60	14	53
#Verified feedback	4	14	2	17	4	15	7	15	0	4	2	16
ICL w/ verified only	67.4	70.6	36.0	50.8	37.5	54.3	55.4	55.0	58.9	61.4	21.7	35.2
ICL w/ all feedback	68.6	71.1	39.1	51.7	35.4	56.1	53.6	55.6	59.6	61.8	22.7	35.7

Appendix

Table 13: Analysis of LLM-generated error feedback across all six datasets under 16-shot and 64-shot annotation budgets, using a single seed (42) to reduce cost, as variance across seeds is small. “Verified” feedback refers to feedback that directly corrects the original error when added to the ICL prompt. Using all feedback beats using only verified feedback in 10 of 12 settings, so feedback provides task-level guidance beyond instance-specific correction.

	NCBI		ATIS
Method	16-shot	64-shot	16-shot	64-shot
Random	67.1	68.6	28.5	39.7
+ EF	67.6 +0.5	69.3 +0.7	29.4 +0.9	42.9 +3.2
K-Means	63.4	69.3	28.2	45.7
+ EF	65.7 +2.3	71.4 +2.1	28.4 +0.2	48.8 +3.1
vote- k	64.1	66.7	29.6	42.5

Appendix

Table 14: Effect of applying error feedback (EF) as a post-processing step on top of different selection strategies, computed with GPT-4o-mini at a single seed (42). The claim that feedback helps regardless of selection method rests on the within-row EF gain, which compares the same selected examples with and without feedback and is unaffected by the selection seed.

	Accuracy		Avg. length
Method	NCBI	ATIS	NCBI	ATIS
Random	67.7 0.3	42.5 2.7	23.8	10.9
BoW-coverage	68.6 0.2	41.0 0.1	64.7	19.3
Longest- $n$	65.5 0.3	37.6 0.1	75.5	27.4
Verb-ICL w/o EF	69.6 0.1	48.7 0.5	42.0	13.1

Appendix

Table 15: Token-level selection baselines at a 64-example budget with GPT-4o-mini , averaged over three seeds. NCBI reports mention-level F1 and ATIS exact match. Average length is the word count of the

M

selected examples after whitespace tokenization and lowercasing.

	NCBI		ATIS
Method	16	64	16	64
Random	68.6 2.9	73.5 1.2	33.2 0.8	50.8 3.3
K-Means	70.7 0.7	74.5 1.0	36.3 1.9	52.0 2.7
IDDS	72.8 0.9	72.9 0.3	41.2 0.8	53.1 0.4
Verb-ICL w/o EF	73.2 0.4	75.7 0.4	45.9 0.3	58.7 0.5
Verb-ICL	74.7 0.1	76.9 0.3	46.0 1.2	59.7 0.7

Appendix

Table 16: ICL performance with GPT-5.4-mini under 16- and 64-example annotation budgets, averaged over three seeds. NCBI reports mention-level F1 and ATIS exact match. Feedback is generated by GPT-4.1 , as in the main experiments.

Method	Error source	Grounding	Selection	Position
LEAP	zero-shot	gold	all	global
Self-Refine	test-time output	label-free	–	–
Verb-ICL	leave-one-out	gold	retrieved	inline

Appendix

Table 17: Existing error feedback methods situated across the four design dimensions.

Dataset	Task	Error feedback
NCBI	IE	Only include the core disease term in the entity span, and exclude preceding descriptive adjectives unless they are part of the canonical disease name.
TweetNER7	IE	Government bodies and administrations should be classified as “corporation”, not “group”.
PHEE	IE	Only extract subject_disorder when the text explicitly states that the patient has the condition, not when it is described as an outcome or a risk.
GeoQuery	SP	Use state abbreviations (e.g., “wa”) in cityid predicates, not full state names.
ATIS	SP	Add the predicates needed to represent the target entity type (“city”) and the relationship (“to”) that links flights to cities.
MTOP	SP	When the sentence instructs the system to send a message or ask someone something, use SEND_MESSAGE as the intent rather than ASK .

Appendix

Table 18: Representative error feedback generated by GPT-4.1 on GPT-4o-mini predictions at a 64-example budget, one item per dataset. IE denotes information extraction and SP semantic parsing.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 404 papers

Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data
Jun 10, 2026Antonio Pelusi, Stefano Braghin, Alberto TrombettaLLM Fine-Tuning In-Context Learning
University of Insubria, Italy · IBM Research, Ireland
A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling
Jun 27, 2026Zhenyu Liu, Huaze Tang, Shao-Lun HuangIn-Context Learning Exponential Family
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning
Sep 30, 2026Gunmay JhingranRepresentation Learning In-Context Learning
Delhi Technological University

Jun 10, 2026J/K move · Enter open · S save

Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data

Antonio Pelusi, Stefano Braghin, Alberto Trombetta

University of Insubria, Italy · IBM Research, Ireland

LLM Fine-Tuning In-Context Learning

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as conditional generators for structured data, relying on in-context learning (ICL) to adapt to new distributions without parameter updates. We investigate the limits of ICL for structured generation under distribution mismatch, using high-cardinality tabular data as a controlled test case, and identify a structural failure mode we term categorical prior lock-in: the inability of ICL to update the model's prior over token distributions inherited from pre-training. Across two 7B-parameter open-weight models, ICL improves numerical fidelity with additional examples but exhibits a sharp ceiling on categorical distributions, failing to reproduce rare classes entirely. Parameter-efficient fine-tuning (LoRA) overcomes these limitations but introduces measurable memorization risk and, in some cases, destabilizes structured output generation, highlighting a fundamental trade-off between adaptability and privacy.

Abstract

In-Context Learning

Explore similar work

Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data

A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling

Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning