POS-Signature Demonstration Retrieval for In-Context Dependency Parsing

Muhammad Azam Afridi; Rajab Ali; Marco Passarotti

cs.CLOct 22, 2026

POS-Signature Demonstration Retrieval for In-Context Dependency Parsing

Authors: Muhammad Azam Afridi, Rajab Ali, Marco Passarotti

Organizations: Università Cattolica del Sacro Cuore, Milan, Italy · University of Zaragoza, Zaragoza, Spain · Universidade NOVA de Lisboa, Lisbon, Portugal · Université libre de Bruxelles, Brussels, Belgium

Abstract

Demonstrations for in-context learning are often retrieved by semantic similarity, which may be a poor proxy for relevance in structured prediction. We study demonstration selection for Universal Dependencies parsing with a POS-signature policy that represents sentences with weighted part-of-speech statistics and retrieves three length-compatible, gold-parsed examples from the labeled training split. We evaluate two open-weight language models without fine-tuning on all 2,077 sentences of the UD English Web Treebank test set, comparing the policy with shot- and format-matched fixed and semantic three-shot baselines, as well as zero-shot, chain-of-thought, and critique-refinement systems. POS-signature retrieval yields the highest labeled attachment score among the tested prompting systems for both models (72.4 and 73.2), outperforming semantic retrieval by 1.9 and 2.7 points; paired-bootstrap 95% confidence intervals exclude zero for both contrasts. Chain-of-thought and refinement comparisons vary across models. These exploratory results favor the tested POS-signature retrieval approach on English-EWT. We release the prompts, archived outputs, and evaluation code.

Figures & tables

Condition	Demonstrations	Selection	Free-text reasoning	Passes
Zero-Shot	0	—	No	1
Fixed Few-Shot	3	fixed, manually selected	No	1
POS-Signature Few-Shot	3	POS signature + length window	No	1
Semantic Few-Shot	3	full-pool sentence embeddings	No	1
Chain-of-Thought	1	fixed worked example	Yes	1
Critique-Refine	—	revises CoT output	Internal review	2

Table 1: Prompting conditions. The three few-shot policies are shot- and format-matched; comparisons involving CoT or Critique-Refine are system-level.

Figure 1: POS-Signature retrieval. Predicted query UPOS selects three length-compatible, gold-parsed demonstrations but is not shown to the LLM; sentences are illustrative.

Condition	gpt-oss		Qwen
	UAS (%)	LAS (%)	UAS (%)	LAS (%)
Zero-Shot	75.1	64.7	75.7	62.9
Fixed Few-Shot	76.5	67.3	76.3	65.3
POS-Signature Few-Shot	80.4	72.4	82.1	73.2
Semantic Few-Shot	79.6	70.5	80.8	70.5
Chain-of-Thought	78.6	70.0	72.4	61.7

Table 2: Corpus UAS/LAS (%) on 2,077 EWT test sentences and 21,998 non-punctuation tokens; unscorable outputs receive zero. Bold and underline mark the best and second-best LLM results, not significance. The duplicated Stanza values represent one reference run.

Comparison (first $-$ second)	Status	gpt-oss		Qwen
		$\Delta$ LAS pp [95% CI]	$p$	$\Delta$ LAS pp [95% CI]	$p$
POS-Signature $-$ Semantic	Primary	$+1.87$ [1.06, 2.69]	$.0008^{\dagger}$	$+2.68$ [1.97, 3.41]	$.0008^{\dagger}$
POS-Signature $-$ Fixed	Primary	$+5.09$ [4.21, 5.98]	$.0008^{\dagger}$	$+7.86$ [7.05, 8.71]	$.0008^{\dagger}$
CoT $-$ Zero-Shot	Exploratory	$+5.30$ [4.41, 6.19]	.0002	$-1.18$ [ $-2.22$ , $-0.21$ ]	.0158
Critique-Refine $-$ CoT	Exploratory	$+0.44$ [ $-0.02$ , 0.91]	.0634	$+1.90$ [1.53, 2.28]	.0002

Table 3: Paired-bootstrap LAS differences (pp) with percentile 95% CIs from 10,000 paired sentence resamples.

\dagger

Primary

p

-values are Holm-adjusted across four tests; exploratory values are unadjusted. All tests are two-sided; Appendix C.2 reports the remaining comparisons.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Relation	Gold support	gpt-oss LAS (%)		Qwen LAS (%)
		POS-Signature	Semantic	POS-Signature	Semantic
nsubj	1,950	78.4	76.4	79.8	77.4
nmod:poss	387	73.1	59.2	84.2	61.2
cc	750	73.7	68.1	84.5	82.4
cop	584	44.3	38.4	52.9	48.6
nmod:unmarked	111	34.2	28.8	43.2	36.9

Appendix

Table 5: LAS (%) for six diagnostic dependency relations. Gold-token counts are shared across conditions.

Critique outcome	gpt-oss	Qwen
Analyzed non-punctuation tokens	21,998	21,801
Token prediction changed	2,260 (10.3%)	2,084 (9.6%)
Wrong $\rightarrow$ correct	703	756
Correct $\rightarrow$ wrong	606	338
Wrong $\rightarrow$ different wrong	951	990
Net LAS-correct tokens	$+97$	$+418$

Appendix

Table 7: Changes from CoT to Critique-Refine. Qwen transition counts use 21,801 tokens with schema-valid parses at both stages; full-corpus gains use 21,998 tokens.

Model	Condition	Schema-valid	Root warnings	Tree-conforming
gpt-oss	Zero-Shot	2,077	5	2,017
gpt-oss	Fixed Few-Shot	2,077	1	2,035
gpt-oss	POS-Signature Few-Shot	2,077	1	2,005
gpt-oss	Semantic Few-Shot	2,077	3	2,009
gpt-oss	Chain-of-Thought	2,077	2	2,044
gpt-oss	Critique-Refine	2,077	2	2,031

Appendix

Table 8: Output validity, out of 2,077 sentences per condition. Root warnings count schema-valid responses with no token or multiple tokens having head=0 ; tree checks impose the additional requirements described in the text.

In-Context Example Selection

In-context example selection chooses demonstrations that help language and multimodal models perform a query-specific task without changing model weights.

See all 29 papers

Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Language
Sep 24, 2026Toqeer Ehsan, Miriam Butt, Sarmad Hussain +2Multi-Task Learning Low-Resource Language Processing
Reliable Intelligence Team, VTT Technical Research Centre of Finland Ltd., 02150 Espoo, Finland. · Department of Linguistics, University of Konstanz, 78464 Konstanz, Germany. · Center for Language Engineering, Al-Khawarizmi Institute of Computer Science, University of Engineering and Technology, 54890 Lahore, Pakistan. +2
THIVLVC: Retrieval Augmented Dependency Parsing for Latin
Apr 7, 2026Luc Pommeret, Thibault Wagret, Jules DeretDependency Parsing Low-Resource Language Processing
STL · Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France · ENS de Lyon, HiSoMA +1
RAG-RST: Retrieved Exemplars as In-Context Discourse Schema for RST Parsing
Oct 15, 2026Mohammadreza Sediqin, Shlomo Engelson ArgamonRetrieval-Augmented ICL
Illinois Institute of Technology New York, USA · Touro University New York, USA

Sep 24, 2026J/K move · Enter open · S save

Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Language

Toqeer Ehsan, Miriam Butt, Sarmad Hussain, Hassan Alhuzali+1

Reliable Intelligence Team, VTT Technical Research Centre of Finland Ltd., 02150 Espoo, Finland. · Department of Linguistics, University of Konstanz, 78464 Konstanz, Germany. · Center for Language Engineering, Al-Khawarizmi Institute of Computer Science, University of Engineering and Technology, 54890 Lahore, Pakistan.+2

Multi-Task Learning Low-Resource Language Processing Dependency Parsing Constituency Parsing

Open paper AskPDF arXiv

We address the challenge of syntactic parsing for Urdu, a morphologically rich language, and present state-of-the-art results for both constituency and dependency parsing. This paper offers four major contributions: 1) the conversion of the CLE-UTB phrase structure treebank into a dependency treebank by developing language-specific head-word and phrase-to-dependency label mapping rules; 2) a novel sequence labeling scheme that transforms the parsing task into a unified representation; 3) the training of contextualized word representations on a large 220 million tokens Urdu corpus collected from the web; and 4) development of parsing framework using two learning paradigms, single-task and multi-task learning. Several post-processing rules are applied to improve the quality of the automatically converted dependency structure treebank. The proposed sequence labeling scheme enables the use of a shared architecture that learns the syntactic structures from both grammatical structures simultaneously and hence improves generalization. Experiments show that the multi-task learning setup significantly enhances parsing performance, achieving an F1 score of 91.39 for constituency parsing (an improvement of 3.29 points) and a labeled attachment score of 85.69 for dependency parsing (an improvement of 1.49 points). These results demonstrate that learning cross-task representations provides measurable benefits and advances the state of syntactic parsing for Urdu.

Abstract

In-Context Example Selection

Explore similar work

Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Language

THIVLVC: Retrieval Augmented Dependency Parsing for Latin

RAG-RST: Retrieved Exemplars as In-Context Discourse Schema for RST Parsing