POS-Signature Demonstration Retrieval for In-Context Dependency Parsing
Authors: Muhammad Azam Afridi, Rajab Ali, Marco Passarotti
Organizations: Università Cattolica del Sacro Cuore, Milan, Italy · University of Zaragoza, Zaragoza, Spain · Universidade NOVA de Lisboa, Lisbon, Portugal · Université libre de Bruxelles, Brussels, Belgium
Demonstrations for in-context learning are often retrieved by semantic similarity, which may be a poor proxy for relevance in structured prediction. We study demonstration selection for Universal Dependencies parsing with a POS-signature policy that represents sentences with weighted part-of-speech statistics and retrieves three length-compatible, gold-parsed examples from the labeled training split. We evaluate two open-weight language models without fine-tuning on all 2,077 sentences of the UD English Web Treebank test set, comparing the policy with shot- and format-matched fixed and semantic three-shot baselines, as well as zero-shot, chain-of-thought, and critique-refinement systems. POS-signature retrieval yields the highest labeled attachment score among the tested prompting systems for both models (72.4 and 73.2), outperforming semantic retrieval by 1.9 and 2.7 points; paired-bootstrap 95% confidence intervals exclude zero for both contrasts. Chain-of-thought and refinement comparisons vary across models. These exploratory results favor the tested POS-signature retrieval approach on English-EWT. We release the prompts, archived outputs, and evaluation code.
Figures & tables
Condition
Demonstrations
Selection
Free-text reasoning
Passes
Zero-Shot
0
—
No
1
Fixed Few-Shot
3
fixed, manually selected
No
1
POS-Signature Few-Shot
3
POS signature + length window
No
1
Semantic Few-Shot
3
full-pool sentence embeddings
No
1
Chain-of-Thought
1
fixed worked example
Yes
1
Critique-Refine
—
revises CoT output
Internal review
2
Table 1: Prompting conditions. The three few-shot policies are shot- and format-matched; comparisons involving CoT or Critique-Refine are system-level.
Figure 1: POS-Signature retrieval. Predicted query UPOS selects three length-compatible, gold-parsed demonstrations but is not shown to the LLM; sentences are illustrative.
Condition
gpt-oss
Qwen
UAS (%)
LAS (%)
UAS (%)
LAS (%)
Zero-Shot
75.1
64.7
75.7
62.9
Fixed Few-Shot
76.5
67.3
76.3
65.3
POS-Signature Few-Shot
80.4
72.4
82.1
73.2
Semantic Few-Shot
79.6
70.5
80.8
70.5
Chain-of-Thought
78.6
70.0
72.4
61.7
Table 2: Corpus UAS/LAS (%) on 2,077 EWT test sentences and 21,998 non-punctuation tokens; unscorable outputs receive zero. Bold and underline mark the best and second-best LLM results, not significance. The duplicated Stanza values represent one reference run.
Comparison (first − second)
Status
gpt-oss
Qwen
Δ LAS pp [95% CI]
p
Δ LAS pp [95% CI]
p
POS-Signature − Semantic
Primary
+1.87 [1.06, 2.69]
.0008†
+2.68 [1.97, 3.41]
.0008†
POS-Signature − Fixed
Primary
+5.09 [4.21, 5.98]
.0008†
+7.86 [7.05, 8.71]
.0008†
CoT − Zero-Shot
Exploratory
+5.30 [4.41, 6.19]
.0002
−1.18 [ −2.22 , −0.21 ]
.0158
Critique-Refine − CoT
Exploratory
+0.44 [ −0.02 , 0.91]
.0634
+1.90 [1.53, 2.28]
.0002
Table 3: Paired-bootstrap LAS differences (pp) with percentile 95% CIs from 10,000 paired sentence resamples. † Primary p -values are Holm-adjusted across four tests; exploratory values are unadjusted. All tests are two-sided; Appendix C.2 reports the remaining comparisons.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Relation
Gold support
gpt-oss LAS (%)
Qwen LAS (%)
POS-Signature
Semantic
POS-Signature
Semantic
nsubj
1,950
78.4
76.4
79.8
77.4
nmod:poss
387
73.1
59.2
84.2
61.2
cc
750
73.7
68.1
84.5
82.4
cop
584
44.3
38.4
52.9
48.6
nmod:unmarked
111
34.2
28.8
43.2
36.9
Appendix
Table 5: LAS (%) for six diagnostic dependency relations. Gold-token counts are shared across conditions.
Critique outcome
gpt-oss
Qwen
Analyzed non-punctuation tokens
21,998
21,801
Token prediction changed
2,260 (10.3%)
2,084 (9.6%)
Wrong → correct
703
756
Correct → wrong
606
338
Wrong → different wrong
951
990
Net LAS-correct tokens
+97
+418
Appendix
Table 7: Changes from CoT to Critique-Refine. Qwen transition counts use 21,801 tokens with schema-valid parses at both stages; full-corpus gains use 21,998 tokens.
Model
Condition
Schema-valid
Root warnings
Tree-conforming
gpt-oss
Zero-Shot
2,077
5
2,017
gpt-oss
Fixed Few-Shot
2,077
1
2,035
gpt-oss
POS-Signature Few-Shot
2,077
1
2,005
gpt-oss
Semantic Few-Shot
2,077
3
2,009
gpt-oss
Chain-of-Thought
2,077
2
2,044
gpt-oss
Critique-Refine
2,077
2
2,031
Appendix
Table 8: Output validity, out of 2,077 sentences per condition. Root warnings count schema-valid responses with no token or multiple tokens having head=0 ; tree checks impose the additional requirements described in the text.
In-context example selection chooses demonstrations that help language and multimodal models perform a query-specific task without changing model weights.
We address the challenge of syntactic parsing for Urdu, a morphologically rich language, and present state-of-the-art results for both constituency and dependency parsing. This paper offers four major contributions: 1) the conversion of the CLE-UTB phrase structure treebank into a dependency treebank by developing language-specific head-word and phrase-to-dependency label mapping rules; 2) a novel sequence labeling scheme that transforms the parsing task into a unified representation; 3) the training of contextualized word representations on a large 220 million tokens Urdu corpus collected from the web; and 4) development of parsing framework using two learning paradigms, single-task and multi-task learning. Several post-processing rules are applied to improve the quality of the automatically converted dependency structure treebank. The proposed sequence labeling scheme enables the use of a shared architecture that learns the syntactic structures from both grammatical structures simultaneously and hence improves generalization. Experiments show that the multi-task learning setup significantly enhances parsing performance, achieving an F1 score of 91.39 for constituency parsing (an improvement of 3.29 points) and a labeled attachment score of 85.69 for dependency parsing (an improvement of 1.49 points). These results demonstrate that learning cross-task representations provides measurable benefits and advances the state of syntactic parsing for Urdu.
Toqeer Ehsan, Miriam Butt, Sarmad Hussain +2
Reliable Intelligence Team, VTT Technical Research Centre of Finland Ltd., 02150 Espoo, Finland. · Department of Linguistics, University of Konstanz, 78464 Konstanz, Germany. · Center for Language Engineering, Al-Khawarizmi Institute of Computer Science, University of Engineering and Technology, 54890 Lahore, Pakistan. +2
We describe THIVLVC, a two-stage system for the EvaLatin 2026 Dependency Parsing task. Given a Latin sentence, we retrieve structurally similar entries from the CIRCSE treebank using sentence length and POS n-gram similarity, then prompt a large language model to refine the baseline parse from UDPipe using the retrieved examples and UD annotation guidelines. We submit two configurations: one without retrieval and one with retrieval (RAG). On poetry (Seneca), THIVLVC improves CLAS by +17 points over the UDPipe baseline; on prose (Thomas Aquinas), the gain is +1.5 CLAS. A double-blind error analysis of 300 divergences between our system and the gold standard reveals that, among unanimous annotator decisions, 53.3% favour THIVLVC, showing annotation inconsistencies both within and across treebanks.
Luc Pommeret, Thibault Wagret, Jules Deret
STL · Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France · ENS de Lyon, HiSoMA +1
In Rhetorical Structure Theory (RST), each edge of a discourse tree carries a nuclearity label, marking which span holds the central content, and a relation drawn from a closed inventory of functional categories. Neither label is a surface property: the same two clauses can be Elaboration or Background depending on rhetorical intent, and the boundary between them is fixed by annotation convention rather than by lexical cues. Predicting these labels therefore requires a formal discourse schema, and exposure to discourse-rich pretraining text does not supply it: given only a target EDU pair, a strong LLM recovers nuclearity at 0.794 F1 and relation at 0.738 F1. We propose RAG-RST, a retrieval-augmented framework that supplies this schema as in-context demonstration rather than through parameter updates, raising nuclearity to 0.846 F1 and relation to 0.788 F1 with no training. Two configurations are compared: a precomputed strategy that labels all EDU pairs before parsing, and a dynamic strategy that retrieves at each reduce step, so that the exemplars reaching the generator depend on the reductions the parser has already committed to. Both outperform existing RST parsers across span, nuclearity, relation, and full tree metrics, with the dynamic variant consistently strongest. Two properties of the retrieved schema emerge: its utility is bounded in number, saturating and then declining beyond fifty exemplars, and it depends on structural position. Both indicate that a formal linguistic schema can be supplied to a foundation model as a demonstration rather than trained into its parameters.
Mohammadreza Sediqin, Shlomo Engelson Argamon
Illinois Institute of Technology New York, USA · Touro University New York, USA