Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails

Sai Krishna Arthanari; JaeHyeong Chang; Chengzhe Sun; Siwei Lyu

cs.CLAug 21, 2026

Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails

Authors: Sai Krishna Arthanari, JaeHyeong Chang, Chengzhe Sun, Siwei Lyu

Organizations: Institute for Artificial Intelligence and Data Science (IAD), University at Buffalo Buffalo, NY, USA

Abstract

Splitting a bidirectional LSTM's contextual representation into a forward-only $F_i$ (strictly a function of tokens $1..i$ ) and a backward-only $B_i$ (strictly a function of tokens $i..n$ ) beats either alone and beats a fused self-attention representation for dependency relation-type classification. But a specific, natural extension of this idea -- pairing a token's forward state against a candidate's backward state (``cross-direction'' pairing, $F_i$ vs.\ $B_j$ ) -- consistently underperforms same-direction pairing, and the penalty grows, not shrinks, with token distance, both paired-bootstrap significant. We diagnose why using a frozen-trunk methodology: architectural information leakage between directions is impossible by construction (a single-layer BiLSTM, verified by code inspection); 93% of the same-vs-cross gap survives freezing the trunk and training only fresh heads, ruling out training-co-adaptation as the primary cause; linear regression shows partial representational redundancy between $F_i$ and $B_i$ ( $R^2{=}0.324$ vs.\ $0.028$ for a shuffled control) and a linear probe shows partial anticipatory encoding of upcoming tokens in $F_i$ (36.5% vs.\ 17.2% majority baseline) -- real effects, but neither alone, nor combined, cleanly explains the full gap. Extended frozen-trunk diagnostics (a positional probe and a distance-decay probe) show directional information is genuinely stored but not exactly positioned, and propagates only a few tokens before decaying to baseline -- consistent with, and mechanistically underneath, the distance-growth finding.

Figures & tables

Panel of Figure 1 — Fig. 1: Trunk architecture. A single-layer BiLSTM over token embeddings exposes $F_{i}$ and $B_{i}$ directly, with no attention or mixing layer between the LSTM and the split. $P_{i}$ is computed independently of the LSTM. $H_{i}=\mathrm{concat}(F_{i},B_{i},P_{i})$ is the per-token diagnostic representation used by the positional and distance-decay probes (Section V C).

Variant	Test Acc.	5-seed Acc.	Macro-F1	Params
f_only	0.9667	0.9650 $\pm$ .0016	0.9654	1.39M
b_only	0.9625	0.9622 $\pm$ .0011	0.9617	1.39M
f_plus_b	0.9725	–	0.9720	1.52M
f_plus_b_plus_p	0.9733	0.9716 $\pm$ .0020	0.9732	1.54M
standard_attention	0.9652	0.9646 $\pm$ .0009	0.9645	2.34M

TABLE I: Relation-type classification, test set. Single-seed columns are the primary run; 5-seed column (mean

\pm

std, seeds 42–46) confirms it.

Variant	Test Acc.	Test F1	Params
same_f (FF)	0.9362 $\pm$ 0.0004	0.9364 $\pm$ 0.0003	1.39M
same_b (BB)	0.9351 $\pm$ 0.0014	0.9354 $\pm$ 0.0015	1.39M
cross_fb (FB)	0.9050 $\pm$ 0.0018	0.9058 $\pm$ 0.0025	1.39M
cross_bf (BF)	0.8588 $\pm$ 0.0022	0.8582 $\pm$ 0.0025	1.39M
combined_e	0.9543 $\pm$ 0.0004	0.9547 $\pm$ 0.0004	1.40M
full_ref	0.9522 $\pm$ 0.0015	0.9524 $\pm$ 0.0016	1.54M

TABLE II: Binary edge existence, test set, mean

\pm

std over 3 seeds.

$\|i-j\|$ bucket	1–2	3–5	6–10	11–20	21+
$n$ (test pairs)	19,571	11,639	7,576	5,188	1,922
Accuracy gap	$-$ 0.0233	$-$ 0.0300	$-$ 0.0384	$-$ 0.0502	$-$ 0.0538
F1 gap	$-$ 0.0166	$-$ 0.0324	$-$ 0.0606	$-$ 0.1045	$-$ 0.1137

TABLE III: cross_fb

-

same_f accuracy/F1 gap by token distance (test, mean over 3 seeds).

Fig. 2: cross_fb

-

same_f accuracy and F1 gap by token-distance bucket (test set, mean over 3 seeds). Both gaps are monotonically negative and grow with distance – the opposite of the “masked by short-range dominance” prediction.

Condition	same_f Acc.	cross_fb Acc.
Frozen trunk (fresh heads, 3 seeds)	0.9240 $\pm$ 0.0008	0.8943 $\pm$ 0.0006
End-to-end	0.9367 $\pm$ 0.0001	0.9049 $\pm$ 0.0017

TABLE IV: Frozen-trunk vs. end-to-end same_f

-

cross_fb accuracy gap.

Source	Dim	MLP Acc.	Linear Acc.	MLP Frac. MAE
$F_{i}$	128	0.3310	0.3103	0.1920
$B_{i}$	128	0.3314	0.2983	0.1919
$F_{i}{+}B_{i}$ (no $P$ )	256	0.4830	0.4072	0.1436
$H_{i}$ (with $P_{i}$ )	320	0.6679	0.5140	0.1350

TABLE V: Position-probe bucket accuracy, MLP vs. linear head, mean over 3 seeds (majority baseline 0.1367). MAE columns are for the MLP head’s normalized-position regression (median baseline MAE 0.2718).

$\Delta$	Fwd. probe	Fwd. maj.	Bwd. probe	Bwd. maj.
1	0.3669	0.1721	0.3518	0.1722
2	0.2240	0.1729	0.1985	0.1543
3	0.1889	0.1764	0.1686	0.1552
5	0.1814	0.1814	0.1541	0.1540
10	0.1806	0.1847	0.1549	0.1532
20	0.1797	0.1805	0.1457	0.1535

TABLE VI: Distance-decay probe accuracy vs. majority baseline, frozen trunk, seed 42.

Backbone	Params	Dev UAS	Test UAS
BiLSTM	1.535M	0.8580 $\pm$ .0012	0.8609 $\pm$ .0014
Transformer (unmatched)	2.339M	0.8308 $\pm$ .0047	0.8305 $\pm$ .0043
Transformer (matched)	1.490M	0.8337 $\pm$ .0029	0.8349 $\pm$ .0037

TABLE VII: Full-pairwise arc scorer backbone comparison, mean

\pm

std over 5 seeds.

Comparison	Point	95% CI	Cohen’s $d$
BiLSTM $-$ Transf. (unmatched)	+0.0386	(+0.0331, +0.0444)	0.216
BiLSTM $-$ Transf. (matched)	+0.0234	(+0.0180, +0.0287)	0.085

TABLE VIII: Paired bootstrap, BiLSTM

-

Transformer UAS (seed 42, sentence-level).

Claim	EWT	GUM
$f\_plus\_b-$ standard_attn. (acc. gap)	+0.0073	+0.0077
same_f $-$ cross_fb (3-seed acc. gap)	+0.0312	+0.0397

TABLE IX: EWT vs. GUM, core claims (test set).

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 798 papers

The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse
Mar 13, 2026Julian Coda-Forno, Jane X. Wang, Arslan ChaudhryDecoder-Only Language Models Masked Language Modeling
TUM, Helmholtz Munich · Google DeepMind
Concept Direction Reliability Across Languages with Different Tokenizer Fertility
Sep 28, 2026Muhammad Abdullahi Said, Abass Oguntade, Elisha Komolafe +3Multilingual Sentiment Analysis Cross-Lingual Consistency in Language Models
African Institute for Mathematical Sciences · University of Cape Town · Bayero University Kano +2
Readout Blindness: VLM Scores Miss the Spatial Direction Their Frozen Encoders Retain
Oct 5, 2026Guangyuan Li, Tianming Du, Yan Jiang +2VLM Evaluation Vision-Language Grounding
ELLIS Institute Finland · Aalto University · University of Oulu +1

Mar 13, 2026J/K move · Enter open · S save

The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse

Julian Coda-Forno, Jane X. Wang, Arslan Chaudhry

TUM, Helmholtz Munich · Google DeepMind

Decoder-Only Language Models Masked Language Modeling Representation Geometry in Language Models

Open paper AskPDF arXiv

The reversal curse describes a failure of autoregressive language models to retrieve a fact in reverse order (e.g., training on $A > B$'' but failing on

B < A

''). Recent work shows that objectives with bidirectional supervision (e.g., bidirectional attention or masking-based reconstruction for decoder-only models) can mitigate the reversal curse. We extend this evaluation to include a vanilla masked language modeling (MLM) objective and compare it to decoder-only masking-based training across four reversal benchmarks and then provide a minimal mechanistic study of how these objectives succeed. We show that reversal accuracy requires training signal that explicitly makes the source entity a prediction target, and we find little evidence that success corresponds to a single direction-agnostic representation of a fact. Instead, representation distances and linear probes are consistent with storing forward and reverse directions as distinct entries, with different indexing geometry for MLM versus decoder-only masking-based training. Our results caution that objective-level ``fixes'' can improve reversal behavior without necessarily inducing the kind of latent generalization one might expect from a unified concept.

Abstract

Representation Learning

Explore similar work

The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse

Concept Direction Reliability Across Languages with Different Tokenizer Fertility

Readout Blindness: VLM Scores Miss the Spatial Direction Their Frozen Encoders Retain