Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings

Logan Hallee; Jason P. Gleghorn

q-bio.QMApr 9, 2026

Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings

Authors: Logan Hallee, Jason P. Gleghorn

Organizations: Center for Bioinformatics and Computational Biology, University of Delaware · Synthyra, Newark, DE · Department of Biomedical Engineering, University of Delaware

Abstract

Bidirectional transformers are the foundation of many sequence modeling tasks across natural, biological, and chemical language domains, but they are permutation-invariant without explicit positional embeddings. In contrast, unidirectional attention inherently encodes positional information through its triangular mask, enabling models to operate without positional embeddings altogether. Here, we introduce Dual Triangle Attention, a novel bidirectional attention mechanism that separates the query-key subspace of each attention head into two complementary triangular masks: one that attends to past-and-self positions and one that attends to future-and-self positions. This design provides bidirectional context while maintaining the causal mask's implicit positional inductive bias in both directions. Using PyTorch's flex_attention, Dual Triangle Attention is implemented as a single compiled kernel call with no additional parameters beyond standard multi-head attention. We evaluated Dual Triangle Attention across three settings: (1) a synthetic argmax position probe, (2) masked language modeling (MLM) on natural language, and (3) MLM on protein sequences. In the argmax task, both Dual Triangle Attention and causal attention learn positional information without explicit positional embeddings, whereas standard bidirectional attention cannot. In the MLM experiments, Dual Triangle Attention with Rotary Positional Embeddings (RoPE) achieved the best context extension performance and strong performance across the board. These findings suggest that Dual Triangle Attention is a viable attention mechanism for bidirectional transformers, with or without positional embeddings.

Figures & tables

Figure 1: Argmax position probe accuracy across attention types, positional encoding strategies, and model configurations. Each cell shows the best evaluation accuracy (mean across three seeds) for a given combination of hidden size (columns) and number of layers (rows). Dual Triangle Attention and causal attention achieve high accuracy across nearly all configurations, including without explicit positional embeddings. Bidirectional attention fails completely without explicit positional embeddings.

Figure 2: Argmax probe accuracy trend, varying number of layers and hidden size. (a) Without positional embeddings, Dual Triangle and causal attention learn the task, while bidirectional attention cannot. Error bars show 95% confidence intervals across three seeds. (b) At the largest model size (768 hidden, 12 layers), Dual Triangle Attention matches or exceeds both causal and bidirectional attention across all positional encoding strategies.

Panel of Figure 3 — Figure 3: Masked language modeling on natural language (FineWeb-Edu). (a) Validation loss, accuracy, MCC, and F1 at training context length (256 tokens). (b) Validation loss, accuracy, MCC, and F1 at extended context length (1,024 tokens). Shaded regions represent $\pm$ 1 standard deviation across three seeds. (c) Best validation loss, accuracy, MCC, and F1 at training context length. (d) Test loss, accuracy, MCC, and F1 at extended context length. Significance brackets indicate Welch’s $t$ -test results: ∗ $p<0.05$ , ∗∗ $p<0.01$ , ∗∗∗ $p<0.001$ . DTA + RoPE achieves the best performance at both training and extended context lengths. Without positional embeddings, DTA significantly outperforms bidirectional and causal attention.

Panel of Figure 4 — Figure 4: Masked language modeling on protein sequences (OMG-Prot50). (a) Validation loss, accuracy, MCC, and F1 at training context length (256 tokens). (b) Validation loss, accuracy, MCC, and F1 at extended context length (1,024 tokens). Shaded regions represent $\pm$ 1 standard deviation across three seeds. (c) Best validation loss, accuracy, MCC, and F1 at training context length. (d) Test loss, accuracy, MCC, and F1 at extended context length. Significance brackets indicate Welch’s $t$ -test results: ∗ $p<0.05$ , ∗∗ $p<0.01$ , ∗∗∗ $p<0.001$ . Trends mirror the natural language experiments: DTA + RoPE performs best, and DTA without PE outperforms bidirectional without PE.

Panel of Figure 5 — Figure 5: DroPE recovery analysis. (a) NLP extended-context validation loss, accuracy, MCC, and F1 before and after dropping positional embeddings at 70% of training. (b) Protein extended-context validation loss, accuracy, MCC, and F1. The vertical dashed line marks the drop point. Shaded regions represent $\pm$ 1 standard deviation across three seeds. (c) NLP final test loss, accuracy, MCC, and F1 comparing RoPE (kept throughout) vs. RoPE-off (dropped at 70%). (d) Protein final test loss, accuracy, MCC, and F1. Significance brackets indicate Welch’s $t$ -test results: ∗ $p<0.05$ , ∗∗ $p<0.01$ , ∗∗∗ $p<0.001$ . Dropping positions degrades test performance across all attention types for both NLP and protein MLM, in contrast to the success of DroPE in autoregressive settings.

Panel of Figure 6 — Figure 6: Dual Triangle Attention. ( a ) Each logical head’s $Q$ and $K$ tensors ( $l\times d_{h}$ ) are split at $d_{h}/2$ : the down half ( $Q_{\downarrow},K_{\downarrow}$ , blue) computes the lower triangle ( $j\leq i$ ) and the up half ( $Q_{\uparrow},K_{\uparrow}$ , orange) computes the upper triangle ( $j\geq i$ ). The diagonal ( $i=j$ ) is shared by both sub-heads. ( b ) Attention mask comparison. Bidirectional attention permits all position pairs. Causal attention masks future tokens. Dual Triangle Attention provides full bidirectional coverage through complementary triangular masks from separate $Q$ / $K$ subspaces, via a single flex_attention call with head-routed block masks.

Panel of Figure 7 — Figure 7: U-Net transformer architecture for masked language modeling. The encoder ( $\frac{n_{\text{layers}}}{2}$ blocks) feeds forward through the left path, and the decoder ( $\frac{n_{\text{layers}}}{2}$ blocks) ascends through the right path. Dashed arrows indicate skip connections with learnable scalar weights $w_{\text{skip}}$ . Per-layer value embeddings $\text{ve}(x)$ ( nn.Embedding ) from raw token indices are injected into each block. The original input representation $x_{0}$ is mixed into every block via learnable scalars $\lambda_{0},\lambda_{1}$ . Shown with four blocks; generalizes to any even $n_{\text{layers}}$ .

Figure 8: Argmax control experiment with random labels. All attention types achieve near-chance accuracy when labels are randomized, confirming that high accuracy in the main experiments reflects genuine positional learning.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 511 papers

May 10, 2026

Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases

Positional encoding in transformers is commonly implemented through positional embeddings, attention masks, or bias terms, but formal connections between these mechanisms remain limited. We study attention with positional bias through the lens of locality-sensitive hashing (LSH), focusing on Attention with Linear Biases (ALiBi). We show that the ALiBi bias matrix is the expectation of contiguous block-diagonal binary masks induced by a ``positional LSH'' scheme. The empirical mean of masks sampled from this scheme yields spectral norm and max-norm approximation guarantees with bounded block sizes with high probability. This structural theorem implies a uniform approximation theorem for ALiBi-biased attention: with high probability over the sampled masks, the approximate attention output is accurate simultaneously for all query-key-value inputs and can be computed in near-linear time in the context length, reducing long-context ALiBi to a collection of randomized short-context regular (positionally unbiased) attention operations. Conceptually, this connects positional bias, masks, and positional embeddings in a single formal framework and suggests an approach to efficient ALiBi-biased attention. Experiments on large language models validate our theoretical findings.

Daniel Wolfson, Tal Wagner

Blavatnik School of Computer Science and AI, Tel Aviv University

Self-Attention Long-Context Language Modeling Efficient Attention Positional Encoding Attention Mechanisms

Ask PDF

Oct 30, 2023

HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding

In Transformer-based architectures, unmasked self-attention without positional information is permutation-equivariant with respect to the input sequence's tokens. To impose sequential order, token positions are typically encoded using a scheme with either fixed or learnable parameters. We introduce Hyperbolic Positional Encoding (HyPE), a method that utilizes hyperbolic functions' properties to encode tokens' relative positions. This approach biases the attention mechanism without the necessity of storing the

O(L^2)

values of the bias tensor, with

L

being the length of the input sequence. HyPE leverages preliminary concatenation operations and matrix multiplications, encoding relative distances by indirectly incorporating biases into the softmax computation. This design supports gradient backpropagation through the encoding and can be used with FlashAttention-2 under the width and scaling conditions below. We analytically demonstrate that, by careful hyperparameter selection, HyPE can approximate the causal attention bias of ALiBi \citep{press2021train} on a bounded range; model generalization beyond the pretraining length remains to be evaluated. The experimental evaluation of HyPE is proposed as a direction for future research.

Giorgio Angelotti

Independent Researcher.

Relative Positional Encoding Positional Encoding Transformer Attention Memory-Efficient Attention

Ask PDF

Aug 9, 2026

Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling

Self-attention models content-dependent interactions between tokens but does not by itself encode token order. Position encoding addresses this limitation by introducing absolute coordinates, relative distances, or position-dependent rotations into Transformer representations and attention scores. This technical survey develops a unified account of sinusoidal and learned absolute position embeddings, Shaw-style relative position representations, Transformer-XL, T5 relative position bias, ALiBi, and Rotary Position Embeddings (RoPE). We derive how RoPE converts absolute position indices into relative phase differences in Query-Key inner products and compare these methods in terms of where position is injected, computational cost, compatibility with KV caching, and length extrapolation. We then examine long-context extensions, including Position Interpolation, RoPE scaling laws, NTK-aware scaling, Dynamic NTK, NTK-by-parts, YaRN, LongRoPE, and LongRoPE2, with emphasis on frequency allocation, attention rescaling, training length, and target context length. We also summarize implementation considerations, evaluation protocols, and position-encoding choices in representative large language models. A central conclusion is that the ability to compute positional features beyond the training length does not imply reliable long-context generalization; context extension must be evaluated through short-context retention, position-wise perplexity, retrieval, reasoning, and long-context code tasks.

Jiguo Li

Rotary Positional Embeddings Long-Context Modeling Positional Encoding LLM Context Management Relative Positional Encoding

Ask PDF

Abstract

Attention Mechanisms

Explore similar work

Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases

HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding

Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling