Avey-B

Devang Acharya; Mohammad Hammoud

cs.CLFeb 17, 2026

Avey-B

Authors: Devang Acharya, Mohammad Hammoud

Organizations: Avey AI

Abstract

Compact pretrained bidirectional encoders remain the backbone of industrial NLP under tight compute and memory budgets. Their effectiveness stems from self-attention's ability to deliver high-quality bidirectional contextualization with sequence-level parallelism, as popularized by BERT-style architectures. Recently, Avey was introduced as an autoregressive, attention-free alternative that naturally admits an encoder-only adaptation. In this paper, we reformulate Avey for the encoder-only paradigm and propose several innovations to its architecture, including decoupled static and dynamic parameterizations, stability-oriented normalization, and neural compression. Results show that this reformulated architecture compares favorably to four widely used Transformer-based encoders, consistently outperforming them on standard token-classification and information-retrieval benchmarks while scaling more efficiently to long contexts.

Figures & tables

Figure 1: A simple illustration of coupled (a) and decoupled (b) parameterizations (

e_{i}=

embedding

i

;

s_{ij}=

cosine similarity score between

e_{i}

and

e_{j}

;

n_{i}=

neuron

i

{n_{i}}^{(d)}=

neuron

i

in dynamic layer

d

;

{n_{i}}^{(s)}=

neuron

i

in static layer

s

; and

w_{ij}=

weight corresponding to

e_{i}

{n_{i}}^{(d)}

used in the weighted sum of

n_{j}

{n_{j}}^{(s)}

, respectively).

Question	Answer	Experiments
What is the most effective arrangement of static (S) and dynamic (D) layers?	Interleaved layers with a repeating $S\rightarrow D$ pattern	Appendix D
What is the most effective normalization technique within the dynamic layers, Softmax, RMS Norm, or Row-wise normalization?	Row-wise normalization	Appendix E
What are the best values for sequence length $N$ , split size $S$ , and top $k$ splits?	$N=2048$ , $S=256$ , $k=3$	Appendix F
Should the ranker operate bidirectionally as well?	No	Appendix C
What is the best masking rate?	20%	Appendix G

Table 1: Design and masked language modeling (MLM) choices.

	Model	SC				TC				QA				IR
	Model	MNLI	QQP	SST-2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuAD v2	Avg.	MLDR	MS MARCO	NQ	Avg.
Base	Avey - B	$83.58$	89.81	92.94	$88.78$	92.88	93.80	94.10	93.59	$44.03$	$74.44$	$68.88$	$62.45$	63.83	88.14	83.62	78.53
	BERT	$81.92$	$88.57$	$90.94$	$87.14$	$90.25$	$91.03$	$88.20$	$89.82$	$36.76$	$72.20$	$63.99$	$57.65$	$57.42$	$81.15$	$80.66$	$73.08$
	RoBERTa	$86.42$	$89.12$	$92.78$	$89.44$	$90.55$	$92.11$	$88.16$	$90.27$	67.86	80.68	$76.62$	75.05	$56.07$	$86.47$	$80.30$	$74.28$
	ModernBERT	86.72	$89.81$	$92.32$	89.61	$92.30$	$93.74$	$92.30$	$92.78$	$65.73$	$80.23$	77.36	$74.44$	$54.29$	$88.09$	$75.24$	$72.54$
M	NeoBERT	$82.53$	$88.88$	$84.69$	$85.36$	$87.55$	$88.88$	$88.17$	$88.20$	$37.74$	$64.84$	$64.42$	$55.67$	$39.98$	$70.76$	$59.43$	$56.72$

Table 2: Effectiveness results for several encoders at different scales (M = Medium).

Figure 2: Throughput of Avey - B, ModernBERT, and NeoBERT on NVIDIA B200 GPUs with mixed precision (BF16). We use Avey - B base , ModernBERT base , and NeoBERT medium (the only publicly available size). Avey - B is shown in (a) as optimized using torch.compile (no fused-kernel implementation is available yet) and in (b) as unoptimized (eager). For ModernBERT and NeoBERT, throughput is shown for system–optimized ( with FlashAttention) and system–unoptimized (eager) variants in (a) and (b), respectively.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	BERT		RoBERTa		ModernBERT		NeoBERT	Avey-B
	base	large	base	large	base	large	medium	base	large
Parameters	120M	350M	125M	355M	149M	395M	250M	165M	391M
Data Sources	BooksCorpus Wikipedia		BooksCorpus OpenWebText Stories / CC-News		Undisclosed		RefinedWeb	FineWeb
Pre-training Context Width	512		512		$1{,}024\rightarrow 8{,}192$		$1{,}024\rightarrow 4{,}096$	$2{,}048$
Inference Sequence Length	512		512		8,192		4,096	$\infty$
Masking Rate	15%		15%		30%		20%	20%

Appendix

Table 3: A comparison of all the evaluated encoders across different dimensions.

Ranker Type	SC				TC				QA				IR
Ranker Type	MNLI	QQP	SST-2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuADv2	Avg.	MLDR	MS MARCO	NQ	Avg.
Unidirectional ranker	81.39	88.92	91.86	87.39	92.96	93.21	$93.97$	93.38	30.22	62.26	60.72	51.07	60.27	87.48	76.71	74.82
Bi-directional ranker	$80.54$	$88.45$	$91.74$	$86.91$	$92.62$	$92.91$	94.35	$93.29$	$10.64$	$45.34$	$53.56$	$36.51$	$60.05$	$90.02$	$48.54$	$66.20$

Appendix

Table 4: Effectiveness results comparing unidirectional vs. bi-directional rankers.

Pattern	SC				TC				QA				IR
Pattern	MNLI	QQP	SST-2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuADv2	Avg.	MLDR	MS MARCO	NQ	Avg.
Interleaved, $S\!\rightarrow\!D\!\rightarrow\!\cdots$	81.39	88.92	91.86	87.39	92.96	93.21	$93.97$	93.38	30.22	62.26	60.72	51.07	$60.27$	$87.48$	$76.71$	$74.82$
Interleaved, $D\!\rightarrow\!S\!\rightarrow\!\cdots$	$77.89$	$87.51$	$90.37$	$85.26$	$91.73$	$92.48$	$93.24$	$92.48$	$21.31$	$56.93$	$55.77$	$44.67$	$52.67$	$88.47$	$68.63$	$69.92$
Single dynamic as a head	$73.69$	$86.55$	$91.06$	$83.77$	$92.42$	$92.75$	$93.31$	$92.83$	$22.42$	$56.94$	$54.87$	$44.74$	$60.22$	89.70	$73.32$	$74.41$
Single dynamic as a tail	$73.19$	$87.90$	$91.63$	$84.24$	$92.58$	$93.10$	$93.80$	$93.16$	$24.64$	$55.29$	$54.06$	$44.66$	$60.55$	$87.72$	$75.31$	$74.53$
Two - stage stack, $S^{L/2}\!\rightarrow\!D^{L/2}$	$75.70$	$86.56$	$90.25$	$84.17$	$92.51$	$92.65$	94.24	$93.13$	$15.29$	$52.18$	$55.09$	$40.85$	$54.97$	$85.92$	$67.54$	$69.48$

Appendix

Table 5: Effectiveness results across different static (S) and dynamic (D) layering patterns.

Normalization Scheme	SC				TC				QA				IR
Normalization Scheme	MNLI	QQP	SST-2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuADv2	Avg.	MLDR	MS MARCO	NQ	Avg.
Divide-by-sum norm	81.39	88.92	91.86	87.39	92.96	93.21	$93.97$	93.38	30.22	62.26	60.72	51.07	$60.27$	$87.48$	76.71	$74.82$
RMS norm	$64.70$	$87.21$	$88.76$	$80.22$	$90.82$	$92.12$	$91.89$	$91.61$	$21.37$	$56.09$	$56.33$	$44.60$	$50.27$	$89.33$	$66.68$	$68.76$
Softmax	$79.31$	$88.16$	$91.06$	$86.18$	$92.39$	$92.96$	$93.45$	$92.93$	$27.70$	$59.29$	$58.55$	$48.51$	$61.83$	89.75	$74.43$	75.34
Scaled softmax	$76.70$	$87.24$	91.86	$85.27$	$92.63$	$93.02$	94.07	$93.24$	$24.14$	$58.79$	$56.23$	$46.39$	62.24	$87.63$	$74.39$	$74.75$

Appendix

Table 6: Effectiveness results across different normalization schemes.

N	S	$\bm{k}$	SC				TC				QA				IR
			MNLI	QQP	SST-2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuADv2	Avg.	MLDR	MS MARCO	NQ	Avg.
512	128	1	$80.09$	$88.82$	$91.63$	$86.846\,666\,67$	$92.69$	93.08	93.85	93.20666667	$27.65$	$60.44$	$58.82$	$48.97$	$60.06$	$88.89$	$76.11$	$75.02$
	128	3	80.95	88.89	$91.63$	87.15666667	92.74	$93.03$	$93.55$	$93.106\,666\,67$	$28.11$	$60.66$	$58.79$	$49.186\,666\,67$	60.72	90.11	$76.06$	75.63
	256	1	$79.68$	$88.75$	92.32	$86.916\,666\,67$	$92.27$	$92.94$	$93.69$	$92.966\,666\,67$	43.70	71.25	64.06	59.67	$60.71$	$88.61$	76.29	$75.203\,333\,33$
1024	128	1	80.79	$88.80$	$91.63$	$87.073\,333\,33$	$92.87$	$93.03$	$93.81$	$93.236\,666\,67$	$27.42$	$61.14$	$58.85$	$49.136\,666\,67$	$60.44$	$88.95$	$76.05$	$75.146\,666\,67$
		3	$80.02$	$88.71$	$91.63$	$86.786\,666\,67$	92.89	$93.06$	94.43	93.46	$28.74$	$61.55$	$60.49$	$50.26$	$61.17$	90.26	$75.66$	$75.696\,666\,67$

Appendix

Table 7: Effectiveness results across different sequence length

N

, split size

S

, and top-

k

values.

Masking %	SC				TC				QA				IR
Masking %	MNLI	QQP	SST-2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuADv2	Avg.	MLDR	MS MARCO	NQ	Avg.
10%	$78.07$	$88.02$	$91.86$	$85.98$	$91.61$	$92.54$	$93.50$	$92.55$	$36.64$	$68.57$	$60.65$	$55.29$	$51.02$	$85.75$	$71.50$	$69.42$
20%	80.18	88.91	$91.74$	86.94	$92.33$	93.05	93.75	$93.04$	42.91	71.99	64.81	59.90	$60.02$	90.36	$76.58$	$75.65$
30%	$78.62$	$88.49$	91.97	$86.36$	$92.45$	$93.01$	93.75	93.07	$42.80$	$71.26$	$63.85$	$59.30$	62.16	$89.72$	$76.19$	$76.02$
40%	$77.05$	$88.02$	$91.51$	$85.53$	$92.26$	$92.90$	$93.49$	$92.88$	$39.70$	$69.84$	$62.45$	$57.33$	$59.67$	$90.33$	$74.56$	$74.85$
50%	$66.12$	$88.32$	$91.06$	$81.83$	92.86	$92.62$	$93.16$	$92.88$	$42.15$	$70.44$	$62.90$	$58.50$	$62.03$	$90.06$	77.69	76.59

Appendix

Table 8: Effectiveness results at different masking rates for Avey-B’s base model.

Masking %	SC				TC				QA				IR
Masking %	MNLI	QQP	SST-2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuADv2	Avg.	MLDR	MS MARCO	NQ	Avg.
10%	$81.01$	$88.54$	$92.09$	$87.21$	$92.39$	$92.91$	$93.32$	$92.87$	$42.46$	$71.53$	$64.63$	$59.54$	$54.58$	$88.73$	$75.25$	$72.85$
20%	82.12	$89.19$	92.32	87.88	$92.76$	$92.94$	$93.65$	$93.12$	$47.93$	$72.84$	$65.79$	$62.19$	$63.53$	91.54	$80.47$	$78.51$
30%	$81.54$	89.43	$91.74$	$87.57$	$92.59$	$92.98$	93.83	$93.13$	48.16	73.44	66.52	62.71	$61.72$	$89.31$	81.76	$77.60$
40%	$70.21$	$89.11$	$92.20$	$83.84$	$92.51$	93.13	$93.65$	$93.10$	$46.29$	$73.36$	$65.71$	$61.79$	64.09	$90.96$	$81.35$	78.80
50%	$78.19$	$89.02$	$92.20$	$86.47$	92.83	$92.95$	93.83	93.20	$46.99$	$71.28$	$63.89$	$60.72$	$61.63$	$90.49$	$79.37$	$77.16$

Appendix

Table 9: Effectiveness results at different masking percentages for Avey-B’s large model.

Model	SC				TC				QA				IR
Model	MNLI	QQP	SST - 2	Avg.	CONLL	Onto.	UNER	Avg.	ReCoRD	SQuAD	SQuAD v2	Avg.	MLDR	MS MARCO	NQ	Avg.
Avey - B (full design)	$80.74$	$88.91$	91.97	87.2	91.84	$93.25$	93.09	92.72	$39.6$	$68.52$	60.48	$56.2$	$57.49$	90.38	$75.64$	$74.5$
Avey - B w/o normalization	$77.47$	$84.6$	$90.25$	$84.1$	$90.98$	$92.65$	$92.12$	$91.91$	$29.72$	$67.15$	$58.83$	$51.90$	$46.43$	$82.89$	$59.92$	$63.08$
Avey - B w/o decoupling	$79.94$	$88.6$	$89.33$	$85.95$	$89.1$	$92.11$	$91.06$	$90.75$	$36.86$	$67.89$	$59.57$	$54.77$	$55$	$82.77$	$69.18$	$68.98$
Avey - B w/o compression	80.8	89.03	$91.17$	$87$	$91.52$	93.29	$92.97$	$92.59$	42.8	70.54	$59.77$	57.7	60.95	$89.13$	76.92	75.66
Avey - B w/o residual	$77.53$	$87.48$	$90.37$	$85.12$	$90.8$	$92.44$	$91.17$	$91.47$	$34.71$	$66.08$	$56.02$	$52.27$	$55.22$	$86.74$	$71.72$	$71.22$

Appendix

Table 10: Ablations of Avey - B, removing one component at a time while holding all others fixed. (1) w/o normalization : removes row - wise normalization in the dynamic layers; (2) w/o decoupling : reverts to coupled static and dynamic parameterizations; (3) w/o compression : omits the neural compressor; (4) w/o residual : discards the residual connection between the compressor output and the current split’s tokens; and (5) w/o ranker : disables the ranker entirely.

Figure 3: The throughput of Avey-B with and without the neural compressor.

Figure 4: Latency of Avey - B, ModernBERT, and NeoBERT on NVIDIA B200 GPUs with mixed precision (BF16). We use Avey - B base , ModernBERT base , and NeoBERT medium (the only publicly available size). Avey - B is shown in (a) as optimized using torch.compile (no fused-kernel implementation is available yet) and in (b) as unoptimized (eager). For ModernBERT and NeoBERT, latency is shown for system–optimized ( with FlashAttention) and system–unoptimized (eager) variants in (a) and (b), respectively.

	Model	SC			TC			QA			IR
	Model	MNLI	QQP	SST-2	CONLL	Onto.	UNER	ReCoRD	SQuAD	SQuADv2	MLDR	MSMARCO	NQ
Base	Avey - B	$0.92$	$0.12$	$0.97$	$0.71$	$0.12$	$2.65$	$0.67$	$0.17$	$0.47$	$0.67$	$1.34$	$0.75$
	BERT	$0.17$	$0.16$	$0.31$	$1.10$	$0.31$	$0.69$	$3.70$	$0.53$	$0.32$	$1.22$	$0.73$	$1.44$
	RoBERTa	$0.13$	$0.10$	$0.38$	$0.25$	$0.14$	$0.60$	$0.42$	$0.11$	$0.19$	$0.32$	$1.02$	$0.72$
	ModernBERT	$0.37$	$0.12$	$0.53$	$0.24$	$0.11$	$0.45$	$0.70$	$2.36$	$0.29$	$1.40$	$1.39$	$2.18$
M	NeoBERT	$0.40$	$0.14$	$1.20$	$0.24$	$0.17$	$0.47$	$5.98$	$0.91$	$0.66$	$4.70$	$4.30$	$9.48$

Appendix

Table 11: Standard deviations across 10 random seeds for all evaluated encoders and benchmarks.

Figure 5: Learned static cross - embedding projection matrices for the (a) coupled configuration (left or red) with 15 matrices uniformly subsampled from 30 static layers and (b) decoupled configuration (right or blue) with all 15 static matrices (dynamic and static layers are interleaved, hence, only 15 static matrices exist). For comparability, we display 15 layers per panel. The coupled setting exhibits diffuse, more homogeneous patterns (e.g., see layers 14, 22, 24, and 26) suggestive of redundancy, whereas the decoupled setting shows sharper, more heterogeneous structure and variability in spread, indicating greater representational diversity.

Coupled
Layer	Mean	Std	Min	Median	Max	Abs. Mean	L1 Norm	L2 Norm	Frac. Pos.	Frac. Neg.
1	$0.0011$	$0.1198$	$-1.6666$	$-0.0035$	$1.2015$	$0.0514$	$3369.19$	$30.68$	$0.4675$	$0.5325$
2	$-0.0105$	$0.1078$	$-1.5488$	$0.0023$	$0.3019$	$0.0358$	$2345.75$	$27.73$	$0.5335$	$0.4665$
3	$0.0849$	$0.0894$	$-0.2293$	$0.0640$	$0.6771$	$0.0901$	$5906.35$	$31.56$	$0.8965$	$0.1035$
4	$0.0040$	$0.0953$	$-1.1896$	$0.0077$	$0.2948$	$0.0337$	$2211.32$	$24.42$	$0.6145$	$0.3855$
5	$-0.0261$	$0.1295$	$-1.0887$	$0.0042$	$0.1540$	$0.0568$	$3724.35$	$33.82$	$0.5495$	$0.4505$

Appendix

Table 13: Layer statistics for coupled vs. decoupled settings. For comparability, we display 15 layers per setting. For the coupled setting, we uniformly subsampled 15 layers from 30 static layers. For the decoupled setting, all the 15 static layers are shown (dynamic and static layers are interleaved, hence, only 15 static layers exist). The coupled setting exhibits positivity bias (see the “fraction of positive” column), while the decoupled setting demonstrates more balanced positive and negative weights, indicating greater representational diversity.

	Model	NIAH-1
	Model	1k	2k	4k	8k	16k	32k	64k	96k
Base	Avey - B	$79.405$	$79.210$	$78.940$	$79.185$	$78.905$	$77.725$	$77.175$	$75.715$
Base	ModernBERT	$67.735$	$67.640$	$68.305$	$70.670$	–	–	–	–
M	NeoBERT	$79.650$	$79.130$	$74.725$	–	–	–	–	–
Large	Avey - B	$79.690$	$79.235$	$79.030$	$79.580$	$79.440$	$78.440$	$76.760$	$76.055$
Large	ModernBERT	$68.800$	$67.515$	$67.200$	oom	–	–	–	–

Appendix

Table 14: Needle-in-a-haystack (NIAH-1) accuracy across sequence lengths from 1k to 96k for several encoders at different scales (M = Medium; OOM = Out-of-Memory).

	Model	NIAH-2
	Model	1k	2k	4k	8k	16k	32k	64k	96k
Base	Avey - B	$78.285$	$79.395$	$79.770$	$78.525$	$78.695$	$75.500$	$73.780$	$71.860$
Base	ModernBERT	$66.990$	$67.250$	$69.485$	$70.480$	–	–	–	–
M	NeoBERT	$79.605$	$79.520$	$80.070$	–	–	–	–	–
Large	Avey - B	$78.935$	$79.480$	$79.990$	$78.705$	$79.070$	$78.310$	$74.470$	$74.535$
Large	ModernBERT	$66.960$	$67.285$	$68.070$	oom	–	–	–	–

Appendix

Table 15: Needle-in-a-haystack (NIAH-2) accuracy across sequence lengths from 1k to 96k for several encoders at different scales (M = Medium; OOM = Out-of-Memory).

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

UltraBERT: Beyond the Limit of BERT Pre-Training
Oct 10, 2026Jungwoo Park, Taewhoo Lee, Jaewoo KangLanguage Model Pretraining Masked Language Modeling
Korea University · AIGEN Sciences · Amazon
Wave-Attractor-Tree: A Hierarchical Binary Tree Reduction Architecture for Efficient Sequence Modeling
Feb 28, 2026Igor BerezkinLong-Range Dependency Modeling Sequence Modeling
Independent Researcher
AraSSM: A bidirectional state-space encoder for Arabic masked language modeling
Aug 8, 2026Ahmed Amine Aliane, Hassina Aliane, Nasredine SemmarLanguage Model Pretraining Arabic NLP
Arabic Institute for Translation, Algiers, Algeria · CERIST, Algeria · CEA LIST, Paris-Saclay, France

Oct 10, 2026J/K move · Enter open · S save

UltraBERT: Beyond the Limit of BERT Pre-Training

Jungwoo Park, Taewhoo Lee, Jaewoo Kang

Korea University · AIGEN Sciences · Amazon

Language Model Pretraining Masked Language Modeling Information Retrieval Text Classification Language Model Distillation

Open paper AskPDF arXiv

Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance can be pushed by revisiting core design choices in training, revealing three key findings: (1) Encoders saturate on downstream tasks beyond a few trillion tokens, indicating that additional compute past this point does not help. (2) Across model sizes, lower masking ratios benefit classification while higher ones benefit retrieval, contradicting prior work suggesting larger ratios for larger models. (3) Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones. Guided by these findings, we present UltraBERT, a single encoder that assigns separate LM heads to low and high masking ratios within a batch and restricts the distillation objective to the low ratio. To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR. Together, these results show how far encoder performance can be pushed and provide a principled basis for compute allocation, masking ratio design, and knowledge transfer in bidirectional models.

Abstract

Language Modeling

Explore similar work

UltraBERT: Beyond the Limit of BERT Pre-Training

Wave-Attractor-Tree: A Hierarchical Binary Tree Reduction Architecture for Efficient Sequence Modeling

AraSSM: A bidirectional state-space encoder for Arabic masked language modeling