A Scaled-Up Empirical Study of Syntactic Language Models

Pengcheng Wang; Zhexuan Zhang; Chao Lou; Kewei Tu

cs.CLOct 12, 2026

A Scaled-Up Empirical Study of Syntactic Language Models

Authors: Pengcheng Wang, Zhexuan Zhang, Chao Lou, Kewei Tu

Organizations: School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging · Independent researcher

Abstract

Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks. Prior studies, however, have mostly evaluated smaller models trained on less pretraining data and tested on narrower task suites. In addition, the design space of tree linearization and attention masks remains underexplored. We conduct a large-scale empirical study of 12 controlled configurations---one terminal baseline and 11 SLM variants spanning different linearizations and masks---at scales up to 1B parameters and 100B training tokens, evaluated across 11 downstream tasks and two syntactic benchmarks. The experimental results show that SLMs with standard causal attention match or exceed structural-mask variants across downstream and syntactic benchmarks. They require no architectural modifications and remain compatible with FlashAttention. We also observe that when scaling up to the 1B-parameter scale, the top-performing SLM variant no longer shows a clear advantage over a compute-matched non-structural baseline, suggesting that SLMs may become less competitive with scale. Our code is available at https://github.com/qlwpc/TG-Interpolation.

Figures & tables

Name	Left( $\mathbf{v}$ )	Right( $\mathbf{v}$ )	Post-process	Example
Linearization
$\textsc{Lin}^{1}$	ONT( $v$ )	CNT( $v$ )	-	(S Cats (VP chase mice VP) S)
$\textsc{Lin}^{2}$	ONT( $v$ )	2 $\times$ CNT( $v$ )	-	(S Cats (VP chase mice VP) VP) S) S)
$\textsc{Lin}^{3}$	ONT( $v$ )	3 $\times$ CNT( $v$ )	-	(S Cats (VP chase mice VP) VP) VP) S) S) S)
$\textsc{Lin}^{1}_{\text{--ont}}$	$\emptyset$	CNT( $v$ )	-	Cats chase mice VP) S)
$\textsc{Lin}^{1}_{\text{--merge}}$	ONT( $v$ )	CNT( $v$ )	Merge CNTs	(S Cats (VP chase mice )

Table 1: Overview of linearizations and non-structural token sequences used in this paper. “Post-process” refers to operations applied after linearization. ONT and CNT denote opening (e.g., (S and (VP ) and closing non-terminal tokens (e.g., S) and VP) ), respectively.

v

indicates that the emitted token carries a node label. “Merged CNT” signifies that consecutive CNTs (ignoring labels) are collapsed into a single ) token. Finally,

\textsc{Lin}^{1}_{\text{shuf}}

is generated by shuffling the

\textsc{Lin}^{1}

token sequence while preserving the original terminal order.

Dataset	$\textsc{Lin}^{0}$	$\textsc{Lin}^{1}$	$\textsc{Lin}^{2}$
BBC News	0.801B	1.963B	2.545B
FineWeb-Edu	$\sim$ 98B	$\sim$ 233B	$\sim$ 301B

Table 2: Corpus tokens by linearization, including special tokens. BBC counts use the deduplicated reference corpus. Historical training exposure, including repeated articles, is reported in Appendix D .

#	Variant	Linearization	Attention
1	Terminal	$\textsc{Lin}^{0}$	Causal
2	Tree	$\textsc{Lin}^{1}$	Causal
3	TGTree	$\textsc{Lin}^{2}$	Causal
4	TG	$\textsc{Lin}^{2}$	STACK/COMPOSE
5	TGNomask	$\textsc{Lin}^{2}$	Nomask
6	TGNomask-Aug ∗	$\textsc{Lin}^{2}$	Nomask

Table 3: The 12 configurations in our controlled design space: one terminal baseline and 11 SLM variants. ONT = opening non-terminal, CNT = closing non-terminal. “Hybrid” denotes mixing-head models. ∗ TGNomask-Aug differs from TGNomask: CNT 2 remains visible to subsequent attention in Aug, while TGNomask prevents subsequent tokens from attending to it.

BBC 100M: original training corpus
Model	XSum $\uparrow$	BoolQ $\uparrow$	SG $\uparrow$	BLiMP $\uparrow$	Doc-PPL $\downarrow$
Terminal-100M	22.02 $\pm$ 0.06	66.70 $\pm$ 0.36	69.80	70.77	15.369
Tree-100M	22.84 $\pm$ 0.03	68.07 $\pm$ 0.35	81.40	80.95	14.736
TGTree-100M	23.03 $\pm$ 0.02	68.12 $\pm$ 0.62	79.47	81.46	14.709
TG-100M	20.15 $\pm$ 0.04	64.50 $\pm$ 0.40	79.19	83.28	16.405
TGNomask-100M	21.91 $\pm$ 0.04	66.66 $\pm$ 0.45	78.62	81.78	14.658

Table 4: BBC results at 100M and dedup 500M. XSum reports R-AVG; XSum and BoolQ show mean

\pm

sample SD over five fine-tuning seeds. Other metrics are single evaluations. Best values within each training regime are in bold . All Doc-PPL values use the same reserved-clean test; NoONT/Compress candidates are deduplicated after linearization. The shaded Pushdown row is an architectural reference.

\dagger

marks model-specific PPL protocols;

\ddagger

marks BLiMP scores whose projected candidate uniqueness remains unchecked (Appendix B.4 ).

Model	WG $\uparrow$	HS $\uparrow$	BQ $\uparrow$	MM $\uparrow$	MMR $\uparrow$	OBQA $\uparrow$	CSQA $\uparrow$	SIQA $\uparrow$	ARCE $\uparrow$	ARCC $\uparrow$	PIQA $\uparrow$	AVG $\uparrow$
Terminal	58.17	61.53	58.65	36.79	37.61	43.60	53.24	48.98	69.30	43.48	73.88	53.20
Tree	61.64	54.10	63.85	36.88	37.53	44.00	56.02	48.46	68.42	45.15	73.07	53.56
TGTree	63.38	63.51	67.55	37.76	38.42	45.80	58.15	50.31	70.00	44.82	74.97	55.88
Pause-1	58.96	63.05	65.41	37.70	38.81	42.80	56.18	51.28	74.04	43.48	75.14	55.17
Pause-2	62.59	64.45	63.39	38.38	39.46	44.00	56.76	48.67	72.98	48.16	74.97	55.80

Table 5: Primary content-focused terminal-score accuracy at 1B on FineWeb-Edu, based on the scoring definition in Section 4 . WG = WinoGrande, HS = HellaSwag, BQ = BoolQ, MM = MMLU, MMR = MMLU-Redux, OBQA = OpenBookQA, CSQA = CommonsenseQA, SIQA = SocialIQA, ARCE = ARC-Easy, ARCC = ARC-Challenge. Best per task in bold ; Table 10 reports 95% bootstrap intervals.

Model	BLiMP $\uparrow$	SG $\uparrow$
Terminal	79.65	88.01
Tree	81.79	89.95
TGTree	80.98	89.00
Pause-1	80.83	86.54
Pause-2	80.36	86.59

Table 6: Syntactic evaluation at 1B on FineWeb-Edu. For Tree and TGTree, SG uses word-synchronous beam search with a maximum linearized decoding length of six times the terminal sequence length.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Scale	$d_{\text{model}}$	MLP ratio	Layers	Heads	Params
100M	768	4	12	12	$\sim$ 100M
500M	1408	8	16	16	$\sim$ 500M
1B	2048	8	16	16	$\sim$ 1B

Appendix

Table 7: Model configurations at each scale.

Model	Training tokens $D$	Peak LR ( $\times 10^{-4}$ )	Global batch	Length	Final step
Terminal	800,848,405	6.012086	34	2048	11,501
Tree	1,963,388,153	7.917501	58	2048	16,529
TGTree	2,544,658,027	8.573605	66	2048	18,825
Pause-1	1,601,696,810	7.437747	52	2048	15,039
Pause-2	2,402,545,215	8.423672	64	2049	18,321
Tree-NoONT	1,382,118,279	7.108576	46	2048	14,670

Appendix

Table 8: Training settings for the ten BBC dedup 500M models. Batch counts sequences, length counts input positions, and steps identify the final checkpoints used for evaluation. Pause budgets include inserted pause tokens.

Scale	Causal	Structural	Ratio
100M	62,175	37,975	1.6 $\times$
500M	12,550	8,316	1.5 $\times$

Appendix

Table 9: Historical H800 training throughput (token/s), using FlashAttention-2 for causal and FlexAttention for structural masks. The historical 100M mixing-head measurement was 3,375 token/s (approximately 18

\times

slower than causal). These measurements do not characterize the new RTX 5090 typed-kernel runs.

Task	Terminal	Tree	TGTree	Pause-1	Pause-2
WG	58.17 [55.49, 60.93]	61.64 [58.96, 64.33]	63.38 [60.69, 66.06]	58.96 [56.20, 61.64]	62.59 [59.91, 65.27]
HS	61.53 [60.57, 62.48]	54.10 [53.14, 55.09]	63.51 [62.57, 64.45]	63.05 [62.09, 63.99]	64.45 [63.51, 65.39]
BQ	58.65 [56.97, 60.34]	63.85 [62.23, 65.50]	67.55 [65.93, 69.11]	65.41 [63.82, 67.06]	63.39 [61.71, 65.08]
MM	36.79 [35.99, 37.59]	36.88 [36.06, 37.69]	37.76 [36.96, 38.56]	37.70 [36.88, 38.51]	38.38 [37.57, 39.18]
MMR	37.61 [36.37, 38.89]	37.53 [36.28, 38.77]	38.42 [37.18, 39.70]	38.81 [37.56, 40.07]	39.46 [38.23, 40.72]
OBQA	43.60 [39.20, 48.00]	44.00 [39.60, 48.40]	45.80 [41.40, 50.20]	42.80 [38.40, 47.00]	44.00 [39.60, 48.40]

Appendix

Table 10: Accuracy point estimates and 95% bootstrap confidence intervals for the terminal-format 1B evaluation. Per-task rows use paired resampling of evaluation examples. † Macro-average intervals resample tasks and therefore measure task-suite sensitivity rather than uncertainty within a fixed suite. Task abbreviations follow Table 5 .

Task	TGTree $-$ Terminal	Tree $-$ Pause-1	TGTree $-$ Pause-2
WG	+5.21 [2.29, 8.13]; $p<.001$	+2.68 [ $-$ 0.32, 5.68]; $p=.082$	+0.79 [ $-$ 2.05, 3.63]; $p=.604$
HS	+1.98 [1.22, 2.74]; $p<.001$	$-$ 8.94 [ $-$ 9.96, $-$ 7.89]; $p<.001$	$-$ 0.94 [ $-$ 1.68, $-$ 0.19]; $p=.014$
BQ	+8.90 [6.48, 11.31]; $p<.001$	$-$ 1.56 [ $-$ 3.27, 0.12]; $p=.074$	+4.16 [2.02, 6.30]; $p<.001$
MM	+0.97 [0.26, 1.67]; $p=.009$	$-$ 0.82 [ $-$ 1.53, $-$ 0.09]; $p=.027$	$-$ 0.62 [ $-$ 1.30, 0.08]; $p=.078$
MMR	+0.81 [ $-$ 0.32, 1.96]; $p=.172$	$-$ 1.28 [ $-$ 2.40, $-$ 0.18]; $p=.024$	$-$ 1.04 [ $-$ 2.14, 0.05]; $p=.069$
OBQA	+2.20 [ $-$ 1.40, 5.80]; $p=.262$	+1.20 [ $-$ 2.80, 5.20]; $p=.592$	+1.80 [ $-$ 1.80, 5.40]; $p=.367$

Appendix

Table 11: Paired bootstrap differences in percentage points with 95% confidence intervals and uncorrected two-sided

p

-values. Positive values favor the model named first. Tree–Pause-1 and TGTree–Pause-2 are approximate budget comparisons.

Historical model	XSum $\uparrow$	BoolQ $\uparrow$	SG $\uparrow$	BLiMP $\uparrow$	Clean Doc-PPL $\downarrow$
Terminal-500M	20.75 $\pm$ 0.06	69.69 $\pm$ 0.60	71.11	64.74	17.303
Tree-500M	22.25 $\pm$ 0.05	70.72 $\pm$ 0.35	63.13	76.09	16.001
TGTree-500M	22.01 $\pm$ 0.08	70.63 $\pm$ 0.32	65.95	77.65	15.895
TGNomask-Aug-500M	21.77 $\pm$ 0.06	70.78 $\pm$ 0.74	57.58	75.97	15.484

Appendix

Table 12: Supplementary 500M models trained on the original BBC corpus. Doc-PPL is reevaluated on reserved-clean; the other metrics retain their original evaluations. XSum and BoolQ show means

\pm

sample SD over five fine-tuning seeds. Best values are in bold . These models use a different training corpus from the dedup 500M models in Table 4 .

Model	Agr $\uparrow$	CtrEmb $\uparrow$	GrdnPath $\uparrow$	GrsSyn $\uparrow$	Lic $\uparrow$	LDD $\uparrow$	SG $\uparrow$
Terminal	64.91	91.07	78.29	94.57	78.95	77.52	80.88
Tree	75.44	92.86	80.26	91.30	80.45	80.28	83.43
TGTree	75.44	85.71	85.53	85.87	78.57	72.94	80.68
Pause-1	66.67	82.14	80.92	96.74	73.68	73.85	79.00
Pause-2	64.91	91.07	79.61	85.87	73.31	74.31	78.18
Tree-NoONT	68.42	91.07	88.16	92.39	87.22	74.77	83.67

Appendix

Table 13: SG subtask breakdown at 500M on deduplicated BBC News. Agr = Agreement, CtrEmb = Center Embedding, GrdnPath = Garden Path Effects, GrsSyn = Gross Syntactic Expectation, Lic = Licensing, LDD = Long-Distance Dependencies. Best per metric in bold .

Model	Agr $\uparrow$	CtrEmb $\uparrow$	GrdnPath $\uparrow$	GrsSyn $\uparrow$	Lic $\uparrow$	LDD $\uparrow$	SG $\uparrow$
Terminal	87.72	94.74	93.42	98.91	74.81	78.44	88.01
Tree	85.96	100.00	98.03	100.00	78.20	77.52	89.95
TGTree	80.70	100.00	94.08	100.00	78.95	80.28	89.00
Pause-1	75.44	100.00	94.08	98.91	73.31	77.52	86.54
Pause-2	82.46	94.74	95.39	100.00	72.18	74.77	86.59

Appendix

Table 14: SG subtask breakdown at 1B on FineWeb-Edu. Best per metric in bold ; abbreviations follow Table 13 .

	Tree			TGTree
Task	Full	Term	$\Delta$	Full	Term	$\Delta$
WinoGrande	61.17	61.64	+0.47	64.64	63.38	$-$ 1.26
HellaSwag	37.83	43.92	+6.08	47.11	49.45	+2.34
OpenBookQA	33.60	33.60	+0.00	34.20	34.00	$-$ 0.20
CommonsenseQA	54.79	55.86	+1.06	57.17	57.49	+0.33
SocialIQA	43.65	45.60	+1.94	44.42	45.80	+1.38

Appendix

Table 15: Accuracy (%) using raw full and terminal scores on the eight-task sensitivity suite;

\Delta=\mathrm{Term}-\mathrm{Full}

. Macro averages are unweighted. Paired 95% bootstrap intervals for the macro

\Delta

are [1.35, 2.99] for Tree and [

-

0.09, 1.35] for TGTree (10,000 resamples; seed 2026).

Model	Macro accuracy	$\Delta$ vs. Terminal (95% CI)
Terminal	54.43	reference
Tree	54.75	+0.32 [ $-$ 0.67, 1.30]
TGTree	57.35	+2.92 [1.95, 3.84]
Pause-1	56.47	+2.05 [1.15, 2.93]
Pause-2	57.09	+2.66 [1.82, 3.49]

Appendix

Table 16: Terminal-score accuracy (%) on Validation-10. The paired TGTree–Pause-2 difference is +0.26 points (95% CI [

-

0.68, 1.20]).

Syntactic Generalization in Language Models

See all 13 papers

Jan 9, 2026

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on over 3,000 datapoints spanning a wide range of syntactic phenomena, languages, models, and methods. We take the data to collectively show that TLMs encode a non-trivial amount of syntactic knowledge. Behavioral evidence shows strong performance on formal syntactic phenomena, but weaker and more variable performance on phenomena at the syntax-semantics interface. Performance is also consistently lower for languages with less digital support. Probing and mechanistic studies further support the presence of syntactic knowledge in TLMs. Yet, because most work remains observational and methodologically heterogeneous, insight into the detailed computational mechanisms underlying syntactic processing remains limited. At the same time, the literature remains heavily concentrated on English and BERT-like models. We discuss the implications of our results and provide recommendations for future research.

Nora Graichen, Iria de-Dios-Flores, Gemma Boleda

Universitat Pompeu Fabra1 · ICREA2

Transformer Interpretability LLM Evaluation

Ask PDF

Sep 30, 2026

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned during PPT that transfers to natural language grammar. However, PPT has only been tested on models of at most 1B parameters and PT budgets below 2B tokens on predominantly web text. It is unknown whether PPT is effective at larger scales and under more realistic PT data mixtures that combine diverse sources (e.g., code and math). We therefore present a comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens. Our results demonstrate that the downstream performance and token efficiency gains of PPT persist at scale, e.g., saving at least 21B PT tokens at the 3B scale. However, in contrast to prior work, we find no consistent evidence that these gains stem from a grammatical prior. Downstream performance does not consistently align with grammatical acceptability across model sizes. Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval. Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent. Overall, PPT is a low-cost addition to PT, and future PPT task design should target long-range retrieval rather than natural language grammar.

Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3

University of Sheffield, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Hugging Face +3

Language Model Pretraining Synthetic Data Pretraining Linguistic Bias in Language Models

Ask PDF

Aug 28, 2026

Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy

Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations. They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus (as measured by undirected unlabeled attachment score). Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately, establishing important differences among relations that overlap linguistic distinctions. Moreover, we identify two factors that predict most of UASL's variability across relations: (i) the mean and dispersion of the linear distance (on a log scale) between the related words, and (ii) the diversity (similarity-aware entropy) of the syntactic relation's head. These results, which hold across a range of model sizes and architectures, shed light on the degree of abstraction of the representation of syntax in language models and the dependence of such representation on geometric properties of the embedding space.

Juan Pablo Vigneaux, Mary Kennedy, Khalil Iskarous +2

Northwestern University · University of Southern California · Yale University +1

Language Model Probing Representation Geometry in Language Models

Ask PDF

Abstract

Syntactic Generalization in Language Models

Explore similar work

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy