GatedV: Self-Gated Values for Transformer Language Models

Li-Ming Yu; Hongyu Zhou

cs.CLNov 1, 2026

GatedV: Self-Gated Values for Transformer Language Models

Authors: Li-Ming Yu, Hongyu Zhou

Organizations: Independent Researcher · Nanyang Technological University, Singapore

Abstract

Standard attention forms query (Q), key (K), and value (V) matrices through learned linear projections of input hidden states. Architectures such as GAU and Mega incorporate nonlinear transformations into attention. In this paper, we propose GatedV, a simple attention variant that applies SiLU only to V before aggregation without introducing additional trainable parameters. We evaluate GatedV against a standard decoder-only Transformer using matched parameter counts, paired initializations, and equal training-token budgets. At 2.33M parameters, GatedV reduces mean held-out perplexity from 59.1 to 57.9 (2.0%), with improvements across all five paired seeds. At 26.16M parameters, following an equal-budget learning-rate search, mean held-out perplexity falls from 20.2 to 19.7 (2.4%), with improvements across all three paired seeds. Small-model ablations show that the tested affine rescaling of values does not fully reproduce GatedV's gains. A short interleaved benchmark on NVIDIA DGX Spark measures approximately 0.7% higher full-context forward latency for the larger model, based on synchronized wall-clock timings. These findings suggest that value-only self-gating can improve language modeling performance without increasing parameter count or redesigning the attention-weighting mechanism.

Figures & tables

Figure 1: Transformer block and GatedV value transformation. The baseline aggregates the projected values

V

, whereas GatedV applies

\operatorname{SiLU}(V)

before aggregation. The value box shows the two alternatives; each model uses one.

Setting	Small	Medium
Layers / model width	6 / 128	12 / 384
Heads / head width	4 / 32	6 / 64
SwiGLU hidden width	384	1,152
Trainable parameters	2,328,192	26,158,464
Training-corpus tokens	40,001,310	512,000,068
Training-corpus documents	27,830	212,052

Table 1: Main confirmation settings. Both models use a vocabulary of 8,192, context length 256, and batch size 8. Target exposures count sampled prediction targets, including repetitions.

Variant	LR 0.00001	LR 0.0001	LR 0.0003	LR 0.0006
Baseline	36.853	20.938	20.558	21.558
GatedV	37.451	20.632	20.062	21.005

Table 2: Medium-model learning-rate selection: final development PPL at the shared selection seed 20270201, with 327,680 updates per run. Both architectures select 0.0003.

Model size	Baseline	GatedV	$\Delta_{P}$	Improved pairs
2.33M	$59.118\pm 0.530$	$57.916\pm 0.782$	$-2.03\%$	5/5
26.16M	$20.233\pm 0.068$	$19.748\pm 0.047$	$-2.40\%$	3/3

Table 3: Held-out confirmation results. Values are arithmetic mean PPL

\pm

sample SD. The studies use different held-out sets and training configurations; the medium model uses peak LR 0.0003.

Figure 2: Relative development PPL,

\Delta\mathrm{PPL}/P_{\mathrm{base}}

, shown as a percentage;

\Delta\mathrm{PPL}

denotes GatedV minus baseline. Thin gray curves show individual seed pairs; the thick blue curve uses the arithmetic mean PPL of each model across seeds. Negative values favor GatedV.

Figure 3: Development PPL over the final 25% of training. Copper denotes baseline and blue denotes GatedV. Thin light curves show individual runs; thick curves show arithmetic means across seeds. Held-out endpoints are reported separately in Table 3 .

Variant or peak LR	Baseline PPL	Variant PPL	$\Delta_{P}$	Improved pairs
Activation placement; LR = 0.0048
Q-only	$58.580\pm 0.557$	$59.522\pm 0.337$	$+1.61\%$	0/3
GatedV	$58.580\pm 0.557$	$58.039\pm 0.525$	$-0.92\%$	3/3
All Q/K/V	$58.580\pm 0.557$	$58.458\pm 0.485$	$-0.21\%$	2/3
Small-model LR study; variant is GatedV
0.0012	$56.441\pm 0.548$	$55.269\pm 1.120$	$-2.08\%$	2/3

Table 4: Development-set comparisons. The two groups of rows use different sets of three paired seeds.

\Delta_{P}

compares the variant with the baseline.

Control	Peak LR	PPL $\pm$ SD	vs baseline	vs GatedV
Affine match	0.0024	$55.551\pm 0.498$	$-1.12\%$	$+2.02\%$
Moment-matched SiLU	0.0024	$55.983\pm 0.787$	$-0.35\%$	$+2.82\%$
Affine match	0.0048	$58.920\pm 0.298$	$+0.26\%$	$+2.68\%$
Moment-matched SiLU	0.0048	$58.399\pm 0.540$	$-0.63\%$	$+1.77\%$

Table 5: Final development PPL for fixed moment controls, with three paired seeds. Relative changes compare each control with baseline or GatedV at the same peak LR.

Measure	Baseline	GatedV
Full-context forward time ratio	1.000000	1.007003
Training-step time ratio	1.000000	1.011433
CUDA peak allocated (GiB)	1.7525	1.8569

Table 6: Medium-model time ratios relative to baseline and median peak CUDA allocated memory over the forward and training-step measurements combined.

Variant	Parameters (M)	LR 0.0006	LR 0.0003	LR 0.0001
Baseline	26.158	21.558	20.558	20.938
GatedV	26.158	21.005	20.062	20.632
Headwise G2	26.186	21.568	20.630	21.189
Full-width GLU	27.928	21.197	20.309	20.930
Reduced-width GLU	26.186	21.522	20.657	21.382
$V\odot\tanh(V)$	26.158	21.337	20.325	20.986

Table 7: Exploratory medium-model value-gate comparison: final development PPL with one shared seed (20270201), reused development data, and 327,680 updates per run.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluation set	Peak LR	Baseline PPL	GatedV PPL	$\Delta_{P}$
Development	0.0006	$21.804\pm 0.182$	$21.039\pm 0.045$	$-3.51\%$
Development	0.0003	$20.640\pm 0.082$	$20.139\pm 0.047$	$-2.43\%$
Held-out	0.0006	$21.384\pm 0.166$	$20.600\pm 0.018$	$-3.66\%$
Held-out	0.0003	$20.233\pm 0.068$	$19.748\pm 0.047$	$-2.40\%$

Appendix

Table 8: Medium-model confirmation at two peak learning rates. PPL is the arithmetic mean

\pm

sample SD across seeds 20270211–20270213;

\Delta_{P}

compares GatedV with baseline within each row.

Training	Linear at inference	SiLU at inference
Linear (baseline)	$20.233\pm 0.068$	$102.565\pm 31.053$
SiLU (GatedV)	$4{,}300.232\pm 899.037$	$19.748\pm 0.047$

Appendix

Table 9: Held-out PPL (mean and sample standard deviation across three paired seeds).

Transformer Attention

Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.

See all 331 papers

Exclusive Self Attention
Mar 10, 2026Shuangfei ZhaiSelf-Attention Transformer
Apple
Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Jun 20, 2026Xin GaoTransformer Inference KV Caching
Department of Mathematics and Statistics York University Toronto, M3J 1P3
Depth-Attention: Cross-Layer Value Mixing for Language Models
Jun 3, 2026Boyi Zeng, Yiqin Hao, Zitong Wang +7Efficient Transformer Inference Cross-Layer Attention
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute +2

Abstract

Transformer Attention

Explore similar work

Exclusive Self Attention

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

Depth-Attention: Cross-Layer Value Mixing for Language Models