Representation Collapse in Machine Translation Through the Lens of Angular Dispersion

Evgeniia Tokarchuk; Maya K. Nachesa; Sergey Troshin; Vlad Niculae

cs.CLFeb 19, 2026

Representation Collapse in Machine Translation Through the Lens of Angular Dispersion

Authors: Evgeniia Tokarchuk, Maya K. Nachesa, Sergey Troshin, Vlad Niculae

Organizations: Language Technology Lab, University of Amsterdam

Abstract

Modern neural translation models based on the Transformer architecture are known for their high performance, particularly when trained on high-resource datasets. A standard next-token prediction training strategy, while widely adopted in practice, may lead to overlooked artifacts such as representation collapse. Previous works have shown that this problem is especially pronounced in the representation of the deeper Transformer layers, where it often fails to efficiently utilize the geometric space. Representation collapse is even more evident in end-to-end training of continuous-output neural machine translation, where the trivial solution would be to set all vectors to the same value. In this work, we analyze the dynamics of representation collapse at different levels of discrete and continuous NMT transformers throughout training. We incorporate an existing regularization method based on angular dispersion and demonstrate empirically that it not only mitigates collapse but also improves translation quality. Furthermore, we show that quantized models exhibit similar collapse behavior and that the benefits of regularization are preserved even after quantization.

Figures & tables

Panel of Figure 1 — Figure 1: Average cosine similarity, Rényi entropy and spherical variance for decoder output, decoder embeddings and encoder outputs for Transformer-big and Transformer-base models.

model	en-de				de-en
	newstest18		newstest19		newstest18		newstest19
	BLEU	COMET	BLEU	COMET	BLEU	COMET	BLEU	COMET
Transformer-big	41.9	0.803	31.7	0.790	43.0	0.811	39.3	0.776
+ $R(\bm{H})$	41.6	0.803	31.7	0.789	42.5	0.808	39.0	0.774
+ $R(\bm{E})$	42.6	0.806	32.6	0.793	43.2	0.812	39.2	0.776
Transformer-base	40.9	0.792	30.9	0.782	41.7	0.799	37.8	0.762

Table 1: BLEU and COMET on newstest18 and newstest19 for en-de end en-de with different dispersion regularizers. Values in bold indicate results that are statistically significant compared to the baseline (p-value < 0.05)

Figure 2: F1 score for each vocabulary token’s frequency bucket for en-de . Note that the F1 score for tokens with frequency

<100

for the model without regularization is 0.

Figure 3: BLEU score for the de-en development set newstest17 of Transformer-base model for different parameters

\gamma

. Horizontal blue line indicates the BLEU score of the unregularized Transformer-base model.

Figure 4: Comparison of the encoder outputs representation collapse metrics for Transformer-big and Transformer-big fp16 de-en .

prec.	reg.	BLEU	COMET
float32	✗	31.7	0.790
float32	$R(\bm{E})$	32.6	0.793
float16 *	✗	31.7	0.787
float16	✗	31.7	0.845
float16	$R(\bm{E})$	32.6	0.847
int8_float16	✗	27.5	0.745

Table 2: Comparison of quantization-aware training ( float* ) and post-training quantization on the Transformer-big en-de model with and without dispersion regularization on newstest19 .

Figure 5: Representation collapse in target representations of the CoNMT models. CoNMT static refers to the model trained with frozen target embeddings.

objective	init.	train $\bm{E}$	BLEU	COMET
$L_{\text{CoNMT}}$	rand.	✗	33.9	0.713
$L_{\text{CoNMT}}$	rand.	✓	0.0	0.000
$L_{\text{CoNMT}}$	NMT	✗	32.9	0.704
$L_{\text{CoNMT}}$	NMT	✓	0.0	0.000
$L_{\text{CoNMT}}$	RNMT	✗	36.6	0.749
$L_{\text{CoNMT}}$	RNMT	✓	0.0	0.000

Table 3: de-en translation scores on newstest2019 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	lang.	#sent.	#toks.	#vocab.
train	de	34M	916M	42024
newstest2017	de	3004	78553
newstest2018	de	2998	81875
newstest2019	de	2000	47035
train	en	34M	888M	42024
newstest2017	en	3004	75011

Appendix

Table 4: Data statistics for training and test splits.

Figure 6: F1 score for de-en for each vocabulary token’s frequency bucket.

architecture	precision	size
transformer-big	float32	$100\%$
transformer-big	float16	$50\%$
transformer-big	int8_float16	$25\%$
transformer-base	float32	$30\%$

Appendix

Table 5: Transformer-big size on disk for the original models and quantized versions. Percentages based on disk storage.

model	ro-en
	newsdev16		newstest16
	BLEU	COMET	BLEU	COMET
Transformer-big	32.6	0.786	31.2	0.790
+ $R(\bm{H})$	32.7	0.785	31.0	0.788
+ $R(\bm{E})$	33.0	0.787	31.4	0.791
Transformer-base	33.2	0.793	31.8	0.795

Appendix

Table 6: BLEU and COMET on newsdev and newstest for ro-en with different dispersion regularizers. Values in bold indicate results that are statistically significant compared to the baseline (p-value < 0.05)

Representation Collapse

Representation collapse is the loss of diversity in learned features, causing distinct inputs to map to similar or low-dimensional embeddings.

See all 47 papers

Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
Mar 17, 2026Wenhao Zhao, Qiran Zou, Rushi Shah +3Vector Quantization Representation Collapse
National University of Singapore · Zhejiang University · Shanghai Jiao Tong University
Force without transmission: a depth-induced rank collapse that no loss on the representation reopens
Oct 7, 2026Martin Hofmann, Patrick MäderTransformer Attention
Data-intensive Systems and Visualization Group (dAI.SY), Technische Universität Ilmenau, Max-Planck-Ring 14, 98693 Ilmenau, Thuringia, Germany · German Centre for Integrative Biodiversity Research (iDiv) Halle–Jena–Leipzig, Deutscher Platz 5e, 04103 Leipzig, Saxony, Germany · Faculty of Biological Sciences, Friedrich Schiller University, Fürstengraben 1, 07745 Jena, Thuringia, Germany
Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
Apr 26, 2026Giansalvo CirrincioneTransformer Interpretability Representation Collapse
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France

Mar 17, 2026J/K move · Enter open · S save

Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization

Wenhao Zhao, Qiran Zou, Rushi Shah, Yudi Wu+2

National University of Singapore · Zhejiang University · Shanghai Jiao Tong University

Vector Quantization Representation Collapse

Open paper AskPDF arXiv

Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data representations for large language models, diffusion models, and other generative models. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored. In this study, we systematically investigate the issue of collapses in vector quantization, where collapsed representations are observed across discrete codebook tokens and continuous latent embeddings. By leveraging both synthetic and real datasets, we identify the severity of each type of collapses and triggering conditions. Our analysis reveals that random initialization and limited encoder capacity result in tokens collapse and embeddings collapse. Building on these findings, we propose potential solutions aimed at mitigating each collapse. To the best of our knowledge, this is the first comprehensive study examining representation collapsing problems in vector quantization.

Abstract

Representation Collapse

Explore similar work

Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization

Force without transmission: a depth-induced rank collapse that no loss on the representation reopens

Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers