cs.CLNov 11, 2026

What Does Each Part of Speech Need? Contexts, Subspaces and Dimensions in Static Word Embeddings

Authors: Jintang Xue, Chengwei Wei, Yun-Cheng Wang, C. -C. Jay Kuo

Organizations: University of Southern California, Los Angeles, California, USA

Abstract

Static word embeddings give a word one vector, ignoring its different functions as noun, verb or adjective. Training on part-of-speech-tagged text gives each function its own vector, at the cost of a larger table. Using nine controlled corpora, we ask which context words each part of speech needs, whether it needs its own subspace, and whether the number of dimensions a word keeps should follow its part of speech or its frequency. The tags raise verb similarity on SimVerb-3500 at every dimension from 50 to 300 but help nouns much less. Added to verb-only text, adverbs help verbs more than as many random words; adjectives help nouns only slightly. At a fixed dimension, per-part-of-speech PCA bases preserve verb and adjective similarity better than a global basis or frequency-band bases. Nouns need more dimensions than verbs only among frequent words. We therefore allocate dimensions by frequency with these bases: at a third of the original size, only rare words lose dimensions, SimVerb-3500 and SimLex-999 nouns match the uncompressed table, and verb-to-noun retrieval stays within 0.02. Part of speech decides what a word's vector should be; frequency decides how much of it to keep.

Explore similar work

Aug 15, 2026

Semantic Space of Parts of Speech

Parts of speech categorization is understood in the European linguistic tradition as crisp categorization, which is also reflected in corpus linguistics, where each disambiguated token is assigned exactly one POS. However, the assigned categories are largely determined by arbitrary decisions distilled into annotation manuals. Since some words stand between parts of speech in their semantics or typical syntax, and some parts of speech are closer to each other than others, POS categorization seems inherently fuzzy. We analyze this fuzziness using word2vec embeddings, training a neural network to reduce their high dimensionality to three dimensions relevant for determining parts of speech. This creates a three-dimensional space onto which we map several thousand words, revealing which are prototypical and which lie on the boundaries, and visualizing relationships between parts of speech. The study uses Universal Dependencies POS tags for French, Czech, Finnish, Russian, and English.
Jul 26, 2026

An empirical investigation into the properties of standard word embeddings

The embedding of word sequences into continuous vector spaces has been one of the most important developments in Natural Language Processing in the recent past. Such embeddings have found application in areas such as Automatic Speech Recognition, Machine Translation, Sentiment Analysis and many more. This essay reviews the various mechanisms that have been proposed for the calculation of word embeddings, investigates popular toolkits and embedding matrices that are available in the public domain, and experiments with one or more selected implementations to better understand their characteristics. La représentation vectorielle continue de mots a été l'un des développements les plus importants dans le domaine du traitement automatique du langage naturel au cours des dernières années. Ces représentations ont trouvé application dans des domaines tels que la reconnaissance vocale, la traduction automatique, l'analyse des sentiments, etc. Ce travail passe en revue les différents mécanismes proposés pour le calcul de ces vecteurs de mots, étudie les kits d'outils populaires et les matrices disponibles publiquement en ligne, et expérimente avec une ou plusieurs implémentations sélectionnées pour mieux comprendre leurs caractéristiques.
Jun 15, 2026

Examining the Limits of Word2Vec with Toki Pona

Word2Vec's effectiveness at generating semantic embeddings has been widely validated, yet it has been tested almost exclusively on languages with large vocabulary inventories. This study examines whether Word2Vec can successfully capture semantic relationships within an extremely reduced vocabulary using data from Toki Pona, a constructed language with approximately 130 words. We sourced 1.4 million sentences (7.95 million tokens) from the Toki Pona community for training. Approximately 23% of sentences in the corpus contain non-Toki Pona tokens such as named entities, loanwords, and neologisms. To investigate whether this linguistic noise enhances or hinders performance -- a topic rarely addressed in word embedding literature -- we trained two distinct models: one retaining these incidental tokens and another filtering them out completely. Evaluation was conducted using quantitative methods measuring word proximity to semantic category centroids, automated silhouette scores via agglomerative clustering, and qualitative analysis utilizing representational similarity matrices compared against English. The results indicate that while sparse, non-core tokens do not affect the relative structure of the learned embeddings, they actually draw similar words closer together in the vector space. Importantly, Word2Vec's effectiveness depends more on distributional patterns than lexicon size even at this extreme lower bound.