cs.CLNov 11, 2026

What Does Each Part of Speech Need? Contexts, Subspaces and Dimensions in Static Word Embeddings

Authors: Jintang Xue, Chengwei Wei, Yun-Cheng Wang, C. -C. Jay Kuo

Organizations: University of Southern California, Los Angeles, California, USA

Abstract

Static word embeddings give a word one vector, ignoring its different functions as noun, verb or adjective. Training on part-of-speech-tagged text gives each function its own vector, at the cost of a larger table. Using nine controlled corpora, we ask which context words each part of speech needs, whether it needs its own subspace, and whether the number of dimensions a word keeps should follow its part of speech or its frequency. The tags raise verb similarity on SimVerb-3500 at every dimension from 50 to 300 but help nouns much less. Added to verb-only text, adverbs help verbs more than as many random words; adjectives help nouns only slightly. At a fixed dimension, per-part-of-speech PCA bases preserve verb and adjective similarity better than a global basis or frequency-band bases. Nouns need more dimensions than verbs only among frequent words. We therefore allocate dimensions by frequency with these bases: at a third of the original size, only rare words lose dimensions, SimVerb-3500 and SimLex-999 nouns match the uncompressed table, and verb-to-noun retrieval stays within 0.02. Part of speech decides what a word's vector should be; frequency decides how much of it to keep.

Explore similar work

CardsList
  1. Semantic Space of Parts of Speech

    Aug 15, 2026Jiří Milička, Ivan Kraus, Arnold Stanovský +5Morphosyntactic TaggingRepresentation Geometry in Language Models

  2. An empirical investigation into the properties of standard word embeddings

    Jul 26, 2026Salomon KabongoText EmbeddingsWord Embeddings

  3. Examining the Limits of Word2Vec with Toki Pona

    Jun 15, 2026Daniel Zhenhan Huang, Hongchen WuText Embedding EvaluationDistributional Semantics