Orthogonal Representation Learning

Orthogonal representation learning studies how to arrange learned features into perpendicular directions or subspaces so that different factors carry less redundant information. Common approaches include orthogonality and cross-covariance penalties, Stiefel-manifold or Householder/Cayley parameterizations, and decompositions into shared and task-specific or common and style components; related methods use orthogonal projections and Jacobian constraints. These designs are applied in contrastive and multimodal learning, continual learning, recommendation, scientific modeling, and generative disentanglement, with evaluations spanning robustness, domain transfer, compositional generalization, and task accuracy. A central tension is that stronger separation can improve isolation and stability but may also discard useful interactions or restrict capacity, making the choice of subspace dimension and constraint strength an ongoing challenge.