Effective Dimension

Effective dimension quantifies the true complexity of a model, going beyond simply counting parameters, by focusing on the number of parameters actually contributing to its function. Current research investigates how effective dimension relates to generalization error across various model architectures, including deep neural networks and evolutionary algorithms, and explores its use in bounding generalization performance and analyzing the impact of factors like data dimensionality and training algorithms (e.g., SGD vs. NGD). Understanding effective dimension offers valuable insights into model capacity, enabling better prediction of generalization performance and potentially guiding the design of more efficient and robust machine learning models.