Skill Neuron

Skill neurons are specific neurons within large language models (LLMs), like RoBERTa and T5, whose activation patterns strongly correlate with the model's successful performance on particular tasks after prompt tuning or other fine-tuning methods. Current research focuses on identifying these neurons, understanding their role in task execution (including robustness to adversarial examples), and exploring their potential for model optimization, such as network pruning and improved transfer learning. The discovery and characterization of skill neurons offer valuable insights into the internal workings of LLMs, potentially leading to more efficient and robust AI systems.