Paper ID: 2408.14332

One-layer transformers fail to solve the induction heads task

Clayton Sanford, Daniel Hsu, Matus Telgarsky

A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.

Submitted: Aug 26, 2024