Zero-Dimensional Attention Topology for Linguistic Acceptability
Organizations: Khalifa University · Department of Engineering Science and Management, Khalifa University of Science and Technology, Abu Dhabi, UAE
Abstract
Topological summaries of Transformer attention provide sentence representations for linguistic acceptability classification. We examine a zero-dimensional persistent-homology () representation that summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. The retrospective evaluation uses 8,517 downstream training sentences and 521 in-domain and 497 out-of-domain CoLA validation sentences after data-integrity filtering. The recorded out-of-domain Matthews correlation coefficients are 0.438 for the topological representation and 0.508 for attention statistics, compared with 0.545 for BERT. Attention-feature results are retained aggregate measurements; BERT predictions are rescored and length controls are refitted from the accompanying records. Analytical results show that the representation retains token-count information and discards a class of directional attention differences. Source resampling of BERT predictions also demonstrates the sensitivity of uncertainty estimates to the sampling unit. The reported results do not demonstrate a predictive advantage for this fixed topological representation over simpler attention summaries. They identify representation properties and evaluation controls relevant to subsequent studies of attention topology.
Figures & tables
| Training | In-domain | Out-of-domain | |
| Original rows | 8,551 | 527 | 516 |
| Training-exposure matches removed | – | 6 | 12 |
| Conflicting-label rows removed | 22 | 0 | 6 |
| Repeated normalized-text rows removed | 10 | 0 | 1 |
| Additional token-equivalent rows removed | 2 | 0 | 0 |
| Primary retained rows | 8,517 | 521 | 497 |
| Partition | Model | Status | MCC [95% interval] | Accuracy | Bal. acc. |
|---|---|---|---|---|---|
| ID ( ) | R | 0.585 [0.508, 0.661] | 0.829 | 0.779 | |
| Attention statistics | R | 0.639 [0.562, 0.708] | 0.852 | 0.794 | |
| BERT | V | 0.599 [0.525, 0.674] | 0.835 | 0.785 | |
| Token length | N | 0.000 [0.000, 0.000] | 0.693 | 0.500 | |
| Text length | N | 0.000 [0.000, 0.000] | 0.693 | 0.500 | |
| Majority | N | 0.000 [0.000, 0.000] | 0.693 | 0.500 |
| Partition | (R) | Attention statistics (R) | BERT (V) | |
|---|---|---|---|---|
| In-domain | 518 | 0.588 | 0.642 | 0.602 |
| Out-of-domain | 495 | 0.435 | 0.511 | 0.548 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Partition | Policy | TN | FP | FN | TP | |
|---|---|---|---|---|---|---|
| In-domain | Primary | 521 | 105 | 55 | 31 | 330 |
| In-domain | Strict | 518 | 105 | 54 | 31 | 328 |
| Out-of-domain | Primary | 497 | 94 | 64 | 30 | 309 |
| Out-of-domain | Strict | 495 | 94 | 63 | 30 | 308 |
Transformer Attention
Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.
See all 345 papers