cs.CLMar 14, 2026
SaveThe GELATO Dataset for Legislative NER
Organizations: Brandeis University
Abstract
This paper introduces GELATO (Government, Executive, Legislative, and Treaty Ontology), a dataset of U.S. House and Senate bills from the 118th Congress annotated using a novel two-level named entity recognition ontology designed for U.S. legislative texts. We fine-tune transformer-based models (BERT, RoBERTa) of different architectures and sizes on this dataset for first-level prediction. We then use LLMs with optimized prompts to complete the second level prediction. The strong performance of RoBERTa and relatively weak performance of BERT models, as well as the application of LLMs as second-level predictors, support future research in legislative NER or downstream tasks using these model combinations as extraction tools.
Figures & tables
| Annotators | House Bills | Senate Bills | ||
|---|---|---|---|---|
| 1 | 2 | 3 | ||
| ✓ | ✓ | 141-170 | 169, 174, 176, 183, 187, 193, 196, 198, 199, 200 | |
| ✓ | ✓ | 171-200 | 102, 104, 107, 109, 112, 117, 121, 126, 128, 143 | |
| ✓ | ✓ | 100-140 | 144, 145, 148, 158, 160, 163, 164, 165, 167, 168 | |
Table 1: Bills annotated by each annotator
| Annotators | IO F1 | Level 1 F1 | ||
|---|---|---|---|---|
| 1 | 2 | 3 | ||
| ✓ | ✓ | 66.35 | 62.53 | |
| ✓ | ✓ | 77.27 | 73.50 | |
| ✓ | ✓ | 80.09 | 79.68 | |
Table 2: Inter-Annotator Agreement (F1)
| Train | Dev | Test | Total | |
|---|---|---|---|---|
| Bills | 80 | 21 | 30 | 131 |
| Tokens | 77,372 | 23,819 | 31,740 | 132,931 |
| Unique Mentions | 1,108 | 448 | 556 | 2,112 |
| Total Mentions | 3,388 | 1,029 | 1,393 | 5,810 |
Table 3: GELATO Summary
| Ontology | Locale | Person | Organization | Location | Legal Text | Time/Date | Other |
|---|---|---|---|---|---|---|---|
| GELATO | American English | ✓ | ✓ | ✓ | Abstraction, Act, Class | ||
| Cardellino et al. | American English | ✓ | ✓ | ✓ | Abstraction, Act | ||
| Leitner et al. | German | ✓ | ✓ | ✓ | ✓ | Case-by-case Regulation | |
| Kalamkar et al. | Indian English | ✓ | ✓ | ✓ | ✓ | ✓ | Court, Case Number |
| LeNER-Br | Brazilian Portuguese | ✓ | ✓ | ✓ | ✓ | ✓ | |
| E-NER | American English | ✓ | ✓ | ✓ | ✓ | Court, Miscellaneous |
Table 4: Comparison of legal-domain NER ontologies across different papers and languages
| Model | LR | BS | E | WD | WR | Lvl 1 F1 | Lvl 2 F1 |
|---|---|---|---|---|---|---|---|
| bert-base-cased | 3 e- 5 | 1 | 44 | 0.4 | 0.5 | 68.09 | 53.26 |
| bert-large-cased | 1 e- 5 | 1 | 34 | 0.0 | 0.2 | 67.16 | 53.14 |
| roberta-base | 1 e- 4 | 2 | 47 | 0.1 | 0.2 | 84.87 | 64.79 |
| roberta-large | 5 e- 5 | 1 | 48 | 0.5 | 0.1 | 88.72 | 67.44 |
| Gold Level One Labels | - | - | - | - | - | - | 76.60 |
Table 5: Best F1 scores and hyperparameter configuration for each model; LR = learning rate, BS = batch size, E = epochs, WD = weight decay, WR = warmup ratio
| Type | F1 | Type | F1 | ||
|---|---|---|---|---|---|
| Abstraction | 67.80 | Code | 5.26 | ||
| Case | – | Parenthetical | 13.33 | ||
| Doctrine | 0 | Reference | 61.54 | ||
| Fund | 66.67 | Report | 0 | ||
| Infrastructure | 61.54 | Treaty | – | ||
| Misc | 0 | Organization | 90.64 |
Table 6: F1 scores for level-two predictions using roberta-large and Qwen/Qwen3-32B; – indicates the label was not present
Figure 1: Level one confusion matrices
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Top Tag | Subclass | Description | Examples |
|---|---|---|---|
| Person | Member MBC | A member of Congress, typically the sponsor or co-sponsor of the legislation. | Mr. Biggs MBC |
| Title TTL | Mentions of individuals by their official position or office. | Speaker TTL ; President TTL | |
| Individual IND | Any specific named person who is neither a Member of Congress nor a Title. | John Doe IND |
Table 7: Tag Definitions: Person Subclasses
| Top Tag | Subclass | Description | Examples |
|---|---|---|---|
| Organization | Nation NAT | National-level geopolitical entities (includes American Indian nations). | United States NAT |
| State STA | State-level geopolitical entities within the U.S. | Illinois STA | |
| Locality LOC | Local geopolitical entities, such as cities, counties, or municipalities. | Cook County LOC | |
| Committee COM | Named Congressional committees and subcommittees. | Ways and Means COM | |
| Agency AGC | Federal departments, bureaus, or executive administrations. | Dept. of Justice AGC | |
| Leg. Body LEG | Primary legislative chambers (Senate, House, Congress). | Senate LEG |
Table 8: Tag Definitions: Organization Subclasses
| Top Tag | Subclass | Description | Examples |
|---|---|---|---|
| Document | Code CODE | Foundational legal codes and structured statutory bodies. | U.S. Code CODE |
| Bill BILL | Specific bill identifiers and tracking numbers. | H.R. 189 BILL | |
| Reference REF | Structural subdivisions of a legal text (sections, titles, paragraphs). | section 102 REF | |
| Parenthetical PAR | Legal citations provided in parenthetical format. | (42 U.S.C. 4332) PAR | |
| Treaty TRE | Named international treaties and conventions. | Geneva Convention TRE | |
| Report REP | Official reports, budget documents, or findings. | President’s Budget REP |
Table 9: Tag Definitions: Document Subclasses
| Top Tag | Subclass | Description | Examples |
|---|---|---|---|
| Act | Public Act PA | Named acts or legislation referred to by their popular title. | Green New Deal PA |
| Amendment AMD | Specific amendments to the U.S. Constitution. | Second Amendment AMD |
Table 10: Tag Definitions: Act Subclasses
| Top Tag | Subclass | Description | Examples |
|---|---|---|---|
| Abstraction | Program PRO | Named government-managed programs and initiatives. | Medicare PRO |
| System SYS | Intangible government frameworks, databases, or digital networks. | Forest System SYS | |
| Infrastructure INF | Tangible or physical frameworks established by government action. | Interceptor INF | |
| Fund FUND | Specific, named accounts or monetary trust funds. | General Fund FUND | |
| Doctrine DTR | Abstract legal policies, principles, or constitutional rights. | Free Speech DTR | |
| Case CASE | Court cases and judicial precedents. | McCulloch v. MD CASE |
Table 11: Tag Definitions: Abstraction Subclasses
| Top Tag | Subclass | Description | Examples |
|---|---|---|---|
| Class | Protected PC | Groups defined by law for special protections (Race, Disability, etc.). | veterans PC |
| Non-Prot. NPC | Other groups that are the subject of legislation but not protected. | refugees NPC |
Table 12: Tag Definitions: Class Subclasses
| Type | Train | Dev | Test | Total | |
|---|---|---|---|---|---|
| Abstraction | 217 | 91 | 79 | 387 | |
| Case | 0 | 0 | 0 | 0 | |
| Doctrine | 8 | 0 | 1 | 9 | |
| Fund | 39 | 11 | 1 | 51 | |
| Infrastructure | 3 | 0 | 21 | 24 | |
| Misc | 4 | 0 | 0 | 4 |
Table 13: Counts of total level-one and level-two mentions by split
More in this topic
Named Entity Recognition
Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.
See all 110 papers