How Green Is Your Manifesto? Scoring Environmental Positions in the 2024 EU Elections with an LLM Panel
Organizations: CRIEP, Interuniversity Research Centre on Public Economics, Padova, Italy · eXplainable Artificial Intelligence (XAI) Hub, University of Leeds, Leeds, United Kingdom · Leeds University Business School, University of Leeds, Leeds, United Kingdom
Abstract
Locating political parties on environmental and climate policy is central to the study of party competition, yet the established instruments, expert surveys and the hand coding of manifestos, are costly, periodic, and slow to extend to new documents. We ask whether a panel of large language models can serve as a cheaper and reproducible alternative. Treating each manifesto as a set of sentences, three models from independent providers score every sentence on two dimensions, environmental protection and climate change, separating a party's position on an issue from the salience it gives it, and the judgments are averaged to the manifesto level. Agreement across the models is retained as a built-in reliability signal. We apply the panel to 41 manifestos from the 2024 European Parliament election, covering Austria, France, Germany, Italy, Spain, and the European party families, and validate it against the 2024 Chapel Hill Expert Survey for the 25 parties with a like-for-like expert benchmark, using the other manifestos as diagnostic cases. The panel recovers the expert ordering of parties with strong rank agreement on both dimensions, while reading them as systematically greener than experts do, a gap we interpret as the distance between manifesto promise and governing record. Inter-model disagreement modestly flags where the instrument errs, and position and salience prove related for environmental protection but independent for climate change. The result is a transparent, low-cost, and multilingual instrument that complements rather than replaces expert and hand-coded approaches; the prompt, code, and manifesto-level results are released openly.
Figures & tables
| Metric | Environmental protection | Climate change |
| Spearman rank correlation | 0.82 | 0.86 |
| Pearson linear correlation | 0.86 | 0.82 |
| Mean bias (ensemble minus expert) | ||
| Mean absolute error | 0.82 | 1.00 |
| Share within one expert standard deviation | 57% | 39% |
| Ratio of ensemble to expert dispersion across parties | 1.01 | 1.15 |
| Relationship | Environmental protection | Climate change |
| Inter-model standard deviation vs absolute error | 0.36 | 0.43 |
| Position vs salience | 0.45 | 0.11 |
| Ensemble salience vs expert salience | 0.45 | 0.04 |
| Environmental protection | Climate change | ||||
| Parties included | Spearman | Bias | Spearman | Bias | |
| All validation parties | 25 | 0.82 [0.61, 0.92] | 0.86 [0.68, 0.94] | ||
| Excluding unverified sources a | 24 | 0.82 [0.60, 0.92] | 0.88 [0.71, 0.95] | ||
| and corrupted OCR b | 23 | 0.80 [0.55, 0.92] | 0.87 [0.67, 0.95] | ||
| and merged bullet lists c | 21 | 0.83 [0.59, 0.93] | 0.86 [0.64, 0.96] | ||
| and truncated or very short d | 19 | 0.79 [0.52, 0.92] | 0.82 [0.55, 0.95] | ||
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Country | Acronym | Party or list | CHES id | Role | Sentences | On topic | Note |
|---|---|---|---|---|---|---|---|
| AT | FPO | Freiheitliche Partei Österreichs | 1303 | V | 99 | 18 / 3 / 19 | |
| AT | Grune | Die Grünen – Die Grüne Alternative | 1304 | V | 1,409 | 437 / 63 / 462 | |
| AT | KPO | Kommunistische Partei Österreichs | – | X | 200 | 83 / 11 / 77 | |
| AT | NEOS | NEOS – Das Neue Österreich | 1306 | V | 614 | 56 / 12 / 66 | |
| AT | OVP | Österreichische Volkspartei | 1302 | V | 478 | 60 / 6 / 66 | |
| AT | SPO | Sozialdemokratische Partei Österreichs | 1301 | V | 282 | 48 / 4 / 48 |
Language Model Ensembles
Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.
See all 154 papers