Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation
Organizations: Group · Acta Language Services, LLC · CFB Scientific Translations LLC
Abstract
In this paper, we present 1020 Group's submission to the WMT26 shared task on automated quality evaluation. Our system, Pterodactyl, was engineered specifically for subtasks 2 and 3, that is, segment-level quality score prediction and the detection of error-free segments. Pterodactyl is a highly customizable, ultra-low cost system optimized for binary classification in translation triage and data curation. It is based on an LLM-as-a-judge approach, with prompt engineering and model selection guided by ROC analysis. We also developed several variants of Pterodactyl that were specifically adapted for subtasks 3.
Figures & tables
| Pterodactyl | Austriadactylus | ||
| Description | Primary system, | Variant, | |
| optimized for subtasks 2 & 3 | optimized for subtasks 2 & 3 | ||
| Foundational LLM | OpenAI gpt-5.4 | OpenAI gpt-5.4 | |
| Reasoning effort | None | None | |
| Language-specific | None | OpenAI gpt-5.6-terra, | |
| exceptions | low reasoning, for | ||
| Wukongopterus | Austriadraco | |
| Description | Variant, | Variant, differently |
| optimized for subtask 3 | optimized for subtask 3 | |
| Foundational LLM | OpenAI gpt-5.4 | OpenAI gpt-5.4 |
| Reasoning effort | None | None |
| Language-specific | None | None |
| exceptions |
MT Evaluation
Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.
See all 181 papers