Paper ID: 2412.03242 • Published Dec 4, 2024
Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus
Anastasiia Bezobrazova, Miriam Seghiri, Constantin Orasan
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
This paper compares the accuracy of the terms extracted using SketchEngine,
TBXTools and ChatGPT. In addition, it evaluates the quality of the definitions
produced by ChatGPT for these terms. The research is carried out on a
comparable corpus of fashion magazines written in English and Russian collected
from the web. A gold standard for the fashion terminology was also developed by
identifying web pages that can be harvested automatically and contain
definitions of terms from the fashion domain in English and Russian. This gold
standard was used to evaluate the quality of the extracted terms and of the
definitions produced. Our evaluation shows that TBXTools and SketchEngine,
while capable of high recall, suffer from reduced precision as the number of
terms increases, which affects their overall performance. Conversely, ChatGPT
demonstrates superior performance, maintaining or improving precision as more
terms are considered. Analysis of the definitions produced by ChatGPT for 60
commonly used terms in English and Russian shows that ChatGPT maintains a
reasonable level of accuracy and fidelity across languages, but sometimes the
definitions in both languages miss crucial specifics and include unnecessary
deviations. Our research reveals that no single tool excels universally; each
has strengths suited to particular aspects of terminology extraction and
application.