Paper ID: 2407.11591

AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization

Anum Afzal, Ribin Chalumattu, Florian Matthes, Laura Mascarell

Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale.

Submitted: Jul 16, 2024