AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions

Nils Constantin Hellwig; Jakob Fehle; Udo Kruschwitz; Christian Wolff

cs.CLMar 2, 2026

AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions

Authors: Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz, Christian Wolff

Organizations: Media Informatics Group, University of Regensburg, Regensburg, Germany · Information Science Group, University of Regensburg, Regensburg, Germany

Abstract

We introduce AnnoABSA, the first web-based annotation tool to support the full spectrum of Aspect-Based Sentiment Analysis (ABSA) tasks. The tool is highly customizable, enabling flexible configuration of sentiment elements and task-specific requirements. Alongside manual annotation, AnnoABSA provides optional Large Language Model (LLM)-based retrieval-augmented generation (RAG) suggestions that offer context-aware assistance in a human-in-the-loop approach, keeping the human annotator in control. To improve prediction quality over time, the system retrieves the ten most similar examples that are already annotated and adds them as few-shot examples in the prompt, ensuring that suggestions become increasingly accurate as the annotation process progresses. Released as open-source software under the MIT License, AnnoABSA is freely accessible and easily extendable for research and practical applications.

Figures & tables

Technical requirements
Feature	Label Studio	INCEpTION	BRAT	AnnoABSA
Open source	$\sqrt{}$ (Apache 2.0)	$\sqrt{}$ (Apache 2.0)	$\sqrt{}$ (MIT)	$\sqrt{}$ (MIT)
Installation	Docker, Python 3	Docker, Tomcat, Java	Python 2	Python 3
Regular updates	$\sqrt{}$	$\sqrt{}$	$\times$ Oct. 2021	$\sqrt{}$
Web-based	$\sqrt{}$	$\sqrt{}$	$\sqrt{}$	$\sqrt{}$
Multi-user support	$\sqrt{}$	$\sqrt{}$	$\times$	$\sqrt{}$

Table 1: Comparison of Annotation Tools for ABSA. Detailed comparison of four annotation platforms across technical requirements, usability features, data management capabilities, and ABSA-specific functionalities. The table constitutes an extension of the comparison provided by Colucci Cante et al. (2024, p. 359) , who evaluated non-semantic textual annotation tools based on the following criteria: multi-labelling functionality, annotation suggestions, relationship tagging, label customization, and team collaboration.

Figure 1: UI of AnnoABSA. The UI consists of four components: (1) top navigation bar to change the currently displayed example, (2) annotated text with highlighted sentiment annotations, (3) aspect addition panel, and (4) editable list of added annotations.

Figure 3: F1 score comparison of RAG-based and random sampling approaches. RAG consistently outperforms random sampling across all configurations. Statistical tests (paired t-test or Wilcoxon signed-rank test with Holm-Bonferroni correction) confirm all differences are significant.

M_{diff}

shows mean performance differences.

Pool	LLM Execution time (seconds)
	ACD		TASD		ASQP
	RAG	Random	RAG	Random	RAG	Random
100	0.816	0.851	1.292	1.240	1.708	1.622
200	0.848	0.852	1.291	1.204	1.723	1.626
300	0.846	0.871	1.291	1.211	1.736	1.642
400	0.862	0.856	1.298	1.221	1.743	1.622

Table 2: LLM inference time comparison for ABSA tasks. Average execution time per prediction (in seconds) comparing RAG-based versus random sampling strategies across varying pool sizes for ACD, TASD, and ASQP tasks.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task Type	Task Name	Elements	Gold Label (Example)
Single	Aspect Term Extraction (ATE)	a	[’waiter’]
	Aspect Category Detection (ACD)	c	[’food quality’, ’service speed’]
	Opinion Term Extraction (OTE)	o	[’delicious’, ’way too slow’]
Compound	Aspect Sentiment Classification (ASC)	a , s	[(’waiter’, ’negative’)]
	Aspect-Opinion Pair Extraction (AOPE)	a , o	[(’waiter’, ’way too slow’)]
	End-to-End ABSA (E2E-ABSA)	a , s	[(’NULL’, ’positive’), (’waiter’, ’negative’)]

Appendix

Table 3: Overview of ABSA tasks supported by AnnoABSA. For each task, a gold label is presented for the sentence “It was really delicious, but the waiter was way too slow” . Aspect categories are commonly selected from a predefined set ({food quality, service speed, …}). Notation: a = aspect term, c = aspect category, o = opinion term, s = sentiment polarity. In case of an implicit aspect, aspect term a is set to ’NULL’ . The listed tasks are equivalent to those reported in the literature review by Zhang et al. (2022)

Server Configuration
Option	Description	Default
--backend	Start only backend server	–
--backend-port	Backend server port	8000
--frontend-port	Frontend server port	3000
--backend-ip	Backend server IP address	localhost
--frontend-ip	Frontend server IP address	localhost

Appendix

Table 4: CLI options for AnnoABSA. AnnoABSA offers extensive customization across server settings, annotation logic, and AI integration.

Figure 4: Prompt used for RAG-based suggestion prediction. The prompt includes a task description with explanations of sentiment elements, ten in-context demonstrations, and the target text for aspect prediction. The few-shot examples shown are taken from the Coursera dataset and include annotations for ASQP

Aspect-Based Sentiment Analysis

Aspect-Based Sentiment Analysis (ABSA) identifies what people express opinions about and the sentiment or emotion attached to each aspect.

See all 51 papers

ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation
Jun 2, 2026Ana-Maria Luisa Mocanu, Ciprian-Octavian Truica, Elena-Simona ApostolHuman-in-the-Loop Annotation Aspect-Based Sentiment Analysis
1National University of Science and Technology POLITEHNICA Bucharest, Romania · 2Academy of Romanian Scientists, Ilfov 3, Bucharest, 050044, Romania
LLM-as-an-Annotator: Training Lightweight Models with LLM-Annotated Examples for Aspect Sentiment Tuple Prediction
Mar 2, 2026Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz +1Aspect-Based Sentiment Analysis LLM-Assisted Annotation
Media Informatics Group, University of Regensburg, Regensburg, Germany · Information Science Group, University of Regensburg, Regensburg, Germany
Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model
May 5, 2026Niklas Donhauser, Jakob Fehle, Nils Constantin Hellwig +3Annotator Disagreement LLM-Assisted Annotation
Media Informatics Group, University of Regensburg, Regensburg, Germany · Information Science Group, University of Regensburg, Regensburg, Germany

Jun 2, 2026J/K move · Enter open · S save

ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation

Ana-Maria Luisa Mocanu, Ciprian-Octavian Truica, Elena-Simona Apostol

1National University of Science and Technology POLITEHNICA Bucharest, Romania · 2Academy of Romanian Scientists, Ilfov 3, Bucharest, 050044, Romania

Human-in-the-Loop Annotation Aspect-Based Sentiment Analysis

Open paper AskPDF arXiv

Aspect-Based Sentiment Analysis (ABSA) requires high-quality datasets to train reliable models. However, existing annotation tools treat output as flat files, leaving researchers to manually consolidate multi-annotator data, reconstruct relational structures, and compute reliability metrics through custom scripts. This paper introduces ACAT (Aspect-based sentiment analysis Collaborative Annotation Tool), a web-based platform natively supporting four ABSA workflows: (1) Aspect-Category Sentiment Analysis, (2) Clause-Level Segmentation, (3) Aspect-Term Sentiment Analysis with character-level position tracking, and (4) Aspect Sentiment Triplet Extraction with dual span offset preservation. Its core contribution is an automated Extract, Transform, Load (ETL) pipeline that aligns collaborative annotations and computes Inter-Annotator Agreement (IAA) metrics directly at export, yielding training-ready datasets. In a preliminary validation on 1,002 restaurant reviews with two annotators of differing expertise, ACAT achieves a median annotation time of 31.58 seconds and a raw IAA ranging from 0.78 to 0.86 across all tasks.

Abstract

Aspect-Based Sentiment Analysis

Explore similar work

ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation

LLM-as-an-Annotator: Training Lightweight Models with LLM-Annotated Examples for Aspect Sentiment Tuple Prediction

Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model