cs.CLApr 4, 2026

AI Appeals Processor: A Deep Learning Approach to Automated Classification of Citizen Appeals in Government Services

Authors: Vladimir Beskorovainyi

Organizations: Besk Tech · Moscow Institute of Physics and Technology (MIPT)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Government agencies must register, classify and route every citizen appeal within statutory time limits, and much of this work is still done by hand. We describe AI Appeals Processor, a classification and routing component deployed in a CPU-only government environment, and report what its evaluation and deployment taught us. On 10,000 real Russian-language appeals from a cross-domain dataset, we compare Bag-of-Words and TF-IDF with SVM, fastText, Word2Vec+LSTM and multilingual BERT on a three-way appeal-type task. On a held-out test set of 1,500 appeals, BERT reaches 82% accuracy and Word2Vec+LSTM 78%, against 67% for individual operators measured on an expert-adjudicated gold standard. We deployed the LSTM: in a workflow where an operator verifies every prediction, its lower training cost made frequent retraining on operator-verified labels practical, while the four-point accuracy gap did not change the operator's task. End-to-end handling time fell by 53-56% across four appeal-length bands (unweighted mean 22.5 to 10.25 minutes); model inference takes under two seconds of this. Most residual errors trace to the label taxonomy rather than the model: the statutory definitions of complaints and applications overlap, and many appeals carry two intents. A post-deployment audit of production classifications, made after several retraining cycles by operators who saw the assigned category, judged more than 95% correct; we explain why this figure is not comparable with the test-set result.

Figures & tables

ModelAccuracy95% CIPrecisionRecallF1Train (min)
Operators (manual)6764.6–69.3676566—
BoW+SVM7269.7–74.27370718
TF-IDF+SVM7572.7–77.176737412
fastText7673.8–78.17775763
Word2Vec+LSTM7875.8–80.078787895
mBERT8280.0–83.9838081240
Table 1: Appeal-type classification on the test set ( n=1,500n=1{,}500 ), %. P, R and F1 are macro-averaged. Train: minutes, including model selection, on the experimental server.
True \\backslash PredictedComplaintApplicationProposal
Complaint (570)4458639
Application (540)7241058
Proposal (390)3342315
Table 2: Confusion matrix of Word2Vec+LSTM on the test set.
ConfigurationAccuracyF1Train (min)
Embedding dimensionality (hidden = 64, dropout = 0.3)
6475%75%18
128 (selected)78%78%24
25677%77%38
LSTM hidden units (dimension = 128, dropout = 0.3)
3276%76%16
Table 3: Sensitivity of Word2Vec+LSTM, varying one factor at a time around the selected configuration. Training times are for single runs.
DomainF1
Housing & utilities84%
Healthcare80%
Education79%
Transportation77%
Urban development76%
Social welfare74%
Table 4: Per-domain F1 of Word2Vec+LSTM on the seven-class domain task (test set), %. Accuracy is 77%.
Length (words)ManualAssistedReduction
<100<10015.07.053%
100–30020.09.055%
300–50025.011.056%
>500>50030.014.053%
Unweighted mean22.510.2554%
Table 5: Average end-to-end handling time per appeal (minutes), including operator review and routing.
Concurrent requestsAverage handling time (min)
110.25
510.50
1011.20
2012.80
5015.30
10018.60
Table 6: Average end-to-end handling time per appeal in the assisted workflow under concurrent load.
More in this topic

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 408 papers

Explore similar work

CardsList
Apr 20, 2026

LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC), collected via the DataJud API (CNJ) and annotated across five legal areas through LLM-assisted labeling with heuristic validation. On a class-balanced test set, BERTimbau-LoRA, updating only 0.3% of model parameters, achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini). The gap is most striking on administrativo (administrative law): GPT-4o mini scores F1 = 0.00 and Claude 3.5 Haiku scores F1 = 0.08 on this class, while the fine-tuned model reaches F1 = 0.91. Both commercial LLMs exhibit a systematic bias toward civel (civil law), absorbing ambiguous classes rather than discriminating them, a failure mode that domain-adapted fine-tuning eliminates. These results demonstrate that general-purpose LLMs cannot substitute for domain-adapted models in Brazilian legal classification, even when the task is a simple 5-class problem, and that LoRA fine-tuning on a consumer GPU closes the gap at zero marginal inference cost. We release the full dataset, model, and pipeline to enable reproducible research in Portuguese legal NLP.

Pedro Barbosa de Carvalho Neto

Teresina, PI, Brazil

Legal NLPLLM EvaluationText ClassificationLanguage Model Bias EvaluationClassification
AskPDF
Sep 30, 2026

TTLab at Daleel 2026: STAR-Ar, Sequence Tagging for Argument Recognition in Arabic

Argument Mining (AM) is a critical NLP task that remains significantly under-resourced in Arabic. This paper presents \testttSTAR−Ar\testtt{STAR-Ar}, a BERT-BiLSTM-CRF architecture for argument discourse detection and classification, as our system for Daleel 2026, the inaugural Arabic argument mining shared task. The task requires the identification and classification of argumentative discourse units (ADUs) in debate and editorial texts.We jointly model these two objectives as a token-level sequence labeling task using a BERT-BiLSTM-CRF architecture that combines contextual transformer embeddings with structural transition constraints to support accurate span detection. \testttSTAR−Ar\testtt{STAR-Ar} achieves an F1-score of 72.69 on validation and 73.7 on test data. Our domain-specific analysis shows that models trained exclusively on editorials underperform those trained on debates, a disparity we primarily attribute to the smaller size of the editorial dataset. The code for \testttSTAR−Ar\testtt{STAR-Ar} is available at [\faGithub TTLabatDaleel2026](https://github.com/ENTAILab/daleel2026Arabic−Argumentative−Discourse−Mining){[\faGithub~TTLab at Daleel 2026](https://github.com/ENTAILab/daleel_2026_Arabic-Argumentative-Discourse-Mining)}

Bhuvanesh Verma, Ali Abusaleh, Alexander Mehler

Text Technology Lab (TTLab), Goethe University Frankfurt

Argument MiningArabic NLPClassificationLow-Resource Language Processing
AskPDF
Sep 29, 2026

StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams

Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. First, we release StreamDecisionBench (SDB), a dataset of eight streaming scenarios in four application families, with executable reference decisions derived from public rules. Second, we propose an evaluation protocol and a metric, in-force accuracy: the share of time the applied decision is correct across update intervals of 0.5-8 s. It reflects accuracy and latency jointly, attributing each error to judgment, latency or both. Third, we evaluate fifteen single-model settings, and this attribution separates speed-limited from judgment-limited models: slower, more accurate models lose 42-47% of the time to outdated answers, a fast model 35% to wrong ones. We therefore test hybrids in which a slow model corrects a fast one; with the right pairing and configuration, a hybrid outperforms every single model. However, even the best evaluated system keeps a correct decision in force only about two-thirds of the time, leaving a substantial gap for real-time use.

Jhen-Ke Lin, Chung Chun Wang

National Yang Ming Chiao Tung University

LLM EvaluationLLM InferenceLLM Decision-MakingStreaming Inference
AskPDF