HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries

Shangxuan Tian; Yanhui Chen; Carlos Queiroz

cs.AIAug 22, 2026

HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries

Authors: Shangxuan Tian, Yanhui Chen, Carlos Queiroz

Organizations: Standard Chartered Bank Guangzhou, China · OCBC Singapore

Abstract

Document classification in regulated industries is constrained by data residency, limited cold-start labels, scarce review capacity, and costly model-governance procedures. We present HIRA, a training-free, on-premises retrieval-augmented cascade for document classification in regulated deployments that combines BM25 over OCR text, dense text embeddings, and image-level representations through validation-calibrated weighted reciprocal-rank fusion. Confident documents are classified directly by retrieval; uncertain or visually confusable documents are passed to a locally hosted LLM verifier, which receives the OCR text, retrieved exemplars, label descriptions, and confusion-specific terms. When the verifier remains uncertain, the document is sent to human review. Each correction is stored as a margin-weighted retrieval exemplar and updates a Dirichlet-smoothed confusion graph, letting the system improve without updating model weights. On a private 80-class trade-finance corpus, HIRA processes the full 30,233-document production stream while requesting human correction for only 1,945 documents (6.4%), improving Macro-F1 from 0.6218 to 0.8548. On the corrected Tobacco-3482 benchmark, HIRA reaches 0.9423 Macro-F1 with a locally hosted DeepSeek-R1-Distill-Qwen-32B verifier, 17.4 percentage points above the zero-shot LLM baseline, while invoking the verifier for only about 40% of documents and reducing LLM calls by approximately 60%. With 518 human corrections (24.8% of the pool), HIRA matches the fully labelled pool oracle, in which all 2,086 pool documents are indexed with their ground-truth labels. These results show that selective human feedback and retrieval-memory adaptation can be a practical alternative to repeated model retraining for long-tail document classification in regulated deployments.

Figures & tables

Figure 1. The proposed HIRA cascade framework.

Method	Macro-F1	Acc.	Head-F1	Med-F1	Tail-F1
BM25 only	0.4758	0.5161	0.623	0.365	0.477
MiniLM only	0.4418	0.4588	0.518	0.333	0.511
SigLIP2 only	0.7718	0.7670	0.731	0.789	0.790
RRF fusion	0.7736	0.7778	0.778	0.798	0.737
DS 0-shot	0.7683	0.8172	0.928	0.730	0.660
DiT-100 †	0.8194	0.8065	0.839	0.841	0.971

Table 1. Static baselines on corrected Tobacco-3482 (seed-only index, 10 docs/class). HIRA at HITL convergence (N=518) is reported in Table 4 .

System	Macro-F1	Verifier call rate
DS 0-shot	0.7683	100%
RRF fusion	0.7736	0%
HIRA	0.9423	$\sim$ 40%
$\Delta$ (HIRA $-$ DS 0-shot)	$+$ 17.4 pp	$-60$ %

Table 2. HIRA cascade vs. LLM zero-shot on corrected Tobacco-3482.

Figure 2. HIRA HITL improvement curve on corrected Tobacco-3482 (primary axis, green). Secondary axis (red): HITL dynamic queue rate.

System	EDRM	Macro-F1	Acc.	Head	Med	Tail
kNN+Naive HITL	✗	0.8857	0.8823	0.875	0.854	0.936
HIRA	✓	0.9135	0.8961	0.871	0.907	0.964

Table 3. EDRM contribution ablation on corrected Tobacco-3482 (Tier-1 only).

$N$	Macro-F1	Acc.	Head	Med	Tail
DS 0-shot	0.7683	0.8172	0.928	0.730	0.660
0	0.8819	0.8925	0.935	0.859	0.859
100	0.9132	0.9176	0.952	0.875	0.953
200	0.9157	0.9211	0.936	0.892	0.948
518	0.9423	0.9355	0.937	0.926	0.970
DiT-618 †	0.9193	0.9211	0.956	0.915	0.978

Table 4. HITL learning trajectory with end-to-end evaluation on corrected Tobacco-3482.

Config ( $\tau_{\mathrm{gate}}$ , $\tau_{\mathrm{index}}$ )	Corr.	Ann. rate	T1-only	C-peak	C-final
V0: aggressive decay (0.9, 300)	862	41%	0.8337	0.9479	0.9304
V1: lower Tier-2 gate (0.9, 10k)	265	13%	0.8689	0.9248	0.9069
V2: tighter Tier-1 gate (0.85, 10k)	298	14.3%	0.8667	0.9080	0.9056
V3: deployed (0.95, 10k)	518	24.8%	0.9135	0.9423	0.9423

Table 5. Four-way HITL configuration comparison on corrected Tobacco-3482.

Figure 3. Class-count distribution of Financial-80 dataset.

Method	Macro-F1	Acc.	Head-F1	Med-F1	Tail-F1
BM25 only	0.351	0.484	0.517	0.435	0.307
MiniLM only	0.332	0.476	0.463	0.505	0.242
SigLIP2 only	0.391	0.554	0.681	0.514	0.316
RRF fusion	0.502	0.663	0.749	0.623	0.438
DS 0-shot	0.622	0.824	0.945	0.740	0.557

Table 6. Static baselines on Financial-80 dataset (seed-only index, 648 seed documents,

\sim

8 docs/class).

Figure 4. HIRA HITL improvement curve on Financial-80 (primary axis, green). Secondary axis (red): HITL dynamic queue rate.

$N$	Macro-F1	Acc.	Head	Med	Tail
DS 0-shot	0.6218	0.8238	0.945	0.740	0.557
0	0.7920	0.8804	0.950	0.836	0.695
200	0.8101	0.8932	0.965	0.848	0.687
600	0.8263	0.9119	0.969	0.878	0.693
1000	0.8447	0.9271	0.982	0.895	0.667
1945	0.8548	0.9271	0.976	0.904	0.701

Table 7. HIRA HITL trajectory on Financial-80 with end-to-end evaluation.

Text Classification

Text classification assigns text to categories such as topic, sentiment, intent, stance, or risk, supporting search, moderation, and decisions.

See all 226 papers

Apr 9, 2026

Retrieval Augmented Classification for Confidential Documents

Unauthorized disclosure of confidential documents demands robust, low-leakage classification. In real work environments, there is a lot of inflow and outflow of documents. To continuously update knowledge, we propose a methodology for classifying confidential documents using Retrieval Augmented Classification (RAC). To confirm this effectiveness, we compare RAC and supervised fine tuning (FT) on the WikiLeaks US Diplomacy corpus under realistic sequence-length constraints. On balanced data, RAC matches FT. On unbalanced data, RAC is more stable while delivering comparable performance--about 96% Accuracy on both the original (unbalanced) and augmented (balanced) sets, and up to 94% F1 with proper prompting--whereas FT attains 90% F1 trained on the augmented, balanced set but drops to 88% F1 trained on the original, unbalanced set. When robust augmentation is infeasible, RAC provides a practical, security-preserving path to strong classification by keeping sensitive content out of model weights and under your control, and it remains robust as real-world conditions change in class balance, data, context length, or governance requirements. Because RAC grounds decisions in an external vector store with similarity matching, it is less sensitive to label skew, reduces parameter-level leakage, and can incorporate new data immediately via reindexing--a difficult step for FT, which typically requires retraining. The contributions of this paper are threefold: first, a RAC-based classification pipeline and evaluation recipe; second, a controlled study that isolates class imbalance and context-length effects for FT versus RAC in confidential-document grading; and third, actionable guidance on RAC design patterns for governed deployments.

Yeseul E. Chang, Rahul Kailasa, Simon Shim +2

Department of Security Convergence, Chung-Ang University, Seoul, South Korea · Department of Applied Data Science, San Jose State University, San Jose, CA, United States · Department of Industrial Security, Chung-Ang University, Seoul, South Korea

Retrieval-Augmented Classification Text Classification Privacy-Preserving ML Privacy Leakage in Language Models Class-Imbalanced Learning

Ask PDF

Jul 12, 2026

Constraint-Aware Hierarchical Search for Regulation-Driven Fine-Grained Classification

Tasks such as customs tariff classification, export control categorization, and standards-based equipment coding require assigning an input instance to a fine-grained class under an explicit regulatory hierarchy. Unlike standard text classification, the correct label in these tasks is not determined by semantic similarity alone, but by rule-defined boundaries, threshold conditions, exclusion clauses, definitions, and local exceptions. As a result, two highly similar inputs may require different labels, while a retrieved passage that appears relevant may still be inapplicable under the governing rules. Existing flat classifiers, hierarchical text classification methods, and retrieval-augmented LLM systems are not designed to jointly enforce hierarchical validity, rule consistency, and fine-grained boundary reasoning. In this paper, we formulate this setting as regulation-driven fine-grained hierarchical classification, where an external instance must be assigned to a fine-grained class through a valid path in a regulatory hierarchy and supported by auditable evidence. We construct four benchmark datasets from representative regulation-intensive scenarios and validate the annotations through an expert-in-the-loop process. We further propose a constraint-aware hierarchical search framework that converts regulatory documents into a searchable tree, retrieves only valid local candidate nodes, and uses structured regulatory fields with evidence snippets to guide each next-hop decision. Experiments show that our method achieves the best mean accuracy on all four datasets and provides interpretable decision paths, with the largest gains on cases involving fine-grained neighboring categories and rule-based boundary conditions.

Siyu Wang, Wei Tan, Lulu Chen

aGusu Laboratory of Materials, Suzhou, China · cSuzhou Digital China Wuxin Intelligent Technology Co., Ltd., Suzhou, China

Hierarchical Classification Retrieval-Augmented Classification Classification Rule-Based Classification

Ask PDF

Jul 20, 2026

A Classifier That Teaches Itself: Self-Improving, Frozen-gate Training (SIFT) for Dynamic Document Classification

Document classification is a solved problem in the laboratory and an unsolved one in the enterprise. The blocker is rarely model architecture; it is the labeling project that must precede a model and the institutional fear of letting a model retrain itself once one exists. We present SIFT (Self-Improving, Frozen-gate Training), a dynamic classifier service, which attacks both. SIFT serves classification from a deliberately cheap, CPU-bound pipeline, a SPLADE sparse encoder feeding a LightGBM head, and escalates only the low-confidence minority of pages to an LLM judge. The judge's verdicts are written back into a labeled corpus, so the expensive model continuously teaches the cheap one: the escalation rate falls, the corpus grows from production traffic rather than from an up-front annotation effort, and accuracy compounds with use. Onboarding a new document family requires only a declarative bundle, label space, anchor phrases, and a judge glossary, not a labeling project. The harder problem is safety: an autonomously retraining classifier can silently regress. SIFT resolves this with a two-part promote gate, a critical-label F1 regression check plus a frozen golden regression set the model is never trained on, either of which vetoes promotion. This turns "retrain monthly without a human" from reckless into routine. We describe the architecture, the self-feeding corpus loop, the frozen-gate promotion mechanism, and an illustrative multi-domain deployment, and we discuss the economics of a classifier whose marginal labeling cost trends toward zero.

Bogdan Raduta, Horia Velicu, Alexandru Preda +1

LLM-Assisted Annotation Text Classification Classification Pseudo-Labeling

Ask PDF

Abstract

Text Classification

Explore similar work

Retrieval Augmented Classification for Confidential Documents

Constraint-Aware Hierarchical Search for Regulation-Driven Fine-Grained Classification

A Classifier That Teaches Itself: Self-Improving, Frozen-gate Training (SIFT) for Dynamic Document Classification