OpenAutoNLU: Open Source AutoML Library for NLU

Grigory Arshinov; Aleksandr Boriskin; Sergey Senichev; Ayaz Zaripov; Daria Galimzianova; Daniil Karpov; Leonid Sanochkin

cs.CLMar 2, 2026

OpenAutoNLU: Open Source AutoML Library for NLU

Authors: Grigory Arshinov, Aleksandr Boriskin, Sergey Senichev, Ayaz Zaripov, Daria Galimzianova, Daniil Karpov, Leonid Sanochkin

Organizations: MWS AI · ITMO University · MBZUAI

Abstract

OpenAutoNLU is an open-source automated machine learning library for natural language understanding (NLU) tasks, covering both text classification and named entity recognition (NER). Unlike existing solutions, we introduce data-aware training regime selection that requires no manual configuration from the user. The library also provides integrated data quality diagnostics, configurable out-of-distribution (OOD) detection, and large language model (LLM) features, all within a minimal lowcode API. The demo app is accessible here https://openautonlu.dev.

Figures & tables

Figure 1: Text Classification Training Pipeline and Inference Pipeline flow

	H2O	Light AutoML	Auto Gluon	Auto Intent	OpenAuto NLU
Approach	TAML & Word2Vec	TAML & embeddings	Encoder finetuning	Embeddings	Encoder finetuning
Scarce train data	$\times$	Has small-data modes	Adaptable	Adapted for small datasets	Designed for classes with $\geq$ 2 examples
Custom search pipeline	Flexible API	HP presets	HP presets	Presets & customizable configs	Universal customizable preset
LLM Integrations	$\times$	$\times$	$\times$	Zero-shot classification	Train data augmentations and test set generation
OOD detection	$\times$	$\times$	$\times$	Train examples required, one method	Train examples are optional, configurable
Optimized for NLP production	Requires preprocessing; Java, Python, R support	$\times$	✓	$\times$	ONNX native; inference in two lines

Table 1: Comparison of AutoML frameworks for NLP tasks.

Figure 2: This graph illustrates the ratio between performance in macro F1 on classification tasks and time in seconds took to train the solution. Measures are averaged between four different text classification datasets.

Framework	Preset	Banking77		HWU64		MASSIVE		SNIPS
		Macro	OOD	Macro	OOD	Macro	OOD	Macro	OOD
OpenAutoNLU	supervised	0.905	0.362	0.893	0.276	0.875	0.471	0.928	0.782
OpenAutoNLU	unsupervised	0.912	0.433	0.890	0.378	0.876	0.515	0.921	0.761
AutoIntent	CL & unsupervised	0.869	-	0.829	-	0.755	-	0.786	-
AutoIntent	CL & supervised	0.774	0.156	0.728	0.196	0.683	0.366	0.771	0.662
AutoIntent	CM & supervised	0.819	0.179	0.728	0.196	0.683	0.366	0.829	0.707

Table 2: F1-Macro and OOD F1-Score for frameworks with automatic OOD detection. In unsupervised regime no OOD train samples are provided, for supervised method OOD samples are selected as described in Section 4.2 . 2 best performing presets for AutoIntent are reported ( CL for classic-light and CM for classic-medium ). Full datasets were used. Best results per dataset are shown in bold .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

N-shot	Original	Generated	$\|\Delta\|$ (pp)
$[5,\,10]$	0.667	0.699	+0.032
$[10,\,20]$	0.655	0.637	-0.017
$[20,\,40]$	0.779	0.759	-0.02
$[40,\,80]$	0.808	0.790	-0.018
$[81,\,100]$	0.709	0.561	-0.148
Full	0.759	0.690	-0.689

Appendix

Table A.1: Macro-F1 on original vs. LLM-generated test sets. The horizontal rule separates regimes where the generated set is a reliable proxy (

|\Delta|<5

pp) from those where it is not.

Framework	Banking77	HWU64	MASSIVE	SNIPS
OpenAutoNLU	0.912	0.890	0.876	0.921
AutoGluon	0.920	0.902	0.861	0.763
AutoIntent	0.869	0.829	0.755	0.786
LightAutoML	0.907	0.900	0.841	0.761
H2O	0.645	0.642	0.701	0.778

Appendix

Table C.1: F1-Macro on full datasets under the OOD-unaware regime. OOD samples are present at test time but excluded from the classification report; scores reflect in-domain classification quality only. Best results per dataset are shown in bold .

Framework	Banking77			HWU64			MASSIVE			SNIPS
	5-10	40-80	Full	5-10	40-80	Full	5-10	40-80	Full	5-10	40-80	Full
OpenAutoNLU	0.727	0.881	0.920	0.747	0.753	0.902	0.666	0.711	0.886	0.607	0.890	0.953
AutoIntent	0.808	0.848	0.782	0.782	0.830	0.737	0.660	0.751	0.689	0.813	0.826	0.789
AutoGluon	0.598	0.911	0.935	0.625	0.899	0.923	0.508	0.824	0.885	0.696	0.868	0.915
LightAutoML	0.524	0.891	0.922	0.555	0.891	0.920	0.425	0.824	0.865	0.659	0.867	0.913
H2O	0.159	0.612	0.654	0.272	0.632	0.652	0.398	0.616	0.635	0.549	0.810	0.908

Appendix

Table C.2: In-scope macro-F1 score comparison across different N-shot settings for intent classification task. Default presets for all frameworks are used.

Primary dataset	Far OOD source
Banking77	HWU64
HWU64	Banking77
MASSIVE	Banking77
SNIPS	Banking77

Appendix

Table E.1: Far OOD source for each primary intent benchmark.

Automated Machine Learning

Automated machine learning (AutoML) builds and tunes data-to-model pipelines with minimal manual effort, balancing predictive quality, compute, and deployment constraints.

See all 96 papers

RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules
Jul 1, 2026Ádám Kovács, Nadia Verdha, Gábor RecskiLLM Grounding LLM-Based Program Synthesis
KR Labs
OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research
Jul 18, 2026Tavish Mankash, Vardhaman Kalloli, Keshava Prasad +1Language Model Pretraining Small Language Models
FAIRC · Plaksha University
ltzGLUE: Luxembourgish General Language Understanding Evaluation
Apr 20, 2026Alistair Plum, Felicia Körner, Anne-Marie Lutgen +8LLM Evaluation Text Classification
University of Luxembourg, Luxembourg · LMU Munich, Germany · Munich Center for Machine Learning, Germany +2

Jul 1, 2026J/K move · Enter open · S save

RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules

Ádám Kovács, Nadia Verdha, Gábor Recski

KR Labs

LLM Grounding LLM-Based Program Synthesis Human-in-the-Loop AI Rule-Based Classification

Open paper AskPDF arXiv

We present RuleChef, a framework that uses large language models (LLMs) to generate executable rules for NLP tasks such as text classification, Named Entity Recognition (NER), or relation extraction. Rules are generated based on a task description and a set of labeled examples, then they are iteratively improved based both on additional examples and on human feedback overexisting rules. RuleChef can also be used to bootstrap rules using the observed input-output pairs from any existing model for a given task. LLMs are used only at learning time, synthesizing rules and iteratively patching them based on failures measured on a held-out split. The result of this process is a fast, deterministic, and inspectable rule system. Preliminary evaluation is performed on both classification and NER tasks. We release RuleChef as open-source software under an Apache 2.0

Abstract

Automated Machine Learning

Explore similar work

RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules

OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

ltzGLUE: Luxembourgish General Language Understanding Evaluation