OpenAutoNLU: Open Source AutoML Library for NLU

Grigory Arshinov; Aleksandr Boriskin; Sergey Senichev; Ayaz Zaripov; Daria Galimzianova; Daniil Karpov; Leonid Sanochkin

cs.CLMar 2, 2026

OpenAutoNLU: Open Source AutoML Library for NLU

Authors: Grigory Arshinov, Aleksandr Boriskin, Sergey Senichev, Ayaz Zaripov, Daria Galimzianova, Daniil Karpov, Leonid Sanochkin

Organizations: MWS AI · ITMO University · MBZUAI

Abstract

OpenAutoNLU is an open-source automated machine learning library for natural language understanding (NLU) tasks, covering both text classification and named entity recognition (NER). Unlike existing solutions, we introduce data-aware training regime selection that requires no manual configuration from the user. The library also provides integrated data quality diagnostics, configurable out-of-distribution (OOD) detection, and large language model (LLM) features, all within a minimal lowcode API. The demo app is accessible here https://openautonlu.dev.

Figures & tables

Figure 1: Text Classification Training Pipeline and Inference Pipeline flow

	H2O	Light AutoML	Auto Gluon	Auto Intent	OpenAuto NLU
Approach	TAML & Word2Vec	TAML & embeddings	Encoder finetuning	Embeddings	Encoder finetuning
Scarce train data	$\times$	Has small-data modes	Adaptable	Adapted for small datasets	Designed for classes with $\geq$ 2 examples
Custom search pipeline	Flexible API	HP presets	HP presets	Presets & customizable configs	Universal customizable preset
LLM Integrations	$\times$	$\times$	$\times$	Zero-shot classification	Train data augmentations and test set generation
OOD detection	$\times$	$\times$	$\times$	Train examples required, one method	Train examples are optional, configurable
Optimized for NLP production	Requires preprocessing; Java, Python, R support	$\times$	✓	$\times$	ONNX native; inference in two lines

Table 1: Comparison of AutoML frameworks for NLP tasks.

Figure 2: This graph illustrates the ratio between performance in macro F1 on classification tasks and time in seconds took to train the solution. Measures are averaged between four different text classification datasets.

Framework	Preset	Banking77		HWU64		MASSIVE		SNIPS
		Macro	OOD	Macro	OOD	Macro	OOD	Macro	OOD
OpenAutoNLU	supervised	0.905	0.362	0.893	0.276	0.875	0.471	0.928	0.782
OpenAutoNLU	unsupervised	0.912	0.433	0.890	0.378	0.876	0.515	0.921	0.761
AutoIntent	CL & unsupervised	0.869	-	0.829	-	0.755	-	0.786	-
AutoIntent	CL & supervised	0.774	0.156	0.728	0.196	0.683	0.366	0.771	0.662
AutoIntent	CM & supervised	0.819	0.179	0.728	0.196	0.683	0.366	0.829	0.707

Table 2: F1-Macro and OOD F1-Score for frameworks with automatic OOD detection. In unsupervised regime no OOD train samples are provided, for supervised method OOD samples are selected as described in Section 4.2 . 2 best performing presets for AutoIntent are reported ( CL for classic-light and CM for classic-medium ). Full datasets were used. Best results per dataset are shown in bold .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

N-shot	Original	Generated	$\|\Delta\|$ (pp)
$[5,\,10]$	0.667	0.699	+0.032
$[10,\,20]$	0.655	0.637	-0.017
$[20,\,40]$	0.779	0.759	-0.02
$[40,\,80]$	0.808	0.790	-0.018
$[81,\,100]$	0.709	0.561	-0.148
Full	0.759	0.690	-0.689

Appendix

Table A.1: Macro-F1 on original vs. LLM-generated test sets. The horizontal rule separates regimes where the generated set is a reliable proxy (

|\Delta|<5

pp) from those where it is not.

Framework	Banking77	HWU64	MASSIVE	SNIPS
OpenAutoNLU	0.912	0.890	0.876	0.921
AutoGluon	0.920	0.902	0.861	0.763
AutoIntent	0.869	0.829	0.755	0.786
LightAutoML	0.907	0.900	0.841	0.761
H2O	0.645	0.642	0.701	0.778

Appendix

Table C.1: F1-Macro on full datasets under the OOD-unaware regime. OOD samples are present at test time but excluded from the classification report; scores reflect in-domain classification quality only. Best results per dataset are shown in bold .

Framework	Banking77			HWU64			MASSIVE			SNIPS
	5-10	40-80	Full	5-10	40-80	Full	5-10	40-80	Full	5-10	40-80	Full
OpenAutoNLU	0.727	0.881	0.920	0.747	0.753	0.902	0.666	0.711	0.886	0.607	0.890	0.953
AutoIntent	0.808	0.848	0.782	0.782	0.830	0.737	0.660	0.751	0.689	0.813	0.826	0.789
AutoGluon	0.598	0.911	0.935	0.625	0.899	0.923	0.508	0.824	0.885	0.696	0.868	0.915
LightAutoML	0.524	0.891	0.922	0.555	0.891	0.920	0.425	0.824	0.865	0.659	0.867	0.913
H2O	0.159	0.612	0.654	0.272	0.632	0.652	0.398	0.616	0.635	0.549	0.810	0.908

Appendix

Table C.2: In-scope macro-F1 score comparison across different N-shot settings for intent classification task. Default presets for all frameworks are used.

Primary dataset	Far OOD source
Banking77	HWU64
HWU64	Banking77
MASSIVE	Banking77
SNIPS	Banking77

Appendix

Table E.1: Far OOD source for each primary intent benchmark.

Automated Machine Learning

Automated machine learning (AutoML) builds and tunes data-to-model pipelines with minimal manual effort, balancing predictive quality, compute, and deployment constraints.

See all 96 papers

Jul 1, 2026

RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules

We present RuleChef, a framework that uses large language models (LLMs) to generate executable rules for NLP tasks such as text classification, Named Entity Recognition (NER), or relation extraction. Rules are generated based on a task description and a set of labeled examples, then they are iteratively improved based both on additional examples and on human feedback overexisting rules. RuleChef can also be used to bootstrap rules using the observed input-output pairs from any existing model for a given task. LLMs are used only at learning time, synthesizing rules and iteratively patching them based on failures measured on a held-out split. The result of this process is a fast, deterministic, and inspectable rule system. Preliminary evaluation is performed on both classification and NER tasks. We release RuleChef as open-source software under an Apache 2.0

Ádám Kovács, Nadia Verdha, Gábor Recski

KR Labs

LLM Grounding LLM-Based Program Synthesis Human-in-the-Loop AI Rule-Based Classification

Ask PDF

Jul 18, 2026

OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models while keeping their machinery visible. In OLM, model code reads like the architecture: components are ordinary modules, while Block, Residual, Repeat, and Parallel describe how they are wired. The resulting model can move unchanged from a teaching notebook to a complete pretraining run or a research ablation. OLM connects this readable model layer to tokenizers, local and streaming datasets, optimization, mixed precision, callbacks, checkpoints, and hardware-aware CPU, single-GPU, and single-node multi-GPU execution. We demonstrate the full path by tracing GPT-2 from diagram to code, launching a FineWeb-Edu training script, replacing one attention component, and letting AutoTrainer configure the available machine. The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research. Validation shows close agreement with independent reference implementations, 90.6% four-GPU weak-scaling efficiency for a 348M-parameter workload, compact architecture edits, and positive early usability results. OLM is MIT-licensed and available through PyPI, GitHub, and its documentation site.

Tavish Mankash, Vardhaman Kalloli, Keshava Prasad +1

FAIRC · Plaksha University

Language Model Pretraining Small Language Models Efficient Language Model Training

Ask PDF

Apr 20, 2026

ltzGLUE: Luxembourgish General Language Understanding Evaluation

This paper presents ltzGLUE, the first Natural Language Understanding (NLU) benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English. Although NLU tasks are available for many European languages nowadays, LTZ is one of the official national languages that is often overlooked. We construct new tasks and reuse existing ones to introduce the first official NLU benchmark and accompanying evaluation of encoder models for the language. Our tasks include common natural language processing tasks in binary and multi-class classification settings, including named entity recognition, topic classification, and intent classification. We evaluate various pre-trained language models for LTZ to present an overview of the current capabilities of these models on the LTZ language.

Alistair Plum, Felicia Körner, Anne-Marie Lutgen +8

University of Luxembourg, Luxembourg · LMU Munich, Germany · Munich Center for Machine Learning, Germany +2

LLM Evaluation Text Classification Low-Resource Language Processing

Ask PDF

Abstract

Automated Machine Learning

Explore similar work

RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules

OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

ltzGLUE: Luxembourgish General Language Understanding Evaluation