MONETA: Multimodal Industry Classification through Geographic Information with Multi Agent Systems

Arda Yüksel; Gabriel Thiem; Susanne Walter; Patrick Felka; Gabriela Alves Werb; Ivan Habernal

cs.AIApr 9, 2026

MONETA: Multimodal Industry Classification through Geographic Information with Multi Agent Systems

Authors: Arda Yüksel, Gabriel Thiem, Susanne Walter, Patrick Felka, Gabriela Alves Werb, Ivan Habernal

Organizations: Trustworthy Human Language Technologies · Technical University of Darmstadt, Germany · Deutsche Bundesbank · Frankfurt University of Applied Sciences, Germany ·

Abstract

Industry classification schemes are integral parts of public and corporate databases as they classify businesses based on economic activity. Due to the size of the company registers, manual annotation is costly, and fine-tuning models with every update in industry classification schemes requires significant data collection. We replicate the manual expert verification by using existing or easily retrievable multimodal resources for industry classification. We present MONETA, the first multimodal industry classification benchmark with text (Website, Wikipedia, Wikidata) and geospatial sources (OpenStreetMap and satellite imagery). Our dataset enlists 1,000 businesses in Europe with 20 economic activity labels according to EU guidelines (NACE). Our training-free baseline reaches 62.10% and 74.10% with open and closed-source Multimodal Large Language Models (MLLM). We observe an increase of up to 22.80% with the combination of multi-turn design, context enrichment, and classification explanations. We will release our dataset and the enhanced guidelines.

Figures & tables

Figure 1: Zero-Shot vs Multi-Turn comparison for MONETA . (1) Zero-shot pipeline (left): Available resources are forwarded into the industry classifier MLLM together. Explanations and classifications are obtained. (2) Multi-Turn pipeline (right): Each resource is processed by separate specialized agents. Intermediate clues from these agents are processed by the decision-making agent, returning explanation and classification.

Dataset	Classes	Samples	Text Source	Image Source	Industry Scheme
Remote Sensing Classification Benchmarks
UC Merced Yang and Newsam (2010)	21	2,100	✗	Satellite	✗
AID Xia et al. (2017)	30	10,000	✗	Satellite	✗
AID++ Jin et al. (2018)	46	400,000+	✗	Satellite	✗
CLRS Li et al. (2020)	25	15,000+	✗	Satellite	✗
Industry Classification Benchmarks

Table 1: Comparison of datasets across remote sensing and industry classification tasks. ✗ indicates the absence of that modality or label scheme.

Figure 2: Overview of the dataset preparation process. (1) NACE section XMLs are initially converted to the OSM tags and manually checked by the authors. (2) We added custom filters for data quality and queried Europe OSM data with tag list. (3) Samples are grouped by NACE codes to form the gold dataset.

Figure 3: Example with ground truth NACE sections K and prediction G. Clues and predictions are obtained via InternVL-3 8B. Clue Analysis Methodology: (1) Sources are forwarded into separate MLLM Agents for clue extraction. (2) Keywords based on predefined Economic Activities are extracted and grouped into NACE sections. (3) For each resource (OSM, Satellite, Wikidata, Wikipedia and Website), normalized count vectors are formed. The grouped keywords are first placed to a vector with 21 dimensions (number of sections) and then divided by the total number of keywords for inference. (4) Ground-Truth and Final Prediction NACE sections are used to form $V_{G}$ and $V_{P}$ . Both of these vectors have the dimensions of 5 (number of inputs).

Model	Size (B)	None	Satellite	External	Satellite + OSM	Satellite + External	All
Open Source Models
InternVL 2.5	1	4.20	2.20	1.00	2.50	0.90	0.20
	4	8.70	6.60	11.10	4.70	13.50	6.40
	38	46.30	49.80	58.40	51.40	61.40	60.10
InternVL 3	8	43.60	34.90	48.10	30.10	46.90	41.70
	14	45.00	49.30	56.10	48.30	55.60	53.00

Table 2: Baseline ( Zero-Shot pipeline, Simple prompt, Text output) accuracy for NACE industry classification. Columns after model and size denote input configurations. Image inputs are highlighted. Bold indicates the best performance for the model and size pair for the open-source model, and the best performance among closed-source models. GPT 5 Mini and InternVL 3-78B are the best-performing closed and open source models.

Figure 4: Baseline unknown ratio of InternVL 3 for input configurations. Unknown ratio corresponds to UNK (uncertain) responses over all inferences.

Model	Size	Baseline	Explanation	Extended Prompt	Multi-Turn	Extended Prompt +	Mixture
Open Source Models
InternVL 2.5	4B	6.40	23.10 ( 16.70 )	8.30 ( 1.90 )	22.00 ( 15.60 )	10.60 ( 4.20 )	29.20 ( 22.80 )
InternVL 2.5	38B	60.10	61.80 ( 1.70 )	64.20 ( 4.10 )	58.20 ( -1.90 )	65.00 ( 4.90 )	60.20 ( 0.10 )
InternVL 3	8B	41.70	42.30 ( 0.60 )	36.90 ( -4.80 )	49.80 ( 8.10 )	38.00 ( -3.70 )	45.70 ( 4.00 )
InternVL 3	38B	58.30	59.80 ( 1.50 )	61.30 ( 3.00 )	61.60 ( 3.30 )	64.10 ( 5.80 )	62.60 ( 4.30 )
QwenVL 2.5	7B	23.10	30.50 ( 7.40 )	27.30 ( 4.20 )	38.90 ( 15.80 )	31.00 ( 7.90 )	45.90 ( 22.80 )

Table 3: Selected model accuracies with: Explanations, prompt context enrichment, multi-turn pipeline. Extended Prompt + is the combination of the extended prompt with explanations. Mixture is the combined setting with all the advancements. In these results, MLLMs used all inputs. The best result for a given model and size is shown in bold .

Figure 5: InternVL 3 (8B and 38B) correctness and effectiveness scores for each input. In these experiments, multi-turn pipeline with extended prompt and classification explanations is used.

Example 1	Example 2
Inputs	Inputs
Kieswerk Bahrdorf is a producer and wholesaler of bulk materials such as sand and gravel , supplying the greater Wolfsburg area.	AnconAmbiente provides urban waste collection and environmental services for municipalities in the Province of Ancona.
Clues	Clues
Satellite: [quarrying] Excavation and heavy machinery. Website: [wholesale] Producer and wholesaler of bulk materials.	Satellite: [manufacturing] Large industrial buildings. Website: [waste] Waste collection services.
Rationale and Label	Rationale and Label
Satellite imagery shows excavation and heavy machinery consistent with quarrying. The term “Kieswerk” explicitly refers to a gravel pit. Label: Mining and Quarrying	Observed infrastructure and website content indicate organized waste collection rather than manufacturing. Label: Water Supply; Sewerage, Waste Management and Remediation Activities

Table 4: Qualitative MONETA examples for correct inferences with contradicting clues Satellite image and website summary are followed by extracted clues. LLM clues, rationale, and decision are obtained via InternVL 3-38B in the mixture configuration. Green indicates content supporting ground truth while orange contents do not match the ground truth label.

Configuration	Company Websites	NAICS
Zero-Shot
— 7B	57.93	50.19
— 32B	62.79	57.45
Few-Shot
— 7B	58.16	51.22
— 32B	68.86	56.72

Table 5: Qwen 2.5 accuracies on text-only benchmarks: ExioNAICS Guo et al. (2025) and Company Websites Jagrič and Herman (2024) .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Average number of associated resources (OSM images, satellite images, website text, Wikidata, and Wikipedia) per NACE sections in the MONETA dataset

Attribute	Description
id	Unique identifier for the object.
type	The object type (e.g., node, way, relation).
name	Human-readable name given in OSM.
bbox	Bounding box representing the spatial extent of the object.
osm_tags	OpenStreetMap tags associated with the object.
category	NACE Rev 2 sector classification of the entity (A to U).

Appendix

Table 6: Dataset entry contents. Each entry of MONETA contains these attributes. Id, type, name, bbox and OSM tags are retrieved from OSM. Sources are retrieved online from existing OSM tags.

Attribute	Value
id	122563530
type	way
name	Heim Kieswerk
bbox	[12.4893727, 50.9761359, 12.5089029, 50.9916218]
category	B (Mining and quarrying)
osm_tags	addr:city = Nobitz addr:country = DE addr:housenumber = 14c addr:postcode = 04603 addr:street = Altenburger Straße landuse = quarry resource = sand operator = Heim Kieswerk Nobitz GmbH & Co. KG

Appendix

Table 7: MONETA entry for Heim Kieswerk derived from OpenStreetMap and external sources.

Figure 7: Spatial distribution of MONETA entries across Europe, showing the geographic coverage of OSM-derived entities used for NACE classification.

Figure 8: Distribution of OpenStreetMap (OSM) tags across NACE economic activity sections. The horizontal bar chart illustrates the mapping density for each category, with bar length and labels indicating the total number of unique OSM tags associated with a specific NACE section.

Panel of Figure 9 — Figure 9: RDF/XML NACE official guideline content for Section K (Financial and Insurance Activities).

Panel of Figure 10 — Figure 10: Gemini-generated OpenStreetMap tag candidates for the NACE Financial and Insurance Activities section (K). Relevant tags align with official NACE definitions, while other tags are either rare, non-standard, or weakly related.

Section	Title	Description	Keywords
A	Agriculture, Forestry and Fishing	This section covers the utilization of plant and animal natural resources through farming, animal husbandry, and harvesting from natural environments.	[agriculture], [forestry], [fishing], [crops], [livestock], [timber]
B	Mining and Quarrying	This section includes the extraction of naturally occurring minerals in solid, liquid, or gaseous forms, using various methods such as underground mining, surface mining, and well operations, along with related preparation activities.	[mining], [quarrying], [oil], [coal], [ores]
C	Manufacturing	This section includes the physical or chemical transformation of raw materials or components into new products, typically resulting in outputs ready for use or as inputs to further manufacturing.	[manufacturing], [processing], [assembly], [fabrication]
D	Electricity, Gas, Steam and Air Conditioning Supply	This section covers the provision and distribution of electricity, natural gas, steam, hot water, and air conditioning through a permanent infrastructure of networks such as lines, mains, and pipes.	[electricity], [gas], [steam]
E	Water Supply; Sewerage, Waste Management and Remediation Activities	This section includes the collection, treatment, and disposal of waste and sewage, as well as the management of contaminated sites and the supply of water for various uses.	[water], [sewerage], [waste], [remediation]
F	Construction	This section covers general and specialised construction activities for buildings and civil engineering works, including new projects, repairs, additions, and temporary structures, whether performed directly or through subcontracting.	[construction], [building], [infrastructure]

Appendix

Table 8: NACE Section Codes, Titles, AI-generated descriptions and keywords (from official guidelines). During the multi-turn inference, MLLMs will generate economic activity clues based on provided keywords.

Figure 11: NACE section distribution for MONETA-10K . Sections less than 2% are grouped into other for demonstration.

Text Source	Entry Count
Website	9,015
Wikidata	276
Wikipedia	13
Wikidata + Website	315
Wikidata + Wikipedia	147
Wikipedia + Website	13

Appendix

Table 9: MONETA-10K external resource counts. All denotes the existence of Wikidata, Wikipedia, and Website

Metric	MONETA-10K	MONETA
Macro F1-score	39.40	38.70
Weighted F1-score	45.90	40.70
Precision	52.30	52.30
Recall	39.40	39.70

Appendix

Table 10: Macro and Weighted F1, recall, and F1-score for the MONETA-10K and MONETA datasets with InternVL 8B using Zero-Shot pipeline, Text output, Simple prompt and all available resources.

Figure 12: NACE Section-wise analysis for InternVL 2.5 (4B and 38B), InternVL 3 (8B and 38B), and QwenVL 2.5 (7B and 32B). Rows indicate experiment configurations. Mixture denotes Multi-Turn pipeline with classification explanations and prompt enrichment. Columns are the NACE section letters given in Table 8 .

Figure 13: Experiment configuration results given in Accuracy, Precision, and Recall for InternVL 2.5 (4B and 38B), InternVL 3 (8B and 38B), and QwenVL 2.5 (7B and 32B). At each setting, the given configuration is added. The order of configurations is: Baseline, Explanation, Explanation + Extended Prompt, Mixture (with Multi-Turn).

Figure 14: Clue Keywords confusion matrices obtained for InternVL 3 (8B and 38B), and QwenVL 2.5 (7B and 32B). Extracted keywords are grouped based on NACE sections in the rows. The columns are resources: OSM, Satellite, Wikidata, Wikipedia and Website.

Model	Size (B)	OSM		Satellite		Wikidata		Wikipedia		Website
InternVL 2.5	4	1.74	2.62	6.26	8.46	6.56	0.82	14.54	7.85	7.61	6.47
InternVL 2.5	38	8.15	16.05	6.58	8.12	54.10	56.28	37.58	38.83	30.03	43.45
InternVL3	8	9.31	20.02	11.02	13.77	18.85	14.75	17.15	17.47	29.32	39.03
	14	10.22	19.55	7.83	9.02	40.57	47.95	31.25	37.11	34.23	45.69
	38	14.77	23.40	11.04	14.62	58.61	57.65	33.92	41.23	34.58	45.72
	78	10.58	13.56	9.73	11.34	57.38	55.33	35.08	34.62	39.18	51.04

Appendix

Table 11: Performance comparison in the Mixture setting, reporting correctness (left) and effectiveness (right) for each model across input modalities. Image -based inputs are highlighted. Bold indicates the best performance in correctness/effectiveness for the model and size pair.

Model	Size (B)	OSM	Satellite	Wikidata	Wikipedia	Website
InternVL 2.5	4	24.50	35.20	13.11	51.92	22.85
InternVL 2.5	38	59.20	20.70	71.31	84.62	77.47
InternVL 3	8	84.60	77.50	31.15	67.31	91.82
	14	67.90	44.10	73.77	82.69	90.65
	38	87.10	78.40	78.69	90.38	79.38
	78	73.40	30.00	74.59	80.77	85.76

Appendix

Table 12: Information Discovery of inputs. Image inputs are highlighted. Bold denotes the highest information discovery from a model-size pair.

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 407 papers

Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy
May 11, 2026Denghao Ma, Qing Liu, Zulong Chen +5Hierarchical Classification Text Classification
Beijing Information Science and Technology University, China · Alibaba Group, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China +2
IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products
Jun 12, 2026Haonan Qi, Jin Cao, Yongqi Zhang +9Multimodal Large Language Models Multimodal Model Evaluation
Multimodal and Industrial AI TeamTaobao&Tmall, Alibaba Group
Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?
Oct 8, 2026Xingwu Zhang, Duanyang Du, Huiling Zhu +5Industrial Anomaly Detection Multimodal Anomaly Detection
Hunan University · University of Aberdeen · South China Normal University +1

May 11, 2026J/K move · Enter open · S save

Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy

Denghao Ma, Qing Liu, Zulong Chen, Chuanfei Xu+4

Beijing Information Science and Technology University, China · Alibaba Group, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China+2

Hierarchical Classification Text Classification Multimodal Classification Classification

Open paper AskPDF arXiv

Document classification forms the backbone of modern enterprise content management, yet existing benchmarks remain trapped in oversimplified paradigms -- single domain settings with flat label structures -- that bear little resemblance to the hierarchical, multi-modal, and cross-domain nature of real-world business documents. This gap not only misrepresents practical complexity but also stifles progress toward industrially viable document intelligence. To bridge this gap, we construct the first Multi-level, Multi-domain, Multi-modal document classification Benchmark (MMM-Bench). MMM-Bench includes (1) a deeply hierarchical taxonomy spanning five levels that capture the authentic organizational logic of business documentation; and (2) 5,990 real-world multi-modal documents meticulously curated from 12 commercial domains in Alibaba. Each document is manually annotated with a complete hierarchical path by domain experts. We establish comprehensive baselines on MMM-Bench, which consists of open-weight models and API-based models. Through systematic experiments, we identify four fundamental challenges within MMM-Bench and propose corresponding insights. To provide a solid foundation for advancing research in multi-level, multi-domain document classification, we release all of the data and the evaluation toolkit of MMM-Bench at https://github.com/MMMDC-Bench/MMMDC-Bench.

Abstract

Classification

Explore similar work

Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy

IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products

Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?