ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understanding

Ao Cheng; Xingming Li; Xuanyu Ji; Xixiang He; Qiyao Sun; Chunping Qiu; Runke Huang; Qingyong Hu

cs.CVMar 24, 2026

ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understanding

Authors: Ao Cheng, Xingming Li, Xuanyu Ji, Xixiang He, Qiyao Sun, Chunping Qiu, Runke Huang, Qingyong Hu

Organizations: National University of Defense Technology · Intelligent Game and Decision Lab · The Chinese University of Hong Kong, Shenzhen

Abstract

Electronic Navigational Charts (ENCs) are the safety-critical backbone of modern maritime navigation, yet it remains unclear whether multimodal large language models (MLLMs) can reliably interpret them. Unlike natural images or conventional charts, ENCs encode regulations, bathymetry, and route constraints via standardized vector symbols, scale-dependent rendering, and precise geometric structure -- requiring specialized maritime expertise for interpretation. We introduce ENC-Bench, the first benchmark dedicated to professional ENC understanding. ENC-Bench contains 20,490 expert-validated samples from 840 authentic National Oceanic and Atmospheric Administration (NOAA) ENCs, organized into a three-level hierarchy: Perception (symbol and feature recognition), Spatial Reasoning (coordinate localization, bearing, distance), and Maritime Decision-Making (route legality, safety assessment, emergency planning under multiple constraints). All samples are generated from raw S-57 data through a calibrated vector-to-image pipeline with automated consistency checks and expert review. We evaluate 10 state-of-the-art MLLMs such as GPT-4o, Gemini 2.5, Qwen3-VL, InternVL-3, and GLM-4.5V, under a unified zero-shot protocol. The best model achieves only 47.88% accuracy, with systematic challenges in symbolic grounding, spatial computation, multi-constraint reasoning, and robustness to lighting and scale variations. By establishing the first rigorous ENC benchmark, we open a new research frontier at the intersection of specialized symbolic reasoning and safety-critical AI, providing essential infrastructure for advancing MLLMs toward professional maritime applications.

Figures & tables

Figure 1: Overview of ENC-Bench. Our benchmark evaluates MLLMs across three hierarchical tiers: Perception (L-1), Spatial Reasoning (L-2), and Maritime Decision-Making (L-3). Example tasks are shown on the left with corresponding visual elements on an authentic NOAA chart (right), demonstrating the progression from basic symbol interpretation to complex multi-constraint decision-making required in professional maritime navigation.

		Standardized	Precise Geospatial	Multi-Scale	Multi-Light
Benchmark	Domain	Symbol Recognition	Reasoning (Numerical)	(Cartographic)	(Rendered Modes)
MME [ 16 ]	General	✗	✗	✗	✗
MMBench [ 32 ]	General	✗	✗	✗	✗
MMMU [ 64 ]	General	$\blacktriangle$	$\blacktriangle$	✗	✗
DocVQA [ 40 ]	Chart/Doc	✗	$\blacktriangle$	✗	✗
ChartQA [ 38 ]	Chart/Doc	✗	✓	✗	✗

Table 1: Comparison of ENC-Bench with existing benchmarks. ✓ ,

\blacktriangle

, and ✗ separately represent full support (supports the core capability), partial support (e.g., informal/math symbols, Euclidean/layout reasoning, or multi-resolution), and no support .

Figure 2: ENC-Bench Data Generation Pipeline. Four-stage process transforms 840 NOAA S-57 charts into 20,490 validated samples: (1) Rendering & Parsing produces multi-condition images and extracts GeoJSON features; (2) Image Registration establishes pixel-to-geo coordinate conversion; (3) Feature Annotation marks point/line/polygon features with expert verification; (4) Question Generation applies templates to create structured QA pairs with validated ground truth.

Statistics	Number
Total Questions	20,490
Total Charts	840
Task Categories	10
Task Hierarchy
Perception	14,886 (72.6%)
- Symbol Recognition	384

Table 2: Key statistics of the ENC-Bench benchmark. ∗ Excluding 384 Symbol Recognition samples (isolated symbols without contextual rendering). Contextual tasks total: 20,106 samples.

	Perception				Decision-Making
Model	Symbol	Point	Line	Polygon	Track	Safety	Anchor	Average
Gemini-2.5-Pro [ 10 ]	69.53	45.38	30.05	39.95	63.12	57.55	29.55	47.88
Gemini-2.5-Flash [ 10 ]	63.80	41.89	29.08	39.44	59.57	55.84	24.35	44.85
GPT-4o [ 24 ]	50.78	36.39	21.58	23.97	45.39	45.44	20.57	34.87
Qwen3-VL-235B-Instruct [ 61 ]	57.03	51.79	29.70	29.93	74.47	58.97	26.48	46.91
Qwen3-VL-235B-Thinking [ 61 ]	54.17	49.34	28.18	27.83	75.18	61.54	30.50	46.68

Table 3: Performance on Perception and Decision-Making tasks. Results in accuracy (%). Bold indicates best performance.

	Coordinate (Geo)		Coordinate (Pixel)		Direction		Distance
Model	Acc@200px (%) $\uparrow$	Err (px) $\downarrow$	Acc@200px (%) $\uparrow$	Err (px) $\downarrow$	Acc@20° (%) $\uparrow$	Err (°) $\downarrow$	[email protected] (%) $\uparrow$	Err (%) $\downarrow$
Gemini-2.5-Pro [ 10 ]	17.36	495.2	21.43	480.5	46.86	29.95	25.67	42.31
Gemini-2.5-Flash [ 10 ]	16.77	502.1	21.03	466.0	28.77	66.89	25.93	42.72
GPT-4o [ 24 ]	12.20	540.5	6.94	552.1	25.15	73.88	19.29	48.78
Qwen3-VL-235B-Instruct [ 61 ]	11.31	570.4	12.20	510.9	36.95	33.53	16.71	56.36
Qwen3-VL-235B-Thinking [ 61 ]	11.01	555.2	12.90	501.7	55.64	34.15	20.33	51.02

Table 4: Performance on Spatial Reasoning tasks. Results in Acc@T (%) and Mean Error. Bold indicates best performance.

Task Category	Day	Dusk	Night
Perception & Decision-Making (Accuracy % $\uparrow$ )
Point Features	42.88	41.76 ( -1.12 )	42.27 ( -0.61 )
Linestring Features	27.01	24.86 ( -2.15 )	25.26 ( -1.75 )
Polygon Features	27.32	29.92 ( +2.60 )	27.70 ( +0.38 )
Track Direction	65.35	61.27 ( -4.08 )	58.92 ( -6.43 )
Safety Assessment	57.80	55.99 ( -1.81 )	57.52 ( -0.28 )

Table 5: Average model performance across lighting conditions. Delta shows change relative to day mode baseline. green indicates improvement, Red indicates degradation.

Task Category	Large Scale	Intermediate	Small Scale
	(1:50k, 1:70k)	(1:100k, 1:130k)	(1:200k, 1:300k)
Perception Tasks (Accuracy % $\uparrow$ )
Point Features	42.24	47.72 ( +5.48 )	38.19 ( -4.05 )
Linestring Features	28.83	20.23 ( -8.60 )	20.26 ( -8.57 )
Polygon Features	28.93	26.73 ( -2.20 )	25.37 ( -3.56 )
Decision-Making Tasks (Accuracy % $\uparrow$ )

Table 6: Average model performance across scale levels. Delta shows change relative to large scale baseline. green indicates improvement, Red indicates degradation.

Figure 3: Error distribution of Gemini-2.5-Pro’s incorrect results and example errors in its responses.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Domain Contrast: Consumer Mapping vs. Professional Hydrography. Comparison of the same geographic region across (a) Google Maps Standard View, (b) Google Maps Satellite Imagery, and (c) NOAA Electronic Navigational Chart (ENC). While consumer maps focus on road networks and vague water representations, ENCs illustrate a complex, vector-based hydrographic reality. Note how the ENC explicitly renders critical depth contours, shipping channels, and navigational aids absent in consumer views, prioritizing safety-critical information over visual realism.

Figure 5: Standardized IHO S-57 Symbology Types. ENC features are classified into three geometric primitives with rigorous semantic definitions. (a) Point Features: Complex composite symbols where shape and topmark denote function (e.g., Beacon tower , Non-dangerous wreck ). (b) Linestring Features: Define boundaries with directionality constraints, such as Pipelines or Traffic separation lines . (c) Polygon Features: Denote regional regulations, such as Anchoring prohibited areas or Incompletely surveyed areas . Recognizing these symbols requires distinguishing fine-grained visual details that carry heavy semantic weight.

Figure 6: Operational Lighting Modes. Demonstration of the ECDIS standardized color palettes. (a) Day Mode: High contrast with white/blue backgrounds. (b) Dusk Mode: Reduced glare with grey backgrounds. (c) Night Mode: Black background with non-linear color shifts. Note how the blue shallow water in Day mode transforms into dark grey/black tones in Night mode, and text labels shift colors to maintain visibility. This drastic “palette swapping” challenges MLLMs reliant on natural image color statistics.

Figure 7: Scale-Dependent Rendering and Cartographic Generalization. The same chart region rendered at three standard scale levels. (a) Large Scale (1:50k): Maximum detail with dense soundings (e.g., depth values like 15 or 16 feet) and all navigational aids visible. (b) Intermediate Scale (1:100k): Partial generalization where minor soundings are suppressed to reduce clutter. (c) Small Scale (1:200k): High-level abstraction where only critical features remain, and dense depth numbers are removed. Models must handle this dynamic appearance where features logically “exist” but visually vanish based on zoom level.

Task	Prompt Template
L-1 Perception
Symbol Recognition	The image displays a symbol used to represent ENC (Electronic Navigational Chart) data on an ECDIS. According to the description provided, this symbol specifically portrays what type of object or feature? Please choose the correct answer from the following options: {options} Please respond with only the letter (A, B, C, or D) of the correct answer.
Point Feature	What is the nautical symbol located above point labeled “ {label} ”? Please choose the correct answer from the following options: {options} Please respond with only the letter (A, B, C, or D) of the correct answer.
Linestring Feature	What does the linestring between point {label_start} and point {label_end} represent? Please choose the correct answer from the following options: {options} Please respond with only the letter (A, B, C, or D) of the correct answer.
Polygon Feature	What is the most likely meaning of the area {bbox_label} ? Please choose the correct answer from the following options: {options} Please respond with only the letter (A, B, C, or D) of the correct answer.
L-2 Spatial Reasoning

Appendix

Table 7: Task-Specific Prompt Templates. We provide distinct prompts for each task. Note that for Coordinate Localization, we evaluate both Geographic (Latitude/Longitude) and Pixel-space prediction using separate prompts.

Model	Acc@5%	Acc@10%	Acc@15%	Acc@20%
Gemini-2.5-Flash	6.92	11.76	17.99	25.93
Gemini-2.5-Pro	8.76	12.99	19.64	25.67
Qwen3-VL-235B-Thinking	6.10	10.50	15.20	20.33
GPT-4o	3.95	8.33	13.16	19.29
Qwen3-VL-235B-Instruct	5.95	9.52	13.69	16.71
InternVL-3-38B	3.50	6.80	10.20	13.09

Appendix

Table 8: Fine-Grained Distance Measurement Accuracy (Acc@T). Evaluation of model performance across varying relative error tolerances. Bold indicates the best performance for each threshold.

Model	Acc@5 ∘	Acc@10 ∘	Acc@20 ∘	Acc@30 ∘
Qwen3-VL-235B-Thinking	16.20	31.50	55.64	70.20
Gemini-2.5-Pro	13.35	27.95	46.86	62.42
Qwen3-VL-32B-Thinking	11.50	24.80	44.10	59.50
InternVL-3-38B	10.80	22.50	41.09	56.80
Qwen3-VL-235B-Instruct	9.72	19.75	36.95	55.17
GLM-4.5V	5.52	12.99	28.60	52.60

Appendix

Table 9: Fine-Grained Bearing Calculation Accuracy (Acc@T). Evaluation of angular precision with error thresholds of

5^{\circ},10^{\circ},20^{\circ},

and

30^{\circ}

. Bold indicates the best performance for each threshold.

	Acc@50px		Acc@100px		Acc@200px		Acc@500px
Model	Geo	Pix	Geo	Pix	Geo	Pix	Geo	Pix
Gemini-2.5-Pro	0.00	2.07	8.70	8.28	17.36	21.43	56.52	50.34
Gemini-2.5-Flash	0.76	1.45	3.05	5.80	16.77	21.03	58.02	56.52
InternVL-3-38B	0.50	0.80	2.50	2.50	13.19	10.52	48.20	35.40
GPT-4o	0.00	1.74	1.22	2.61	12.20	6.94	50.00	43.48
Qwen3-VL-235B-Instruct	4.29	2.04	4.29	2.04	11.31	12.20	45.71	38.78

Appendix

Table 10: Comparative Analysis of Coordinate Localization Modalities (Geo vs. Pixel). Accuracy at varying pixel error thresholds for Geographic (Geo) and Pixel-based (Pix) localization. Geo requires symbolic grid interpretation; Pix requires only visual localization. Bold indicates the best performance for each column.

Figure	L-1/2/3 Task	Specific Sub-task	Gemini-2.5-Pro	GPT-4o	Qwen3-VL
Fig. 8	L-1 Perception	Symbol Recognition	Reasoning Error	Visual Perception Error	Visual Perception Error
Fig. 9	L-1 Perception	Symbol Recognition (Cardinal)	Knowledge Error	Visual Perception Error	Knowledge Error
Fig. 10	L-1 Perception	Symbol Recognition (Area)	Knowledge Error	Knowledge Error	Knowledge Error
Fig. 11	L-1 Perception	Point Feature (Day)	Visual Perception Error	Visual Perception Error	Knowledge Error
Fig. 12	L-1 Perception	Point Feature (Dusk)	Visual Perception Error	Visual Perception Error	Correct
Fig. 13	L-1 Perception	Point Feature (Night)	Correct	Visual Perception Error	Correct

Appendix

Table 11: Index of case study figures by sub-tasks (L-1 to L-3) with associated error categories for representative MLLMs. Errors are color-coded: Visual Perception Error , Reasoning Error , Knowledge Error , Calculation Error , and Instruction Following Error . Correct indicates a successful prediction.

Figure 8: A sample case of Symbol Recognition . The model must identify an ”Obscured Light” symbol.

Figure 9: A sample case of Symbol Recognition . The model must distinguish between East and North cardinal marks.

Figure 10: A sample case of Symbol Recognition . Identifying a symbolized Anchorage Area.

Figure 11: A sample case of Point Feature Understanding (Day Mode). Identifying a Caution Area symbol.

Figure 12: A sample case of Point Feature Understanding (Dusk Mode).

Figure 13: A sample case of Point Feature Understanding (Night Mode).

Figure 14: A sample case of Point Feature Understanding (Small Scale 1:200k).

Figure 15: A sample case of Linestring Feature Understanding (Day Mode). Identifying a Submarine Pipeline.

Figure 16: A sample case of Linestring Feature Understanding (Dusk Mode).

Figure 17: A sample case of Linestring Feature Understanding (Night Mode).

Figure 18: A sample case of Linestring Feature Understanding (Small Scale 1:300k). Identifying a Traffic Separation Scheme boundary.

Figure 19: A sample case of Polygon Feature Understanding (Day Mode). Identifying a Dumping Ground.

Figure 20: A sample case of Polygon Feature Understanding (Dusk Mode).

Figure 21: A sample case of Polygon Feature Understanding (Night Mode).

Figure 22: A sample case of Polygon Feature Understanding (Small Scale 1:300k).

Figure 23: A sample case of Coordinate Localization (Day Mode). Comparison of Geographic vs. Pixel localization errors.

Figure 24: A sample case of Coordinate Localization (Dusk Mode).

Figure 25: A sample case of Coordinate Localization (Night Mode).

Figure 26: A sample case of Bearing Calculation (Day Mode).

Figure 27: A sample case of Bearing Calculation (Dusk Mode).

Figure 28: A sample case of Bearing Calculation (Night Mode).

Figure 29: A sample case of Distance Measurement (Day Mode).

Figure 30: A sample case of Distance Measurement (Dusk Mode).

Figure 31: A sample case of Distance Measurement (Night Mode).

Figure 32: A sample case of Track Direction Recognition (Large Scale). Determining legal traffic flow direction.

Figure 33: A sample case of Track Direction Recognition (Small Scale).

Figure 34: A sample case of Safety Passage Assessment . Evaluating depth constraints for a vessel draft.

Figure 35: A sample case of Safety Passage Assessment (Small Scale).

Figure 36: A sample case of Anchorage Selection . Choosing the nearest valid anchorage in an emergency.

Figure 37: A sample case of Anchorage Selection (Small Scale).

Multimodal Spatial Reasoning

See all 44 papers

Electronic Navigational Chart Change Classification
Aug 20, 2026Jacob Arndt, Abhishek Potnis, Alexandre SorokineGeospatial Representation Learning Maritime Navigation
Oak Ridge National Laboratory Oak Ridge, Tennessee, USA
MARINER: A 3E-Driven Benchmark for Fine-Grained Perception and Complex Reasoning in Open-Water Environments
Apr 9, 2026Xingming Liao, Ning Chen, Muying Shu +5VLM Evaluation Visual Question Answering
Guangdong University of Technology, Guangzhou, China
Exploring LLM Capabilities for Situational Understanding and COLREG compliance on real-world maritime navigation scenarios
Aug 8, 2026Julius Wirbel, P. Nicholas Hansen, Line K. H. Clemmensen +1LLM Evaluation Maritime Navigation
Technical University of Denmark, Kongens Lyngby, Denmark · University of Copenhagen, Copenhagen, Denmark

Aug 20, 2026J/K move · Enter open · S save

Electronic Navigational Chart Change Classification

Jacob Arndt, Abhishek Potnis, Alexandre Sorokine

Oak Ridge National Laboratory Oak Ridge, Tennessee, USA

Geospatial Representation Learning Maritime Navigation

Open paper AskPDF arXiv

Electronic Navigational Charts (ENCs) are geospatial vector datasets used in maritime navigation systems that represent hydrographic and navigational information such as depths, navigational aids, traffic schemes, and hazards. A major challenge for hydrographic offices is determining whether a given chart change poses a critical or non-critical risk to maritime safety. Existing workflows rely heavily on manual review and verification, which is labor-intensive, scales poorly with the volume of incoming chart updates, and introduces inter-analyst inconsistencies. To address this challenge, we propose a method for automated classification of ENC changes. We establish a baseline encoding scheme to translate complex vector data changes into a structured tabular format for classification models. The two crucial components of the encoding scheme include a spatial context encoder to enrich the change representations with surrounding geographic features, and an ENC attribute encoder to represent nuanced attribute-value descriptions of the modified objects. We evaluate the proposed approach across two distinct operational datasets, comprising 1,308 chart pairs containing over 100,000 individual chart modifications. Tuned gradient-boosted trees leveraging the proposed encoding schemes achieve accuracies of 90% and 94% on the two datasets, yielding a 5-7% improvement over default hyperparameterized models trained on encodings without spatial context and attribute embeddings. These results demonstrate the viability of integrating machine learning into operational geospatial pipelines to improve ENC maintenance and enhance maritime safety. Finally, our experiments demonstrate the effectiveness of simple location and spatial aggregation methods, providing a foundation for evaluating more sophisticated spatial representation learning techniques for this application.

Abstract

Multimodal Spatial Reasoning

Explore similar work

Electronic Navigational Chart Change Classification

MARINER: A 3E-Driven Benchmark for Fine-Grained Perception and Complex Reasoning in Open-Water Environments

Exploring LLM Capabilities for Situational Understanding and COLREG compliance on real-world maritime navigation scenarios