ChartGalaxy++: A Richly Annotated Dataset for Chart Understanding and Generation
Organizations: Tsinghua University · Canva
Abstract
Infographic charts communicate data through cohesive compositions of visual elements such as charts, text, and images. Existing infographic chart datasets primarily annotate individual elements with bounding boxes and category labels. Such element-level annotations do not capture hierarchical and spatial relationships, limiting their use for understanding how elements are organized and related within an infographic and for generating well-organized designs. To address this limitation, we introduce ChartGalaxy++, a richly annotated dataset of 217,195 infographic charts. At the core of this dataset is a scene graph representation, in which nodes represent visual elements and semantic groups, and edges encode their hierarchical and spatial relationships. For real infographics, we construct scene graphs by detecting and progressively grouping visual elements; for synthetic ones, we derive them directly from the generation process. We demonstrate the utility of ChartGalaxy++ through three applications: 1) image-to-scene-graph prediction, 2) scene-graph-augmented infographic question answering, and 3) evaluation of scene graph preservation in image generation. Together, these applications highlight the value of ChartGalaxy++ for advancing chart understanding and generation.
Figures & tables
| Node-level | Edge-level | |||||||
| Model | Overall | Type | Box | Text | Color | Description | Hierarchical | Spatial |
| Molmo2-8B | 8.7 | 67.3 | 32.0 | 43.2 | 65.0 | 34.9 | 4.0 | 0.9 |
| InternVL3.5-8B | 27.7 | 79.6 | 34.6 | 75.1 | 68.7 | 36.2 | 9.8 | 10.0 |
| Gemma 4-12B-IT | 45.8 | 94.1 | 51.1 | 91.8 | 77.5 | 53.7 | 18.1 | 29.5 |
| Qwen3.5-4B | 27.8 | 90.5 | 25.3 | 86.4 | 80.1 | 51.3 | 9.0 | 10.6 |
| Qwen3.8-27B | 60.0 | 95.6 | 65.7 | 96.5 | 87.3 | 54.8 | 27.3 | 47.1 |
| Open-source models | Proprietary models | |||||||||
| Prompting | Molmo2 8B | InternVL3.5 8B | Gemma 4 12B-IT | Qwen3.5 4B | Qwen3.8 27B | Qwen3.8 Flash-Next FP8 | GLM-5.3 Flash | Gemini 3.8 Flash | GPT-5.6 Sol | GPT-6 Astra |
| Direct | 37.5 | 31.9 | 32.4 | 35.9 | 75.1 | 72.7 | 74.0 | 90.0 | 81.4 | 95.9 |
| Evidence-guided | 40.4 | 33.8 | 40.2 | 46.7 | 76.9 | 76.0 | 79.0 | 88.5 | 85.1 | 94.0 |
| Scene-graph-augmented | 46.7 | 45.3 | 67.4 | 53.4 | 86.9 | 86.1 | 83.9 | 90.6 | 91.6 | 97.6 |
| Annotations | Accuracy |
| No scene graph | 76.9 |
| + node types & + bounding boxes | 77.8 |
| + relationships | 78.9 |
| + labels | 80.2 |
| + text without digits | 81.5 |
| + text with digits (best) | 86.9 |
| Node-level | Edge-level | |||||||
| Model | Overall | Type | Box | Text | Color | Description | Hierarchical | Spatial |
| HiDream-O1-Image | 18.7 | 92.1 | 32.8 | 20.8 | 75.9 | 40.1 | 16.6 | 6.6 |
| GLM-Image | 35.9 | 83.7 | 25.0 | 19.1 | 74.3 | 35.9 | 20.8 | 14.0 |
| Qwen-Image-2512 | 40.1 | 82.5 | 28.2 | 15.9 | 77.8 | 36.4 | 28.0 | 19.1 |
| Qwen-Image-SP | 44.9 | 85.9 | 31.7 | 24.9 | 78.7 | 38.1 | 35.2 | 25.3 |
| ERNIE-Image-Turbo | 45.3 | 89.6 | 30.3 | 34.2 | 78.6 | 45.1 | 36.9 | 23.2 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Field | Applies to | Definition |
| label | All | Unique node identifier within the image, combining its type and one-based position in the node array, e.g., <text 4> . |
| parent | All | Label of the direct parent group, or the reserved token ROOT . |
| node_kind | All | element or group . |
| element_type | Element | text , image , or shape . |
| group_type | Group | One of the seven semantic group types. |
| bbox | All | Four integers in the range . |
| Group type | Meaning |
| chart | A chart component containing data marks or data items and their associated reading references. |
| axis | An axis and its associated components, such as ticks, tick labels, and an axis title. |
| legend | A legend containing its entries and any associated title. |
| legend_item | One legend entry associating a visual key with its label. |
| data_item | Elements jointly representing one observation, such as a mark with its category label, value, or identifying icon. |
| series | Elements representing one data series, including its marks and associated labels. |
| Stage | Input | Output |
| Element annotation | Image, crops, and localized candidates | Leaf nodes with type, box, palette, description, and text where applicable. |
| Hierarchical relationship extraction | Image and annotated elements | Semantic groups and parent–child assignments. |
| Spatial relationship extraction | Element and group boxes | Direction, alignment, overlap, and proximity relationships. |
| Iterative refinement | Image, current graph, and validation findings | Corrected annotations, rechecked graph, and remaining unresolved findings. |
| Component | Setting |
| HunyuanOCR-1.5 | spotting_json ; temperature 0, top- 1, repetition penalty 1.08; at most 32,768 output tokens. |
| DAD | Fixed layer-localization instruction; temperature 0, top- 0.9, top- 20; deployed output limit 16,384 tokens. |
| Image and crop inputs | Up to ownership cores; grid size depends on image dimensions and candidate count, with a 900-pixel size threshold and 80-pixel context padding. |
| Element attributes / scene description | Gemini 3.8 Flash; at most 64 candidates per attribute request; LOW thinking; at most 32,768 output tokens. |
| Localization recovery / crop correction | Gemini 3.8 Flash; LOW thinking; at most 32,768 output tokens. |
| Whole-image review | Gemini 3.8 Flash; MEDIUM thinking; at most 32,768 output tokens; at most two audit rounds in the recorded extraction configuration. |
| Complexity | Images |
| Low ( nodes) | 191 |
| Medium (51–91 nodes) | 344 |
| High ( nodes) | 465 |
| Total | 1000 |
| Measure | Result |
| Non-root nodes | 99,010 |
| Element nodes | 82,042 |
| Element precision | 99.81% |
| Element recall | 99.78% |
| Mean box IoU | 0.9976 |
| Text transcription errors | 134/34,242 |
| Name | URL | License |
| Statista (Daily Data) | statista.com | CC BY-ND 4.0 |
| Visual Capitalist | visualcapitalist.com | Customized |
| Our World in Data | ourworldindata.org | CC BY |
| OECD | oecd.org | CC BY 4.0 |
| Office for National Statistics | ons.gov.uk | OGL |
| Kaiser Family Foundation | kff.org | CC BY-NC-ND 4.0 |
| Node kind | Type | Count | Percentage |
| Element | text | 6,044,031 | 38.76% |
| image | 1,107,791 | 7.11% | |
| shape | 8,439,834 | 54.13% | |
| Group | chart | 221,006 | 7.16% |
| axis | 72,699 | 2.35% | |
| legend | 88,976 | 2.88% |
| Quantity | Min | P25 | Median | P75 | P90 | P99 | Max | Mean |
| Elements | 1 | 35 | 54 | 85 | 121 | 336 | 3,682 | 71.79 |
| Non-root groups | 0 | 7 | 10 | 18 | 31 | 57 | 234 | 14.22 |
| Total nodes | 2 | 44 | 67 | 108 | 151 | 344 | 3,725 | 87.01 |
| Hierarchy depth | 2 | 4 | 4 | 5 | 6 | 7 | 8 | 4.54 |
| Model (revision prefix) | Context | Output | Decoding |
| Molmo2-8B ( e28fa28 ) | 36,864 | 24,576 | Sampling; inventory first |
| InternVL3.5-8B ( 9bb6a56 ) | 32,768 | 24,576 | Temperature 0 |
| Gemma 4-12B-IT ( 707f0a3 ) | 65,536 | 24,576 | Temperature 0 |
| Qwen3.5-4B ( 851bf6e ) | 32,768 | 24,576 | Sampling; thinking off |
| Qwen3.8-27B ( 1d4bf0f ) | 65,536 | 24,576 | Temperature 0; thinking off |
| Qwen3.8-Flash-Next-FP8 ( 236dfdf ) | 65,536 | 24,576 | Temperature 0; thinking off |
| Model | Checkpoint | Sampling |
| HiDream-O1-Image | HiDream-ai/HiDream-O1-Image | 50 steps; CFG 5; shift 3; UniPC |
| GLM-Image | zai-org/GLM-Image | 50 steps; CFG 1.5 |
| Qwen-Image-2512 | Qwen/Qwen-Image-2512 | 50 steps; CFG 4 |
| Qwen-Image-SP | Qwen/Qwen-Image with heheyas/Qwen-Image-SP transformer and text-encoder overlays | 25 steps; CFG 4 |
| ERNIE-Image-Turbo | baidu/ERNIE-Image-Turbo | 8 steps; CFG 1; prompt enhancement disabled |
| SenseNova-U1 Infographic V3 | sensenova/SenseNova-U1-8B-MoT-Infographic-V3 | 50 steps; CFG 4; shift 3; thinking disabled |
| Model | Direction | Alignment | Overlap | Proximity |
| HiDream-O1-Image | 6.7 | 3.5 | 13.5 | 5.0 |
| GLM-Image | 16.0 | 3.6 | 19.7 | 7.5 |
| Qwen-Image-2512 | 21.9 | 5.2 | 25.8 | 11.1 |
| Qwen-Image-SP | 28.2 | 9.1 | 34.1 | 15.2 |
| ERNIE-Image-Turbo | 25.5 | 9.5 | 34.0 | 15.2 |
| SenseNova-U1 Infographic V3 | 28.2 | 11.1 | 36.3 | 16.8 |
Chart Understanding
Chart understanding enables AI systems to read visualizations, recover their data, answer questions, and reason about what charts show.
See all 63 papers