SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning Traces
Organizations: Department of Artificial Intelligence Kathmandu University Dhulikhel, Nepal · Department of Computer Science and Engineering Kathmandu University Dhulikhel, Nepal
Abstract
Computer science papers rely heavily on diagrams: architecture drawings, system flowcharts, and pipeline schematics that often carry more information than the text around them. There is currently no public dataset that pairs this specific kind of figure with captions, context, questions, answers, and step-by-step reasoning, which is exactly what is needed to train a vision-language model to understand them. We present SCAFFOLD\footnote{https://github.com/theranjitraut/scaffold}, a large-scale structured dataset of computer science research figures with diagram QA and Chain-of-Thought reasoning traces. This dataset consists of (image, caption, context, question-answer, chain-of-thought) tuples from arXiv computer science papers prepared using layout detection and PDF parsing, with an AI-assisted question-generation step. The resulting large-sized SCAFFOLD-157K dataset spans 3,058 papers with 29,887 figures (157,387 pairs), a medium-sized SCAFFOLD-37K dataset (36,797 pairs), and a small-sized SCAFFOLD-12K dataset (12,000 pairs). We used SCAFFOLD-12K for baseline experiments on Qwen2.5-VL-3B-Instruct.
Figures & tables
| Field | Type | Description |
|---|---|---|
| figure_id | str | Unique identifier for the record, e.g. facenet_p03_fig00 |
| paper_id | str | Source paper identifier, e.g. facenet |
| figure_number | str | Figure number as parsed from the caption text |
| caption | str | Full caption text extracted from the detected caption region |
| ref_sentence | str | First sentence in the paper body that mentions this figure |
| ref_sentence_found | bool | True if a real referencing sentence was found, False otherwise |
| Field | Example value |
|---|---|
| figure_id | attn_p03_fig03 |
| paper_id | attention-is-all-you-need |
| figure_number | 3 |
| caption | "The proposed cross-attention module. Keys and values are derived from the encoder output; queries from the decoder hidden state." |
| ref_sentence | The first body sentence found to cite Figure 3, describing the module’s role in aligning encoder and decoder representations |
| ref_sentence_found | True |
| Dataset version | Train | Validation | Total |
|---|---|---|---|
| Scaffold-157K | 120,225 | 36,797 | 157,387 |
| Scaffold-37K | 29,397 | 7,400 | 36,797 |
| Scaffold-12K (used for baseline training) | 10,000 | 2,000 | 12,000 |
| Question type | What it asks |
|---|---|
| component | Identifies or describes a specific part of the diagram |
| relationship | Asks how two parts of the diagram connect or interact |
| process | Asks about the order or flow of steps shown |
| result | Asks about outcomes shown in a results-style figure (e.g., a plot or table) |
| comparison | Asks the reader to compare two elements within the figure |
| architecture | Asks about the overall structure or design shown |
| Source value | Meaning |
|---|---|
| qa_source = "gemini" | Question and answer generated by the Gemini API, conditioned on the figure image, caption, and context |
| qa_source = "synthetic" | Question and answer generated by the deterministic template fallback, used when Gemini generation is unavailable or fails |
| cot_source = "gemini" | Reasoning trace generated by Gemini alongside the question/answer |
| cot_source = "synthetic" | Reasoning trace generated by the template fallback |
| cot_source = "" (empty) | No reasoning trace was produced for this record ( has_cot = False ) |
| Metric | Score |
|---|---|
| BLEU-4 (full output) | 0.237 |
| ROUGE-L (full output) | 0.448 |
| Token-F1 (full output) | 0.527 |
| Answer-only ROUGE-L | 0.419 |
| Answer-only Token-F1 | 0.433 |
| Numeric accuracy | 0.638 |
| Category | Metric | Value |
|---|---|---|
| ChartQA | ||
| relaxed accuracy (overall) | 0.455 | |
| relaxed accuracy numeric | 0.500 | |
| relaxed accuracy categorical | 0.270 | |
| answer containment | 0.670 | |
| numeric accuracy | 0.503 | |
Document Image Understanding
Document image understanding turns page images into recognized text, layout, structured data, and answers grounded in visual evidence.
See all 148 papers