cs.CVSep 9, 2026
SaveVectorHarness: Recovering Editable, Relation-Preserving Structure from Scientific Graphics
Organizations: CSU-JPG, Central South University · National University of Singapore
Abstract
Converting scientific graphics into editable representations remains a challenging problem for image-to-code generation because of their heterogeneous elements and complex layouts. Recent multi-agent reconstruction systems have advanced this line of work, but often follow a copy-paste paradigm: the reconstructed image closely resembles the original, while complex regions remain effectively uneditable. We instead formulate a different objective, raster-to-authoring reconstruction, which aims to recover an authoring representation that supports native, customized editing rather than mere visual replication. To this end, we present VectorHarness, a multi-agent framework for raster-to-authoring reconstruction that recovers heterogeneous components using type-appropriate native representations. Text, formulas, shapes, connectors, icons, charts, and tables are reconstructed as natively editable objects, while intrinsically image-based regions remain raster content. To systematically evaluate reconstruction quality, we introduce VectorHarness-Bench, which jointly assesses rendering fidelity, raster fallback coverage, executable object edits, and relation-preserving edits. Experiments show that VectorHarness improves executable edit success and relation preservation, reduces avoidable raster fallback, and maintains high visual fidelity across heterogeneous graphics.
Figures & tables
Figure 1: Copy-Paste vs. Native Editability . Existing systems [ 1 ] may preserve complex regions using raster patches or non-native primitives, producing visually similar outputs that do not support the intended editing operations. In contrast, VectorHarness reconstructs semantic components as native authoring objects, enabling localized edits to text styles, shape geometry, colors, and connectors while preserving overall visual fidelity.
| Method | AP | Diag. | DP | Tab. | Chart | Raster | Fmt. |
| AutoFigure-Edit | – | – | – | Yes | SVG | ||
| Edit-Banana | – | – | – | Yes | XML | ||
| CraftEditor | – | – | Yes | SVG | |||
| VectorHarness | ✓ | ✓ | ✓ | ✓ | ✓ | Intrinsic | SVG/XML/PPTX |
Table 1: Comparison of supported graphic categories, native recovery, raster policy, and export formats.
Figure 2: Overview of VectorHarness. Given a flattened raster graphic, VectorHarness first analyzes its global layout and routes the input to an appropriate reconstruction branch. Specialized parsing and recovery agents then reconstruct text, formulas, shapes, connectors, icons, charts, tables, and raster regions using type-appropriate representations. The recovered objects and their grouping, containment, association, alignment, hierarchy, and attachment relations are assembled into a shared Semantic Authoring Representation (SAR). Stage-local verification checks object validity, relation consistency, and rendering fidelity, and routes localized errors back for targeted repair. Finally, deterministic format-specific compilers translate the verified SAR into layered SVG, diagrams.net-compatible XML, and editable PPTX outputs.
Figure 3: Benchmark composition and human-rated difficulty. Left: Category distribution of 250 graphics across five categories. Right: Category-wise distribution of difficulty scores from 1 (easiest) to 5 (hardest).
Figure 4: Qualitative comparison on complex diagrams and academic posters. Existing methods may leave text, formulas, arrows, or shapes as raster patches or fragmented primitives, limiting native editing operations. In contrast, VectorHarness reconstructs recoverable visual components as native editable objects, enabling localized edits while maintaining high visual fidelity.
| Method | Fidelity | Automatic Editability | Human Evaluation | ||||||
| LPIPS | VLM-Fid. | Text-CER | RFC | Edit-SR | Rel-SR | Fid. | Edit-SR | Ease | |
| Edit-Banana | 0.61 | 0.63 | 0.30 | 0.45 | 0.60 | 0.53 | 0.55 | 0.25 | 0.35 |
| AutoFigure-Edit | 0.56 | 0.67 | 0.21 | 0.37 | 0.69 | 0.42 | 0.51 | 0.36 | 0.42 |
| CraftEditor | 0.48 | 0.71 | 0.06 | 0.21 | 0.72 | 0.64 | 0.63 | 0.67 | 0.58 |
| Direct MLLM | 0.85 | 0.26 | 0.62 | 0.50 | 0.41 | 0.24 | 0.15 | 0.18 | 0.14 |
| VectorHarness | 0.41 | 0.76 | 0.12 | 0.13 | 0.79 | 0.88 | 0.72 | 0.76 | 0.66 |
Table 2: Overall comparison and component ablation on VectorHarness-Bench. LPIPS, VLM-Fid., and Text-CER assess perceptual, holistic, and textual fidelity on standardized renderings. RFC measures raster fallback on the vector-dominant subset. Auto Edit-SR evaluates predefined object-level edits, while Rel-SR tests whether attachment, grouping, and containment are preserved after editing. Editability metrics use each method’s native outputs; multi-format scores are averaged across supported formats without per-sample selection. Human evaluation reports visual fidelity, editing success, and ease.
Figure 5: Qualitative reconstruction results for scientific diagrams. The outputs are opened in Inkscape, where blue selection outlines indicate independently editable objects and are not part of the rendered output. VectorHarness accurately reconstructs diagrams with complex nested layouts and conventional multi-box, multi-arrow structures while maintaining high visual fidelity.
Figure 6: Qualitative reconstruction results for academic posters. VectorHarness decomposes the complete poster into independently editable text, diagrams, backgrounds, and graphical elements while retaining intrinsically image-based regions as separate raster objects. Blue selection outlines in Inkscape indicate the recovered editable components.
Figure 7: Qualitative reconstruction results for design posters. VectorHarness separates typography, backgrounds, decorative graphics, and raster image regions into distinct layers, reconstructing recoverable elements as native editable objects and preserving genuine raster content as independent image assets.
Figure 8: Qualitative reconstruction results for charts. In additio n to editable SVG, diagrams.net-compatible XML, and PPTX artifacts, VectorHarness recovers Python visualization code when applicable, supporting both object-level modification and data- or program-level editing.
Figure 9: Qualitative reconstruction results for tables. VectorHarness recovers cell content, row–column organization, borders, fills, typography, and alignment as structured editable content, enabling direct modification of both table values and visual formatting.
| Agent | Primary Role | Structured Output |
| Graphic router | Select a category-specialized branch | Category, confidence, and routing evidence |
| Parsing agent | Identify candidates and aggregate multi-source evidence | Regions, provisional types, geometry, evidence, and relation proposals |
| Recovery agents | Select representations and recover editable payloads | SAR object records and recovery confidence |
| Relation assembler | Instantiate and resolve cross-object relations | Typed relation records, ordering information, and conflict-resolution log |
| Stage-local verifier | Localize object, relation, and rendering errors | Responsible stage, affected IDs, error type, evidence, and confidence |
| Format compiler | Serialize verified SAR deterministically | Native artifact and export/approximation log |
Table 3: Agent roles and structured interfaces in VectorHarness.
| SAR Content | Layered SVG | diagrams.net XML | PPTX |
| Text | Editable text elements with typography | Editable text cells with style attributes | Editable text frames and runs |
| Formula | Editable formula group with symbolic payload | Editable formula group with symbolic payload | Editable equation or structured formula object |
| Shape | Independently editable primitives or vector paths | Independently styled vertex geometries | Editable AutoShapes or freeform shapes |
| Icon | Editable primitive/group; otherwise masked raster icon | Editable grouped cells; otherwise background-removed image cell | Editable shape group; otherwise background-removed picture |
| Connector | Native editable connector with endpoint and attachment metadata | Native edge with source/target references | Native connector with endpoint attachments |
| Group | Group element with shared transform | Parent–child cell structure | Native group shape |
Table 4: Format-specific mappings from SAR to native or structured editable authoring objects. Chart exports preserve the recovered data and visualization specification, while icon rasterization is used only when faithful editable reconstruction cannot be achieved.
| Component | Role | Implementation / Version | Key Configuration |
| Graphic router | Five-category branch selection | GPT-5.5 (API, July 2026 snapshot) | Prompt v1.0; temperature 0; JSON output; deterministic top-1 routing. |
| Backbone MLLM | Parsing, representation selection, and recovery | GPT-5.5 (API, July 2026 snapshot) | Longest image side 2048 px; max 16,384 output tokens; temperature 0.2; max 3 retries. |
| Text/OCR recovery | Text detection, recognition, and typography recovery | PaddleOCR 3.7.0 with PP-OCRv5 server models | English/Chinese recognition; confidence threshold 0.60; orientation and unwarping disabled. |
| Formula recovery | Iterative editable symbolic reconstruction | GPT-5.5 with a custom formula-validation module | LaTeX output; max 5 attempts; syntax parsing and render validation; no raster fallback. |
| Shape recovery | Decomposition into independently editable shapes | GPT-5.5 with a custom Python/SVG geometry module | Native primitives preferred; vector paths for irregular shapes; 2-px geometry tolerance; max 3 retries. |
| Icon recovery | Editable icon reconstruction or isolated raster preservation | GPT-image-2 with GPT-5.5 visual verification | Max 3 reconstruction attempts; acceptance score ; otherwise background-removed raster icon. |
Table 5: Implementation details of VectorHarness. All model snapshots, prompts, decoding parameters, software versions, and verification thresholds are fixed across benchmark samples.
| Category | # Samples | Typical Components |
| Scientific diagrams | 100 | Text, formula, shape, connector |
| Charts | 25 | Axes, marks, legends, labels |
| Tables | 25 | Cells, headers, spans, borders |
| Academic posters | 50 | Sections, figures, tables, formulas |
| Design posters | 50 | Typography, icons, decorative shapes |
| Total | 250 | – |
Table 6: Category distribution of VectorHarness-Bench.
| Method | Version | Native Outputs | Inference Configuration | # Runs |
| Edit-Banana | official repository, July 2026 checkout; GPT-5.5 (July 2026 API snapshot) | diagrams.net XML | official reconstruction pipeline; GPT-5.5 for multimodal reasoning; recommended preprocessing; maximum 3 retries | 3 |
| AutoFigure-Edit | v1.1; GPT-5.5 (July 2026 API snapshot) | SVG | official editable-SVG pipeline; GPT-5.5 for all MLLM calls; official prompt and decoding configuration; maximum 3 retries | 3 |
| CraftEditor | official repository, July 2026 checkout; GPT-5.5 (July 2026 API snapshot) | SVG | official raster-to-SVG pipeline; GPT-5.5 for all VLM calls; default SAM3 grounding configuration; maximum 3 retries | 3 |
| Direct MLLM | GPT-5.5 (July 2026 API snapshot) | SVG | fixed direct-generation prompt; temperature 0; maximum 16,384 output tokens; no iterative repair | 3 |
| VectorHarness | July 2026 implementation; GPT-5.5 and GPT-image-2 | SVG; diagrams.net XML; PPTX | fixed prompts and schemas; representation-aware recovery; ; fixed verification thresholds; maximum 3 retries per stage | 3 |
| Potrace | v1.16 | SVG | official default tracing configuration; SVG backend; no semantic post-processing | 1 |
Table 7: Baseline versions and inference configurations. All MLLM-based methods use the same GPT-5.5 July 2026 API snapshot and are evaluated over three independent runs. Potrace is deterministic and is evaluated once. No per-sample best-run selection is performed.
| Operation | Applicable Target | Modification | Required Native Capability |
| Text replacement | detected text | replace with length-matched text | editable text content |
| Formula editing | detected formula | replace one symbol or number | symbolic or semantic formula payload |
| Fill modification | filled shape or region | change fill to fixed contrast color | editable fill attribute |
| Stroke modification | vector shape or line | change width or style | editable stroke attribute |
| Object resizing | shape, icon, or panel | scale by a fixed factor | editable geometry |
| Object translation | shape, icon, or panel | translate by fixed offset | independent positioning |
Table 8: Executable object-level editing operations.
| Method | Attachment | Grouping | Containment | Rel-SR |
| Edit-Banana | 0.61 | 0.53 | 0.45 | 0.53 |
| AutoFigure-Edit | 0.48 | 0.42 | 0.36 | 0.42 |
| CraftEditor | 0.72 | 0.65 | 0.55 | 0.64 |
| Direct MLLM | 0.29 | 0.24 | 0.19 | 0.24 |
| VectorHarness | 0.94 | 0.89 | 0.81 | 0.88 |
Table 9: Relation-preserving edit success by relation type. Attachment measures whether connector endpoints remain attached after object transformations; Grouping measures whether group membership and relative transforms are preserved; Containment measures whether contained objects remain spatially valid within their containers. Rel-SR denotes the macro-average over the three relation types.
| Configuration | Attachment | Grouping | Containment | Rel-SR |
| Full VectorHarness | 0.94 | 0.89 | 0.81 | 0.88 |
| w/o Representation Selection | 0.82 | 0.73 | 0.64 | 0.73 |
| w/o Stage-Local Verification | 0.72 | 0.64 | 0.56 | 0.64 |
Table 10: Relation-wise ablation results. Removing representation selection reduces the availability of operation-compatible objects, while removing stage-local verification leaves attachment, grouping, and containment inconsistencies unresolved.
| Method | Edit-SR | Ease | Time |
| AutoFigure-Edit | 0.36 | 0.41 | 0.34 |
| CraftEditor | 0.45 | 0.52 | 0.67 |
| Direct MLLM | 0.17 | 0.06 | 0.93 |
| VectorHarness-SVG | 0.57 | 0.64 | 0.26 |
Table 11: Controlled human editing results in a common SVG environment.
| Format | Export Valid | VLM-Fid. | Edit-SR | Rel-SR | Median Size (MB) |
| Layered SVG | 0.99 | 0.78 | 0.78 | 0.84 | 1.67 |
| diagrams.net XML | 0.98 | 0.74 | 0.81 | 0.93 | 1.43 |
| Editable PPTX | 0.97 | 0.76 | 0.78 | 0.87 | 2.67 |
Table 12: Validity, fidelity, editability, and relation preservation across VectorHarness output formats. Export Valid denotes the fraction of artifacts that can be successfully opened and rendered. VLM-Fid., Edit-SR, and Rel-SR are evaluated independently for each format using the same protocol as the main evaluation. Median Size denotes the median file size of valid artifacts. The mean scores across SVG, diagrams.net XML, and PPTX correspond to the multi-format VectorHarness results reported in Table 2 .
| Input Category | # Samples | Avg. Input Tokens | Avg. Output Tokens | Avg. Total Tokens | Aggregate Tokens | Avg. API Cost |
| Scientific diagrams | 100 | 56,468 | 20,229 | 76,697 | 7,669,700 | $2.50 |
| Academic posters | 50 | 205,000 | 72,000 | 277,000 | 13,850,000 | $9.00 |
| Design posters | 50 | 160,000 | 55,000 | 215,000 | 10,750,000 | $6.50 |
| Tables | 25 | 14,800 | 5,200 | 20,000 | 500,000 | $0.50 |
| Charts | 25 | 12,500 | 4,800 | 17,300 | 432,500 | $0.50 |
| Overall | 250 | 98,317 | 34,492 | 132,809 | 33,202,200 | $4.20 |
Table 13: Token consumption and estimated API cost of VectorHarness across the five input categories. Per-sample token counts include all recorded GPT-5.5 calls during routing, parsing, representation-aware recovery, relation assembly, and stage-local verification. API cost includes both GPT-5.5 and GPT-image-2 calls. Because GPT-image-2 usage is not represented using the same text-token accounting, API cost is not strictly proportional to the reported token counts. The overall values are weighted by the number of samples in each category.
Figure 10: Qualitative reconstruction results for scientific diagrams. The outputs are opened in Inkscape, where blue selection outlines indicate independently editable objects and are not part of the rendered output. VectorHarness accurately reconstructs diagrams with complex nested layouts and conventional multi-box, multi-arrow structures while maintaining high visual fidelity.