cs.CVAug 19, 2026

FRAGMENT: Factorized Graph Representations for Document Generation and Editing via Entity-Aware Transformations

Authors: Ayoub El Bouchtili, Guilhaume Leroy-Meline

Organizations: IBM Consulting

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Structured documents such as invoices, forms, reports, and scientific articles derive meaning from the interplay between spatial layout, textual content, and logical structure. Generative models operating at the pixel or token level often struggle to capture these dependencies effectively. We explore FRAGMENT, a generative framework that represents a document as a typed relational graph and factorizes its distribution as p(structure, content) = p(structure) * p(content | structure). The framework consists of two stages. The first stage, the Architect, is a causally masked Transformer conditioned on document category that autoregressively generates the graph topology and typed spatial relations. The second stage, the Builder, is a GATv2-based graph attention network that enriches the graph with normalized bounding boxes, text, and visual style attributes. Both stages define explicit likelihood models, yielding a tractable document-level likelihood that serves as an anomaly score for forgery detection. For controlled editing, a prompt-conditioned extension injects instruction embeddings into the Builder through cross-attention, enabling semantic and entity-aware modifications. We describe training on DocLayNet and fine-tuning on FUNSD and SROIE. Experiments on DocLayNet, FUNSD, and SROIE evaluate FRAGMENT alongside representative autoregressive, layout-only, and graph-based baselines, providing an empirical analysis of the characteristics and trade-offs of the proposed factorized graph generation framework.

Figures & tables

Figure 1: Fragment pipeline. The Architect autoregressively generates a typed document graph conditioned on a category prefix; the Builder contextualizes the scaffold with GATv2 message passing and realizes layout, style, and text through dedicated heads. Dashed paths show the two auxiliary capabilities: prompt-conditioned editing via cross-attention, and the composite log-likelihood used for forgery detection.
ModelDoc-EMD ↓\downarrowLayout FID ↓\downarrowBERTScore ↑\uparrowRender FID ↓\downarrowForgery AUROC ↑\uparrow
LayoutTransformer [ 7 ]0.14218.7n/an/an/a
LayoutDM [ 8 ]0.11811.2n/an/an/a
GNN layout generator [ 9 ]0.13116.4n/an/an/a
DocSynthv2 [ 17 ]0.10914.80.84231.6n/a
Fragment (GCN backbone)0.10414.10.83330.20.89
Fragment (no edge types)0.10113.60.83829.50.90
Table 1: Evaluation results on DocLayNet under the protocol of Section 8 . Values represent measured results from benchmark evaluations. Layout-only baselines are not applicable (n/a) to text, render, and forgery metrics.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Intermediate artifacts produced by the generation pipeline; (a) graph scaffold, (b) instantiated layout, and (c) rendered document.
Figure 3: Instruction-guided editing. A source document is modified according to a text instruction while preserving untouched content.
Figure 4: Likelihood-based forgery localization. A manipulated field produces an anomalous negative log-likelihood contribution, enabling localization of suspicious document regions.
More in this topic

Graph Generation

Graph generation builds models and algorithms that create graphs with desired structures, properties, or distributions.

See all 67 papers

Explore similar work

CardsList
Apr 27, 2026

Factual and Edit-Sensitive Graph-to-Sequence Generation via Graph-Aware Adaptive Noising

Fine-tuned autoregressive models for graph-to-sequence generation (G2S) often struggle with factual grounding and edit sensitivity. To tackle these issues, we propose a non-autoregressive diffusion framework that generates text by iterative refinement conditioned on an input graph, named as Diffusion Language Model for Graphs (DLM4G). By aligning graph components (entities/relations) with their corresponding sequence tokens, DLM4G employs an adaptive noising strategy. The proposed strategy uses per-token denoising error as a signal to adaptively modulate noise on entity and relation tokens, improving preservation of graph structure and enabling localized updates under graph edits. Evaluated on three datasets, DLM4G consistently outperforms competitive G2S diffusion baselines trained on identical splits across both surface-form and embedding-based metrics. DLM4G further exceeds fine-tuned autoregressive baselines up to 12x larger (e.g., T5-Large) and is competitive with zero-shot LLM transfer baselines up to 127x larger. Relative to the strongest fine-tuned PLM baseline, DLM4G improves factual grounding ([email protected]) by +5.16% and edit sensitivity (ESR) by +7.9%; compared to the best diffusion baseline, it yields gains of +3.75% in [email protected] and +23.6% in ESR. We additionally demonstrate applicability beyond textual graphs through experiments on molecule captioning, indicating the method's generality for scientific G2S generation.

Aditya Hemant Shahane, Anuj Kumar Sirohi, Tanmoy Chakraborty +2

Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India · Indian Institute of Science, Bengaluru, India

Diffusion Language Models
AskPDF
Apr 14, 2026

FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing

Unstructured model editing aims to update models with real-world text, yet existing methods often memorize text holistically without reliable fine-grained fact access. To address this, we propose FABLE, a hierarchical framework that decouples fine-grained fact injection from holistic text generation. FABLE follows a two-stage, fact-first strategy: discrete facts are anchored in shallow layers, followed by minimal updates to deeper layers to produce coherent text. This decoupling resolves the mismatch between holistic recall and fine-grained fact access, reflecting the unidirectional Transformer flow in which surface-form generation amplifies rather than corrects underlying fact representations. We also introduce UnFine, a diagnostic benchmark with fine-grained question-answer pairs and fact-level metrics for systematic evaluation. Experiments show that FABLE substantially improves fine-grained question answering while maintaining state-of-the-art holistic editing performance. Our code is publicly available at https://github.com/caskcsg/FABLE.

Peng Wang, Biyu Zhou, Xuehai Tang +2

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

Model EditingLanguage Model EditingFactual Knowledge in Language Models
AskPDF
Jun 19, 2026

LEDGER: Scaling Agentic Document Editing with Dependency-aware Graph Retrieval

We introduce LEDGER to tackle the novel context engineering challenge of agentic document editing, where localized edits to long, structured documents must be applied efficiently without breaking cross-references or semantic consistency. LEDGER constructs a lightweight dependency graph that explicitly models document structure, including hierarchical organization, explicit references, implicit dependencies, and semantic relationships. For each edit, graph-guided retrieval selects only the necessary context, avoiding full-document processing while preserving consistency. We evaluate LEDGER on a curated benchmark of 1.9k test cases with various document types and lengths, spanning six state-of-the-art models: LEDGER improves consistency from 56% to 76% across all six models and test scenarios while reducing token usage. Notably, LEDGER with low reasoning effort matches baseline performance at high reasoning effort using fewer tokens, showing that explicit dependency representations can partially substitute for expensive internal reasoning in agentic document editing.

Mike Hang Wang, Utkarsh Garg, Reza Davari +5

Microsoft

Graph Retrieval-Augmented GenerationLLM Context ManagementAgentic AI
AskPDF