Structured Output Generation

Follow this topic Ask about this topic

Structured output generation concerns responses such as JSON, tool calls, tables, and domain-specific records that downstream software can parse and use. Common approaches include schema-constrained decoding, prompt and schema design, fine-tuning or reinforcement learning with structural rewards, and validation-and-repair loops; some systems instead separate reasoning from deterministic formatting or compilation. Benchmarks increasingly measure structural validity separately from value accuracy, semantic correctness, and executable behavior across tasks such as extraction, question answering, and workflow generation. A central tension is that constraints can guarantee valid form but may reduce reasoning performance, especially in smaller models, while valid outputs can still contain incorrect or misplaced information.

Latest papers

65 papers9 papers in the last four weeks, up 13% on the four weeks before. 0.1% of all new papers.

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation
Jun 27, 2026Varun KotteConformal Prediction LLM Reliability
Independent Researcher
Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints
Jun 24, 2026Fangzheng Li, Aimin Zhang, Chen LvConstrained Decoding LLM Tool Use
1Focus AI Center, Focus Technology Co., Ltd. · 2Nanjing University of Science and Technology
POTracker: Optimizing Large Language Models for Standard-Compliant Power Outage Report Generation
Jun 22, 2026Hung Phan, Aniroop Naladala, Dubey Avanindra +3LLM Fine-Tuning Structured Output Generation
Iowa State University · Oak Ridge Nat. Laboratory
A Dual-Track Framework for Template-Constrained LaTeX Conversion
Jun 22, 2026Chung Cheuk Hei, Liu LiStructured Output Generation
The Hong Kong University of Science and Technology (Guangzhou)
SCENIC: Semantic-Conditioned Edge-Aware Neural Framework for Structured IoT Command Generation
Jun 21, 2026Luke Ztz Hu, Hongbing Lang, Songping MaiEdge Inference Internet of Things
Tsinghua Shenzhen International Graduate School, Shenzhen, Guangdong, 518055 China · Shenzhen Belon Technology Co., Ltd., Shenzhen, China
Dense Coordinate-List Fine-Tuning Induces a Controllable Interference Surface in Vision-Language Models
Jun 12, 2026Chenyu Zhou, Qiliang Jiang, Boguang PanVLM Adaptation Vision-Language Grounding
School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Graduate School of Information, Production and Systems, Waseda University
Vector Map as Language: Toward Unified Remote Sensing Vector Mapping
Jun 9, 2026Yinglong Yan, Yunkai Yang, Haoyi Wang +7Remote Sensing Image Understanding Vision-Language Models
school of Artificial Intelligence and Robotics, Hunan University, Changsha 410082, China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong · Department of Geomatics Engineering, Changsha University of Science and Technology, Changsha 410114, China +1
Capacity, Not Format: Rethinking Structured Reasoning Failures
Jun 8, 2026Hengxin FanLLM Evaluation Structured Prompting
Tianjin Normal University
Ishigaki-IDS: An Open-Weight Verifier-Aware Model for Information Delivery Specification Drafting in Building Information Modeling
Jun 7, 2026Ryo Kanazawa, Koyo Hidaka, Teppei Miyamoto +9Building Information Modeling Structured Output Generation
ONESTRUCTION Inc. · Tottori, Japan · AWS GenAI Innovation +2
Plans for Evaluating Structured Generative Search Summaries
May 26, 2026Tetsuya Sakai, Jina Lee, Hanpei Fang +1Language Model Generation Evaluation Summarization Evaluation
Waseda University/Naver Corporation · Tokyo, Japan · Waseda University +2
Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
May 21, 2026Ryo Kanazawa, Koyo Hidaka, Teppei Miyamoto +9LLM Evaluation Benchmark Construction
ONESTRUCTION Inc. Tottori, Japan · AWS GenAI Innovation Center Tokyo, Japan
The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models
May 20, 2026Jaideep RayConstrained Decoding Small Language Models
TruncProof: A Guardrail for LLM-based JSON Generation under Token-Length Constraints
May 13, 2026Yoshio Kato, Shuhei TarashimaConstrained Decoding Structured Output Generation
NTT DOCOMO BUSINESS, Inc., Japan
ErrOPD: When Does On-Policy Distillation Transfer Teacher Errors?
May 9, 2026Xin Li, Hao Jiang, Annan Wang +2On-Policy Distillation Structured Output Generation
Nanyang Technological University, Singapore
Uncertainty-Aware Structured Data Extraction from Full CMR Reports via Distilled LLMs
May 8, 2026Yi Yu, Parker Martin, Zhenyu Bu +5Clinical Information Extraction LLM Information Extraction
†The Ohio State University · ‡NetMind.AI
Interactive Critique-Revision Training for Reliable Structured LLM Generation
May 8, 2026Fei Xu Yu, Zuyuan Zhang, Mahdi Imani +2LLM Self-Refinement Group Relative Policy Optimization
The George Washington University · Northeastern University · United States Military Academy
Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs
May 6, 2026Aofan Liu, Jingxiang MengPrompt Sensitivity LLM Evaluation
1Peking University. · University of Chicago.
When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models
May 4, 2026Cosimo Galeone, Minsu Park, Giuseppe Ettorre +1LLM Evaluation LLM Input Optimization
Alomana, Grottaglie, Italy
Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
May 2, 2026Deep Pankajbhai MehtaPrompt Sensitivity LLM Evaluation
Adobe Inc.
From World-Gen to Quest-Line: A Dependency-Driven Prompt Pipeline for Coherent RPG Generation
Apr 28, 2026Dominik Borawski, Marta Szulc, Robert Chudy +2LLM Prompting Narrative Generation
Gda´nsk University of Technology
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
Apr 28, 2026Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani +1LLM Evaluation LLM Information Extraction
1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India
LatticeBridge: Rare-Event Sequential Inference for Faithful Structured Sequence Synthesis
Apr 22, 2026Faruk Alpay, Bugra KilictasConstrained Decoding Grounded Text Generation
Department of Computer Engineering, Bahcesehir University
When Agents Go Quiet: Output Generation Capacity and Format-Cost Separation for LLM Document Synthesis
Apr 17, 2026Justice Owusu Agyemang, Michael Agyare, Miriam Kobbinah +2LLM Inference Efficiency Long-Form Document Generation
Sperix Labs · GCTU, Ghana · NCA, Ghana +1
CSPO: Alleviating Reward Ambiguity for Structured Table-to-LaTeX Generation
Apr 13, 2026Yunfan Yang, Cuiling Lan, Jitao Sang +1Reward Decomposition Structured Output Generation
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University · Microsoft Research Asia · Peng Cheng Laboratory
The Format Tax
Apr 4, 2026Ivan Yee Lee, Loris D'Antoni, Taylor Berg-KirkpatrickLLM Prompting Structured Output Generation
UC San Diego
ATLAS-RTC: Closing the Loop on LLM Agent Output with Token-Level Runtime Control
Mar 29, 2026Christopher CruzLanguage Model Decoding Structured Output Generation
Purdue University
Story2Proposal: A Scaffold for Structured Scientific Paper Writing
Mar 28, 2026Zhuoyang Qian, Wei Shi, Xu Lin +19Long-Form Document Generation AI-Assisted Scientific Research
AgentAlpha Team.
Evidence of Layered Positional and Directional Constraints in the Voynich Manuscript: Implications for Cipher-Like Structure
Mar 26, 2026Christophe PariselStructured Output Generation
SEAR: Schema-Based Evaluation and Routing for LLM Gateways
Mar 20, 2026Zecheng Zhang, Han Zheng, Yue XuLLM Routing LLM Evaluation
Strukto.AI, USA · Infron.AI, USA
Generating Hierarchical JSON Representations of Scientific Sentences Using LLMs
Mar 8, 2026Satya Sri Rajiteswari Nimmagadda, Ethan Young, Niladri Sengupta +2Structured Output Generation Hierarchical Representation Learning
Marshall University · Fractal Analytics Inc · West Virginia State University

Jun 27, 2026J/K move · Enter open · S save

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation

Varun Kotte

Independent Researcher

Conformal Prediction LLM Reliability Conformal Risk Control Adaptive Conformal Inference Structured Output Generation Generalization Bounds

Open paper AskPDF arXiv

Large language models (LLMs) deployed for structured generation (NER, JSON extraction, QA, and classification) lack formal reliability guarantees, and standard heuristic abstention policies miss user-specified risk targets by 7.5--12.5%. We characterize when conformal risk control (CRC) can certify structured LLM outputs and when it provably cannot. First, we prove an impossibility result: when the base risk (μ> α), any distribution-free method must abstain on at least ((μ-α)/(1-α)) examples, yielding a closed-form feasibility test: one can check whether CRC will work before running it. Second, we analyze a certification hierarchy across Hoeffding, empirical Bernstein, and a betting-based e-CRC bound, with strict gains in low-variance/large-sample regimes: the Hoeffding-to-Bernstein step delivers the largest gain (+37% certified configurations), while e-CRC adds value when calibration data is scarce (10% certification at 20% data versus 0% for Hoeffding). Third, we validate adaptive conformal inference (ACI) under cross-dataset shift, reducing risk-target violations from 71% to 21%, with residual failures concentrated exactly where the impossibility bound predicts. Across six open-weight models (3B--72B parameters), eight datasets, four tasks, and six nonconformity scores, hard NER/QA/CLS configurations are uncertifiable at (α= 0.10); relaxing to (α= 0.30--0.40) unlocks practical certification (47% NER, 40% QA, 60% CLS). The framework gives a three-step deployment recipe: check feasibility, select the bound and score, then mitigate shift.