cs.AIOct 5, 2026
SaveIn With the Old: Enhancing 'Classical' Document Automation with Generative AI
Organizations: Capstone Practice Systems, Harvard, Massachusetts · Maastricht Law and Tech Lab, Maastricht University, Maastricht, Netherlands
Abstract
Software-based legal assistance systems have leveraged many different forms of knowledge representation and reasoning. This article explores how document automation services rooted in expert system style and other symbolic approaches can usefully enhance and be enhanced by current generative AI approaches. We discuss the possible benefits and challenges, and report on preliminary experiments in using large language models to identify and fix issues in texts written by laypeople.
Figures & tables
Figure 1 : Screenshot of the domestic violence court order module.
| Instructions | Candidate Text | Questions (Answers) | Corrected Text |
| This is a draft contract for the sale of a good. It should adhere to the following requirements or include this information: • The name of the seller and buyer • The specific good in question • The sale price • The date of the sale • Who shall undertake the delivery • Is the good sold ”as-is” or is the seller liable for a defective good? | I am selling a car to Reed Thomas. He will pick it up on December 20th 2024. | • What is the specific car being sold? 1999 Volkswagen ’Bug.’ • What is the sale price of the car? $475 • Is the car being sold ’as-is’ or will you, as the seller, be liable for any defects? As-is • What is your name as the seller? Jane Smith | Contract for Sale of Goods This contract is entered into on December 20, 2024, by and between Jane Smith (”Seller”) and Reed Thomas (”Buyer”). 1. Description of the Good : The Seller agrees to sell, and the Buyer agrees to purchase, a 1999 Volkswagen ’Bug.’ 2. Sale Price : The agreed sale price for the vehicle is $475. 3. Date of Sale : The sale shall take place on December 20, 2024. 4. Delivery : The Buyer, Reed Thomas, shall undertake the responsibility of picking up the vehicle on the date of sale. 5. Condition of the Good : The vehicle is sold ”as-is,” and the Seller, Jane Smith, makes no warranties or guarantees regarding the condition of the vehicle. The Seller shall not be liable for any defects or issues with the vehicle after the sale is completed. By signing below, both parties agree to the terms outlined in this contract. |
Table 1 : Example of an instructions text, a candidate text, questions (LLM) with answers provided by us, and the final text written by the LLM.
| INSTRUCTIONS: |
| These are instructions on how a document should look. I will provide you with a candidate document, and you will return a list of flaws and questions to ask the user to fix these flaws. |
| Return a JSON object with the keys “questions”. The value of “questions” is a list of strings, each representing a question that should be asked to the user to cure these flaws and capture the missing information. |
| Stick to the specific instructions. Do not ask for information that is not in the instructions. Only address the flaws with one question each. If the document is a template, only ask for the variables that are in the template. |
Table 2 : Prompt for Pipeline Step 1
| INSTRUCTIONS: |
| These are instructions on how a document should look. I will provide you with a candidate document, as well as answers to follow-up questions asked to the user. Based on this information, create a new document that is correct, i.e. directly corresponds to the instructions. Return the answer as a text string. |
Table 3 : Prompt for Pipeline Step 2
| Scenario | Do the questions asked capture all of the missing elements? | Does the corrected text comply with the instructions? | Is the corrected document free of hallucinations, i.e., made up facts not part of the candidate text or questions? |
| S1 - Apartment reclamation | 100% | 100% | 100% |
| S2 - Sale Contract | 80% | 80% | 100% |
| S3 - Employment contract | 100% | 20% | 100% |
Table 4 : Evaluation results for generated questions and corrected texts. In total, 5 examples were evaluated per scenario.
| EMPLOYMENT AGREEMENT |
| This Employment Agreement, by and between {{company_name}} and {{employee_name}} is entered into as per today. As of today, {{company_name}} employs {{employee_name}}, and {{employee_name}} accepts employment, as a full-time {{job_title}}. |
Table 5 : Part of instructions for S3