cs.AIOct 27, 2026
SaveFilingLoop: Evaluating and Evolving LLM Agents for Corporate Disclosure Review and Revision
Organizations: Nanyang Technological University, Singapore · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Abstract
Public disclosure repositories such as SEC EDGAR support informed investment through web access to corporate filings. Supporting their review with language model agents requires discovering unstated deficiencies across related documents, acquiring missing company facts, and verifying that requested revisions satisfy evidence-backed requirements. Existing financial question-answering evaluations provide limited evidence of these interactive capabilities. We introduce FilingLoop, an interactive benchmark and a self-evolving agent framework for corporate disclosure review and revision. Grounded in historical U.S.\ Securities and Exchange Commission correspondence, FilingLoop reconstructs 380 predominantly IPO review tasks from 96 companies under distinct information constraints. An evaluator separate from the acting agents scores final filings and responses against 6,099 eligible requirements and audits unsupported claims and harmful revisions. Our framework couples iterative review and revision within each case with procedural learning across cases, converting requirement-level evaluation feedback into role-specific guidance without updating model weights. Updates affect only subsequent cases, allowing procedural adaptation to be assessed without retrying scored tasks. On the evaluated stream, evolution increases requirement fulfillment from 24.28% to 27.96% and full comment resolution from 16.90% to 20.75%, with a 5.8% increase in total cost over the fixed-policy variant. FilingLoop connects the concerns an agent discovers with the company facts it obtains and the revisions it commits, providing requirement-level feedback for procedures used in later review tasks.
Figures & tables
Figure 1. Disclosure review challenges. (1) Discover omissions, (2) acquire missing facts, and (3) verify revisions. Reviewer (left); issuer counsel (right). A reviewer questions a vague revenue-growth explanation; issuer counsel checks company records and revises the disclosure. Three numbered callouts identify omission discovery, fact acquisition, and verification of document changes. Illustrative contributions of ten percentage points from price and thirty from volume explain forty percent revenue growth.
Figure 2. Benchmark scope and task characteristics. Top: corpus hierarchy and coverage. (a) Requirement disposition shares; (b) cumulative distribution of comments per task; (c) requirements per comment (5+ includes the full tail); (d) company-fact dependence within each disposition. A scope diagram connects \BenchCompanies{} companies, \BenchPackages{} review packages, \BenchTasks{} review tasks, \BenchComments{} scorable comments, and \BenchRequirements{} requirement occurrences. The collection contains \BenchReviewedFilings{} unique reviewed filings, with review dates from \BenchReviewStart{} to \BenchReviewEnd{}. Below, four charts in one row: pie sectors show disposition shares, an empirical cumulative curve shows comments per task, bars show percentages of comments by requirement count, and horizontal bars show company-fact dependence. The fact-dependence bars use a zero-to-one-hundred-percent scale. Colors consistently identify the four requirement dispositions.
Figure 3. FilingLoop benchmark and self-evolving agents. (a) Historical SEC exchanges yield tasks with visible records and hidden references. (b) Reviewer and Filer agents iteratively review and revise disclosures. (c) Structured grading locks scores before feedback updates role skills for later batches, without replaying cases. Three vertical nested panels show benchmark construction, two-role review and revision, and online skill evolution. Government and company cartoons identify the historical participants in the benchmark. PDF reports, text files, Excel workbooks and image cards show source materials. A navy-suited robot with a government building and red-annotated issue clipboard represents the regulatory Reviewer. A teal-suited robot with a company building, briefcase and laptop edits a disclosure with visible red and green revisions, representing the company-counsel Filer. Visible task records enter the document workspace, while an amber dashed path supplies hidden reference evidence directly to independent evaluation. Three case-level grading stages match comments, check requirements, and audit claims and edits. Structured JSON verdicts are validated before deterministic scoring. An upward feedback chain connects locked scores, procedural editing and role-specific skill storage. Blue checklist playbooks with magnifiers depict reusable Reviewer procedures; green editing playbooks with pens depict reusable Filer procedures. These icons appear both after distillation and in the role skill library. Only future batches receive updated guidance.
| Accuracy (%) | Task completion | Total tokens (M) | Cost | Agent time | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Backbone | Requirement Acc. | Comment Acc. | Task Acc. | Company Acc. | (%) | Input | Output | (USD) | Total (h) | Avg. (min/task) |
| Codex | gpt-6.1-sol | 22.13 (1,350/6,099) | 14.84 (447/3,012) | 4.74 (18/380) | 0.00 (0/96) | 99.74 (379/380) | 4,706.25 | 33.07 | 3,498.82 | 673.46 | 106.34 |
| Claude Code | claude-opus-5.5 | 21.18 (1,292/6,099) | 14.21 (428/3,012) | 4.21 (16/380) | 0.00 (0/96) | 99.47 (378/380) | 4,301.81 | 38.14 | 3,713.03 | 696.21 | 109.93 |
| mini-SWE-agent | gpt-6.1-sol | 21.99 (1,341/6,099) | 14.61 (440/3,012) | 4.47 (17/380) | 0.00 (0/96) | 99.74 (379/380) | 3,676.76 | 24.14 | 1,353.33 | 183.61 | 28.99 |
| mini-SWE-agent | claude-opus-5.5 | 23.74 (1,448/6,099) | 16.37 (493/3,012) | 5.26 (20/380) | 0.00 (0/96) | 100.00 (380/380) | 3,382.62 | 28.73 | 2,327.73 | 212.99 | 33.63 |
| mini-SWE-agent | gemini-4-argon | 21.10 (1,287/6,099) | 14.31 (431/3,012) | 4.21 (16/380) | 0.00 (0/96) | 99.47 (378/380) | 3,934.13 | 26.07 | 1,028.53 | 172.59 | 27.25 |
Table 1. End-to-end results and role diagnostics on 380 tasks from 96 companies. Parentheses show successful/eligible counts; Filer rows share the Fixed target pool.
Figure 4. Resolution quality and efficiency. (a) Requirement and Comment Accuracy; (b) Fixed–Evolving comparison; (c) cost versus Requirement Accuracy; (d) relative gains and resource overhead. Results derive from Table 1 . Four colored panels in one horizontal row compare the eight configurations in the main table. Evolving achieves the highest requirement and comment accuracies, 27.96 and 20.75 percent. Relative to Fixed it fulfills 224 more requirements and fully resolves 116 more comments. The cost plot places Evolving above Fixed in accuracy with a modest increase in cost. Relative gains of 15.1 percent in fulfilled requirements and 22.8 percent in resolved comments accompany increases of 5.8 percent in cost and 3.1 percent in recorded Agent time. Cost per fulfilled requirement falls by 8.1 percent.
Figure 5. Effect of evolution. (a) System; (b) role diagnostics. Bars show accuracies (%); + labels show gains (pp). Filer uses the shared Fixed target pool. Two side-by-side white-background panels preserve all eight comparisons in the evolution table. Paired vertical bars show Fixed and Evolving accuracies, with exact values above each bar and percentage-point gains marked with plus signs above each pair. System Requirement: 24.28 to 27.96, gain 3.68; Comment: 16.90 to 20.75, gain 3.85; Task: 3.95 to 5.00, gain 1.05; Company: 1.04 to 2.08, gain 1.04. Reviewer Requirement: 35.45 to 36.86, gain 1.41; Comment: 32.64 to 33.93, gain 1.29. Conditional Filer Requirement: 68.50 to 78.86, gain 10.36; Comment: 51.78 to 63.58, gain 11.80. Both axes start at zero; system and role panels have different ranges.
| Cost/task | Requirements met | ||
| Configuration | (USD) | /\uparrow$ | /hour |
| Codex / GPT | 9.21 | 38.6 | 2.00 |
| Claude Code / Opus | 9.77 | 34.8 | 1.86 |
| SWE / GPT | 3.56 | 99.1 | 7.30 |
| SWE / Opus | 6.13 | 62.2 | 6.80 |
| SWE / Gemini | 2.71 | 125.1 | 7.46 |
Table 2. Resource efficiency over 380 tasks. Throughput counts fulfilled requirements; hours denote recorded Agent time. SWE = mini-SWE-agent; backbones follow Table 1 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. Source records and data types. Five source groups retain distinct identities. Icons mark record categories and archived or processed formats; shared provenance links documents to review rounds and reference evidence. Three illustrated cards connect registration filings, staff correspondence, and issuer correspondence. Distinct document, comment, reply, calendar, and archive icons identify all five source groups. Filing content includes business and risk disclosures, financial statements, and tables; correspondence includes numbered requests, issue references, explanations, commitments, and revision evidence. The annual-report card lists two rounds, four comments, eight requirements, and certification exhibits. Context records include earlier periodic filings and status letters. A format strip identifies SGML, HTML/PDF, TXT, images, XLS/ZIP, and JSON/JSONL as archived or processed representations. Shared provenance retains company, file number, accession, dates, source spans, and content hashes.
| Disposition | Count | Share | Needs facts | Evidence target |
|---|---|---|---|---|
| Disclosure change | 4,668 | 76.5% | 2,093 (44.8%) | Amended filing with the required addition, edit, or removal |
| Explanation to staff | 1,075 | 17.6% | 805 (74.9%) | Formal response with the requested explanation or analysis |
| Supporting material | 173 | 2.8% | 33 (19.1%) | Submission containing the requested document, exhibit, or copy |
| Commitment | 183 | 3.0% | 149 (81.4%) | Formal response with an explicit commitment to future action |
| Total | 6,099 | 100% | 3,080 (50.5%) | Requirement-specific evidence in the designated deliverable |
Table 3. Requirement dispositions and evidence targets. Fact percentages are within each disposition.
| Record or artifact | Reviewer | Filer | Trusted host |
|---|---|---|---|
| Permitted filing and earlier history | Read | Read | Selects sources |
| Current draft and committed edits | Read | Propose edits | Validates and commits |
| Generated review comments | Author | Read | Retains and routes |
| Formal issuer responses | Read | Author | Retains and routes |
| Company questions and released answers | Hidden | Private access | Serves scoped facts |
| Current historical letters and amendments | Hidden | Hidden | Reference evidence |
Table 4. Information access during a task. Private fact-service records and current-task grading judgments remain separated from the public exchange.
Figure 7. Illustrated interaction protocol. Reviewer and Filer exchange grounded comments, responses, and revisions. Company facts remain private to the Filer; committed changes persist through finalization. Re-review controls iteration, while independent grading assesses frozen final artifacts. An illustrated workflow uses a navy Reviewer robot, a teal Filer robot, source documents, marked-up drafts, and magnifiers to depict discovery, response, and re-review. A gold private lane shows a human company representative supplying documented facts and a robot committing a revised filing with red before-text and green after-text. Accepted work is retained while pending responses are finalized. Distinct check, loop, clock, and hourglass symbols mark resolution, another exchange, deferred information, and the exchange limit. Validation, preservation of final artifacts, and independent grading remain distinct controls.
Figure 8. Procedural evolution across batches. Skills remain frozen during execution. Locked Evolving outcomes inform validated role updates used only by subsequent tasks; Fixed guidance remains unchanged. Illustrated Reviewer and Filer robots use frozen role playbooks. A padlock and assessment checklist denote locked first-pass outcomes. A fictional research editor and robot revise two procedural books, and later tasks receive validated guidance with the same models and tools. Arrows point forward without returning to scored tasks.
| Reporting scope | Requirements | Comments | Tasks | Companies |
|---|---|---|---|---|
| System and Reviewer | 6,099 | 3,012 | 380 | 96 |
| Filer (shared Fixed target pool) | 2,162 | 983 | 283 | 95 |
Table 5. Accuracy denominators. Both variants use the same scoped targets.
| Setting | Configuration |
|---|---|
| Evaluation population | 380 tasks, 96 companies; 378 registration and 2 annual-report tasks |
| Task order | Seed 0; interleaved companies, chronological rounds within each company |
| Update batches | At most 16 tasks; no repeated company in the same batch |
| Concurrent execution | Up to 16 tasks; policies frozen throughout each update batch |
| Interaction budget | At most 3 exchanges; 100 reference steps per role invocation |
| Reasoning effort | high by default; provider options retained in request records |
Table 6. Shared settings for the reported evaluation.
Figure 9. Resolution and resource accounting. Missed denotes undiscovered reference items; Open denotes discovered but unfulfilled items. Cost separates attributed roles from shared services. All values derive from Table 1 . Three panels compare Fixed and Evolving with stacked bars. Requirements and comments are partitioned into missed, open, and met counts. A third panel partitions dollar cost into Reviewer, Filer, and shared usage. Reviewer and Filer character icons accompany the resolution panels.
| Fixed | Evolving | |||||
|---|---|---|---|---|---|---|
| Resource | Reviewer | Filer | Shared | Reviewer | Filer | Shared |
| Input tokens (M) | 3,041.21 | 2,848.36 | 209.02 | 3,126.36 | 3,076.23 | 229.92 |
| Output tokens (M) | 15.47 | 16.47 | 3.65 | 16.17 | 17.54 | 4.19 |
| Cost (USD) | 1,144.51 | 740.30 | 495.34 | 1,185.71 | 793.60 | 539.92 |
Table 7. Attributed resources for Fixed and Evolving. Shared values subtract the two role rows from the system total.
Figure 10. Evidence-grounded resolution of an acquisition disclosure concern. Scoped company facts enter the filing and formal reply, satisfying both selected requirements. Broader requests in agent comment R3 remain deferred. Four illustrated stages show Reviewer identifying an acquisition revenue gap, Filer requesting facts, a company service returning Pacific Union and Paragon contributions, and Filer committing a revision and reply. The final assessment links each of two fulfilled requirements to its artifact.
Figure 11. Partial fulfillment of a compound request. Payment amounts and note terms are corrected, but an available royalty withholding rationale is omitted from the reply, leaving two linked requirements unmet. Original eligible requirement identifiers are retained. Illustrated review, fact release, filing revision and reply stages show numerical amendments succeeding while the justification for withholding exact royalty rates is omitted. A five-row assessment marks three requirements met and two unmet, including their conditional dependency.
Figure 12. Procedural transfer to a later task. Contradiction feedback informs an accepted filing-wide verification rule. A later accounting-election case connects that procedure to committed revisions and two fulfilled requirements. Earlier feedback about inconsistent market-maker statements leads through an illustrated editor to a procedural search rule. A later task reconciles conflicting accounting-transition statements in a risk factor and note, and the final assessment marks both selected checks met.
More in this topic
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,217 papers