This paper presents a method for the automated collection and aggregation of unstructured data from diverse web sources, utilizing Large Language Models (LLMs). The primary challenge with existing techniques is their instability when the structure of webpages changes, their limited support for dynamically loaded content during information collection, and the requirement for labor-intensive manual design of data pre-processing processes. The proposed algorithm integrates hybrid web scraping (Goose3 for static pages and Selenium+WebDriver for dynamic ones), data storage in a non-relational MongoDB database management system (DBMS), and intelligent extraction and normalization of information using LLMs into a predetermined JSON schema. A key scientific contribution of this study is a two-stage verification process for the generated data, designed to eliminate potential hallucinations byy comparing the embeddings of multiple LLM outputs obtained with different temperature parameter values, combined with formalized rules for monitoring data consistency and integrity. The experimental findings indicate a high level of accuracy in the completion of key fields, as well as the robustness of the proposed methodology to changes in web page structures. This makes it suitable for use in tasks such as news content aggregation, monitoring, and log analysis in near real-time mode, with the capacity to scale rapidly in terms of the number of sources.
Figures & tables
Approaches
Method
Advantages
Disadvantages
Parsing
XPath/CSS-selectors
High speed, simplicity, low cost
Fragility to structure changes, does not work with dynamic content
Regular expressions
Flexibility, precision on known patterns
Requires manual configuration, does not transfer well
Headless browsers (Selenium, Playwright)
Work with dynamic pages
High cost, low speed, complexity of support
Template methods
Extraction automation, suitable for similar pages
Limited versatility, sensitivity to structure
Manual template generation
High accuracy on small volumes
Labor-intensive, does not scale
Information extraction (IE)
CRF, BERT, transformers
High accuracy, resistance to local changes
Requires markup, low portability
Table 1: Comparison of specified tools and approaches.
Figure 1: DFD schema of the method.
Figure 2: IDEF0 of method.
Figure 3: IDEF1 of method.
Figure 4: BERT Similarity between raw text and aggregated text.
Method
Hallucinations, %
Tokens per object
Latency, ms
Base
18.2
800
420
Multi-stage (N=2)
7.6
1600
780
Multi-stage (N=3)
3.8
3200
1320
Multi-stage (N=5)
1.5
6400
2160
Table 2: Multi-stage comparison.
Metrics
LLM-TKIE (baseline)
Suggested method
Field-level (F-score), %
86.1
98.7
Critical Error Rate (CER), %
12.4
2.5
Hallucination Rate, %
28.7
3.5
Table 3: Comparison between the baseline and suggested approach
Approach
Processing of duplicates
Similar news detection
Automatisation
Processing loop time
Structure selection / Words limitation
[ 16 ]
No
Clusters
Yes
Not stated
LLM + generating three structured summary levels / 4096-token context window, forcing 5–8 sentences per article
[ 6 ]
No
Specific dataset
Yes
Not stated
Refining LLM / Length is implicitly constrained by the underlying LLM and API usage
[ 17 ]
No
The study does not cluster or group similar cases. Each case is classified independently
Mostly yes / manual checking
Not stated
LLM prompt / 200k-token context window, allowing full judgments (some over 8,000 words) to be processed without truncation
[ 18 ]
Yes
No
Yes
Not stated
Sentence embeddings and cosine similarity are used. No keyphrase extraction or article structure parsing is performed / Models impose a 512-token input limit for BERT-based systems
[ 19 ]
Yes
No
Yes
Not stated
The only structure extracted is sentence-level consistency labels (Faithful / Intrinsic / Extrinsic) / Summaries are constrained to about 2–3 sentences
[ 20 ]
No
No
Yes
Training time is about 500 hours for 350k steps on RTX 2080 Ti
No structural extraction; T5 is trained directly / Inputs and summaries are truncated to 512 tokens
Modern web scraping struggles with dynamic, interactive websites that require more than static HTML parsing. Current methods are often brittle and require manual customization for each site. To address this, we introduce Webscraper, a framework designed to handle the challenges of modern, dynamic web applications. It leverages a Multimodal Large Language Model (MLLM) to autonomously navigate interactive interfaces, invoke specialized tools, and perform structured data extraction in environments where traditional scrapers are ineffective. Webscraper utilizes a structured five-stage prompting procedure and a set of custom-built tools to navigate and extract data from websites following the common ``index-and-content'' architecture. Our experiments, conducted on six news websites, demonstrate that the full Webscraper framework, equipped with both our guiding prompt and specialized tools, achieves a significant improvement in extraction accuracy over the baseline agent Anthropic's Computer Use. We also applied the framework to e-commerce platforms to validate its generalizability.
Guan-Lun Huang, Yuh-Jzer Joung
Dept. of Information Management, National Taiwan University, Taipei, Taiwan
LLMs and agents can generate web scrapers from natural-language requirements, but direct generation remains unreliable because of dependency errors, broken selectors, schema mismatches, and heterogeneous page structures. We propose a constrained, verifiable agent framework that shifts LLM output from free-form code to typed JSON collector configurations, combining a six-type collector taxonomy, template and utility-function constraints, static Airflow DAG execution, rule-based quality checking, and structured feedback correction. Experiments on 138 tasks show that the taxonomy supports description-based requirement typing, while confirming that stable instantiation requires completing source, field, and execution constraints beyond the initial description. On 80 independently source-verified tasks, the framework runs with zero execution-stage LLM tokens and the lowest average wall-clock time, trading moderate one-shot quality for a reusable, deterministic, and verifiable execution path suited to repeated scheduled collection. These results position the framework as a reusable, low-cost, and verifiable execution path for repeated open-web data collection.
Bo Chen
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
The abundant and heterogeneous nature of web content necessitates automated information extraction, and generating scrapers that can be reused across similar web pages offers an effective solution for scalable data extraction. In this work, we propose Co-Scraper, a two-stage framework capable of handling the hierarchical complexity of long HTML documents. By integrating a query-aware DOM pruning mechanism with stable extraction strategy induction, Co-Scraper can effectively transforms web content into executable programmatic wrappers using a fine-tuned Qwen3-8B model. On the test set of SWDE, Co-Scraper achieves state-of-the-art performance with an F1 score of 94.78% and a reuse success rate of 90.39%. This framework significantly enhances the accuracy and resilience of data extraction, providing a highly efficient approach for web data acquisition tasks.
Shoupeng Wang, Jiantao Qiu, Wuyang Zhang +1
Shanghai Artificial Intelligence Laboratory, OpenDataLab · University of Science and Technology of China