Organizations: National Taiwan University, Taipei, Taiwan · National Yang Ming Chiao Tung University, Hsinchu, Taiwan · National Dong Hwa University, Hualien, Taiwan
Web applications rely heavily on hyperlinks to connect disparate information resources. However, the dynamic nature of the web leads to link rot, where targets become unavailable, and more insidiously, semantic drift, where a valid HTTP 200 connection exists, but the target content no longer aligns with the source context. Traditional verification tools, which primarily function as crash oracles by checking HTTP status codes, often fail to detect semantic inconsistencies, thereby compromising web integrity and user experience. While Large Language Models (LLMs) offer semantic understanding, they suffer from high latency, privacy concerns, and prohibitive costs for large-scale regression testing. In this paper, we propose SemLink, a novel automated test oracle for semantic hyperlink verification. SemLink leverages a Siamese Neural Network architecture powered by a pre-trained Sentence-BERT (SBERT) backbone to compute the semantic coherence between a hyperlink's source context (anchor text, surrounding DOM elements, and visual features) and its target page content. To train and evaluate our model, we introduce the Hyperlink-Webpage Positive Pairs (HWPPs) dataset, a rigorously constructed corpus of over 60,000 semantic pairs. Our evaluation demonstrates that SemLink achieves a Recall of 96.00%, comparable to state-of-the-art LLMs (GPT-5.2), while operating approximately 47.5 times faster and requiring significantly fewer computational resources. This work bridges the gap between traditional syntactic checkers and expensive generative AI, offering a robust and efficient solution for automated web quality assurance.
Figures & tables
Fig. 1: The automated data collection pipeline for the HWPPs dataset. The system crawls seed URLs, extracts hyperlink context ( Hinfo ) and target page content ( Pinfo ), filters for accessibility, and stores valid pairs.
Fig. 2: The SemLink Feature Extraction Pipeline. The system handles both text-based and image-based links, augmenting them with contextual “Side-Text” derived from the DOM tree structure.
Fig. 3: The SemLink Siamese Network Architecture. Two inputs are processed by identical SBERT backbones and embedding layer with shared weights. Then, semantic difference vector is passed through MLP to predict semantic similarity.
Model/Tool
Accuracy
Precision
Recall
F1-Score
Generative Oracles (LLMs)
Llama-3-8B
86.98%
80.70%
97.20%
88.18%
Llama-3-70B
95.37%
95.44%
95.30%
95.37%
GPT-3.5 Turbo
94.55%
99.37%
89.65%
94.27%
GPT-4o
98.40%
99.95%
96.85%
98.37%
GPT-5.2
99.10%
99.90%
98.30%
99.09%
TABLE I: Performance Comparison with Baselines on Test Set
Feature Configuration
Accuracy
Precision
Recall
F1-Score
Anchor Text Only
92.03%
94.45%
89.30%
91.80%
Anchor + Side-Text
92.65%
90.31%
95.56%
92.86%
Anchor + Image OCR
92.28%
94.11%
90.20%
92.11%
Full Context (SemLink)
92.70%
90.06%
96.00%
92.93%
TABLE II: Ablation Study: Impact of Feature Extraction Strategies
Method
F1-Score
No Image Text (Baseline)
84.07%
Attributes Only ( alt / title )
89.86%
OCR Only
86.46%
Attributes + OCR
90.60%
Attributes + OCR + LabelDroid
88.81%
TABLE III: Ablation: Image Text Extraction Methods (Image Links Only)
Model
Recall
Speed (links/sec)
Speedup
Llama-3-8B
97.20%
0.87
8.7x
Llama-3-70B
95.30%
0.10
1x
GPT-3.5 (API)
89.65%
1.27
12.7x
GPT-4o (API)
96.85%
1.17
11.7x
GPT-5.2 (API)
98.30%
0.65
6.5x
SemLink
96.00%
30.87
308x
TABLE IV: Efficiency Comparison: Throughput vs. Recall
Fig. 4: Efficiency vs. Performance. SemLink (Green Star) is the sole approach within the real-time threshold ( <0.1 s).
Fig. 5: A “Soft Link Rot” example. The server returns HTTP 200, but the content is an error message. Traditional tools mark this as Valid; SemLink correctly marks it as Irrelevant.
Fig. 6: An example of Semantic Drift. The anchor text promises specific news content, but the target page has drifted to a general landing page. SemLink identifies this as Irrelevant due to low semantic overlap.
Fig. 7: Visualizing the Side-Text Heuristic. SemLink extracts the parent header “Campus Life” to disambiguate the generic “Read More” anchor text, correctly validating the link.
Fig. 8: A False Negative case. The link redirects to a login portal. SemLink classifies this as “Irrelevant” due to lack of semantic overlap, though it is functional behavior.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Search Space
Optimal
Triplet Margin ( α )
{0.5,1.0,2.0}
1.0
Triplet Weight ( λ1 )
{0,0.5,1.0}
0
BCE Weight ( λ2 )
{0.5,1.0,2.0}
1.0
Dense Layers
{1,2,3}
2
Appendix
TABLE V: Hyperparameter Search Space and Optimal Values
The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in programming, giving rise to "vibe coding", where users can build complete projects and even control computers using natural language instructions. This paradigm has driven automated webpage development, but it introduces a new requirement about how to automatically verify whether the web functionalities are reliably implemented. Existing works struggle to adapt, relying on static visual similarity or predefined checklists that constrain their utility in open-ended environments. Furthermore, they overlook a vital aspect of software quality, namely latent logical constraints. To address these gaps, we introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing. WebTestBench encompasses comprehensive dimensions across diverse web application categories. We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester, a baseline framework for this task. Evaluating popular LLMs with WebTester reveals severe challenges, including insufficient test completeness, detection bottlenecks, and long-horizon interaction unreliability. These findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands. We hope that WebTestBench provides valuable insights and guidance for advancing end-to-end automated web testing. Our dataset and code are available at https://github.com/friedrichor/WebTestBench.
Recent progress in Natural Language Processing (NLP) has been driven by the emergence of Large Language Models (LLMs), which exhibit remarkable generative and reasoning capabilities. However, despite their success, evaluating the true semantic understanding of these models remains a persistent challenge. Traditional benchmarks such as Word-in-Context (WiC) effectively probe this capability, but their creation is resource-intensive and often limited to high-resource languages. In this paper, we introduce SemBench, a framework for automatically generating synthetic benchmarks that assess the semantic competence of LLMs using only dictionary sense definitions and a sentence encoder. This approach eliminates the need for curated example sentences, making it both scalable and language-independent. We evaluate SemBench in three languages (English, Spanish, and Basque) spanning different levels of linguistic resources, and across a wide range of LLMs. Our results show that rankings derived from SemBench strongly correlate with those obtained from standard WiC datasets. Furthermore, our analysis demonstrates that only a small number of examples is required to achieve stable and meaningful rankings. Overall, SemBench provides a lightweight, adaptable, and data-efficient framework for cross-lingual evaluation of semantic understanding in LLMs.
Mikel Zubillaga, Naiara Perez, Oscar Sainz +1
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
In commercial web search, aligning content freshness with user intent remains challenging due to the highly varied lifespans of information. Traditional industrial approaches rely on static time-window filtering, resulting in "one-size-fits-all" rankings where content may be chronologically recent but semantically expired. To address the limitation, we present a novel Large Language Models (LLMs)-based Query-Aware Dynamic Content Expiration Prediction Framework deployed in Baidu search, reformulating timeliness as a dynamic validity inference task. Our framework extracts fine-grained temporal contexts from documents and leverages LLMs to deduce a query-specific "validity horizon"-a semantic boundary defining when information becomes obsolete based on user intent. Integrated with robust hallucination mitigation strategies to ensure reliability, our approach has been evaluated through offline and online A/B testing on live production traffic. Results demonstrate significant improvements in search freshness and user experience metrics, validating the effectiveness of LLM-driven reasoning for solving semantic expiration at an industrial scale.