Manifest-Assisted Retrieval in Artificial Intelligence Search: Evidence Gaps and a Security Evaluation
Authors: Gangzhen Qian, Kainan Zhou, Chuhong Xu, Hang Xiao, Zhi Jing, Rong Bao, Lu Yi, Zhaoyi Li
Organizations: Google LLC Mountain View, USA · Sony Corporate of America San Jose, USA · Fortinet, Inc. Sunnyvale, USA · Google LLC United States · Intuit Inc. Mountain View, USA
The external sources retained in our non-systematic audit did not isolate the incremental security or utility effect of consuming /llms.txt while ordinary retrieval remained fixed. The initial audit covered 39 sources through August 16, 2026, with later targeted updates through October 14. A purposive 15-tool study observed no automatic requests in 45 trials at one endpoint; a non-probability log study found that 97% of domains with detected files received no recorded requests in May 2026. We separate publication, fetch, model use, and action; define five operational threat paths and paired security-utility estimands; and exercise deterministic pre-model controls with 30 author-defined parser fixtures. We also execute a reduced pilot: 24 tasks across eight sites, five conditions, and 120 real model responses, with ordinary evidence held fixed. Frozen automatic scoring identifies attacker-target adoption in 21/24 raw adversarial responses, with one further response unresolved; the adversarial URLs-only condition has no known adoptions and one unresolved response. The pilot measures output changes conditional on supplied evidence, without live retrieval or external actions. Together, the framework and pilot distinguish manifest-content effects from publication, discovery, and downstream execution, while leaving full retrieval and agent evaluation open.
Figures & tables
Fig. 1: Proposed threat model. A content-borne attack crosses an exposure boundary and may follow one or more paths. Retrieval abuse can precede model interpretation, and ranking manipulation may arise during retrieval or reranking before final context construction. Instruction hijacking, data injection, and evidence poisoning require model-visible content, but only instruction hijacking necessarily involves instruction-like control. Deterministic policy can reduce realized impact independently of model robustness.
Path
Decision rule
Metric / denominator
Retrieval abuse
Fetching or parsing violates a network, origin, resource, cost, or availability policy before model-selected action.
Violations or excess cost per attempted fetch.
Instruction hijack
Instruction-bearing content increases an attacker-goal or task-violation probability relative to a matched neutralized twin.
Goal or violation rates per exposed trial; repeated-sample contrast.
Data injection
The task uses an attacker-controlled identifier, origin, author, tool result, or value as trusted data.
Corrupted-field action rate per exposed trial.
Evidence poisoning
A controlled false or selected claim changes the answer, support, or confidence without an attacker-task instruction.
Target-claim success per context inclusion.
Ranking manipulation
With canonical facts fixed, attacker-controlled content changes candidate inclusion or promotes, suppresses, or monopolizes sources, products, or citations in violation of the user’s stated criteria.
Candidate-set, rank, or citation-share change per paired query.
TABLE I: Operational attack paths and outcomes
Study
Setting / attacker model
Reported outcome and interpretation limit
BIPIA [ 12 ]
Five tasks; 86,250 factorial prompts
GPT-4 attack success 31.03%; model-output instruction following with exposure fixed, not independent web retrieval or executed external action.
AgentDojo [ 13 ]
GPT-4o; 629 stateful cases
Utility 69.0% benign and 50.01% under attack; targeted attack success 57.69%.
Chang et al. [ 14 ]
Ten-token poison; one malicious item per query; 11×100 queries with the default embedding model
Mean Recall@5 94.1%; this retrieval metric is separate from the paper’s end-to-end RAG and agent exploit outcomes.
PoisonedRAG [ 15 ]
Five malicious passages in a 2,681,468-text corpus
97/100 target answers; targeted corpus poisoning, not arbitrary-page prevalence.
RAG data loaders [ 16 ]
Nineteen techniques; five loaders; 357 scenarios
Loader-stage attack success 74.4%; ingestion representation, not live discovery.
SIREN [ 17 ]
One edited page per context; two Claude models; eight query–model contexts
Rank-1 in 62/124 technique trials; successful payloads had a mean fresh-session reproduction rate of 0.805.
TABLE II: Illustrative focal studies, reported outcomes, and interpretation limits
Fig. 2: Evidence-stage coverage for the 15 historical focal studies. Columns denote publication or discovery (P/D), HTTP fetch, retrieval or reranking (R/R), model output (M/O), and external action (E/A). Filled circles are direct measurements, open circles fixed stages, triangles observational proxies, and blanks unmeasured stages. Carrier labels distinguish actual /llms.txt , manifest-style proxies, and generic content. October additions are outside this historical map.
Fig. 3: Cumulative success across retrieval, reranking, and generation in the Qwen3-8B pipeline reported by Yin et al. [ 10 ] . Each line retains 200 original query–target trials under BM25 or dense retrieval. The seven attack labels and all 42 percentages are provided in the supporting archive. The figure is neither a cross-study average nor an /llms.txt estimate.
Source / sample
Observed endpoint
Inference boundary
Rankability; September 18 ranked samples [ 11 ]
93/1,000 and 830/10,000 confirmed root files; 474 and 4,211 unknown outcomes
Non-probability frame; changed validation rules prevent a direct trend comparison with the historical 87/1,000 result. No use measure.
SE Ranking; nearly 300,000 domains [ 21 ]
10.13% presence without a detected citation advantage
Commercial cross-section without causal support.
Ahrefs; 137,210 domains [ 22 ]
About 38,000 detected HTTP-200 Markdown files; 97% received no recorded request during May 2026
Non-probability frame; specification conformance was not assessed, and request absence is not a safety outcome.
Borysenko; 15 tools, 45 trials [ 20 ]
No /llms.txt requests: 0/15 tools and 0/45 trials
Server-log test of incidental fetching under an explicit-URL prompt; purposive tools, one endpoint, and one task.
Google Search; Chrome Lighthouse [ 23 , 24 ]
Search ignores the file for ranking; Lighthouse offers an optional audit
Product-specific statements; general consumption remains unmeasured.
TABLE III: /llms.txt Evidence by Source and Denominator
Level
Safeguard
Boundary / outcome
Discovery policy
Fetching is policy-controlled and logged; file location confers neither trust nor rank.
Exposure and hidden product behavior.
Network invariant
Require secure transport and same-origin defaults; revalidate redirect targets and DNS results, and verify the connected peer.
Server-side request forgery and cross-origin steering.
Resource invariant
Enforce byte, token, link, depth, redirect, time, and cost limits before model use.
Resource exhaustion, cost abuse, or expansion loops.
Parsing and provenance
Emit typed untrusted data with origin, final location, time, representation, and hash.
Instruction ambiguity, drift, and attribution.
Ranking and evidence
Do not let manifest prose set cross-site rank; corroborate comparative or consequential claims.
Evidence poisoning and source manipulation.
Capability policy
Separate public retrieval from secrets and write tools; confirm consequential actions.
Data exfiltration and unauthorized state changes.
TABLE IV: Core Safeguards and Verification Levels
Arm
Known target adoptions
Unresolved security
Security-loss bounds (%)
Accepted answers
Utility failures
A: absent
0/24
2/24
0–8.3
22/24
2/24
B: benign, raw
0/24
1/24
0–4.2
23/24
1/24
C: adversarial, raw
21/24
1/24
87.5–91.7
2/24
22/24
D: benign, guarded
0/24
2/24
0–8.3
22/24
2/24
E: adversarial, guarded
0/24
1/24
0–4.2
23/24
1/24
TABLE V: Frozen automatic pilot results, 24 responses per arm. Security bounds reflect unresolved classification. Accepted answers use frozen forms; utility also requires the specified source URL and no actions.
School of Cyber Science and Technology, University of Science and Technology of China · College of Computing and Data Science, Nanyang Technological University