cs.IRSep 23, 2026

System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer

Authors: Dmitrij Żatuchin

Organizations: Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Audits of AI visibility summarise the brand recommendations of deployed language models into per-system profiles. We test whether such a profile describes the system on one corpus of 6,475 stored responses (6,324 analysable) collected between December 2025 and February 2026 from five deployed endpoints across gift-recommendation, corporate-reputation and category-ownership queries. The collection harness cut many answers short: 83.1% of Gemini 3 Flash answers in category ownership end mid-sentence under a 1,024-token output cap. With every answer cut to its first 800 characters, a character n-gram classifier cross-validated by prompt attributes one response to GPT-5.2, Gemini 3 Flash, Gemini 3 Flash with search, Grok or Perplexity sonar-pro with 97.84% accuracy (5,028 responses, 383 prompts, majority class 31.5%, 30 split seeds). Length alone falls to the majority rate, 24 formatting statistics reach 95.79%, and masking brand names and capitalised tokens leaves 97.72%. Held-out query conditions keep 97.43% weighted by size and 88.0% unweighted; in a retrieval-grounded arm that changes the harness, no Grok answer is attributed to Grok (0/120). Aggregated into 50 model-by-domain-by-condition units, twelve behavioural features separate four systems at 66.53% under grouped cross-validation, against a label-permutation null with mean 33.71% and 95th percentile 46.0%. Across domains the aggregate profile fails: a forest trained on category-ownership units assigns all 22 gift units to the wrong system, consistent with a reversal in brand volume (8.41 against 0.94 brands per response in gifts, 3.01 against 3.91 in category ownership), while single responses transfer at 89.92% balanced accuracy. The surface form of an answer carries the system across the query domains tested; aggregated brand behaviour does not, and the uncrossed design cannot separate the system from the domain or the harness.

Figures & tables

SubsetSystemsResp.PromptsCond.Incomplete (%)
Gifts, ChristmasGem., GPT, Grok904869.2 / 3.5 / 0.0
Gifts, Valentine’sGem., GPT, Grok150550.0 / 0.0 / 0.0
Reputation, groundedGem.-search, Grok, Perpl.36012010.8 / 0.0 / 0.0
Reputation, parametricGem., GPT1,296360170.8 / 2.2
Category ownershipGem., GPT, Perpl.3,614250583.1 / 2.2 / 0.5
Totalfive endpoints6,32474318 
Table 1: The corpus as read from the raw files. Responses are the analysable rows, after the 136 empty and 15 colliding-index rows are set aside. Incomplete is the share of each system’s responses whose text stops before a finished sentence (Section 3.3 gives the test), listed per system in the order of the Systems column.
TaskResp.GroupsMajorityPrefixSDStored
Gifts, 3 systems1,0541334.2%92.37%0.2494.07%
Reputation grounded, 3 systems36012033.3%99.91%0.1799.86%
Category ownership, 3 systems3,61425034.6%100.0%0.099.97%
Pooled, 5 systems5,02838331.5%97.84%0.2198.14%
Pooled, complete responses only3,92138336.7%  97.86%
Reputation parametric, 2 systems1,29636050.0%98.55%0.2898.52%
Table 2: Response-level attribution, mean of 30 split seeds of stratified grouped 5-fold cross-validation with the prompt as the group. Prefix: every text cut to its first 800 characters. Stored: the full stored text. The parametric reputation row is excluded from the headline because its Gemini arm is truncated in storage to a median of 104 characters.
ConditionResp.StoredPrefixGem.GPTGrokPerpl.
Val. boyfriend3063.3363.3310/107/102/10–
Val. girlfriend3070.076.6710/108/105/10–
Val. husband3063.3360.010/108/100/10–
Val. partner3070.060.010/104/104/10–
Val. wife3050.076.6710/109/104/10–
Xmas daughter120100.0100.040/4040/4040/40–
Table 3: Leave-one-condition-out at the response level: each collection condition held out of training in turn, accuracy (%) on the stored text and on 800-character prefixes, and correct attributions per system on the prefixes; in the grounded row the Gem. column is Gemini with search (Val. Valentine’s gifts, Xmas Christmas gifts, Rep. reputation, Cat. category ownership). † The grounded arm is the only held-out condition that changes the harness; its Gemini-with-search label is absent from training, so its accuracy is undefined.
Figure 1: Reliability of the two classifiers. Left, the response-level character nn -gram model on 800-character prefixes, inside the training distribution (grouped cross-validation) and on held-out conditions. Right, the aggregate-level random forest under grouped cross-validation. Points on the diagonal mean that a stated confidence matches the observed accuracy at that confidence. Marker area counts predictions in the bin, and the grouped curves pool the predictions of 30 split seeds, so each response or unit contributes 30 times.
FeaturesStored text800-char prefixComplete only
Character nn -grams98.14 (0.09)97.84 (0.21)97.86 (0.1)
Character nn -grams, masked98.15 (0.07)97.72 (0.25)97.73 (0.06)
24 formatting statistics98.41 (0.09)95.79 (0.16)97.95 (0.25)
Length, logistic regression37.0331.7238.6
Length, random forest36.9331.7241.56
Length, gradient boosting38.7931.5541.84
Table 4: What carries the response-level signal. Pooled five-system task, 5,028 responses (complete only: 3,921), mean over 30 split seeds of prompt-grouped 5-fold cross-validation; SD in parentheses. The formatting forest’s seed varies with the split seed.
Figure 2: Grouped cross-validated accuracy of the pooled five-system task by feature set, on the stored text, on 800-character prefixes and on complete responses only. Length is informative only while the text still shows where an answer stopped.
Figure 3: Feature importance for the four-system aggregate classifier: mean decrease in impurity on the full fit, and permutation importance on held-out folds averaged over 30 split seeds with the forest seed varied (bars one SD).
Figure 4: Four-system accuracy over split seeds on the feature matrix the earlier version computed, under the ungrouped protocol that produced its headline (100 seeds) and the grouped protocol it described (30 seeds). The line marks the published 68.0%, the ungrouped result at seed 42.
TrainTestTrain unitsTest unitsCorrectNote
GiftsCategory22104 
CategoryGifts10220every unit assigned to the other system
ReputationCategory2105predicts one label for all ten
CategoryReputation1022two test units
Table 5: Cross-domain transfer at the aggregate unit, Gemini 3 Flash against GPT-5.2, two classes. Features are the six brand-volume and concentration features defined in both domains; all twelve give the same four results. Each row is one forest fit.
Figure 5: The same cross-domain question at two units of observation, Gemini 3 Flash against GPT-5.2, chance 50%. Left, aggregated behavioural profiles: units correct out of units tested for each of the four train and test pairs, one forest fit each. Right, single responses on the gift and category-ownership pairs, on the stored text, on 800-character prefixes and on masked prefixes.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

SystemPrecisionRecallF1Units
Gemini 3 Flash0.6430.7530.69417
GPT-5.20.7260.7840.75417
Grok0.6960.5330.60411
Perplexity (sys.)0.3840.2530.3055
Macro0.6120.5810.58950
Appendix
Table 6: Per-system precision, recall and F1 for the four-system aggregate random forest, pooled over 30 split seeds of stratified grouped 5-fold cross-validation. Support is aggregated units.
FeatureGemini 3 FlashGPT-5.2GrokPerplexity (sys.)
Mean brand count μb\mu_{b}6.8011.7975.3042.897
Brand-count CV0.4401.0630.5730.594
Brand abstinence α\alpha0.0550.2920.1300.082
Gini GG0.5640.5180.4800.676
HnormH_{\mathrm{norm}}0.8180.6620.8330.722
Top-brand share τ\tau0.1630.4730.3010.211
Appendix
Table 7: Unit-averaged feature values by system, four-system matrix, 50 units, computed from the raw responses under the definitions of Section 4.3 . Grok units are all gift units and Perplexity units all category-ownership units, so these columns mix system and domain.
Train →\rightarrow testStoredPrefixCompleteMasked prefixECE, prefix
Gifts →\rightarrow Category93.6784.4396.2490.170.1156
Category →\rightarrow Gifts94.4495.4199.0395.970.0763
Mean94.0689.9297.6393.070.096
Appendix
Table 8: Response-level cross-domain transfer, Gemini 3 Flash against GPT-5.2, balanced accuracy (%) and expected calibration error. One fit per pair and variant. Masked: dictionary brand spans, capitalised tokens, URLs and digits replaced.
QuantityEarlierCorrectedReason
Raw corpus size6,340 responses6,475 on disk, 6,324 analysablethe loader reached 6,115 rows; 6,340 is not reachable from the files
Gift subset, raw nn1,2791,054every file present was read; 1,279 exceeds the design ceiling
Category subset, raw nn3,7503,750 stored, 3,614 analysable136 empty responses are missing data
Grounded reputation armnot reported360 responses in 72 filesrecovered from the loader gate
Aggregated units per subset30 / 5 / 1533 / 2 / 15recomputed; the Reputation row also listed a system with no rows there
Headline four-system accuracy68.0%66.53% (SD 3.23, 30 grouped seeds)the published value was an ungrouped split at one seed; grouped on the earlier matrix gives 68.13%
Appendix
Table 9: Earlier value against corrected value. Every corrected value is produced by the rerun described in Section 4 .
More in this topic

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

Explore similar work

CardsList
Jun 24, 2026

How Large Language Models Source Brand Reputation Across Languages and Markets

When a large language model (LLM) answers a question about a company, it grounds the answer in retrieved web sources, and those sources decide what the model says. Most analysis of AI brand visibility looks at the answer text. This study looks one step earlier, at the citations. We merge three Rankfor.AI datasets covering 128 brands across 12 home markets and 13 languages, and analyse 167,551 URL-grounded citations (189,974 total attribution rows). We classify each citation by domain and source type and measure where AI gets its brand information, by language and by market. Four patterns hold. First, AI grounds brand answers overwhelmingly in third-party sources: 85.7% of citations point to sites the brand does not own, against 14.3% owned. Second, the source base is concentrated and long-tailed: 80% of citations come from about 18% of domains, fitting a Zipf law (alpha = 0.86, R^2 = 0.983). Third, one reference site dominates almost everywhere: Wikipedia is the most-cited domain in 11 of 12 languages, the exception being Lithuanian, where the business daily vz.lt edges it (4.38%). Fourth, the source mix is market-specific at the margin: for 46 Polish national brands the most-cited domain is YouTube, and four HR and careers portals supply 637 citations against 297 for Polish Wikipedia, about twice as many.

Dmitrij Zatuchin

Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.

Multilingual Language ModelsClaim Attribution in Language ModelsLLM Grounding
AskPDF
Jun 22, 2026

The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languages

Large language models (LLMs) increasingly mediate how people form impressions of organisations, yet most monitoring is done in English, assuming an English query returns a representative picture. We measure how far that holds. We queried three grounded LLMs (GPT-5.4, Gemini 3.1 Pro, Perplexity Sonar Pro) about 66 brands from eleven Northern, Baltic, and Central European markets, in twelve languages across four families (Germanic, Uralic, Baltic, Slavic), generating 35,640 responses. Multilingual embeddings (BGE-M3) allow cross-language comparison without translation. Three results emerge. First, AI-constructed reputation is language-bound: mean cross-language cosine similarity is 0.825, same-family responses are more similar than cross-family (0.844 vs 0.820; d = 0.31), and sentiment varies by language (F = 268.5, eta^2 = 0.077), with Uralic and Baltic languages most positive and Germanic, including English, most critical; clustering recovers the Slavic and Baltic families (cophenetic 0.915). Second, query language shifts which brands are recommended far more than how they are described: moving from an English query to a brand's home language raises recommendation share by 0.80 for local champions but only 0.15 for global multinationals (t = -8.84, p < 0.001), with no comparable reversal in sentiment. An English-only audit therefore understates a local champion's AI visibility. Third, response stability varies more with model choice than with language (eta^2_model = 0.32 vs eta^2_language = 0.01, on a five-iteration replication over a 20-brand subset). These results indicate that English-only AI reputation monitoring leaves a measurable language blind spot, concentrated in the visibility of locally headquartered brands.

Dmitrij Żatuchin

Estonian Entrepreneurship University of Applied Sciences · Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.

Cultural Bias in Language ModelsLinguistic Bias in Language ModelsCross-Lingual Consistency in Language Models
AskPDF
Jul 26, 2026

Who Gets Named: Citation Type Predicts Individual Naming by Grounded Language Models, and a Roster Instrument Captures 0.5% of It

Prior work on AI brand visibility measures the firm: does a model recommend a company, and does that track its reputation. This study asks the question one level down, in categories where the buyer picks a person. It issued 2,400 grounded API calls in one two-hour window on 24 July 2026: 120 buyer-intent prompts, four models (GPT-5.6 Sol, Gemini 3.6 Flash, Perplexity Sonar Pro, Grok 4.5), five iterations each, four European markets and five query languages. Every response was coded for whether it named an individual professional, by a rule cascade that never consults a roster and that drops detections resolving to a same-named American city (precision 96.9%, recall 61.7%, so every rate below is a lower bound). All inference corrects for clustering within prompt: intraclass correlation 0.258, effective n 407 against a nominal 2,400. Models named an individual in 25.8% of responses. Category dominates: real estate 35.4% and car dealerships 32.9% against insurance 9.1% (chi-square 159.3, p = 5.8e-8 after correction). Models differ four-fold, from Grok 38.0% to Gemini 9.3%. Citation type predicts naming and citation volume does not: naming responses cite the individual's own site 2.6 points more often (95% CI +1.4 to +3.9) and category portals 4.3 points more often, and cite firm-owned pages at the same rate (44.1% against 45.5%). On nine matched translation pairs, English prompts named an individual in 36.7% of responses against 15.6% for the same question in the local language (OR 3.14, clustered p = 0.074, so the direction is clear and the design cannot close it). A 939-person roster built from public LinkedIn search matched 128 of 27,293 name-shaped mentions (0.47%), 26 of the 939 people were ever named, and the roster-derived rates of 0.0% to 25.4% measure that overlap. Roster-based measurement of individual AI visibility sees a small and unrepresentative slice of what models do.

Dmitrij Żatuchin

Rankfor.AI OÜ, Tallinn · Rankfor.AI OÜ, Tallinn, Estonia

Generative AI EvaluationLanguage Model Auditing
AskPDF