System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer

Dmitrij Żatuchin

cs.IRSep 23, 2026

System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer

Authors: Dmitrij Żatuchin

Organizations: Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia.

Abstract

Audits of AI visibility summarise the brand recommendations of deployed language models into per-system profiles. We test whether such a profile describes the system on one corpus of 6,475 stored responses (6,324 analysable) collected between December 2025 and February 2026 from five deployed endpoints across gift-recommendation, corporate-reputation and category-ownership queries. The collection harness cut many answers short: 83.1% of Gemini 3 Flash answers in category ownership end mid-sentence under a 1,024-token output cap. With every answer cut to its first 800 characters, a character n-gram classifier cross-validated by prompt attributes one response to GPT-5.2, Gemini 3 Flash, Gemini 3 Flash with search, Grok or Perplexity sonar-pro with 97.84% accuracy (5,028 responses, 383 prompts, majority class 31.5%, 30 split seeds). Length alone falls to the majority rate, 24 formatting statistics reach 95.79%, and masking brand names and capitalised tokens leaves 97.72%. Held-out query conditions keep 97.43% weighted by size and 88.0% unweighted; in a retrieval-grounded arm that changes the harness, no Grok answer is attributed to Grok (0/120). Aggregated into 50 model-by-domain-by-condition units, twelve behavioural features separate four systems at 66.53% under grouped cross-validation, against a label-permutation null with mean 33.71% and 95th percentile 46.0%. Across domains the aggregate profile fails: a forest trained on category-ownership units assigns all 22 gift units to the wrong system, consistent with a reversal in brand volume (8.41 against 0.94 brands per response in gifts, 3.01 against 3.91 in category ownership), while single responses transfer at 89.92% balanced accuracy. The surface form of an answer carries the system across the query domains tested; aggregated brand behaviour does not, and the uncrossed design cannot separate the system from the domain or the harness.

Figures & tables

Subset	Systems	Resp.	Prompts	Cond.	Incomplete (%)
Gifts, Christmas	Gem., GPT, Grok	904	8	6	9.2 / 3.5 / 0.0
Gifts, Valentine’s	Gem., GPT, Grok	150	5	5	0.0 / 0.0 / 0.0
Reputation, grounded	Gem.-search, Grok, Perpl.	360	120	1	0.8 / 0.0 / 0.0
Reputation, parametric	Gem., GPT	1,296	360	1	70.8 / 2.2
Category ownership	Gem., GPT, Perpl.	3,614	250	5	83.1 / 2.2 / 0.5
Total	five endpoints	6,324	743	18

Table 1: The corpus as read from the raw files. Responses are the analysable rows, after the 136 empty and 15 colliding-index rows are set aside. Incomplete is the share of each system’s responses whose text stops before a finished sentence (Section 3.3 gives the test), listed per system in the order of the Systems column.

Task	Resp.	Groups	Majority	Prefix	SD	Stored
Gifts, 3 systems	1,054	13	34.2%	92.37%	0.24	94.07%
Reputation grounded, 3 systems	360	120	33.3%	99.91%	0.17	99.86%
Category ownership, 3 systems	3,614	250	34.6%	100.0%	0.0	99.97%
Pooled, 5 systems	5,028	383	31.5%	97.84%	0.21	98.14%
Pooled, complete responses only	3,921	383	36.7%			97.86%
Reputation parametric, 2 systems	1,296	360	50.0%	98.55%	0.28	98.52%

Table 2: Response-level attribution, mean of 30 split seeds of stratified grouped 5-fold cross-validation with the prompt as the group. Prefix: every text cut to its first 800 characters. Stored: the full stored text. The parametric reputation row is excluded from the headline because its Gemini arm is truncated in storage to a median of 104 characters.

Condition	Resp.	Stored	Prefix	Gem.	GPT	Grok	Perpl.
Val. boyfriend	30	63.33	63.33	10/10	7/10	2/10	–
Val. girlfriend	30	70.0	76.67	10/10	8/10	5/10	–
Val. husband	30	63.33	60.0	10/10	8/10	0/10	–
Val. partner	30	70.0	60.0	10/10	4/10	4/10	–
Val. wife	30	50.0	76.67	10/10	9/10	4/10	–
Xmas daughter	120	100.0	100.0	40/40	40/40	40/40	–

Table 3: Leave-one-condition-out at the response level: each collection condition held out of training in turn, accuracy (%) on the stored text and on 800-character prefixes, and correct attributions per system on the prefixes; in the grounded row the Gem. column is Gemini with search (Val. Valentine’s gifts, Xmas Christmas gifts, Rep. reputation, Cat. category ownership). † The grounded arm is the only held-out condition that changes the harness; its Gemini-with-search label is absent from training, so its accuracy is undefined.

Figure 1: Reliability of the two classifiers. Left, the response-level character $n$ -gram model on 800-character prefixes, inside the training distribution (grouped cross-validation) and on held-out conditions. Right, the aggregate-level random forest under grouped cross-validation. Points on the diagonal mean that a stated confidence matches the observed accuracy at that confidence. Marker area counts predictions in the bin, and the grouped curves pool the predictions of 30 split seeds, so each response or unit contributes 30 times.

Features	Stored text	800-char prefix	Complete only
Character $n$ -grams	98.14 (0.09)	97.84 (0.21)	97.86 (0.1)
Character $n$ -grams, masked	98.15 (0.07)	97.72 (0.25)	97.73 (0.06)
24 formatting statistics	98.41 (0.09)	95.79 (0.16)	97.95 (0.25)
Length, logistic regression	37.03	31.72	38.6
Length, random forest	36.93	31.72	41.56
Length, gradient boosting	38.79	31.55	41.84

Table 4: What carries the response-level signal. Pooled five-system task, 5,028 responses (complete only: 3,921), mean over 30 split seeds of prompt-grouped 5-fold cross-validation; SD in parentheses. The formatting forest’s seed varies with the split seed.

Figure 2: Grouped cross-validated accuracy of the pooled five-system task by feature set, on the stored text, on 800-character prefixes and on complete responses only. Length is informative only while the text still shows where an answer stopped.

Figure 3: Feature importance for the four-system aggregate classifier: mean decrease in impurity on the full fit, and permutation importance on held-out folds averaged over 30 split seeds with the forest seed varied (bars one SD).

Figure 4: Four-system accuracy over split seeds on the feature matrix the earlier version computed, under the ungrouped protocol that produced its headline (100 seeds) and the grouped protocol it described (30 seeds). The line marks the published 68.0%, the ungrouped result at seed 42.

Train	Test	Train units	Test units	Correct	Note
Gifts	Category	22	10	4
Category	Gifts	10	22	0	every unit assigned to the other system
Reputation	Category	2	10	5	predicts one label for all ten
Category	Reputation	10	2	2	two test units

Table 5: Cross-domain transfer at the aggregate unit, Gemini 3 Flash against GPT-5.2, two classes. Features are the six brand-volume and concentration features defined in both domains; all twelve give the same four results. Each row is one forest fit.

Figure 5: The same cross-domain question at two units of observation, Gemini 3 Flash against GPT-5.2, chance 50%. Left, aggregated behavioural profiles: units correct out of units tested for each of the four train and test pairs, one forest fit each. Right, single responses on the gift and category-ownership pairs, on the stored text, on 800-character prefixes and on masked prefixes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

System	Precision	Recall	F1	Units
Gemini 3 Flash	0.643	0.753	0.694	17
GPT-5.2	0.726	0.784	0.754	17
Grok	0.696	0.533	0.604	11
Perplexity (sys.)	0.384	0.253	0.305	5
Macro	0.612	0.581	0.589	50

Appendix

Table 6: Per-system precision, recall and F1 for the four-system aggregate random forest, pooled over 30 split seeds of stratified grouped 5-fold cross-validation. Support is aggregated units.

Feature	Gemini 3 Flash	GPT-5.2	Grok	Perplexity (sys.)
Mean brand count $\mu_{b}$	6.801	1.797	5.304	2.897
Brand-count CV	0.440	1.063	0.573	0.594
Brand abstinence $\alpha$	0.055	0.292	0.130	0.082
Gini $G$	0.564	0.518	0.480	0.676
$H_{\mathrm{norm}}$	0.818	0.662	0.833	0.722
Top-brand share $\tau$	0.163	0.473	0.301	0.211

Appendix

Table 7: Unit-averaged feature values by system, four-system matrix, 50 units, computed from the raw responses under the definitions of Section 4.3 . Grok units are all gift units and Perplexity units all category-ownership units, so these columns mix system and domain.

Train $\rightarrow$ test	Stored	Prefix	Complete	Masked prefix	ECE, prefix
Gifts $\rightarrow$ Category	93.67	84.43	96.24	90.17	0.1156
Category $\rightarrow$ Gifts	94.44	95.41	99.03	95.97	0.0763
Mean	94.06	89.92	97.63	93.07	0.096

Appendix

Table 8: Response-level cross-domain transfer, Gemini 3 Flash against GPT-5.2, balanced accuracy (%) and expected calibration error. One fit per pair and variant. Masked: dictionary brand spans, capitalised tokens, URLs and digits replaced.

Quantity	Earlier	Corrected	Reason
Raw corpus size	6,340 responses	6,475 on disk, 6,324 analysable	the loader reached 6,115 rows; 6,340 is not reachable from the files
Gift subset, raw $n$	1,279	1,054	every file present was read; 1,279 exceeds the design ceiling
Category subset, raw $n$	3,750	3,750 stored, 3,614 analysable	136 empty responses are missing data
Grounded reputation arm	not reported	360 responses in 72 files	recovered from the loader gate
Aggregated units per subset	30 / 5 / 15	33 / 2 / 15	recomputed; the Reputation row also listed a system with no rows there
Headline four-system accuracy	68.0%	66.53% (SD 3.23, 30 grouped seeds)	the published value was an ungrouped split at one seed; grouped on the earlier matrix gives 68.13%

Appendix

Table 9: Earlier value against corrected value. Every corrected value is produced by the rerun described in Section 4 .

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

How Large Language Models Source Brand Reputation Across Languages and Markets
Jun 24, 2026Dmitrij ZatuchinMultilingual Language Models Claim Attribution in Language Models
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.
The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languages
Jun 22, 2026Dmitrij ŻatuchinCultural Bias in Language Models Linguistic Bias in Language Models
Estonian Entrepreneurship University of Applied Sciences · Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.
Who Gets Named: Citation Type Predicts Individual Naming by Grounded Language Models, and a Roster Instrument Captures 0.5% of It
Jul 26, 2026Dmitrij ŻatuchinGenerative AI Evaluation Language Model Auditing
Rankfor.AI OÜ, Tallinn · Rankfor.AI OÜ, Tallinn, Estonia

Jun 24, 2026J/K move · Enter open · S save

How Large Language Models Source Brand Reputation Across Languages and Markets

Dmitrij Zatuchin

Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Tallinn, Estonia.

Multilingual Language Models Claim Attribution in Language Models LLM Grounding

Open paper AskPDF arXiv

When a large language model (LLM) answers a question about a company, it grounds the answer in retrieved web sources, and those sources decide what the model says. Most analysis of AI brand visibility looks at the answer text. This study looks one step earlier, at the citations. We merge three Rankfor.AI datasets covering 128 brands across 12 home markets and 13 languages, and analyse 167,551 URL-grounded citations (189,974 total attribution rows). We classify each citation by domain and source type and measure where AI gets its brand information, by language and by market. Four patterns hold. First, AI grounds brand answers overwhelmingly in third-party sources: 85.7% of citations point to sites the brand does not own, against 14.3% owned. Second, the source base is concentrated and long-tailed: 80% of citations come from about 18% of domains, fitting a Zipf law (alpha = 0.86, R^2 = 0.983). Third, one reference site dominates almost everywhere: Wikipedia is the most-cited domain in 11 of 12 languages, the exception being Lithuanian, where the business daily vz.lt edges it (4.38%). Fourth, the source mix is market-specific at the margin: for 46 Polish national brands the most-cited domain is YouTube, and four HR and careers portals supply 637 citations against 297 for Polish Wikipedia, about twice as many.

Abstract

LLM Auditing

Explore similar work

How Large Language Models Source Brand Reputation Across Languages and Markets

The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languages

Who Gets Named: Citation Type Predicts Individual Naming by Grounded Language Models, and a Roster Instrument Captures 0.5% of It