Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers
Organizations: Rankfor.AI Sp. z o.o., Wrocław, Poland
Abstract
Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replication. The same 162 requests (54 questions in English, Estonian and Finnish about three software categories, four of twelve candidate brands per question, three repeats) were sent to gemini-3.1-pro-preview in three waves within about ten hours on 22 and 23 September 2026, and again on 13 October 2026, 20.3 days after the third wave, with identical request bodies. The plan and its decision rule were frozen, hashed and registered on OSF before the first call. The outcome is a lexical mention label per answer and brand, 648 per wave. The primary forecast, each brand-and-question cell's mention rate over the nine earlier answers, scored Brier 0.0281 and log loss 0.126, against Brier 0.0259 for the six-answer forecast of the 8-hour wave and a registered ceiling of 0.0559. It beat brand history (0.104), the overall rate (0.234) and a constant 0.5 (0.250) on both scores and in all nine question families (exact sign-flip p = 0.0039), so durability holds under the registered rule. A forecast frozen from the first wave alone scored 0.0599, 0.0587 and 0.0604 at horizons of 1.5 hours, 9.6 hours and 20.7 days. Of the 202 cells unanimous in the 8-hour wave, 194 were unchanged at D1 and none reversed. The latest wave alone was nearly as accurate as the pooled history (Brier 0.0292; pooled lower in 5 of 9 families, p = 0.28). Every answer reported the requested model version, and none issued a search query although search was enabled. A blind human check of 96 labels agreed with the rule on 94; both disagreements were adjudicated as rater omissions, so after adjudication the rule matched on 96 of 96. The test covers one model, one interval and twelve known brands, and estimates no decay rate.
Figures & tables
| Wave | Collected (UTC) | Positive labels | Forecast scored | Horizon | Freeze |
|---|---|---|---|---|---|
| v0.2 | 22 Sep 2026, 19:58 to 20:16 | 247 | none (first history) | development wave of another experiment | |
| v0.3 | 22 Sep, 21:41 to 21:54 | 246 | question history (from v0.2) | 1.5 h | manifest hash before collection |
| v0.4 | 23 Sep, 05:51 to 06:09 | 244 | pooled rate, six answers (v0.2, v0.3) | 7.9 h | manifest hash before collection |
| D1 | 13 Oct, 14:03 to 14:18 | 243 | pooled rate, nine answers (v0.2 to v0.4) | 487.9 to 488.2 h (20.3 days) | hash, commit and OSF registration before call 1 |
| Forecast | Role | Brier | Log loss |
|---|---|---|---|
| Pooled cell rate, nine answers | primary | 0.0281 | 0.126 |
| Latest wave alone (v0.4) | comparator | 0.0292 | 0.130 |
| Oldest wave alone (v0.2) | comparator | 0.0346 | 0.146 |
| Question history, frozen at v0.3 | comparator | 0.0604 | 0.238 |
| v0.2 context model, frozen | comparator | 0.0705 | 0.242 |
| Brand history | rule baseline | 0.104 | 0.322 |
| Forecast | History | v0.3 (1.5 h) | v0.4 (9.6 h) | D1 (20.7 days) |
|---|---|---|---|---|
| Question history, frozen | v0.2 | 0.0599 | 0.0587 | 0.0604 |
| v0.2 context model, frozen | v0.2 | 0.0692 | 0.0723 | 0.0705 |
| Pooled cell rate | all earlier waves | 0.0259 (7.9 h) | 0.0281 (20.3 days) | |
| Brand history | all earlier waves | 0.1131 | 0.1051 | 0.104 |
| Overall rate | all earlier waves | 0.2355 | 0.2348 | 0.234 |
| v0.4 \ D1 | 0 | 1 | 2 | 3 | Cells |
|---|---|---|---|---|---|
| 0 | 125 | 3 | 1 | 0 | 129 |
| 1 | 1 | 2 | 0 | 0 | 3 |
| 2 | 0 | 3 | 6 | 2 | 11 |
| 3 | 0 | 0 | 4 | 69 | 73 |
| Language | Answers | Labels | Human positives | Automatic positives | Agreement [95% CI] | Precision [95% CI] | Recall [95% CI] | Disagree- ments |
|---|---|---|---|---|---|---|---|---|
| English | 8 | 32 | 11 | 13 | 30/32, 93.8% [79.9, 98.3] | 11/13, 84.6% [57.8, 95.7] | 11/11, 100% [74.1, 100] | 2 |
| Estonian | 8 | 32 | 15 | 15 | 32/32, 100% [89.3, 100] | 15/15, 100% [79.6, 100] | 15/15, 100% [79.6, 100] | 0 |
| Finnish | 8 | 32 | 11 | 11 | 32/32, 100% [89.3, 100] | 11/11, 100% [74.1, 100] | 11/11, 100% [74.1, 100] | 0 |
| All | 24 | 96 | 37 | 39 | 94/96, 97.9% [92.7, 99.4] | 37/39, 94.9% [83.1, 98.6] | 37/37, 100% [90.6, 100] | 2 |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 802 papers