Point-in-Time Encoders for Earnings Calls
Organizations: Pratt School of Engineering, Duke University, Durham, NC, USA
Abstract
Text encoders turn earnings calls into vectors for search, for language models that answer from retrieved passages, and for forecasting models. A live retriever can use an encoder trained once on all available text, as finance retrievers are, but a backtest needs an encoder that read no text written after the date of each vector. To address this look-ahead bias, from 412,248 calls we train vintages under four pretraining recipes, with yearly cutoffs from 2010 to 2024, and each vintage reads only text dated on or before its cutoff. We then fine-tune each vintage on the question-answer pairs of the calls themselves. The vintages we train on calls alone, with 33.8M parameters, retrieve answers on held-out calls better than the general retrievers we tested in every test quarter. Their vectors also raise by about half the rank correlation between next-day returns and forecasts built from past returns and firm size. When each vintage continues the previous one, a later vintage retrieves better on earlier quarters than the vintage available at the time, and a point-in-time language model that answers from the passages the later vintage retrieves inherits the gain. The 2024 vintage of our families also forecasts earnings surprises better, although the interval of this gain includes zero on calls that no vintage read. It forecasts returns no better, so point-in-time vintages matter most for retrieval and little for forecasts of returns.
Figures & tables
| Cutoff | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Calls (k) | |||||||||||||||
| Held out (k) | |||||||||||||||
| Chunks (M) | |||||||||||||||
| Tokens (B) | |||||||||||||||
| Pairs (M) |
| Model | Params (M) | Dim. | Pretraining text | Tokens | Tokenizer | Fine-tuning | Cutoff |
|---|---|---|---|---|---|---|---|
| References | |||||||
| BM25 | n/a | n/a | n/a | n/a | word counts | none | n/a |
| bge-small | 33.4 | 384 | general web | not stated | WordPiece | general pairs | unknown |
| bge-base | 109.5 | 768 | general web | not stated | WordPiece | general pairs | unknown |
| bge-large | 335.1 | 1024 | general web | not stated | WordPiece | general pairs | unknown |
| e5-base | 109.5 | 768 | general web | not stated | WordPiece | general pairs | unknown |
| Setting | Pretraining | DAPT | Fine-tuning |
|---|---|---|---|
| Objective | masking, 30% | masking, 30% | InfoNCE, |
| Length | 1.2B tokens | 1.2B tokens | 4,000 steps |
| Batch | 98,304 tokens | 98,304 tokens | 32 calls up to 16 pairs |
| Sequence | 768 tokens | 768 tokens | 256 and 512 tokens |
| Peak rate | |||
| Warmup | 6% | 6% | 5% |
| Difference | Within | 95% interval | Across | |
|---|---|---|---|---|
| Ours, from scratch, minus e5-base | 15 | [ , ] | ||
| Ours, from scratch, minus FinBERT | 6 | [ , ] | ||
| ChronoBERT + contrastive fine-tuning minus ChronoBERT | 15 | [ , ] | ||
| ChronoBERT + DAPT minus ChronoBERT + contrastive fine-tuning | 3 | [ , ] | ||
| Ours, from scratch, minus ChronoBERT + contrastive fine-tuning | 15 | [ , ] | ||
| Ours, chained, minus ours, from scratch | 14 | [ , ] |
| IC, or increment in IC ( ) | Sharpe ratio | Text alone | |||||
| Panel A | Next day | 5 days | Session after | Next quarter | Next day | Session after | Next day |
| Price baselines, IC | ( ) | ( ) | ( ) | ( ) | |||
| Text added to the baselines: increment in IC | |||||||
| Dictionary tone | ( ) | ( ) | ( ) | ( ) | ( ) | ||
| TF-IDF | ( ) | ( ) | ( ) | ( ) | ( ) | ||
| Ours, from scratch | ( ) | ( ) | ( ) | ( ) | ( ) | ||
| Nearest neighbors | Forecast with baselines | |||||||
| Retrieval | Next day | Surprise | Next day | Session after | Next quarter | Surprise | Volatility | |
| Panel A: the 2024 vintage | ||||||||
| Ours, from scratch | ||||||||
| Ours, chained | ||||||||
| ChronoBERT + contrastive fine-tuning | ||||||||
| ChronoBERT | n/a | |||||||
| Own answer (%) | |||||
|---|---|---|---|---|---|
| Family | Vintage | 2024 | Premium | 95% interval | Pos. |
| Ours, chained | [ , ] | 13/14 | |||
| Ours, from scratch | [ , ] | 8/14 | |||
| Ours, chained, own answer removed | n/a | n/a | [ , ] | 12/14 | |