What Does an Observability Foundation Model Know?

Dhyey Dharmendrakumar Mavani; Rian Atri; Tairan Ji

cs.LGOct 4, 2026

What Does an Observability Foundation Model Know?

Authors: Dhyey Dharmendrakumar Mavani, Rian Atri, Tairan Ji

Organizations: Independent · Keiji AI

Abstract

A linear probe can show that a label is recoverable from a model's hidden states, but not whether that goes beyond what the input already reveals, or whether the model uses it. We audit Toto, an observability forecasting foundation model, on the Benchmark of Observability Metrics (BOOM) across five series-disjoint resplits, comparing linear probes on its frozen residual stream with models that read the raw input window and with Toto's architecture stripped of its trained configuration. Short-vs-medium cadence and metric type are more linearly recoverable from Toto's residuals than from the strongest raw-window model in every resplit (macro-F1 0.766 vs. 0.633 and 0.545 vs. 0.498). Domain is nearly tied, and series cardinality is recovered far better from the raw window. MOMENT-base shows related cadence, metric-type, and domain readouts. Recoverability is not use: exchanging Toto's residuals with those of high-burst donors moves a future-burstiness readout as intended but does not make forecasts consistently burstier than a randomized donor. A BOOM-trained coordination probe has negative zero-shot R^2 on the tested external benchmarks. We report each label against its strongest baseline.

Figures & tables

Panel of Figure 1 — Figure 1 : Held-out test macro- $F_{1}$ of Toto, its baselines, and the shuffled-label floor for each taxonomy label (means $\pm$ half-widths over five resplits; § 4 ). The dashed line marks Toto’s mean; the orange square is the strongest raw-window model for that label.

Panel of Figure 2 — Figure 2 : Method. (a) Linear probes on frozen Toto residual views (pretrained backbone, or the randomly initialized and block-permuted backbone baselines) versus the input baselines (raw-window models and a six-statistic probe), compared as paired gaps on held-out series. (b) Layer-11 donor exchange, read by the probe and the forecast.

A. Input baselines
Label	Toto	Six-stat. probe	Strongest raw-window model	Paired gap	Wins
Cadence	$0.766\pm 0.024$	$0.409\pm 0.047$	FNO $0.633\pm 0.047$	$+0.133\pm 0.063$	5/5
Metric type	$0.545\pm 0.037$	$0.347\pm 0.030$	GBDT $0.498\pm 0.007$	$+0.047\pm 0.036$	5/5
Domain	$0.484\pm 0.011$	$0.371\pm 0.037$	GBDT $0.482\pm 0.040$	$+0.002\pm 0.040$	2/5
Cardinality	$0.471\pm 0.029$	$0.635\pm 0.027$	FNO $0.961\pm 0.016$	$-0.490\pm 0.044$	0/5
B. Backbone baselines and shuffled-label floor

Table 1: Held-out test macro-

F_{1}

on BOOM (means

\pm

half-widths over five resplits). A: Toto against the input baselines. The strongest raw-window model is chosen by mean validation macro-

F_{1}

; the paired gap is Toto minus that model, and wins count the resplits in which it is positive. B: backbone baselines and the shuffled-label floor. All raw-window models are in App. D .

Panel of Figure 3 — Figure 3 : Held-out test macro- $F_{1}$ in each resplit for Toto and the strongest raw-window model (thin lines: solid where Toto is higher, dashed where the raw-window model is higher) and their five-resplit means (bold). Each panel has its own vertical scale.

Target	Fixed stratum	Toto	Random init.	Six-stat.	Wins
Cadence	Infrastructure/gauge	$0.697\pm 0.060$	$0.586\pm 0.121$	$0.485\pm 0.137$	5/5; 5/5
Metric type	Application Usage/Short	$0.502\pm 0.053$	$0.395\pm 0.048$	$0.371\pm 0.034$	5/5; 5/5
Domain	Short/gauge	$0.406\pm 0.073$	$0.350\pm 0.091$	$0.376\pm 0.087$	4/5; 3/5

Table 2: Common-support probes: held-out test macro-

F_{1}

within one fixed stratum of the other two labels (means

\pm

half-widths over five resplits; classes balanced by series). Test support is 11–19 (cadence), 17–26 (metric type), and 11–18 (domain) series per class. Wins are against random initialization and against the six-statistic probe.

Label	MOMENT	Random init.	Six-stat.	Gap to six-stat. (wins)
Cadence	$0.632\pm 0.113$	$0.556\pm 0.105$	$0.401\pm 0.099$	$+0.231\pm 0.044$ (5/5)
Metric type	$0.499\pm 0.022$	$0.452\pm 0.029$	$0.351\pm 0.042$	$+0.148\pm 0.030$ (5/5)
Domain	$0.437\pm 0.014$	$0.409\pm 0.020$	$0.337\pm 0.034$	$+0.100\pm 0.023$ (5/5)
Cardinality	$0.259\pm 0.010$	$0.257\pm 0.018$	$0.668\pm 0.013$	$-0.409\pm 0.012$ (0/5)

Table 3: MOMENT-base held-out test macro-

F_{1}

on the same five resplits (means

\pm

half-widths; views chosen on validation series). The last column is the paired gap to the six-statistic probe and the number of resplits in which it is positive.

Panel of Figure 4 — Figure 4 : Probe check versus forecast endpoint across the residual blend $\beta$ . The shaded band is the probe check (mean $\pm$ half-width), which is the same at every blend; points are resplits, and the line joins the forecast-endpoint means $\pm$ half-widths. The dashed line is parity.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Parameters	Input	Training	Checkpoint selection
CNN	39.5k	masked ctx + coverage	AdamW, 20 ep, batch 16	best val epoch
FNO	105.9k	masked ctx + coverage	AdamW, 20 ep, batch 16	best val epoch
Patch Transformer	246.7k	masked ctx + coverage	AdamW, 20 ep, batch 16	best val epoch
GBDT	–	15 eng. features	HistGradBoost, 300 iter	none (fixed budget)

Appendix

Table 4: Raw-window baseline models. Neural models see the masked full context plus a coverage channel; GBDT sees 15 engineered features.

Target	Held-out support	Toto macro- $F_{1}$ range	Random init. range
Cadence	Domain $\times$ metric type	$0.865\ [0.627,1.000]$	$0.541\ [0.250,1.000]$
Cadence	Domain	$0.654\ [0.513,0.766]$	$0.511\ [0.420,0.575]$
Metric type	Domain $\times$ cadence	$0.258\ [0.130,0.472]$	$0.179\ [0.135,0.224]$
Metric type	Domain	$0.281\ [0.188,0.505]$	$0.220\ [0.161,0.266]$

Appendix

Table 5: Rotated within-BOOM held-out-combination tests. Each bracket spans scores from distinct held-out values across the five resplits; it is not an uncertainty interval.

Blend $\beta$	Forecast burstier	Resplits $>0.5$	Median WAPE ratio
0.25	$0.480\pm 0.086$	1/5	$0.969\pm 0.062$
0.50	$0.430\pm 0.046$	0/5	$0.952\pm 0.108$
1.00	$0.390\pm 0.127$	1/5	$0.903\pm 0.155$

Appendix

Table 6: Toto donor exchange: fraction of the 40 targets per resplit whose forecast is burstier under the high-burst than under the randomized donor, and the median WAPE ratio (high-burst/randomized). Means

\pm

half-widths across resplits; the same triples are used at every blend. Donors come from other series and are not matched on covariates or taxonomy. Probe check:

0.655\pm 0.136

at every blend.

Benchmark set	Entries	Coordination $R^{2}$
FEV	11	$-0.938$ $\pm$	$0.654$
LSTF	4	$-17.668$ $\pm$	$10.896$

Appendix

Table 7: Zero-shot coordination-probe transfer (BOOM-trained probe, no refitting).

R^{2}

is macro-averaged over the datasets of each set within a resplit, relative to a constant-mean reference; means

\pm

half-widths across resplits.

Dynamic label	MOMENT $R^{2}$		Random init.		Six-stat.		MOMENT above six-stat.
Coordination	$0.092$ $\pm$	$0.047$	$0.083$ $\pm$	$0.041$	$0.071$ $\pm$	$0.041$	3/5
Current burstiness	$0.024$ $\pm$	$0.025$	$0.007$ $\pm$	$0.018$	$0.057$ $\pm$	$0.620$	2/5
Future burstiness	$0.015$ $\pm$	$0.027$	$0.002$ $\pm$	$0.021$	$-0.005$ $\pm$	$0.082$	2/5
Shift risk	$-0.006$ $\pm$	$0.067$	$-0.018$ $\pm$	$0.062$	$0.021$ $\pm$	$0.076$	1/5

Appendix

Table 8: MOMENT dynamic readouts on held-out BOOM series. Views are selected on validation

R^{2}

and evaluated once on test series. The near-constant sparsity targets are omitted; their six-statistic

R^{2}

values are unstable. Means

\pm

half-widths (§ 4 ).

Blend $\beta$	Lower future-MAE fraction	Resplits $>0.5$
0.25	$0.410\pm 0.084$	0/5
0.50	$0.365\pm 0.109$	0/5
1.00	$0.405\pm 0.104$	1/5

Appendix

Table 9: MOMENT matched interchange. Each resplit evaluates 40 held-out target/real/matched-null triples with the same triples reused at each blend. Probe check (blend-invariant):

0.590\pm 0.071

, above 0.5 in 5/5 resplits. The lower future-MAE fraction is the forecast endpoint. Means

\pm

half-widths (§ 4 ).

Target	Toto	Random init.	Block perm.
Cadence	$0.766\pm 0.024$	$0.547\pm 0.040$	$0.567\pm 0.020$
Metric type	$0.545\pm 0.037$	$0.398\pm 0.035$	$0.420\pm 0.016$
Domain	$0.484\pm 0.011$	$0.362\pm 0.017$	$0.394\pm 0.011$
Cardinality	$0.471\pm 0.029$	$0.250\pm 0.025$	$0.428\pm 0.127$

Appendix

Table 10: Held-out test macro-

F_{1}

of Toto and all raw-window and backbone baselines over five resplits. Neural raw-window checkpoints are selected on validation data. Means

\pm

half-widths (§ 4 ).

Explore similar work

Dynamical Parameters: An Interpretability Framework for Time-Series Foundation Models
Sep 28, 2026Kang Yang, Gaofeng Dong, Liying Han +1Time Series Foundation Models Hidden States
Department of Electrical and Computer Engineering University of California, Los Angeles
Toto 2.0: Time Series Forecasting Enters the Scaling Era
May 19, 2026Emaad Khwaja, Chris Lettieri, Gerald Woo +10Time Series Foundation Models Time Series Forecasting
Datadog AI Research · Carnegie Mellon University
From Surface Forecasting to Observability Forecasting: A Latent World Model for Cloud-Aware EO Monitoring
Jul 15, 2026Mohanad AlbughdadiTrustworthy Earth Observation Decisions Partial Observability
European Centre for Medium-Range Weather Forecasts

Sep 28, 2026, cs.AIJ/K move · Enter open · S save

Dynamical Parameters: An Interpretability Framework for Time-Series Foundation Models

Kang Yang, Gaofeng Dong, Liying Han, Mani Srivastava

Department of Electrical and Computer Engineering University of California, Los Angeles

Time Series Foundation Models Hidden States

Open paper AskPDF arXiv

This work studies a central gap in interpreting time-series foundation models (TSFMs): a dynamical property may be accessible in a hidden state even when the forecast fails to respond correctly as that property changes. We formalize these properties as Dynamical Parameters, including trend slope, oscillation frequency, and autoregressive dependence. We compare their representation accessibility, measured by recovery from hidden states, with their forecast response, measured by agreement with the expected forecast change. Across nine frozen TSFMs and thirteen laws, 42 of 63 model-parameter cells achieve accessibility above 0.95, whereas their median reference-aligned response relative to the conditional reference is only 0.46. To explain this gap, causal geometry compares the hidden-state change required to produce the reference response with the change induced by the parameter intervention. Directly modifying the hidden state recovers the reference response, but the parameter intervention often moves the state in a different direction. These results show that accessible parameter information need not be expressed in forecasts when input changes miss the required hidden-state direction.

Abstract

Explore similar work

Dynamical Parameters: An Interpretability Framework for Time-Series Foundation Models

Toto 2.0: Time Series Forecasting Enters the Scaling Era

From Surface Forecasting to Observability Forecasting: A Latent World Model for Cloud-Aware EO Monitoring