BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks

Jeong-Yoon Kim; Tae-Yoon Lee

cs.CLAug 27, 2026

BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks

Authors: Jeong-Yoon Kim, Tae-Yoon Lee

Abstract

Operational facilities accumulate extensive local telemetry, but agent-ready interactive evaluations with source-verifiable answers are costly to construct from such archives. Site-specific measurement histories also motivate a local conversion workflow that can operate where raw operational data must remain under organizational control. We introduce BTS-AgentBench, a deterministic compiler that transforms read-only telemetry and equipment metadata into tool-backed, multi-turn agent tasks, phase-level targets, evidence and executable evaluation contracts. The BTS release includes 532 episodes across nine families; a second-corpus port produces 204 XAI4HEAT episodes. Frozen-input replay reproduces released artifacts byte-for-byte, and independent raw-source checks cover all 89 BTS and 41 XAI4HEAT test episodes. Replayable provenance links 1,170 trial identities to recorded execution and scorer conditions, while 1,125 completed traces retain model responses and tool results. An obligation-level comparison of 216 scored failures found that linked records exposed additional recorded failed obligations in 202 cases (93.5%), including tool arguments, stream identity, state carryover and missing interaction steps. The pipeline thereby supports source-grounded task generation, exact reconstruction and auditable diagnosis of agent executions. Code, released benchmarks, and reproduction instructions are available at https://anonymous.4open.science/r/telemetry-benchmark-reproduction-3D68

Figures & tables

Raw building telemetry Equipment/point metadata and timestamped raw values; e.g., BTS_C Zone 005,

12.9457

at 2022-02-03 07:03:23.640 UTC.

\Downarrow

Telemetry Adapter Corpus-specific parsing and mapping to normalized site, stream, point, equipment/location, time, value, and source-lineage fields.

\Downarrow

Corpus Contract Declared identity, UTC/window, missing-value, tool, aggregation, nearest, quality, and tie semantics.

\Downarrow

Deterministic episode compiler Select source-grounded task; compose dialogue and state revisions; execute tools to derive phase golds and evidence.

Figure 1: Telemetry-to-benchmark pipeline. The concrete source value follows the final-artifact worked example in Table 14 ; numeric values are raw and unitless unless the source supplies authoritative units.

Component	Input / output	Role in the benchmark
Compiler	Raw telemetry + metadata $\rightarrow$ episode contract, tool-derived golds, evidence	Builds the benchmark before evaluation.
Evaluator	Agent trace + frozen episode contract $\rightarrow$ scores	Checks observable behavior and task completion offline.
Controller	Revealed dialogue + public tools + its own state/results $\rightarrow$ trace	Produces deterministic audit traces for programmatic-compressibility analysis.

Table 1: Separation between benchmark construction and scoring.

Source category	Candidate space	Retained tasks
BTS metadata streams	19,665	–
Tool-ready points	14,422	–
Point lookup/disambiguation	4,263	60
Day mean lookup	2,193,431	60
Relative 24h mean lookup	2,193,431	60
Window mean lookup	315,929	53

Table 2: Source pool and retained static task set derived from BTS data before episode lifting. Candidate counts are induced by the tool-store construction and task generator; the retained tasks use a 356/87/89 train/dev/test split.

1	Raw building telemetry: Site_Caa/2254.pickle contains stream c24589e8...9d20 ; at 2022-02-03 07:03:23.640 UTC its raw value is 12.9457.
$\downarrow$
2	Point/equipment catalog: the stream maps to BTS_C, Zone 005, air differential pressure.
$\downarrow$
3	Source-grounded task candidate: resolve that descriptor, then request the exact observation at the recorded timestamp.
$\downarrow$
4	Episode contract: hide the time initially; retain the resolved stream and bind exact-value and evidence targets.

Figure 2: End-to-end construction of test_timestamp_value_lookup_00051 . The dialogue revision and later reportability decision are derived from the source task and deterministic tool results; evaluated agents produce their own calls and responses.

Phase type	Evaluated role
Clarification	Recover missing site or time context before querying.
Initial answer	Execute the source telemetry task with read-only tools.
Goal revision	Reuse resolved state under a revised request.
Timestamp policy	Decide exact, nearest, or insufficient-time reporting.
Quality commitment	Answer or abstain under quality evidence.
Rationale follow-up	Explain a quality or reportability decision.

Table 3: Canonical phase vocabulary used to evaluate converted agent episodes.

Corpus	Model	Trial P/F/U/B	$\mathrm{pass}^{3}$ P/F/R
BTS	GPT	229/30/2/6	76/10/3
BTS	Gemini	138/88/12/29	35/35/19
BTS	Opus	238/21/0/8	67/17/5
XAI4HEAT	GPT	81/42/0/0	27/14/0
XAI4HEAT	Gemini	109/13/0/1	30/10/1
XAI4HEAT	Opus	94/28/0/1	26/14/1

Table 4: Paid executions evaluated with the released scorer. Trial P/F/U/B: verified PASS, FAIL, UNVERIFIED, BLOCKED/UNSCORED. Group P/F/R: PASS, FAIL, UNRESOLVED. Each model has 267 trials/89 groups on BTS and 123 trials/41 groups on XAI4HEAT.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Axis	Reviewer	Valid	Invalid	Undetermined
Semantic	R1	86	1	0
Semantic	R2	87	0	0
Contract	R1	0	2	85
Contract	R2	86	1	0

Appendix

Table 5: Recorded human-review judgments. R2 used a selected-check interpretation of semantic validity; R1 additionally retained unresolved contract-level questions.

Raw telemetry

\rightarrow

Telemetry Adapter

\rightarrow

Corpus Contract

\rightarrow

Episode compiler

\rightarrow

Replayable artifact

\rightarrow

Evaluator

\rightarrow

Optional execution

Equipment and point metadata, timestamps, values

Parse and map records; retain lineage

Identity, time, tool and policy rules

Compile dialogue and execute gold operations

Dialogue, calls, golds, evidence

Rescore traces and verify contracts

Agent sees revealed interaction only

Appendix

Figure 3: Construction and replay boundaries supplementing the main-text pipeline. Gold construction precedes optional execution.

Field	Value
Released BTS benchmark rows	532 total rows
Released BTS split sizes	train 356 / dev 87 / test 89
Families in the final BTS release	point disambiguation, day mean, relative 24h mean, window mean, window pairwise compare, window rank, timestamp value, timestamp nearest, quality gate
Contract preflight	0 findings under the coded executable-contract checks
Exact replay	2 independent raw-to-episode builds with exact logical and release matches
Replay boundary	raw telemetry/metadata $\rightarrow$ tool store $\rightarrow$ static tasks $\rightarrow$ typed contracts $\rightarrow$ operator-facing episodes $\rightarrow$ scored release

Appendix

Table 6: Released BTS artifact snapshot used by the paper.

Family	Rows	Clarify	Revision	Quality	Evidence
Point disambiguation	10	10	10	10	10
Day mean	10	10	10	10	10
Relative 24h mean	10	10	10	10	10
Window mean	10	10	10	10	10
Pairwise compare	10	10	10	10	10
Window rank	10	10	10	10	10

Appendix

Table 7: Interaction obligations encoded in the final test contracts. Explicit reporting/policy branches occur in 54/89 rows; clarification is absent in nearest-timestamp rows by construction.

Contract view	Obligations retained or removed on the 89-row test panel
Static source task	Source identity and base query; no generated dialogue-state obligation.
Single-turn task	Base operation/result; omits 79 clarification slots, 304 revision turns, and 98 post-answer turns.
Multi-turn without revisions	Dialogue remains, but removes 89 row-level revision obligations and downstream state-update checks.
Multi-turn without policy branch	Removes explicit reporting/policy branches in 54 rows.
Full compiled episode	Clarification where required, revisions, quality/reporting, commitment, and evidence contracts.

Appendix

Table 8: Structural contract projections of the 89-row test panel.

Stage	Raw corpus	External LLM	Local execution
Parse and normalize	Yes	No	Yes
Tool store and task generation	Yes	No	Yes
Episode and gold compilation	Yes	No	Yes
Deterministic replay/scoring	Yes	No	Yes
Optional frontier evaluation	No	Optional	Separate from benchmark construction

Appendix

Table 9: Dependency audit after source acquisition. Construction, replay, and deterministic scoring use local Python/SQL components.

Benchmark	Applicable	Direct (coverage)	Fully/partial/not direct
BTS-AgentBench (27 types)	1,352	1,263 (93.4%)	26/0/1
$\tau^{3}$ Airline (6 types)	112	20 (17.9%)	0/1/5
ToolSandbox Base129 (5 types)	268	209 (78.0%)	2/2/1

Appendix

Table 10: Within-benchmark direct task-contract coverage. Denominators and taxonomies are benchmark-specific; percentages do not compare evaluator accuracy or construct validity.

Benchmark / case	Varied task-facing behavior	Compliant	Violating
$\tau^{3}$ / task 0	Refuse / agree to ineligible cancellation	1.0	1.0
$\tau^{3}$ / task 3	Verify membership / omit profile lookup	1.0	1.0
$\tau^{3}$ / task 6	Refuse / offer disallowed insurance	1.0	1.0
ToolSandbox / city	Ask for location / assert unsupported city	1	1
ToolSandbox / holiday	Ask which date / assert unsupported interval	1	1
ToolSandbox / distance	Ask for location / assert unsupported distance	1	1

Appendix

Table 11: Six selected official-evaluator counterexamples. Each pair preserves the evaluator-observed primary condition while changing the stated task-facing behavior.

Check	Verifies	Boundary
Input checksum	Exact raw and retained files	Not upstream collection validity
Selection identity	Unique regenerated retained candidates	Not random representativeness
Static/final equality	Every released object and byte	Not provider repeatability
Contract preflight	Schema, gold, evidence, verifier, runtime alignment	Not every possible annotation error

Appendix

Table 12: Scope of the deterministic validation checks.

Typed construction stage	Rows modified
Timestamp policy / nearest wording	120 / 60
Timestamp-value / alternate-point revision	60 / 60
Clarification / goal revision	60 / 293
Multi-axis composition	472
Terminal reporting commitment	532
Controller-proxy transformation	0

Appendix

Table 13: Overlapping typed-stage applications. Counts show how often each typed construction stage is applied.

Compilation stage	Deterministic rule and emitted artifact
Raw observation	Site_Caa.zip/Site_Caa/2254.pickle contains stream c24589e8_a1f3_4529_b409_5a56761c9d20 ; its value at 2022-02-03T07:03:23.640Z is $12.9457$ .
Adapter output	The stream maps to Air_Differential_Pressure_Sensor on BTS_C Zone 005 ; its source value remains unitless.
Corpus Contract	Stable stream identity; UTC timestamps; half-open calendar windows; deterministic exact/nearest lookup and quality operations; no engineering unit without authoritative source metadata.
Selected static task	timestamp_value_lookup ; resolve Zone 005 and query the source timestamp, preserving stream, value, and evidence.
Mode and visibility	The missing-time request selects clarify_time ; the simulator supplies “07:03:23.64 UTC on February 3, 2022.”
Turns and state updates	Revise the request to the nearest reading around 07:03, assess the week beginning January 31, then decide whether to report, abstain, or re-clarify.

Appendix

Table 14: Field-level compilation of test_timestamp_value_lookup_00051 , complementing the main-text construction diagram.

Stage	Source-derived contract and expected state/result
Static task	Resolve BTS_C / Zone 005 / air differential pressure; query the exact timestamp 2022-02-03 07:03:23.640 UTC.
Initial dialogue	“What was the air differential pressure reading on Zone 005 in BTS_C?” The timestamp is missing, so ask the operator; the revealed clarification supplies it.
Initial execution	resolve_point returns exactly one stream. Exact lookup_observation returns raw value 12.9457 at the requested timestamp.
Revision	“Keep the same signal and site … around 07:03 … give me the nearest available reading.” Retain site, entity, point class, and stream; replace exact time with the 07:03 probe and nearest policy.
Nearest execution	Exact probe at 07:03:00 has no match. Nearest lookup returns the 07:03:23.640 observation, value 12.9457, offset 23.64 seconds; cadence-derived tolerance is 299.761 seconds, so status is acceptable.
Quality follow-up	The operator asks about the week beginning January 31. Retain the resolved stream; add the half-open UTC week [2022-01-31, 2022-02-07). Quality output is coverage 1.0, gap ratio 1.0563, decision answer .

Appendix

Table 15: Worked final-artifact compilation: source identity and observation, dialogue revision, deterministic tool results, and derived reporting/evidence golds.

Independent source check	Records	Matches
Single-stream means	80	80
Pairwise comparisons (40 mean operands)	20	20
Exact/nearest observations	72	72
Window quality statistics	89	89
Total phase records	261	261

Appendix

Table 16: Independent raw-source checks across all 89 test episodes, with zero mismatches. Means use UTC half-open windows; numeric statistics match at four decimals, timestamps exactly, and nearest offsets within 1 ms. Archive lookup uses the frozen raw index.

Recorded observation	Count	Rate
Final-answer component has a failure	93	43.1%
Final-answer component passes	123	56.9%
Answer-only diagnosis complete	14	6.5%
Linked record adds failure information	202	93.5%
Substantive phase failure	181	83.8%
Execution-check failure	132	61.1%

Appendix

Table 17: Failure observability for 216 scored FAIL trajectories.

Fixed query	Eligible	A	B	C
Quality stream ID	1103	106	402	1103
Quality window endpoints	1103	0	0	1103
Lookup stream ID	774	84	250	774
Lookup value literal	774	391	488	774
Lookup timestamp	774	23	76	774

Appendix

Table 18: Recorded execution fields recoverable from the final answer (A), all public assistant responses (B), or tool records (C). The question and denominator are fixed within each row; text retrieval uses exact literal matches.

Task family	Scored	FAIL	FAIL (%)
Day mean	170	7	4.1
Relative 24h mean	179	13	7.3
Window mean	149	6	4.0
Point disambiguation	79	9	11.4
Nearest timestamp	143	38	26.6
Exact timestamp value	145	50	34.5

Appendix

Table 19: Observed full-trajectory outcomes by task family across the scored paid-run archive.

Layer	BTS / shared role	XAI4HEAT result	Reuse status
Source organization	Graph metadata + per-stream archives	Row-oriented SCADA tables + substation/channel mapping	Corpus-specific source adaptation
Normalized interface	Site, stream, point, equipment, time, value, lineage	Same normalized fields	Unchanged
Candidate builders	Shared telemetry task builders	Five supported temporal families	Unchanged; family selection configured
Interaction construction	Typed phases, revisions, reportability	Localized single-stream/reportability adaptation	Two corpus-specific modules
Protocol/runtime	Slot-consumption dependencies and read-only tools	169/169 clarification edges slot-consistent; 134/134 independent-order controls accepted	Unchanged
Scoring/alignment/replay	Deterministic primary scorer and trace alignment	204/204 canonical scorer pass; shared replay interface	Unchanged

Appendix

Table 20: Cross-corpus reuse on XAI4HEAT. Porting localizes corpus-specific interaction and reportability semantics in two adaptation modules while reusing the shared protocol, runtime, scoring, alignment, and replay stack.

Audit item	Final result
Regenerated artifact	204 rows; 132/31/41 train/dev/test
Row/split/family identity vs. archive	204/204 unchanged
Provenance-only augmentation	70 rows
Rows affected by XAI adaptation modules	134 rows
Rows changing dialogue, phase, state, verifier, or tool-call fields	44 rows
Clarification protocol edges	169/169 slot-consistent

Appendix

Table 21: Final XAI4HEAT porting audit. The shared stack is reused with a localized corpus-specific interaction/reportability adaptation layer.

Benchmark Construction

Benchmark construction develops tasks, datasets, environments, and protocols that make AI capabilities measurable and comparisons reliable.

See all 68 papers

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
May 27, 2026Tomer Keren, Nitay Calderon, Asaf Yehudai +3Computer-Use Agent Benchmarks Tool-Use Evaluation
Faculty of Data and Decision Science Technion · IBM Research
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jun 22, 2026Jincheng Zhong, Weizhi Wang, Che Jiang +5LLM Agent Evaluation Tool-Augmented Language Model Agents
Horizon Research, Frontis.AI
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Computer-Use Agent Benchmarks Terminal-Agent Benchmarks
University College London · Nanjing University · Tencent

May 27, 2026J/K move · Enter open · S save

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz+2

Faculty of Data and Decision Science Technion · IBM Research

Computer-Use Agent Benchmarks Tool-Use Evaluation Tool-Augmented Language Model Agents Benchmark Construction LLM Agent Evaluation Synthetic Benchmark Generation+1

Open paper AskPDF arXiv

As agent capabilities advance, existing benchmarks, such as

τ^2

-Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labor-intensive. Moreover, the standard approach, in which scenarios are first written in natural language and then mapped to tool sequences, captures only a narrow subset of the tool-use patterns agents exercise. In this paper, we address these problems by reversing the task construction process. We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage. TASTE utilizes an Adaptive Contrastive

n

-gram model trained on LLM-judged validity signals. This enables sampling valid tool sequences that cover a vast range of tool combinations. TASTE then selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through iterative difficulty evolution. Using TASTE, we construct

τ^c

-Bench, a challenging extension of the three domains of

τ^2

-Bench. We evaluate

11

agent/user LLM pairs and find that models nearly saturating

τ^2

-Bench suffer severe performance drops on our tasks (e.g., Gemini-3-Flash falls from

0.82\!-\!0.94

0.28\!-\!0.61

). Beyond increasing difficulty, our generated tasks more than double the number of unique tool combinations agents must execute. Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability. By automating the generation of difficult, high-coverage benchmarks, TASTE enables continuous, scalable evaluation of future agents.

Abstract

Benchmark Construction

Explore similar work

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks