Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations

Toby D. Pilditch; Konstantinos Voudouris; Alexandra Abbas; Cozmin Ududec

cs.AIOct 6, 2026

Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations

Authors: Toby D. Pilditch, Konstantinos Voudouris, Alexandra Abbas, Cozmin Ududec

Organizations: UK AI Security Institute

Abstract

Frontier AI evaluations increasingly use open-ended, agentic, long-horizon tasks whose transcripts can span hundreds of pages of outputs and actions from complex multi-agent networks. The observability envelop-the range of what evaluators can reliably infer about an agent's behaviours-is therefore narrowing. Language model assistants can help classify and interpret agent behaviour but also afford human evaluators significant analytical degrees of freedom, threatening the reproducibility and auditability of language-model-based transcript analysis. Transect is an open source package built on Inspect Scout to help evaluators understand how a long agent run unfolded, identify behaviour worth investigating, and check interpretations against the transcript. Users specify task context and behavioural vocabulary in a reusable evaluation-family configuration, with judge models and analysis settings supplied separately. Transect's navigable reports align recorded events, token use, sub-agent activity, and model-generated behavioural labels on a common turn-based timeline. Reviewers can quickly grasp a run's narrative, trace any label or event to its source turns, and export the underlying data tables for cross-run analysis. We demonstrate the workflow on an AI R&D evaluation that generated almost 13 million tokens, dividing the agents' work into behavioural phases aligned with research-skill classifications, sub-agent delegations and interactions, and token use. The combined view shows a focus on operational work and manuscript production, with little evidence of a sustained hypothesis generation stage-arguably a necessary component for high-quality scientific outputs. Transect's flexible, customisable transcript-analysis pipeline will enable evaluators to keep pace with longer, more complex, more frequent AI evaluations while supporting scientific rigour, transparency, and reproducibility.

Figures & tables

Panel of Figure 1 — Figure 1: The Transect pipeline: ingest , select , scan , store , and render . The dataframes are used to generate the navigable report but can also be used for downstream cross-run analyses. Grey dotted boxes show inputs and configuration; the Spec draws on evaluation materials and, where needed, expert input. Structural scanners extract recorded information without model calls; judged scanners call the models selected for the analysis. Red dashed marks indicate two extension points. Custom scanners add structural or judged signals to dataframes. Custom report layers use those dataframes to add timeline layers or interface elements to the report without changing the upstream analysis.

Figure 2: The TabPFN epoch on an orchestrator-turn axis: paper-specific activities, research activities, cumulative references, sub-agent deployments, operator messages, and context-window size. Grey strips show pairwise label disagreement across five judge rolls for every turn; darker means greater disagreement.

Figure 3: New-work tokens attributed to paper-specific activities, research activities, and sub-agent categories. In the activity panels, solid segments show the orchestrator’s own expenditure and lighter segments the expenditure of sub-agents launched within the unit. The research activities panel excludes work without an assigned research skill, and the sub-agent panel excludes the orchestrator’s own activity. Token counts should not be summed across panels.

Figure 4: Multi-agent analysis of one TabPFN epoch. A: activity across 131 agent sessions and four shared-file tracks in source-log order. Arcs connect earlier writes (open symbols) to later cross-agent reads (filled symbols); colour and shape denote context type. B: all 922 access records by resource path. C: inferred purposes, with compound labels counted in each category. Access does not establish uptake.

Figure 5: Descriptive analyses of the TabPFN epoch. Top left: observed transitions between paper-specific phases after removing non-research-relevant engineering phases and merging consecutive repeats, with edge width proportional to transition count. Top right: the number of operator messages and context compactions followed by the start of a new phase within 3 or 10 orchestrator tool calls, and compactions after which the current phase continues; these are co-occurrence counts, not effects. Bottom left: registry verification of references in the agent- and human-written manuscripts against OpenAlex [ 46 ] , Crossref [ 47 ] , and arXiv [ 48 ] . Bottom right: the combined citation network, with node size indicating global citation count and edges indicating citations among works in the combined bibliography.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Input	Output
Ingest	Logs in a supported format	Scout transcripts representing agent and sub-agent activity recoverable from the source
Select	Transcripts and sample/epoch settings	The selected sample and one or more epochs
Scan	Selected transcripts, Spec and analysis settings	Results from structural scanners and any enabled judged scanners, saved in a Scout scan store
Store	Stored scanner results	Tables of activities, recorded events, token use and audit information
Render	Dataframes and source transcripts	HTML reports organised by epoch; tables users can analyse or export

Appendix

Table 1: Inputs and outputs of the five pipeline stages. Structural scanners extract or derive information without model calls; judged scanners call language models. Regenerating a report with excerpts requires access to the source transcripts.

Parameter	Default or limit	Control
Judge models	None; built-in judged scanners disabled	Top level
Roll count $k$	1	Top level
Verifier	On for solo and k-roll; unavailable for cohort	Top level
Verifier model	First configured judge model	Top level
Verifier spot-check share	5%; default phase-sampling floor: 3 phases	Top level
Phase digests per classification chunk	40	Scanner

Appendix

Table 2: Selected defaults in the reviewed package. Top-level controls are analysis-function arguments; scanner and helper controls require calls to the corresponding lower-level functions. Fixed values are set in the implementation. None of these controls is a built-in Spec field.

Table	One row represents	Linking fields within a transcript
transcript_info	Transcript identity and setup	No additional field
token_timeline , phase_turns	A model turn: token records or phase attribution	turn
flushes , interventions	A recorded or inferred compaction, or human-input event	turn ; several events can share a turn
lane_activity	Sub-agent tool activity at a turn	turn , agent_span_id
phases	A behavioural phase	phase_index ; turn_start , turn_end locate its range
turn_groups	A group of turns within a phase, with narration or fallback text	phase_index , group_index

Appendix

Table 3: Table units and linking fields. surface identifies the phase, sub-agent or custom-layer vocabulary. A judge member is identified by its model and repetition number ( roll ). Linking fields locate related records; they are not necessarily unique row keys. layer_frames is a collection of custom tables, and turn_tags is optional; both are separate from the built-in tables returned by frames() .

Explore similar work

Sep 30, 2026cs.AI

Staying on Task: Testing the Foundations of Long-Horizon Agent Reliability

Long-horizon agentic workflows require models to sustain repeated state-dependent actions all while the context grows, sub-task complexity changes, and new data arrives. Each situation represents an independent axis along which an agent may fail. An agent reconciling a long ledger, for example, must repeatedly read its state, update the correct record, and preserve alignment across thousands of outputs. A model may accept the entire ledger yet lose its place or stop applying the operation consistently as generation proceeds. We introduce Long-Transduction, a controlled diagnostic that tests a model's ability to stay on task during long generation while continuously reading, mutating, and outputting input-context dependent operations such as arithmetic, sorting, variable lookups, and table transformations. Long-Transduction evaluation independently varies local task complexity, input data formatting, and context length isolate failures along each axis. We evaluate seven open-weight models, finding a 62.8% decrease when scaling context length from 4-128K, a 36.5% decrease when varying input format, and a 39.9% decrease by increasing local task complexity. Together, these failures represent critical liabilities in long-horizon agentic workflows.

Jeffrey Willette, Krishna C. Puvvada, Boris Ginsburg

NVIDIA

Long-Context Language Model Evaluation Long-Horizon Agent Evaluation Sequence Modeling LLM Agent Reliability

Ask PDF

May 11, 2026cs.CL

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.

Shuangrui Ding, Xuanlang Dai, Long Xing +14

1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8

Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents Terminal Agents LLM Agent Evaluation Long-Horizon Agent Evaluation Agent Evaluation+2

Ask PDF

Sep 1, 2026cs.AI

Parsing the Stream: A Live Trace Model for Long-Horizon Agents and Their Observers

A long-horizon agent's trace outgrows both of its consumers: the human observer monitoring the run, and the agent itself, whose bounded context the trace must be folded back into. We present a live trace model, an append-only event ledger folded incrementally into typed run state and compiled into per-consumer views, and evaluate it for both consumers against deterministic ground truth. For the observer side, evaluated with an LLM reader as proxy, the compiled view answers monitoring questions using approximately 14x and 15x fewer input tokens (by reader) and at 5-7x lower cost than a budget-capped single-call reading of the raw trace, with higher accuracy (0.85-0.87 versus 0.48). Because the questions were co-designed with the view schema, we treat the token and cost reduction, conditional on schema coverage, as the transferable result. For the agent, on 120-link sequential-dependency tasks, mechanisms that maintain the task's running statistic in per-step state succeed where full-context prompting fails (30/30 versus 8/30 under a clean protocol, n=30, labeled descriptive owing to benchmark-system co-development); a prompt-level scratchpad matches the fold's accuracy at lower cost, and a two-arm decomposition attributes the fold's accuracy to its deterministic aggregate and its cost advantage to its compactness. The fold's remaining value over cheaper alternatives is deterministic auditability and serving the observer from the same state. We derive eleven candidate requirements for trace folding from observed failures and delimit them with an order-sensitive task family on which the fold ceases to help. Code, benchmarks, a regenerable synthetic corpus, and all workbench traces are released.

Egor Pakhomov, Erik Nijkamp

Salesforce AI Research

Long-Horizon Agent Evaluation AI Agent Monitoring Long-Horizon Agent Tasks

Ask PDF