cs.AIOct 16, 2026

Annotrace: A Workbench for Step-Level Annotation of Agent Traces

Authors: Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu, Sahan Bulathwela

Organizations: Department of Computer Science, University College London, London, United Kingdom · Centre for Artificial Intelligence, University College London, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at κ=0.82κ= 0.82.

Figures & tables

Figure 2: Annotrace workflow. Top: adapters convert six trace formats into nested steps. Middle: a model proposes an annotation schema and answers the same questions as the human annotator. Bottom: adapters can recover and separately label Thinking–Action–Response cycles embedded in a single text field.
FormatStep kindsNested
OpenTelemetryinvoke_agent, chat, execute_toolyes
LangSmithchain, llm, tool, retrieveryes
Langfuseagent, generation, tool, spanyes
Chat messagesmessage, tool call, tool resultno
Who&Whenmessageno
Coordinatordecision, reply, cycle, outputyes
Table 1: The six adapters. Kinds are each format’s own vocabulary; a question can be restricted to any subset of them.
Figure 3: The annotation screen. The sidebar holds the run’s question and the agent’s answer, so both stay in view while the steps scroll. Colour marks the kind of step until a label is chosen, then marks the label. Steps still wanting an answer say so.
More in this topic

Human-in-the-Loop Annotation

Human-in-the-loop annotation combines human judgment and AI assistance to create, review, and improve labeled data.

See all 135 papers

Explore similar work

CardsList

  1. Willful Disobedience: Automatically Detecting Failures in Agentic Traces

    Mar 25, 2026Reshabh K Sharma, Shraddha Barke, Benjamin ZornAI Agent EvaluationAI Agent Auditing

    University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA

  2. REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces

    Jun 8, 2026Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok +4LLM Agent EvaluationLLM Agents

    University of California, Los Angeles, USA · Cisco, USA · University of Mississippi, USA

  3. AgentTrails: Towards Trust and Reuse for Agentic Tasks

    Jul 21, 2026Eden Wu, Sonia Castelo, Yurong Liu +2Data ProvenanceAgent Trajectory Analysis

    New York University