TEC: A Collection of Human Trial-and-error Trajectories for Problem Solving
Authors: Xinkai Zhang, Jingtao Zhan, Yiqun Liu, Qingyao Ai
Organizations: College of AI, Tsinghua University · Quancheng Laboratory · Gaoling School of Artificial Intelligence, Renmin University of China · Institute of Data and Information, Tsinghua Shenzhen International Graduate School · Department of Computer Science and Technology, Tsinghua University
Trial-and-error is a fundamental strategy for humans to solve complex problems and a necessary capability for Artificial Intelligence (AI) systems operating in real-world environments. Although several trial-and-error AI techniques have recently been proposed, most of them rely on simple heuristics designed by researchers and achieve limited performance gains. The core issue is the absence of appropriate data: current models cannot learn from detailed records of how humans actually conduct trial-and-error in practice. To address this gap, we introduce a data annotation platform and a corresponding dataset, termed Trial-and-Error Collection (TEC). The platform records users' complete trajectories across multiple trials and collects their reflections after receiving error feedback. Using this platform, we record the problem-solving processes of 46 participants on 58 tasks, resulting in 5,370 trial trajectories along with error reflections across 41,229 webpages. With this dataset, we observe that humans achieve substantially higher accuracy compared to LLMs, which demonstrates that humans are more effective in trial-and-error than LLMs. We believe that the TEC platform and dataset provide a valuable foundation for understanding human trial-and-error behavior and for developing more capable AI systems. Platform and dataset are publicly available.
Figures & tables
Trial Dimension
Error Dimension
Resource
Task Type
Trials
Behavioral Traces
Explicit Feedback
Error Reflection
Web agent benchmarks
Mind2Web
Web interaction
Single
✓ (DOM actions)
✓ (task success)
✗
WebArena
Web interaction
Single
✗
✓ (task success)
✗
WebVoyager
Web interaction
Single
✓ (screenshots)
✓ (task success)
✗
Information seeking benchmarks
Table 1. Comparison with existing resources along the dimensions of trial and error . The trial dimension captures whether multiple trials are recorded with behavioral trajectories; the error dimension captures whether per-trial correctness and error reflection are provided. TEC is the first resource satisfying both dimensions.
Figure 1. Platform architecture. The Chrome extension captures browsing trajectories from any webpage; the Django frontend supports annotation with replay; and the backend manages task assignment, data ingestion, and evaluation.
Figure 2. Multi-stage replay-based annotation workflow. On error, a reflection precedes the next trial; on success or cancellation, a corresponding assessment captures the outcome.
Figure 3. Platform interfaces for multi-stage replay-based annotation workflow. (a) Participants browse freely while the extension records in the background; (b) evidence marking via right-click menu; (c) DOM-based replay of human trajectories and page changes; (d) reflection form collecting prioritized error diagnosis and corrective plan.
Figure 4. Demographic profile of 46 participants.
Record
Key Fields
Per page
Webpage
URL, title, rrweb DOM recording, interaction events, mouse/scroll trajectory, dwell time, referrer
Selected text, DOM position, source URL, evidence type, relevance/credibility ratings
Reflection (on error)
Error category, corrective plan (prioritized), adjusted difficulty, free-text notes
Table 2. Data schema of the TEC dataset.
Metric
Value
Scale
Questions
58
Tasks trajectories
2,424
Total trials
5,370
Avg trials per task
2.2
Browsing data
Table 3. Dataset statistics of TEC.
Figure 5. Distributions of four key behavioral dimensions.
Figure 6. Conditional probability of corrective plan given error diagnosis, P(corrective plan∣error diagnosis) , over 2,946 reflections. Specific error triggers targeted strategies (e.g., “Format Error” → “Correct Format” at 57%), showing that human reflection is diagnostic, not blind repetition.
Figure 7. Case study: “Who sang Smoke Gets in Your Eyes first?” (answer: Tamara Drasin). Each row shows one method’s reflect-then-act trial sequence. The human diagnoses a misinterpretation and succeeds at T2; Vanilla LLM and RAG both fail all 5 trials; Vanilla Agent succeeds by T4 through iterative search; Browser Agent converges at T5 after prolonged fixation.
Method
Model
SR@1
SR@5
Recovery
Avg T
Human
—
56.6
88.9
74.5
2.14
Vanilla LLM
GPT-4o-mini
34.5
55.2
31.6
3.21
Qwen3-8B
21.4
32.1
13.6
3.89
RAG
GPT-4o-mini
50.0
72.4
44.8
2.38
Qwen3-8B
48.1
68.5
39.3
2.54
Vanilla Agent
GPT-4o-mini
58.6
79.3
50.0
2.17
Table 4. Human vs. LLM performance on 58 questions (up to 5 trials). SR@ k : fraction of tasks solved within k trials. Recovery Rate: P(success≤5∣fail at T1) , measuring the ability to succeed after an initial error. Avg T: average number of trials used. The best LLM matches human first-trial accuracy but recovers from errors at a substantially lower rate.
Figure 8. Query reformulation patterns (GPT-4o-mini). (a) Similarity between queries and original question; (b) pairwise similarity between queries in adjacent trials. We exclude Browser Agent and Vanilla LLM here as they submit few or no search queries. Humans progressively diverge in semantic space, while LLMs show only surface-level lexical changes.