TeleTune: Evolving Agent Skills From Offline Telemetry

Justin Chih-Yao Chen; Elias Stengel-Eskin; Yan Chen; Pol Llado; Scott Counts; Mohit Bansal; Benjamin Van Durme; Harsh Jhamtani

cs.AIOct 4, 2026

TeleTune: Evolving Agent Skills From Offline Telemetry

Authors: Justin Chih-Yao Chen, Elias Stengel-Eskin, Yan Chen, Pol Llado, Scott Counts, Mohit Bansal, Benjamin Van Durme, Harsh Jhamtani, +1 more

Organizations: UNC Chapel Hill · Microsoft

Abstract

Computer-use agents need to capture procedural knowledge of how people use software. User telemetry offers a scalable source of this knowledge. However, learning reusable skills from these logs requires addressing three challenges: (1) Goal Underspecification, since logs do not record the goal behind each action; (2) Non-Replayability, since past activity cannot be replayed to evaluate skill updates; and (3) Interleaved Trajectories, since logs may mix several tasks without marking their boundaries. To address these, we introduce TeleTune, a framework for learning a textual skill library from offline logs without recorded goals, cannot be replayed during optimization, and may interleave tasks. TeleTune uses action-prediction errors on logged trajectories to propose library edits and keep only those that improve held-out action-prediction accuracy, which we call skill-guided progress. The learned workflows also enable retrieval of demonstrations that cover the subgoals of a new task. At test time, the agent is provided with the learned library and the workflow-based retrieved demonstrations. Experiments on WorkArena and Online-Mind2Web show that TeleTune outperforms random retrieval, Agent Workflow Memory (AWM), and their combination. We find that the best baseline varies by setting, whereas TeleTune achieves average success rates of 77.1% and 80.6%, respectively, improving over the strongest baseline on each benchmark by 6.7% and 7.7%. Under the heaviest perturbation of the WorkArena training data,TeleTune keeps the highest average success rate at 68.5%, 6.3% above the strongest baseline. Our analyses show (1) skill optimization and workflow-based retrieval are complementary, (2) optimizing on fixed logs costs 5 to 75 times fewer tokens than validating the same edits with live episodes, (3) skill-guided progress tracks the live success rate.

Figures & tables

Figure 1: Learning Transferable Agent Skills From Offline User Telemetry. (a) Execution-based skill refinement uses explicit task goals and outcome feedback with a live environment. (b) User telemetry does not record the goal behind each action, and revising a skill cannot produce new outcomes for the logged activity. The logs may also interleave tasks. (c) TeleTune infers a goal for each logged trajectory, uses prediction errors on the fixed logs to propose skill edits, and keeps edits that improve skill-guided progress. The learned library is then evaluated on unseen tasks.

Figure 2: Overview of TeleTune . (a) Training trajectories are split into train and dev splits. The predictor runs through each train-split trajectory with the current skill library $\mathcal{M}$ in its prompt. After each prediction, we append the recorded action to the history (teacher forcing), so an error does not derail later steps. The optimizer reflects on the first prediction error in each trajectory and proposes edits, giving a candidate library, and we keep the one with higher skill-guided progress. (b) At inference, workflow-based retrieval selects complementary demonstrations for the new task. The task goal, skill library, and retrieved demonstrations form the agent prompt.

	WorkArena			Online-Mind2Web
Method	L1	L2	Avg.	Easy	Medium	Hard	Avg.
Naive	45.6 ${\scriptstyle\,\pm 0.6}$	48.0 ${\scriptstyle\,\pm 2.0}$	46.8 ${\scriptstyle\,\pm 0.9}$	78.5 ${\scriptstyle\,\pm 6.0}$	66.7 ${\scriptstyle\,\pm 10.8}$	48.7 ${\scriptstyle\,\pm 1.2}$	63.9 ${\scriptstyle\,\pm 4.6}$
Random- $k$	65.1 ${\scriptstyle\,\pm 2.3}$	54.2 ${\scriptstyle\,\pm 2.1}$	59.7 ${\scriptstyle\,\pm 2.1}$	87.3 ${\scriptstyle\,\pm 7.4}$	71.3 ${\scriptstyle\,\pm 1.7}$	64.7 ${\scriptstyle\,\pm 9.4}$	72.9 ${\scriptstyle\,\pm 4.0}$
AWM	75.7 ${\scriptstyle\,\pm 1.7}$	51.9 ${\scriptstyle\,\pm 2.9}$	63.8 ${\scriptstyle\,\pm 1.8}$	88.8 ${\scriptstyle\,\pm 5.4}$	73.4 ${\scriptstyle\,\pm 3.9}$	56.7 ${\scriptstyle\,\pm 5.8}$	71.9 ${\scriptstyle\,\pm 3.0}$
AWM $+$ Random- $k$	80.3 ${\scriptstyle\,\pm 1.1}$	60.4 ${\scriptstyle\,\pm 3.4}$	70.4 ${\scriptstyle\,\pm 1.7}$	85.6 ${\scriptstyle\,\pm 4.6}$	66.5 ${\scriptstyle\,\pm 12.0}$	64.0 ${\scriptstyle\,\pm 5.7}$	70.2 ${\scriptstyle\,\pm 4.4}$
TeleTune	92.3 ${\scriptstyle\,\pm 2.0}$	61.8 ${\scriptstyle\,\pm 2.8}$	77.1 ${\scriptstyle\,\pm 2.1}$	94.5 ${\scriptstyle\,\pm 5.3}$	76.6 ${\scriptstyle\,\pm 9.7}$	75.3 ${\scriptstyle\,\pm 2.8}$	80.6 ${\scriptstyle\,\pm 4.1}$

Table 1: Comparison of TeleTune with retrieval and workflow-induction baselines, using GPT-5.4 as the agent. TeleTune achieves the best success rate in every column. It improves over the strongest baseline by

6.7\%

on WorkArena average and by

7.7\%

on Online-Mind2Web average, with the largest gains on WorkArena L1 and on the hardest Online-Mind2Web tasks. Results are success rates (%) averaged over three seeds. Bold marks the best result in each column.

	Qwen-3.6		Mistral-3.5
Method	L1	L2	L1	L2
Naive	30.4	36.4	39.6	31.3
Random- $k$	55.9	43.9	64.4	38.9
AWM	60.4	43.9	74.7	38.0
AWM $+$ Random- $k$	63.7	44.9	72.2	38.4
TeleTune	75.6	50.0	75.2	37.8

Table 2: TeleTune generalizes to Qwen-3.6 and Mistral-3.5. Each backbone learns its own skill library and uses it at test time.

Figure 3: Skill-guided progress on the dev split during optimization on WorkArena L1 with GPT-5.4. The live success rate rises with skill-guided progress across the four evaluated checkpoints. At step 10, the first skill edit adds six skills, and the skill-guided progress rises from

51.3\%

66.9\%

. We also show the number of workflows and skills in the library at each step (# workflow and # skill). We omit the last 5 of the 32 steps, which accept no edits.

Skill Opt.	Retrieval	L1	L2	OM2W
$\times$	Random	65.1	54.2	72.9
$\times$	Workflow	66.8	52.6	74.3
✓	Random	83.7	60.8	75.1
✓	Workflow	92.3	61.8	80.6

Table 3: Ablation of skill optimization and workflow-based retrieval with GPT-5.4. Without either component, the agent is the Random-

k

baseline (first row). The two main components are complementary, as workflow-based retrieval adds

8.6\%

on L1 with the learned library but

1.7\%

without it. OM2W is the Online-Mind2Web average.

Figure 4: TeleTune has the highest success rate on both WorkArena levels at every interruption rate

q

except L1 at

q=1.0

, where Random-

k

1.1\%

higher. The baselines respond differently as

q

grows. AWM loses

26.4\%

on L1, whereas Random-

k

gains

10.8\%

on L1 and changes non-monotonically on L2. Naive uses no training pool and is omitted. Expanded values are in Table 7 .

Data	Steps	Ckpt.	Opt.	Pred.	Total
L1	32	14	0.24	173.60	173.84
L2	15	4	0.10	23.49	23.59
M2W	18	5	0.23	329.92	330.15

Table 4: Offline cost of skill optimization with GPT-5.4. Steps is the total number of optimization steps. Ckpt. reports the number of accepted steps. Token counts (in millions) are split into the optimizer (Opt., edit proposals and workflow induction) and the predictor (Pred., scoring candidates on the dev split), which accounts for almost all of them. L1 and L2 are both from WorkArena, and M2W represents Mind2Web.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Skill optimization
Stage stopping rule	5 consecutive steps without an accepted edit, or 25 steps
Maximum total steps	100
Failure search	batch size is 12 until at least 6 fail, at most 48 per step
Failure cases per step	At most 6, lowest $Q$ first; each shows the last 8 actions
Edits per step	At most 3

Appendix

Table 5: Hyperparameters of TeleTune . Values are shared across settings and backbones.

	WorkArena L1	WorkArena L2	Mind2Web
Task instances (train / test)	270 / 270	216 / 198	640 / 122
Training actions	4,573	3,561	4,677
Train split (traj. / actions)	132 / 2,405	103 / 1,995	323 / 2,413
Dev split (traj. / actions)	138 / 2,168	78 / 1,566	317 / 2,264

Appendix

Table 6: Training and evaluation data. Trajectory counts are successful oracle (WorkArena) or recorded (Mind2Web) trajectories; action counts are scoreable actions. Train and dev are the trajectory-level hash split used for reflection and edit selection. WorkArena test instances are other seeds of the training task templates. Mind2Web test tasks are live Online-Mind2Web tasks.

	WorkArena L1				WorkArena L2
Method	$q{=}0$	$q{=}0.33$	$q{=}0.66$	$q{=}1.0$	$q{=}0$	$q{=}0.33$	$q{=}0.66$	$q{=}1.0$
Naive	45.6	45.6	45.6	45.6	48.0	48.0	48.0	48.0
Random- $k$	65.1	63.9	69.6	75.9	54.2	61.6	47.3	48.5
AWM	75.7	74.1	60.9	49.3	51.9	52.8	50.5	50.0
AWM $+$ Random- $k$	80.3	82.8	70.4	65.9	60.4	60.6	53.7	55.1
TeleTune	92.3	88.9	83.1	74.8	61.8	65.8	63.3	62.1

Appendix

Table 7: Success rates (%) on WorkArena as the interruption rate

q

grows from

0

(the clean training pool) to

1.0

. Each method uses the same variant at every interruption rate. The

q{=}0

columns and the Naive row are three-seed means from Table 1 , and all other entries are single-seed. Naive uses no training pool and is unaffected by construction. Bold marks the best result in each column.

	WorkArena		Online-Mind2Web
Method	L1	L2	Easy	Medium	Hard
Naive	386.1 / 3.4	1489.0 / 16.3	76.6 / 3.3	166.8 / 7.4	242.9 / 10.6
Random- $k$	369.3 / 2.8	1404.1 / 13.8	90.2 / 3.5	238.1 / 9.3	308.2 / 12.2
AWM	343.4 / 2.5	1155.0 / 13.0	86.1 / 3.5	222.6 / 9.0	314.4 / 12.5
AWM $+$ Random- $k$	376.4 / 2.3	1405.0 / 13.6	86.8 / 3.3	227.2 / 8.4	328.5 / 12.4
TeleTune	636.9 / 3.1	1522.8 / 13.0	130.1 / 3.9	326.5 / 10.3	405.4 / 12.6

Appendix

Table 8: Average tokens per episode, reported as input / output in thousands (K). WorkArena values are per level (L1/L2); Online-Mind2Web values are split by task difficulty.

	WorkArena		Online-Mind2Web
Method	L1	L2	Easy	Medium	Hard
Naive	34.4	121.2	6.7	13.3	17.8
Random- $k$	28.6	105.9	6.5	14.7	19.1
AWM	27.4	94.3	6.8	14.3	19.5
AWM $+$ Random- $k$	30.5	94.5	6.4	13.7	18.8
TeleTune	31.4	90.0	6.7	15.4	18.7

Appendix

Table 9: Average number of steps per episode. WorkArena values are per level (L1/L2); Online-Mind2Web values are split by task difficulty.

Explore similar work

May 7, 2026cs.AI

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

A persistent skill library allows language model agents to reuse successful strategies across tasks. Maintaining such a library requires three coupled capabilities. The agent selects a relevant skill, utilizes it during execution, and distills new skills from experience. Existing methods optimize these capabilities in isolation or with separate reward sources, resulting in partial and conflicting evolution. We propose Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. The policy generates a query to search the skill library, re-ranks candidates to select one, solves the task conditioned on it, and distills a new skill from the trajectory. All learning derives from a single task-outcome signal. Its low-frequency trend credits selection and its high-frequency variation credits distillation. Experiments on ALFWorld and WebShop show that Skill1 outperforms prior skill-based and reinforcement learning baselines. Training dynamics confirm the co-evolution of the three capabilities, and ablations show that removing any credit signal degrades the evolution.

Yaorui Shi, Yuxin Chen, Zhengxi Lu +6

University of Science and Technology of China · 2Meituan · 3National University of Singapore +2

Skills Agentic Reinforcement Learning Language-Model Agents Co-Evolution Evolutionary Optimization Methods

Ask PDF

Jun 12, 2026cs.AI

SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing

Agent skills are structured procedural packages that guide frozen LLM agents in specialized workflows. Skills rarely remain sufficient after deployment: edge cases, API changes, and deployment constraints become visible only through use, making skill evolution a practical necessity. Existing methods depend on privileged feedback such as held-out validation scores, hidden test outcomes, or environment rewards -- signals often unavailable when a practitioner has only a task description and workspace data. We introduce SkillAudit, a framework for evolving agent skills without ground-truth feedback. The key idea is paired trajectory auditing: at each iteration, the same task is executed with and without the candidate skill, isolating how the skill changes agent behavior without external labels. To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation (PACE), a cluster of evaluators that maps trajectory divergences to diagnostic signals linked to specific passages in the skill document. A structural verifier, compiled once from the task specification and then fixed, checks task constraints and rolls back harmful updates. SkillAudit routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task. Across 89 containerized tasks spanning 8 professional domains, SkillAudit achieves 73.9% average task reward, outperforming an agent without skills (40.9%) and the static expert skill (56.7%). These gains are obtained without accessing hidden tests, reference solutions, or external scoring functions during evolution.

Haowen Gao, Haoran Chen, Can Wang +5

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · Tongyi Lab, Alibaba Group, Beijing, China

Skill Evolution Skills

Ask PDF

Jun 1, 2026cs.CL

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

Abundant procedural knowledge on the Web holds great potential for helping agents solve long-horizon tasks. However, such knowledge is often multimodal, heterogeneous, noisy, and implicitly assumes human executors, making it difficult to use directly as the skills required by agents. To bridge the gap between human-oriented guides and agent-executable skills, we formalize this problem as guide-to-skill learning: converting in-the-wild guides into executable skills and continuously improving them from trajectories observable to the agent. To evaluate the capability of existing agents on this task, we introduce MMG2Skill-Bench, the first benchmark designed for this problem. We further propose MMG2Skill, a closed-loop framework that compiles guides into editable skills, conditions a fixed vision-language model (VLM) agent on these skills during execution, and revises the skills from trajectory-level root-cause feedback without using benchmark scores. Across GUI control, open-ended gameplay, and strategic card play with six VLM backbones, MMG2Skill consistently outperforms vanilla baseline agents in every model-domain setting, achieving macro-average gains of +12.8 to +25.3 percentage points across backbones. Ablation studies show that directly prompting agents with raw guides can degrade performance, while both structured skill construction and trajectory-driven revision are necessary for the observed improvements. On success-inferable tasks, analyzer-based early stopping further prevents late-stage performance regressions and saves 25%-53% of attempts when the success signal is properly calibrated.

Xinyu Che, Junqi Xiong, Yunfei Ge +10

Nanjing University · Kuaishou Technology

Skill Evolution Skills Trajectory-Level Credit In-The-Wild Videos

Ask PDF