TeleTune: Evolving Agent Skills From Offline Telemetry

Justin Chih-Yao Chen; Elias Stengel-Eskin; Yan Chen; Pol Llado; Scott Counts; Mohit Bansal; Benjamin Van Durme; Harsh Jhamtani

cs.AIOct 4, 2026

TeleTune: Evolving Agent Skills From Offline Telemetry

Authors: Justin Chih-Yao Chen, Elias Stengel-Eskin, Yan Chen, Pol Llado, Scott Counts, Mohit Bansal, Benjamin Van Durme, Harsh Jhamtani, +1 more

Organizations: UNC Chapel Hill · Microsoft

Abstract

Computer-use agents need to capture procedural knowledge of how people use software. User telemetry offers a scalable source of this knowledge. However, learning reusable skills from these logs requires addressing three challenges: (1) Goal Underspecification, since logs do not record the goal behind each action; (2) Non-Replayability, since past activity cannot be replayed to evaluate skill updates; and (3) Interleaved Trajectories, since logs may mix several tasks without marking their boundaries. To address these, we introduce TeleTune, a framework for learning a textual skill library from offline logs without recorded goals, cannot be replayed during optimization, and may interleave tasks. TeleTune uses action-prediction errors on logged trajectories to propose library edits and keep only those that improve held-out action-prediction accuracy, which we call skill-guided progress. The learned workflows also enable retrieval of demonstrations that cover the subgoals of a new task. At test time, the agent is provided with the learned library and the workflow-based retrieved demonstrations. Experiments on WorkArena and Online-Mind2Web show that TeleTune outperforms random retrieval, Agent Workflow Memory (AWM), and their combination. We find that the best baseline varies by setting, whereas TeleTune achieves average success rates of 77.1% and 80.6%, respectively, improving over the strongest baseline on each benchmark by 6.7% and 7.7%. Under the heaviest perturbation of the WorkArena training data,TeleTune keeps the highest average success rate at 68.5%, 6.3% above the strongest baseline. Our analyses show (1) skill optimization and workflow-based retrieval are complementary, (2) optimizing on fixed logs costs 5 to 75 times fewer tokens than validating the same edits with live episodes, (3) skill-guided progress tracks the live success rate.

Figures & tables

Figure 1: Learning Transferable Agent Skills From Offline User Telemetry. (a) Execution-based skill refinement uses explicit task goals and outcome feedback with a live environment. (b) User telemetry does not record the goal behind each action, and revising a skill cannot produce new outcomes for the logged activity. The logs may also interleave tasks. (c) TeleTune infers a goal for each logged trajectory, uses prediction errors on the fixed logs to propose skill edits, and keeps edits that improve skill-guided progress. The learned library is then evaluated on unseen tasks.

Figure 2: Overview of TeleTune . (a) Training trajectories are split into train and dev splits. The predictor runs through each train-split trajectory with the current skill library $\mathcal{M}$ in its prompt. After each prediction, we append the recorded action to the history (teacher forcing), so an error does not derail later steps. The optimizer reflects on the first prediction error in each trajectory and proposes edits, giving a candidate library, and we keep the one with higher skill-guided progress. (b) At inference, workflow-based retrieval selects complementary demonstrations for the new task. The task goal, skill library, and retrieved demonstrations form the agent prompt.

	WorkArena			Online-Mind2Web
Method	L1	L2	Avg.	Easy	Medium	Hard	Avg.
Naive	45.6 ${\scriptstyle\,\pm 0.6}$	48.0 ${\scriptstyle\,\pm 2.0}$	46.8 ${\scriptstyle\,\pm 0.9}$	78.5 ${\scriptstyle\,\pm 6.0}$	66.7 ${\scriptstyle\,\pm 10.8}$	48.7 ${\scriptstyle\,\pm 1.2}$	63.9 ${\scriptstyle\,\pm 4.6}$
Random- $k$	65.1 ${\scriptstyle\,\pm 2.3}$	54.2 ${\scriptstyle\,\pm 2.1}$	59.7 ${\scriptstyle\,\pm 2.1}$	87.3 ${\scriptstyle\,\pm 7.4}$	71.3 ${\scriptstyle\,\pm 1.7}$	64.7 ${\scriptstyle\,\pm 9.4}$	72.9 ${\scriptstyle\,\pm 4.0}$
AWM	75.7 ${\scriptstyle\,\pm 1.7}$	51.9 ${\scriptstyle\,\pm 2.9}$	63.8 ${\scriptstyle\,\pm 1.8}$	88.8 ${\scriptstyle\,\pm 5.4}$	73.4 ${\scriptstyle\,\pm 3.9}$	56.7 ${\scriptstyle\,\pm 5.8}$	71.9 ${\scriptstyle\,\pm 3.0}$
AWM $+$ Random- $k$	80.3 ${\scriptstyle\,\pm 1.1}$	60.4 ${\scriptstyle\,\pm 3.4}$	70.4 ${\scriptstyle\,\pm 1.7}$	85.6 ${\scriptstyle\,\pm 4.6}$	66.5 ${\scriptstyle\,\pm 12.0}$	64.0 ${\scriptstyle\,\pm 5.7}$	70.2 ${\scriptstyle\,\pm 4.4}$
TeleTune	92.3 ${\scriptstyle\,\pm 2.0}$	61.8 ${\scriptstyle\,\pm 2.8}$	77.1 ${\scriptstyle\,\pm 2.1}$	94.5 ${\scriptstyle\,\pm 5.3}$	76.6 ${\scriptstyle\,\pm 9.7}$	75.3 ${\scriptstyle\,\pm 2.8}$	80.6 ${\scriptstyle\,\pm 4.1}$

Table 1: Comparison of TeleTune with retrieval and workflow-induction baselines, using GPT-5.4 as the agent. TeleTune achieves the best success rate in every column. It improves over the strongest baseline by

6.7\%

on WorkArena average and by

7.7\%

on Online-Mind2Web average, with the largest gains on WorkArena L1 and on the hardest Online-Mind2Web tasks. Results are success rates (%) averaged over three seeds. Bold marks the best result in each column.

	Qwen-3.6		Mistral-3.5
Method	L1	L2	L1	L2
Naive	30.4	36.4	39.6	31.3
Random- $k$	55.9	43.9	64.4	38.9
AWM	60.4	43.9	74.7	38.0
AWM $+$ Random- $k$	63.7	44.9	72.2	38.4
TeleTune	75.6	50.0	75.2	37.8

Table 2: TeleTune generalizes to Qwen-3.6 and Mistral-3.5. Each backbone learns its own skill library and uses it at test time.

Figure 3: Skill-guided progress on the dev split during optimization on WorkArena L1 with GPT-5.4. The live success rate rises with skill-guided progress across the four evaluated checkpoints. At step 10, the first skill edit adds six skills, and the skill-guided progress rises from

51.3\%

66.9\%

. We also show the number of workflows and skills in the library at each step (# workflow and # skill). We omit the last 5 of the 32 steps, which accept no edits.

Skill Opt.	Retrieval	L1	L2	OM2W
$\times$	Random	65.1	54.2	72.9
$\times$	Workflow	66.8	52.6	74.3
✓	Random	83.7	60.8	75.1
✓	Workflow	92.3	61.8	80.6

Table 3: Ablation of skill optimization and workflow-based retrieval with GPT-5.4. Without either component, the agent is the Random-

k

baseline (first row). The two main components are complementary, as workflow-based retrieval adds

8.6\%

on L1 with the learned library but

1.7\%

without it. OM2W is the Online-Mind2Web average.

Figure 4: TeleTune has the highest success rate on both WorkArena levels at every interruption rate

q

except L1 at

q=1.0

, where Random-

k

1.1\%

higher. The baselines respond differently as

q

grows. AWM loses

26.4\%

on L1, whereas Random-

k

gains

10.8\%

on L1 and changes non-monotonically on L2. Naive uses no training pool and is omitted. Expanded values are in Table 7 .

Data	Steps	Ckpt.	Opt.	Pred.	Total
L1	32	14	0.24	173.60	173.84
L2	15	4	0.10	23.49	23.59
M2W	18	5	0.23	329.92	330.15

Table 4: Offline cost of skill optimization with GPT-5.4. Steps is the total number of optimization steps. Ckpt. reports the number of accepted steps. Token counts (in millions) are split into the optimizer (Opt., edit proposals and workflow induction) and the predictor (Pred., scoring candidates on the dev split), which accounts for almost all of them. L1 and L2 are both from WorkArena, and M2W represents Mind2Web.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Skill optimization
Stage stopping rule	5 consecutive steps without an accepted edit, or 25 steps
Maximum total steps	100
Failure search	batch size is 12 until at least 6 fail, at most 48 per step
Failure cases per step	At most 6, lowest $Q$ first; each shows the last 8 actions
Edits per step	At most 3

Appendix

Table 5: Hyperparameters of TeleTune . Values are shared across settings and backbones.

	WorkArena L1	WorkArena L2	Mind2Web
Task instances (train / test)	270 / 270	216 / 198	640 / 122
Training actions	4,573	3,561	4,677
Train split (traj. / actions)	132 / 2,405	103 / 1,995	323 / 2,413
Dev split (traj. / actions)	138 / 2,168	78 / 1,566	317 / 2,264

Appendix

Table 6: Training and evaluation data. Trajectory counts are successful oracle (WorkArena) or recorded (Mind2Web) trajectories; action counts are scoreable actions. Train and dev are the trajectory-level hash split used for reflection and edit selection. WorkArena test instances are other seeds of the training task templates. Mind2Web test tasks are live Online-Mind2Web tasks.

	WorkArena L1				WorkArena L2
Method	$q{=}0$	$q{=}0.33$	$q{=}0.66$	$q{=}1.0$	$q{=}0$	$q{=}0.33$	$q{=}0.66$	$q{=}1.0$
Naive	45.6	45.6	45.6	45.6	48.0	48.0	48.0	48.0
Random- $k$	65.1	63.9	69.6	75.9	54.2	61.6	47.3	48.5
AWM	75.7	74.1	60.9	49.3	51.9	52.8	50.5	50.0
AWM $+$ Random- $k$	80.3	82.8	70.4	65.9	60.4	60.6	53.7	55.1
TeleTune	92.3	88.9	83.1	74.8	61.8	65.8	63.3	62.1

Appendix

Table 7: Success rates (%) on WorkArena as the interruption rate

q

grows from

0

(the clean training pool) to

1.0

. Each method uses the same variant at every interruption rate. The

q{=}0

columns and the Naive row are three-seed means from Table 1 , and all other entries are single-seed. Naive uses no training pool and is unaffected by construction. Bold marks the best result in each column.

	WorkArena		Online-Mind2Web
Method	L1	L2	Easy	Medium	Hard
Naive	386.1 / 3.4	1489.0 / 16.3	76.6 / 3.3	166.8 / 7.4	242.9 / 10.6
Random- $k$	369.3 / 2.8	1404.1 / 13.8	90.2 / 3.5	238.1 / 9.3	308.2 / 12.2
AWM	343.4 / 2.5	1155.0 / 13.0	86.1 / 3.5	222.6 / 9.0	314.4 / 12.5
AWM $+$ Random- $k$	376.4 / 2.3	1405.0 / 13.6	86.8 / 3.3	227.2 / 8.4	328.5 / 12.4
TeleTune	636.9 / 3.1	1522.8 / 13.0	130.1 / 3.9	326.5 / 10.3	405.4 / 12.6

Appendix

Table 8: Average tokens per episode, reported as input / output in thousands (K). WorkArena values are per level (L1/L2); Online-Mind2Web values are split by task difficulty.

	WorkArena		Online-Mind2Web
Method	L1	L2	Easy	Medium	Hard
Naive	34.4	121.2	6.7	13.3	17.8
Random- $k$	28.6	105.9	6.5	14.7	19.1
AWM	27.4	94.3	6.8	14.3	19.5
AWM $+$ Random- $k$	30.5	94.5	6.4	13.7	18.8
TeleTune	31.4	90.0	6.7	15.4	18.7

Appendix

Table 9: Average number of steps per episode. WorkArena values are per level (L1/L2); Online-Mind2Web values are split by task difficulty.

Explore similar work

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning
May 7, 2026Yaorui Shi, Yuxin Chen, Zhengxi Lu +6LLM Agent Skill Learning LLM Agent Skill Retrieval
University of Science and Technology of China · 2Meituan · 3National University of Singapore +2
SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing
Jun 12, 2026Haowen Gao, Haoran Chen, Can Wang +5Algorithmic Auditing LLM Agent Self-Improvement
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · Tongyi Lab, Alibaba Group, Beijing, China
MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?
Jun 1, 2026Xinyu Che, Junqi Xiong, Yunfei Ge +10Continual Learning for LLM Agents LLM Agent Skill Learning
Nanjing University · Kuaishou Technology

May 7, 2026, cs.AIJ/K move · Enter open · S save

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao+5

University of Science and Technology of China · 2Meituan · 3National University of Singapore+2

LLM Agent Skill Learning LLM Agent Skill Retrieval Policy Learning RL for LLM Agents Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

A persistent skill library allows language model agents to reuse successful strategies across tasks. Maintaining such a library requires three coupled capabilities. The agent selects a relevant skill, utilizes it during execution, and distills new skills from experience. Existing methods optimize these capabilities in isolation or with separate reward sources, resulting in partial and conflicting evolution. We propose Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. The policy generates a query to search the skill library, re-ranks candidates to select one, solves the task conditioned on it, and distills a new skill from the trajectory. All learning derives from a single task-outcome signal. Its low-frequency trend credits selection and its high-frequency variation credits distillation. Experiments on ALFWorld and WebShop show that Skill1 outperforms prior skill-based and reinforcement learning baselines. Training dynamics confirm the co-evolution of the three capabilities, and ablations show that removing any credit signal degrades the evolution.

Abstract

Explore similar work

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?