FARL: A Fully Automated Pipeline of RL Interface and Solver Design

Tianyu Jin; Liuyu Xiang; Chaowen Yang; Xingyang Chang; Pei Pei Li; Zhaofeng He

cs.AIOct 11, 2026

FARL: A Fully Automated Pipeline of RL Interface and Solver Design

Authors: Tianyu Jin, Liuyu Xiang, Chaowen Yang, Xingyang Chang, Pei Pei Li, Zhaofeng He

Organizations: Beijing University of Posts and Telecommunications, Beijing, China

Abstract

Existing reinforcement learning (RL) largely focuses on optimizing learning algorithms within a pre-defined task formulation, while applying RL to new tasks still relies heavily on human expertise to design the learning pipeline: what an agent observes, how it is rewarded, which algorithm it uses, and how training is configured. Automating these choices is challenging because they are tightly coupled. We introduce FARL (Fully Automated Reinforcement Learning), a framework that automates RL design by jointly searching observation and reward programs, learning algorithms, and training settings. FARL uses training statistics and observed behavior to diagnose failures, propose changes, and test whether they improve learning. Its search preserves diverse candidates and the controls needed for these tests, using a staged budget to screen designs before investing in longer training. Experimental memory records each finding together with its conditions, so an unsuccessful component can be reconsidered in a different combination. Across expert-withheld and expert-available settings, FARL outperforms the evaluated automated baselines on challenging navigation, locomotion, and manipulation tasks. On Adroit Hand Relocate, FARL achieves 97.1% success at 5M training steps per policy, compared with 0% for Eureka and LiMEN and 1.0% for native Proximal Policy Optimization (PPO) under the same policy training budget.

Figures & tables

Figure 1: Automating the RL design loop. (a) An expert revisits the learning setup when training stalls or becomes unstable. (b) FARL’s LLM agent designs and directs experiments using behavioral evidence and scientific memory. Both receive the same task interface and evaluator.

Figure 2: FARL’s architecture. An adapter exposes the task API, and miniRL collects initial experience. (1) LLMs propose observations, rewards, and solver configurations. (2) Search screens diverse candidates at short budgets and trains selected designs and controls for longer. (3) Diagnosis uses training statistics and behavior to propose the next experiment. Programs enter the candidate bank; findings and test conditions enter scientific memory. A trained policy may help a new learner reach an intermediate state during training. Final evaluation uses the new policy alone from the original resets.

	DoorKey		Ant (m)		Door	Hand (%)
Method	EW	EA	EW	EA	EW	ST
FARL	$\boldsymbol{1.00\pm 0.00}$	$\boldsymbol{1.00\pm 0.00}$	$\boldsymbol{345.8\pm 14.2}$	$\boldsymbol{281.1\pm 74.4}$	$\boldsymbol{1.00}$	$\boldsymbol{97.11\pm 2.99}$
Eureka	$0.00\pm 0.00$	$0.92\pm 0.06$	$94.1\pm 50.9$	$110.8\pm 39.5$	$\boldsymbol{1.00}$	$0.00\pm 0.00$
LiMEN	$0.00\pm 0.00^{\ddagger}$	$\boldsymbol{1.00\pm 0.00}$	$89.9\pm 20.4$	$144.2\pm 4.7^{\dagger}$	$\boldsymbol{1.00}$	$0.00\pm 0.00$
Native design	/	$\boldsymbol{1.00\pm 0.00}$	/	$81.9\pm 19.8$	/	$1.00\pm 1.00$

Table 1: Task outcomes (mean

\pm

SD) and costs (tokens / training steps, both in M). Ant uses meters, Hand percent, other tasks completion rates. Ant entries use 10M for FARL/LiMEN and 5M for Eureka/native PPO. All Hand policies use 5M steps. Ant and Hand costs cover one complete search. Bold marks the highest outcome or lowest cost in each column, including ties. ‘/‘ means inapplicable. ∗ Estimated tokens; † trimmed statistic; ‡ episode SD. Policy horizons and accounting: Appendices A.3 and G .

Figure 3: Outcomes, design diversity, and learning dynamics. (a) DoorKey stage completion. (b) Curve 1 (C1), Curve 2 (C2), and Curve 3 (C3) track three FARL candidates at 300k: C1/C2 use PPO, C3 uses A2C; lines are five-seed mean exit frequencies in 25k-step windows. (c) All 121 Ant designs at 1M: dots are five-seed means; diamonds mark each learner’s best design. (d) Ant outcomes by budget; filled/open markers denote EW/EA. (e) Hand stage completion at 5M steps per policy. (f) The 111-dimensional SAC design across Hand iterations 11–14; lines average relocation frequencies across three training runs in 100k-step windows. Bars and bands show seed SD, clipped to

[0,100]\%

for completion. Appendices F.3 and F.4 give candidate identities and final-evaluation protocols.

Condition	DoorKey (%)	Ant (disp.)	Door (%) †
FARL (fixed reference)	$\boldsymbol{100.00\pm 0.00}$	$\boldsymbol{273.35\pm 15.35}$	$\boldsymbol{100.00}$
A1: obs.–reward search only	$0.00\pm 0.00$	$-3.09\pm 5.38$	$0.00\pm 0.00$
A2: no rich evidence / memory	$0.00\pm 0.00$	$-0.10\pm 0.39$	$0.00\pm 0.00$
A3: no diversity / preservation	$\boldsymbol{100.00\pm 0.00}$	$2.40\pm 2.93$	$0.00\pm 0.00$

Table 2: Three core ablations in EW and frozen FARL design references. DoorKey/Ant policy horizons are 300k/1M; task success (%) or signed displacement, mean

\pm

sample SD. References use five seeds, ablations three. Search protocols are given in Appendix E . † Door reference: 50k, two seeds; ablations: 1M, three seeds.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Benchmark identifier	Observation / action interface
DoorKey	MiniGrid-DoorKey- 5x5-v0	EW: 148 inputs, 7 discrete actions; EA: 83 inputs, 7 actions.
Ant	Ant-v4	27 raw inputs, 8 continuous actions; derived features are design choices.
Door	door-open-v3- goal-observable	Meta-World; 39 inputs, 4 continuous actions.
Hand	AdroitHandRelocate-v2	ST: 84 inputs, 30 continuous actions; native control: 39 inputs.

Appendix

Table A1: Task interfaces. EW means expert-withheld; EA means expert-available. Hand is the stress test (ST) under EA permissions.

Task	Method	Cohort	Selection and replication
DoorKey EW	FARL	$5\times 25\times 1$ M	Seeds 200–204; the same design has five 300k runs, seeds 100–104.
	Eureka	$5\times 25\times 1$ M	Seeds 0–4, fixed selected reward.
	LiMEN	$1\times 20\times 1$ M	Selected short-stage run; the full-stage threshold was not reached.
DoorKey EA	FARL	$3\times 5\times 100$ k	Seeds 0–2; completed early-saturation cohort.
	Eureka	$2\times 25\times 500$ k	Seeds 101–102; 22/25 and 24/25 successes.
	LiMEN	$3\times 20\times 2$ M	Seeds 42–44; aggregate also used in selection.

Appendix

Table A2: Outcome cohorts: runs

\times

evaluation episodes/run

\times

training steps/run. Unless noted, runs train one selected design.

Task	Learner	Permitted probe feedback	Learning	Follow-up
DoorKey EW	DQN	Aggregate native scalar	30k	2k
Ant EW	SAC	Controllable novelty	30k	5k
Ant EA	SAC	Native scalar	30k	5k
Hand ST	SAC	Native scalar	100k	5k

Appendix

Table A3: Executed initial sensing. Follow-up steps collect generator input with the probe policy; they are not extra optimizer updates.

Design axis	Evidence supplied	Generated output
Observation	Coordinate meanings/types, task relations, probe transitions, and earlier feature comparisons	Ordered field selections or raw inputs augmented by deterministic expressions, with source dependencies and formulas.
Reward	Task intent, transition interface, observed behavior, current incentives, and retrieved explanations	Complete transition-reward source and a testable account of its proposed behavioral mechanism.
Learner	Action-space type, representation, supported implementations, and learning-health evidence	Compatible algorithm and network alternatives with learner-conditioned configuration domains.
Training	Selected interface/learner, exploration alternatives, completed learning evidence, and resource protocol	Instantiated hyperparameters, exploration settings, and requested training depth for each execution.

Appendix

Table A4: Executable objects in joint design. Inputs follow the task’s declared access setting; available execution evidence informs later proposals.

Record	Question it makes explicit
Program and family	What incentive is executed, and which proposed mechanism distinguishes this alternative?
Mechanism and behavior	Why might it help, and which events or behavior should change if it works?
Distinguishing prediction	Which observation separates this explanation from a competing explanation?
Falsifier and proxy risks	Which result would challenge the mechanism, and how might reward improve without the intended behavior?
Temporal role	Does the proposal emphasize acquisition, maintenance, regression prevention, or their interaction?
Compatibility and depth	Which learner properties and minimum training duration are proposed for an informative test?

Appendix

Table A5: The prediction record attached to a reward proposal. Source code defines the executed incentive; the other fields guide subsequent tests.

Available evidence	Next use	Question for the next experiment
No compatible source with usable readiness	Joint design or exploration from original resets	Observe partial competence and identify a testable frontier opportunity.
Positive local signal; checkpoint and runtime interface verifiable	Exploratory source-policy attempts mixed with original resets	Whether the source can reliably create that opportunity and the new learner can use it.
The same source also satisfies its declared repeated-use checks	Qualified reuse under the recorded checkpoint/interface and scope	Whether the complete learner succeeds without assistance.
Completed whole-task evidence for one learner	Deeper training of the selected recipe or reserved fixed-design evaluation	Performance at the newly evaluated depth and replication level.

Appendix

Table A6: Different evidence warrants different uses of partial competence.

Observed discrepancy	Next comparison	Discriminating observation
Frontier event never observed	Exploration change with design basis fixed	Increased event coverage or frontier entry.
Useful events observed but poorly represented	Public feature/observation comparison	Better separation of relevant states and subsequent behavior.
Reward improves without task progression	Alternative incentive or learner test	Predicted behavioral transition, not merely larger return.
Behavior improves but varies across seeds	Replication or greater depth	Consistent progress under the stated conditions.
Training progress is absent from saved evaluation	Same-checkpoint, same-episode evaluator check; then retention test	Separate measurement disagreement from irreproducible behavior.
Declared learner updates do not occur	Learner/exploration repair	Changed effective settings and measured optimizer activity.

Appendix

Table A7: Examples of observations and the comparisons they motivate.

Object	Required content	Role in the next decision
Proposed test	Explanation, competing account, changed axis, prediction, falsifier, and requested depth	Select a comparison that can discriminate between explanations.
Bound comparison	Exact reference and treatment artifacts, compatible non-target stack, seed IDs, and declared replication unit	Reserve executable arms and bind learner-appropriate hyperparameter settings.
Completed evidence	Materialized settings, actual updates/depth, evaluation validity, and outcomes attached to the bound arms	Determine whether the requested contrast actually occurred and has interpretable results.
Memory update	Local prediction verdict, its scope, and links to the comparison and observations	Inform subsequent retrieval, revision, or another discriminating test.

Appendix

Table A8: From a proposed diagnostic test to a result that can guide later experiments.

Choice	Executed effect and eligibility
Native exploration	No additional modification to algorithm-native behavior.
Entropy profile	PPO entropy coefficient at least 0.005; SAC selects adaptive-entropy initialization 0.1 or 0.2.
Coverage profile	PPO entropy coefficient at least 0.01; SAC adaptive-entropy initialization 0.2 or 0.5; compatible off-policy warmup proposal $\max(1000,0.25b)$ .
Long warmup	SAC/TD3 or deep deterministic policy gradient (DDPG) warmup proposal $\max(5000,0.20b)$ , subject to the executable learning-start guard.
Controllable novelty	Training-only bonus over measured action-responsive observation coordinates; a targeted variant uses available candidate-goal coordinates.
Public-goal replay descriptor	Requires public goal atoms; the recorded implementation does not activate SB3 hindsight experience replay (HER) merely by selecting this descriptor.

Appendix

Table A9: Supported exploration settings for each learner. Here

b

is the requested number of learner-training steps.

Condition	DoorKey	Ant	Door
FARL reference	72.800	1,825.735	9.100
A1	21.960	56.860	64.055
A2	17.760	8.460	3.060
A3	5.560	51.960	10.060

Appendix

Table A10: Recorded search training for the ablation comparisons, in millions of interactions; final fixed-design training is accounted separately. Each column uses independently run searches with the indicated allocations.

Task	Intervention	Seed 200	Seed 201	Seed 202
DoorKey	A1	$0.00$	$0.00$	$0.00$
	A2	$0.00$	$0.00$	$0.00$
	A3	$100.00$	$100.00$	$100.00$
Ant	A1	$-6.7831$	$3.0899$	$-5.5665$
	A2	$-0.3452$	$0.3557$	$-0.3055$
	A3	$5.7242$	$1.2957$	$0.1823$

Appendix

Table A11: All final seed-level outcomes underlying Table 2 . DoorKey and Door entries are percent; Ant entries are signed displacement.

Intervention	Pick up key	Open door	Reach exit
A1	$18.00\pm 17.09$	$1.00\pm 1.73$	$0.00\pm 0.00$
A2	$46.67\pm 7.23$	$34.33\pm 17.67$	$0.00\pm 0.00$
A3	$98.67\pm 2.31$	$98.67\pm 2.31$	$98.67\pm 2.31$

Appendix

Table A12: DoorKey ablation stage diagnostics, percent, mean

\pm

seed SD. All three seeds train for 300k; each stage evaluation uses 100 episodes.

Setting	PPO	SAC	TD3
Hidden layers	$64,64$ (Tanh)	$256,256$ (ReLU)	$400,300$ (ReLU)
Learning rate / discount	$0.0003/0.99$	$0.0003/0.99$	$0.001/0.99$
Batch size	64	256	256
Rollout / epochs	2048 / 10	—	—
Replay / warmup	—	$10^{6}/100$	$10^{6}/100$
Train frequency / gradient steps	—	$1/1$	$1/1$

Appendix

Table A13: Fixed native-control learner settings. Unlisted settings retain the version-pinned library defaults; full constructor receipts are supplied.

Learner	Steps	Reach	Lift	Transport	Final	Any
PPO	1M	$70.67\pm 9.45$	$26.00\pm 6.93$	$1.67\pm 1.53$	$0.67\pm 1.15$	$1.67\pm 2.89$
	5M	$42.00\pm 3.61$	$9.33\pm 4.04$	$1.00\pm 1.00$	$1.00\pm 1.00$	$3.00\pm 1.00$
SAC	1M	$25.33\pm 8.02$	$4.00\pm 2.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$
	5M	$18.00\pm 9.54$	$5.33\pm 4.16$	$0.67\pm 1.15$	$0.00\pm 0.00$	$0.67\pm 1.15$
TD3	1M	$8.00\pm 10.39$	$0.67\pm 1.15$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.33\pm 0.58$
	5M	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$

Appendix

Table A14: Completed native Hand controls; the PPO 5M stage profile is plotted in Figure 3 e. Stage and Final entries are percent, equal-seed mean

\pm

sample SD. Final requires five consecutive native-success steps; Any is at least one successful step in the episode.

Steps/seed	Seed 0	Seed 1	Seed 2	Displacement
1M	113.08	37.57	36.41	$62.36\pm 43.93$
5M	96.86	59.41	89.40	$\boldsymbol{81.89\pm 19.82}$
10M	21.45	50.34	39.17	$36.99\pm 14.57$

Appendix

Table A15: Native-reward PPO on Ant. Per-seed means use 25 evaluation episodes; the aggregate is the equal-seed mean

\pm

sample SD.

Component	EW	EA
Policy input	27 raw + 30 difference features	27 raw coordinates
Reward	Forward-motion credit, action-diversity terms, coordinate-norm penalties	Speed/control-cost bands plus continuous velocity and cost terms
Learner / network	SAC / $256,256$	SAC / $256,256,128$
Optimizer learning rate	$2.5981\times 10^{-4}$	$3\times 10^{-4}$
Entropy / warmup	Automatic, initial 0.2 / 10k	Automatic, default / 20k
Replay	Ordinary replay	Ordinary replay

Appendix

Table A16: Selected Ant designs and their complete EW/EA configurations.

Setting	Steps/seed	Seeds	Episodes/seed	Displacement
EW	1M	0–4	5	$273.3\pm 15.4$
EW	5M	42–46	5	$313.1\pm 31.7$
EW	10M	42–46	5	$345.8\pm 14.2$
EA	1M	$6\times(0\text{--}4)$	5	$207.1\pm 85.6$
EA	10M	0–6	3	$281.1\pm 74.4$

Appendix

Table A17: Ant outcomes by training budget in Figure 3 d. Rows summarize recorded training cohorts; EA 1M includes six search configurations.

Setting	Selected A2C	PPO comparison
Learning rate	$1.8485447659\times 10^{-4}$	$1.3505226478\times 10^{-4}$
Discount	0.9957992252	0.9974352405
Entropy coefficient	0.00022588495	0.02108114065
GAE parameter	0.9032534459	0.9120453326
Success by seed	$(1,1,1,1,1)$	$(0,1,0,0,0)$

Appendix

Table A18: Same observation, reward, and exploration artifacts; different training configurations. Both use five seeds (100–104), 300k steps and 25 evaluation episodes per seed.

Figure 4: Candidate learning and selection at fixed training budgets. (a) DoorKey: five-seed mean exit frequency and sample SD for each of three 300k candidates. (b) Ant: all 121 candidate evaluations at 1M, with their five-seed SDs; the step line tracks the incumbent. The horizontal axes are policy-training steps in (a) and completed candidate evaluations in (b).

Figure 5: All 30 LiMEN search evaluations in Ant and DoorKey. Faint lines connect candidate results; dark lines show the best observed result. The horizontal axis counts search iterations.

Steps/seed	Design evaluations	Seeds/design	Seed runs	Search role
5k	256	1	256	Broad executable-design probe
100k	12	2	24	Short-budget learning comparisons
500k	26	3	78	Deeper tests of retained designs

Appendix

Table A19: Completed short-budget Hand experiments in the August 29 archive. A design evaluation is one candidate configuration trained on its assigned seeds; repeated evaluations of a component combination are counted separately. The 500k row contains 26 completed evaluations across three rounds. Training steps exclude evaluation interactions.

Figure 6: Early Hand learning within three independently trained cohorts of the retained design 031b6a159bb610d7 : iteration/trial 0/165 at 5k, 1/1 at 100k, and 4/1 at 500k. Curves are marginal stage-completion frequencies from all recorded training episodes, using nonoverlapping 1k-, 10k-, and 25k-step windows, respectively. Each seed contributes 25, 500, or 2,500 episodes in the corresponding panel. Lines average equally across the one, two, or three training seeds; bands show sample SD where multiple seeds are available. Each panel starts with a fresh policy and uses its own executed training configuration. The panels are separate training histories, rather than checkpoints of one continuously trained policy; saved-policy evaluations are reported separately.

Method	Policy depth	Successful seeds	Success (%)	Tokens / steps (M)
FARL	5M	$\boldsymbol{3/3}$	$\boldsymbol{97.11\pm 2.99}$	$2.58^{*}\,/\,408.4$
Eureka	5M	$0/5$	$0.00\pm 0.00$	$0.284\,/\,110.4$
LiMEN	5M	$0/3$	$0.00\pm 0.00$	$0.607\,/\,55.0$
Native PPO	5M	$2/3$	$1.00\pm 1.00$	$\mathbf{0}\,/\,\mathbf{15.0}$
Native SAC	5M	$0/3$	$0.00\pm 0.00$	$\mathbf{0}\,/\,\mathbf{15.0}$
Native TD3	5M	$0/3$	$0.00\pm 0.00$	$\mathbf{0}\,/\,\mathbf{15.0}$

Appendix

Table A20: Hand outcomes and expenditure. A successful seed has at least one successful evaluation episode. Success is the equal-seed mean

\pm

SD; policy depth is per seed. FARL costs cover the complete search; ∗ denotes estimated tokens.

Iteration/trial	Input dim.	Seed 0	Seed 1	Seed 2	Mean $\pm$ SD
11/0	104	4.00	98.67	0.33	$34.33\pm 55.74$
11/1	111	98.00	0.00	24.67	$40.89\pm 50.97$
12/0	111	0.67	97.33	74.67	$57.56\pm 50.55$
12/1	104	1.33	92.33	12.33	$35.33\pm 49.67$
13/0	111	99.00	98.67	93.67	$\boldsymbol{97.11\pm 2.99}$
13/1	104	87.33	94.00	91.33	$90.89\pm 3.36$

Appendix

Table A21: All ten Hand configurations evaluated at iterations 11–15 with 5M training steps per policy. Each row uses three training seeds and 300 final evaluation episodes per seed. Values are sustained relocation percentages; SD is across training seeds. The 104D and 111D designs use SAC; the 84D designs use PPO except iteration 15 / trial 1, which uses SAC.

Evidence	Reach	Lift	Transport	Final
Seed 0	100.00	100.00	100.00	99.00
Seed 1	100.00	99.67	99.00	98.67
Seed 2	100.00	97.33	94.33	93.67
Mean $\pm$ SD	$100.00\pm 0.00$	$99.00\pm 1.45$	$97.78\pm 3.02$	$97.11\pm 2.99$

Appendix

Table A22: The selected Hand cohort: marginal completion of each milestone over all 300 evaluation episodes per seed. These are not conditional transition probabilities.

Setting	Iteration 11	Iteration 12	Iteration 13
Network	$384,384,256$	$384,384,256$	$384,384,256$
Learning rate	$1.21296\times 10^{-4}$	$1.96651\times 10^{-4}$	$1.23911\times 10^{-4}$
Discount	0.987555	0.982773	0.987258
Target update $\tau$	0.003684	0.005543	0.009802
Warmup steps	200k	1M	1M
Batch size / replay capacity	$256/10^{6}$	$256/10^{6}$	$256/10^{6}$

Appendix

Table A23: SAC settings at iterations 11–13. Within each iteration, the two representations share the listed SAC settings.

Record	Observed content
Diagnosis	Transport is the earliest unresolved stage; an observation-feature contrast motivates a representation test.
Experiment	Iteration 12 tests two representations with identical reward, learner, training settings, exploration, 5M depth, and seeds 0–2.
Prediction	The proposed representation improves transport completion.
Measurement	Reference/alternative transport: 60.11/38.44%; final success: 57.56/35.33%.
Conclusion	The result contradicts the predicted improvement; the reference remains champion.
Memory	The associated hypothesis changes from testing to falsified, citing the matched experimental contradiction.

Appendix

Table A24: The Hand representation experiment, from its initial prediction to the measured outcome and the resulting memory update.

Setting	14/0	14/1	15/0	15/1
Learner / input dimension	SAC / 111	PPO / 84	PPO / 84	SAC / 84
Network	$384,384,256$	$512,512,256$	$512,512,256$	$384,384,256$
Learning rate	$4.87439\times 10^{-4}$	$1.95026\times 10^{-4}$	$1.10072\times 10^{-4}$	$1.17979\times 10^{-4}$
Discount	0.990106	0.967368	0.994906	0.994028
Batch size	256	64	128	128
Entropy coefficient	Automatic	0.032534	0.005	Automatic

Appendix

Table A25: Executed settings for the later completed Hand candidates. Every policy trains for 5M learner steps. The generated reward program is shared; prefix interactions are summed over three seeds. A dash marks a setting that does not apply to the learner. Column headings give iteration/trial.

Figure 7: Learning curves for nine of the ten completed 5M Hand configurations. The final raw-input SAC configuration is reported in Table A21 . Thin curves are the three training seeds, the thick curve is their mean, and shading is sample SD. Each point aggregates a 100k-step training window. Iteration 12 panels (c–d) form the matched representation experiment. Panel (e) is the selected design; panels (g–i) show the later completed alternatives.

Window ending	Reach	Lift	Transport	Final
1M	22.40	2.27	2.53	0.07
2M	100.00	99.20	60.13	33.47
3M	98.80	97.20	91.40	90.07
4M	99.53	98.13	93.53	92.07
5M	99.40	97.33	95.53	94.47
Final evaluation	100.0	99.0	97.8	97.1

Appendix

Table A26: Selected Hand design during training. Each row summarizes the 100k-step window ending at the indicated budget, as a percentage averaged equally over three seeds. Final evaluation uses 300 episodes per seed.

Figure 8: Learning progress of the Hand design selected at iteration 13. Lines and bands are three-seed means and SDs of 100k-step training frequencies. The objective completion curve shows how training extends beyond lifting and transport to sustained relocation.

Task	Method	Search steps	Extra training	Input tokens	Output tokens
DoorKey	FARL	72.800	9.500	2,629,000	205,025
	Eureka	80.000	5.000	78,116	170,675
	LiMEN	30.000	0	983,594	111,566
Ant	FARL	1,825.735	75.000	$\approx 0.2$ M total (estimated)
	Eureka	400.000	25.000	47,112	104,420
	LiMEN	900.000	0	376,478	52,042

Appendix

Table A27: EW and Hand ST resource accounting. Training steps are in millions. Tokens are recorded input/output usage, except the rounded Ant FARL total estimate. Hand FARL’s main-table estimate additionally accounts for one missing request.

Paper object	Reconstruction materials	Check
Task and interface	Environment/dependency identities, field meanings, adapter and evaluator definitions	Dynamics, input permissions, and success criteria agree with the declared task.
Co-designed recipe	Observation/reward source, learner constructor, hyperparameters, exploration profile	The reported candidate is one complete executable combination.
Diagnostic test	Generation context, reference/treatment artifacts, prediction, bound arms, and outcomes	A planned intervention can be distinguished from an executed comparison.
Preserved alternatives	Component identities, family keys, selection and retention records, actual training depths	Coverage and continued training refer to actual designs, not only proposed pools.
Reported outcome	Fixed-design seeds, checkpoint identity, evaluation episodes, and aggregation script	Equal-seed means and SD reproduce the tables without candidate or checkpoint splicing.
Figures and controls	Plotting scripts, frozen result registries, native-control checkpoint evaluations	All plotted cohorts and axis units are recoverable from their recorded observations.

Appendix

Table A28: Reconstructing the paper’s mechanisms and measurements. The records distinguish a proposed design, its executed configuration, and the evidence used to report its outcome.

Task	Native observation	Native reward and episode horizon
DoorKey	Symbolic $7\times 7\times 3$ egocentric image (object, color, state), direction, and a textual mission. The numeric EW wrapper uses 147 normalized image entries plus direction, giving 148 inputs.	Zero until reaching the exit, then $1-0.9t/T$ . The $5\times 5$ task has $T=250$ steps. Pickup and door opening receive no separate native reward.
Ant	27 coordinates: torso height, quaternion, eight joint angles, three root linear velocities, three angular velocities, and eight joint velocities. Global root $x,y$ are excluded. Contact-force features are disabled.	Forward velocity plus a survival term, minus $0.5\\|a\\|_{2}^{2}$ ; contact cost is inactive in this configuration. Episodes end on unhealthy state or at 1,000 steps. The 27D vector is already the benchmark’s chosen representation; EW retains it while withholding native reward and reward metadata.
Door	39 coordinates: two 18D frames of hand/gripper and object state, followed by the 3D goal. This includes supplied object and goal geometry, even when native shaping is withheld.	Meta-World’s dense reward combines grasp/readiness shaping with door-opening progress, using gripper effort, hand–handle geometry, and door angle; success sets reward to 10 in the v2 reward implementation. The experimental horizon is 500 steps.
Hand	39 coordinates: 30 hand positions and three 3D relative vectors (palm–object, palm–target, object–target). The experiment’s 84D ST wrapper adds 36 velocities and absolute palm, object, and target positions.	Dense reward is $-0.1d_{po}+\mathbb{1}_{h>0.04}(1-0.5d_{pt}-0.5d_{ot})+10\mathbb{1}_{d_{ot}<0.1}+20\mathbb{1}_{d_{ot}<0.05}$ . Here $h$ is object height and $d$ denotes Euclidean distance. Episodes last 200 steps. Evaluation additionally requires five consecutive successful steps.

Appendix

Table A29: Native observations and expert-designed reward terms. The public wrappers used in our experiments are specified separately below.

Axis	Eureka	LiMEN	FARL
$\phi$	Unchanged 148D public input.	188D: public input plus visible-object counts, nearest-object offsets, front-cell flags, time, and inferred key possession.	Unchanged 148D public input.
$r$	Key pickup/approach, door opening and doorway crossing, goal approach and an exit proxy, with small action/time penalties.	Visible key/door/goal changes and geometry-based shaping computed alongside the expanded observation.	Changes in soft counts of visible key, door, and goal categories, conditioned on pickup/toggle/forward actions; step penalty $0.0015$ .
$L,\lambda$	Fixed PPO; settings in Table A34 .	Fixed PPO; settings in Table A34 .	A2C with selected learning rate, discount, entropy and GAE parameters (Table A18 ).
$e$	PPO policy sampling.	PPO policy sampling with entropy coefficient 0.01.	Policy sampling; the selected coverage profile adds no intrinsic reward or A2C entropy floor.
$b$	$5\times 1$ M selected-reward runs.	One 1M selected run; no candidate passes the full-training threshold.	$5\times 1$ M evaluation runs, following 100k screening and five-seed 300k comparisons.

Appendix

Table A30: Selected DoorKey EW designs. All reported policies use 1M steps. Exact programs are included in the code supplement.

Axis	Eureka	LiMEN	FARL
$\phi$	Unchanged 27D input.	80D: orientation axes, bounded joint/velocity features, pairwise joint coordination, summaries, and time.	57D: all 27 public coordinates plus 30 pairwise coordinate differences.
$r$	Forward velocity gated by height/uprightness, small posture bonuses, and lateral, vertical, angular, joint-motion and action penalties.	Forward velocity with posture-dependent scaling, small upright/facing bonuses and motion/action penalties.	$0.8v_{x}+0.12\operatorname{Var}(a)+0.08\|\bar{a}_{0:4}-\bar{a}_{4:8}\|$ , minus coordinate-norm penalties.
$L,\lambda$	Fixed PPO.	Fixed PPO.	SAC; $256,256$ network, learning rate $2.5981\times 10^{-4}$ , $\gamma=0.991472$ , $\tau=0.007961$ .
$e$	PPO policy sampling.	PPO policy sampling; entropy 0.01.	Automatic SAC entropy initialized at 0.2; 10k random-action warmup.
$b$	$5\times 5$ M reported runs.	$3\times 10$ M reported runs.	Five runs per budget at 1M, 5M, and 10M; search first tests designs at 5k, 50k, 300k, and 1M.

Appendix

Table A31: Selected Ant EW designs. Native input and reward are in Table A29 ; fixed baseline solvers are in Table A34 .

Axis	Eureka	LiMEN	FARL
$\phi$	Unchanged 39D input.	121D: public-state transformations, hand/object/ goal relative vectors, opening-direction projections, alignment features, state summaries and time.	47D: 39 public coordinates plus eight generated distance, offset, magnitude and configuration features.
$r$	Three rewards combine reaching and opening progress, proximity bonuses and action penalties. Two additionally gate opening or motion by proximity to the handle.	Reaching progress, proximity-gated and alignment-weighted opening progress, goal proximity, relative progress and a small action penalty.	Negative hand–goal, object–goal and hand–handle distances; vertical-offset and gripper terms; mean squared-action penalty.
$L,\lambda$	Fixed PPO.	Fixed PPO.	SAC; $256,256$ network, learning rate $4.0943\times 10^{-4}$ , $\gamma=0.981102$ , $\tau=0.009037$ .
$e$	PPO policy sampling.	PPO policy sampling; entropy 0.01.	Automatic SAC entropy; 10k warmup.
$b$	Three designs, each 10M.	Three seeds, each 5M.	Two seeds, each 50k.

Appendix

Table A32: Door EW designs. Eureka’s reported 75/75 outcomes cover three generated rewards, so all three programs are preserved.

Axis	Eureka	LiMEN	FARL
$\phi$	Unchanged 84D public input.	396D: native/public inputs, scaled joint and velocity features, relative vectors in several frames, distances, alignments, geometry summaries, time and native feedback.	111D: 84 public inputs, 20 other derived features and seven selected relational features.
$r$	Native reward plus smooth reach, lift, controlled-lift, target-proximity, transport and late-placement terms, with a small action penalty.	Half native reward plus reach/lift/transport changes normalized by scene scale, contact-weighted progress and state-quality terms.	Half native reward plus bonuses for maintaining control after lifting and bringing the lifted object near target; penalty for losing contact.
$L,\lambda$	Fixed PPO.	Fixed PPO.	SAC; $384,384,256$ network, learning rate $1.2391\times 10^{-4}$ , $\gamma=0.987258$ , $\tau=0.009802$ .
$e$	PPO policy sampling.	PPO policy sampling; entropy 0.01.	Automatic entropy, target $-30$ , and 1M random-action warmup.
$b$	Five seeds, each 5M.	Three seeds, each 5M.	Three seeds, each 5M.

Appendix

Table A33: Selected Hand ST designs. Native controls use 39D inputs; all three automated methods receive the 84D public wrapper. Policy budgets refer to completed cohorts.

Setting	Eureka	LiMEN
Policy / value hidden layers	$64,64$ / $64,64$	$64,64$ / $64,64$
Activation	Tanh	Tanh
Learning rate	$3\times 10^{-4}$	$3\times 10^{-4}$
Discount / GAE	$0.99/0.95$	$0.99/0.95$
Clipping / value coefficient	$0.2/0.5$	$0.2/0.5$
Batch size / optimization epochs	$64/10$	$256/10$

Appendix

Table A34: Fixed automated-baseline PPO settings. These are experiment settings, not hyperparameters supplied by the environments.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

May 5, 2026

Discovering Reinforcement Learning Interfaces with Large Language Models

Reinforcement learning systems rely on environment interfaces that specify observations and reward functions, yet constructing these interfaces for new tasks often requires substantial manual effort. While recent work has automated reward design using large language models (LLMs), these approaches assume fixed observations and do not address the broader challenge of synthesizing complete task interfaces. We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated. We propose LIMEN (Code available at https://github.com/Lossfunk/LIMEN), a LLM guided evolutionary framework that produces candidate interfaces as executable programs and iteratively refines them using policy training feedback. Across novel discrete gridworld tasks and continuous control domains spanning locomotion and manipulation, joint evolution of observations and rewards discovers effective interfaces given only a trajectory-level success metric, while optimizing either component alone fails on at least one domain. These results demonstrate that automatic construction of RL interfaces from raw state can substantially reduce manual engineering and that observation and reward components often benefit from co-design, as single-component optimization fails catastrophically on at least one domain in our evaluation suite.

Akshat Singh Jaswal, Ashish Baghel, Paras Chopra

Lossfunk

Reinforcement Learning LLM-Guided RL Reward Design for RL

Ask PDF

Sep 14, 2026

Specifying Reward Functions for RL Without Environment Sampling

Enabling human stakeholders to specify reward functions that lead to their desired outcomes is a key challenge in deploying reinforcement learning agents. Preference-based methods such as online RLHF can reduce the burden of manual reward design, but they require repeatedly training policies, sampling trajectories from the real world, and eliciting feedback, making them impractical in settings where environment interaction is computationally expensive or unsafe. We introduce Experience-Free Autonomous Reward Specification (EARS), a method for learning reward functions from preferences without environment interaction. Our approach uses a structured LLM-mediated process to construct a small set of expressive reward features from a task description and the environment observation space, then strategically samples imagined trajectories in this feature space and learns feature weights from preferences over the imagined trajectory pairs. We evaluate on three long-horizon domains: pandemic lockdown regulation design, insulin administration for diabetes patients, and autonomous vehicle control on a highway. We compare EARS to baselines that also enable reward specification without environment interaction--namely, methods that directly prompt an LLM to generate a reward function. When learning from either ground-truth preference labels or preferences labeled by a LLM, EARS designs reward functions that are more aligned with the ground truth reward function that produced the preferences or LLM context than these baselines. These results suggest that preference-based reward specification remains effective without environment sampling, enabling practical reward design in settings where collecting real trajectories is costly or infeasible.

Stephane Hatgis-Kessell, W. Bradley Knox, Emma Brunskill

1Computer Science Department, Stanford University · 2Computer Science Department, The University of Texas at Austin

LLM-Guided RL Reward Design for RL Preference-Based RL

Ask PDF

Jun 1, 2026

RDA: Reward Design Agent for Reinforcement Learning

Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions. Recent work, such as Eureka, automates reward design by using an LLM to iteratively generate and refine reward code from task descriptions. However, they rely on coarse feedback signals such as success rate, which provide little semantic insight into the learned behavior. As a result, their trained policies achieve the final goal but are frequently poorly aligned with task instructions. We introduce the Reward Design Agent (RDA), a VLM-based agentic framework that injects semantic understanding into reward design. RDA decomposes tasks, visually evaluates trajectories, summarizes failure modes, and iteratively revises reward code to better align with task instructions. Across 12 tabletop manipulation tasks from ManiSkill and 4 whole-body manipulation tasks from HumanoidBench, RDA produces policies substantially more instruction-aligned than those of other baselines, while achieving comparable task success rates. Videos and the generated reward code are available on https://nitinkamra1992.github.io/reward-design-agent.

Hojoon Lee, Ajay Subramanian, Ben Abbatematteo +4

Meta · Holiday Robotics · Boston Dynamics

Agentic RL Reinforcement Learning Robot Skill Learning Robotic RL Reward Design for RL

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Discovering Reinforcement Learning Interfaces with Large Language Models

Specifying Reward Functions for RL Without Environment Sampling

RDA: Reward Design Agent for Reinforcement Learning