Automated Reinforcement Learning: An Overview

In plain words

Setting up a reinforcement learning (RL) project takes many choices before training starts: how to describe the problem as states, actions and rewards, which learning algorithm to use, and which settings (learning rate, discount, network size) to pick. Today these choices usually rest on expert experience and trial and error, and small changes can make a large difference to the result.

This survey looks at "AutoRL": methods that make some of these choices automatically, in the same way AutoML does for ordinary machine learning. It walks through each choice in turn, from problem design to tuning, and then covers related ideas such as learning a learning rule, searching for network designs, and using large language models to write rewards or state code. It ends with the open problems: fair comparisons between papers, the cost of repeated full training runs, and robustness to random seeds.

The paper is a review, not a new method. It runs no experiments of its own.

TL;DR

• Survey of automated RL, focused on model-free RL §1.
• Organizes automation into MDP modeling (states, actions, rewards), algorithm selection, hyper-parameter optimization, learning-to-learn, neural architecture search and LLM roles §1 fig. 1 tab. 1.
• Gives one comparison table of technique families with key idea, advantages, limitations and settings tab. 1.
• No code and no experiments of its own; all numbers are author-reported by the cited works and are not comparable across papers §8.

Problem

RL needs several design decisions before training: the MDP definition, the algorithm, and its hyper-parameters. Each choice strongly changes results, and most are made by hand §1. AutoML methods cannot be transferred directly, because RL is sequential and each policy evaluation is expensive §1. The survey positions AutoRL as the RL analogue that automates these choices with an outer loop that feeds evaluation results back into each automated component §1 fig. 1.

Method

This is an overview, so the "method" is the taxonomy and the outer loop.

Scope. Discrete-time, fully observable MDPs; model-free RL emphasized §1 §2. The paper defines an MDP as a tuple (S, A, r, T, γ) and describes it as finite-horizon in §2, although §1 lists infinite-horizon MDPs as a variant and says the survey focuses on discrete-time fully observable MDPs §1 §2.

Outer loop (the card's rendering of Fig. 1; Fig. 1 is a diagram, not a published algorithm):

# AutoRL outer loop, as drawn in Fig. 1  [fig. 1] [§1]
# Input:  problem description, candidate sets per component
# Output: a configured agent and its evaluation

def autorl(problem, candidates, budget):
    # Repeat until the budget is spent or settings stabilize  [§1]
    for step in range(budget):
        # Define the MDP: state, action and reward choices  [§2]
        mdp = define_mdp(problem, candidates.mdp)

        # Pick an algorithm and its hyper-parameters  [§3] [§4]
        config = select_and_tune(mdp, candidates.algorithms)

        # Train and evaluate the agent  [fig. 1]
        agent, score = train_and_evaluate(mdp, config)

        # Feed the score back to the automated components  [fig. 1]
        candidates = update_candidates(candidates, score)
    return config, agent

MDP components §2 tab. 1:

• States (§2.1). Expert feature construction (polynomial features, coarse coding, tile coding) or learned embeddings (structure2vec, Pointer Networks), and adaptive discretization or aggregation. For polynomial features the paper gives the state as a product of observation terms, with integer exponents §2.1.1:

  s_i = \prod_{j=1}^{k} o_j^{\,c_{i,j}} §2.1.1

  where c_{i,j} is an integer exponent of the j-th observation in the i-th state feature §2.1.1. Automatic state-representation methods include adaptive tile coding with split heuristics, a genetic algorithm that evolves tilings, and state aggregation with growing neural gas §2.1.2.

• Actions (§2.2). Learned action representations (hyper-graphs, rank r §2.2.1), Gaussian versus Beta policies §2.2.1, and distributional policy optimization. The DPO update is written as §2.2.1:

  \pi_{k+1} = \Gamma\big(\pi_k - \alpha_k \nabla_\pi\, d(\mathcal{D}, \pi)\big|_{\pi=\pi_k}\big) §2.2.1

  where Γ projects onto the set of distributions, d is a distance and D is a distribution over advantage-positive state-action pairs §2.2.1. The paper says the distance is minimized with an implicit quantile network and the Wasserstein metric §2.2.1. Discretization maps each continuous dimension in [-1, 1] to K evenly spaced values, with a softmax over per-dimension logits §2.2.2. The grid is given as A_i = { 2j/(K-1) - 1 : j = 0..K-1 } §2.2.2. Stochastic Q-learning evaluates a random subset of a large discrete action set §2.2.2.

• Rewards (§2.3). Curriculum learning from goal states (reverse learning §2.3.1), bootstrapping from demonstrations or evolutionary populations §2.3.2, and reward shaping. For potential-based shaping the added term is §2.3.3:

  F(s, a, s') = \gamma\,\Phi(s') - \Phi(s) §2.3.3

  where Φ is a potential function and γ the discount factor §2.3.3. Learned or parametric shaping rewards are also covered §2.3.3.

Algorithm selection §3. Filter by problem class first (tabular versus function approximation, discrete versus continuous). Then choose by contextual bandits over algorithms per dataset §3, by epoch-based bandit selection per episode §3, or by auxiliary-task selection §3. The survey notes that algorithm choice interacts with hyper-parameter choice §3.

Hyper-parameter optimization §4.1 fig. 3. The survey groups methods into gradient-based (meta-gradients, §4.1.1), Bayesian optimization §4.1.2, multi-armed bandits and successive halving §4.1.3, evolutionary and population-based search §4.1.4, greedy λ selection for eligibility traces §4.1.5, reinforcement-learning-based tuning §4.1.6, and neural networks that map dataset meta-features to settings §4.1.7. Dynamic configuration, where settings change during a run, is covered in §4.1.6.

Learning to learn §5. A learned optimizer replaces the update rule, with the form given in eq. 1:

\theta_{t+1} = \theta_t + g_t\big(\nabla f(\theta_t), \phi\big) eq. 1 §5

Here g is a recurrent network with parameters φ §5. Other items: learning RL algorithms with a recurrent policy trained across MDPs §5, learning learning rates with a relative-entropy policy search §5, MAML-style meta-learning §5, and environment-design and reset policies §5.

Neural architecture §6. RL controllers that generate architectures §6, neuroevolution (DeepNEAT, CoDeepNEAT) §6, and RL-based choice of per-layer bit-widths §6.

LLMs §7 fig. 4. Four roles, A to D: LLM reward design, with evaluate-and-refine loops for reward code §7.1; LLM-driven algorithm evolution and recipe proposals §7.2; LLM-generated state code and wrappers §7.3; and LLMs as policies with memory, skills or action priors §7.4.

Hyperparameters

The survey has no training runs. Settings quoted from cited works, with the paper's own statement of each:

Results

Author-reported, as quoted by this survey. These come from different papers, with different budgets, seeds and benchmarks. The survey itself says these are not directly comparable §8.

Paper vs code

No official code. The paper has no repository link in its abstract, LaTeX source or arXiv metadata, and a survey has no implementation of its own, so no GitHub search was run for this card. The survey's own sources have their own code; they are not covered by this card.

Pitfalls

• Scope of the MDP definition. §2 defines a finite-horizon MDP, while the introduction lists infinite-horizon MDPs as a variant and says the survey focuses on discrete-time, fully observable MDPs §1 §2. Treat the horizon as unstated for the rest of the survey.
• Section numbers. The paper uses IEEEtran, which shows sections as Roman numerals and subsections as letters. This card writes them as arabic numbers: §2 is "II", §2.2 is "II-B", §4.1.3 is "IV-A, item 3". The mapping was derived from the LaTeX source and not checked against a compiled PDF.
• Only one numbered equation. eq. 1 (learning-to-learn update) is the only numbered equation in the source. The DPO update and the polynomial and potential-shaping formulas are unnumbered display math; they are cited by section.
• Cross-paper claims. The survey's concluding claim that automating components "consistently" gives double-digit gains over expert baselines §7.5 is a summary of separate papers with different setups. The survey itself warns about comparability in §8. Do not use it as evidence for a specific method.
• Vague method details. The DPO distance d is not specified beyond the Wasserstein choice with an implicit quantile network §2.2.1. Hyperband's budget schedule for RL is not stated. Learned state, action and reward methods do not give a generic recipe; the survey says generic state representation is not well studied §2.1.3.
• Reward shaping caveat. The survey reports that potential-based shaping keeps the optimal policy only under specific terminal-state conditions; for finite-horizon tasks with several terminal states, the cited work sets the potential of terminal states to zero §2.3.3.
• LLM claims are proposals. The survey notes that LLM outputs can be inconsistent or reward-hack, and recommends validation in the outer loop §7.5 §8. Reward-code results rely on evaluate-refine loops and expert baselines chosen by the cited authors.
• Evaluation noise. Random seeds and small implementation differences can change ranking; the survey asks for multi-seed reporting and uncertainty §8.
• Cost. With K configurations fully trained, cost grows roughly linearly with K §8. Bayesian optimization, evolutionary search with parallel evaluation and early stopping are the suggested ways to reduce this.

Lineage

• Builds on: AutoML (the survey cites it as the base idea; no card here) §1.
• Compared against: a broader AutoRL survey by Parker et al. (2022), which the survey cites as covering reward and environment design and model-based RL more deeply §1. This card does not use its content.
• Superseded by: not stated in the source.