HG-DAgger: Interactive Imitation Learning with Human Experts

In plain words

Teaching a robot by example is easy at first: a person drives, the robot copies them. The copy goes wrong once the robot drifts into situations the person never showed it, and it has no idea how to get back. A common fix lets the robot drive and asks the person to say what they would have done in the spots the robot reaches. That is awkward for a human. They do not feel in charge while the robot is steering, and their reactions get sloppy.

HG-DAgger changes who holds the controls. The robot drives on its own until the person decides it is heading somewhere unsafe. Then the person takes over, steers back to a safe place, and hands control back. The robot only learns from the moments the person is in full control, so the examples are the person's own choices.

The method also keeps a note of how unsure the robot felt at each takeover. Those notes give a rough cutoff for "safe enough", which can be used to judge the trained policy in different places. The authors test this on a simulated and a real car that must weave between stationary cars.

TL;DR

• Interactive imitation learning where the human expert holds exclusive control during each takeover and decides when control switches, instead of the policy switching at random §II.
• Labels are collected only during expert-controlled recovery, and a doubt threshold is learned from takeover moments §II alg. 1.
• Doubt is the norm of the diagonal of an ensemble's output covariance eq. 3.
• Author-reported: fewer road departures and collisions than DAgger and behavioral cloning in simulation, and 0 of each in the real-vehicle test (five fixed scenarios) §IV-A §IV-C tab. 2.
• Needs a human who can spot and react to danger in real time during training §II footnote.

Problem

Behavioral cloning trains a policy on expert demonstrations alone. It suffers from data mismatch and compounding errors when the learner leaves the demonstrated states §I. DAgger collects expert labels at states the learner visits, by mixing expert and learner control with a probability β §I. That sampling needs the expert to label actions while the learner is in control, which can reduce training-time safety and, with humans, degrade the labels through perceived actuator lag §I. A human's control under shared authority is also hard to tune: β is hard to pick a priori and costly to tune on a physical system §I. The method assumes the human can recognise unsafe states and react quickly §II footnote.

Method

HG-DAgger keeps DAgger's iterative loop: train a novice on the aggregated data, roll out the combined system, and add new expert labels §II alg. 1. The difference is the gate. The expert's judgment, not a coin toss, decides who acts §II. The expert is in control whenever the state lies outside the permitted set of states judged safe §II.

The rollout policy for epoch i is given by eq. 1. The novice sees only the observation o_t = O(x_t), while the expert sees the full state x_t §II eq. 1.

\pi_i(x_t) = g(x_t)\,\pi_H(x_t) + (1 - g(x_t))\,\pi_{N_i}(o_t) eq. 1

g(x_t) = \mathbb{1}[x_t \notin \mathcal{P}] §II

Labels from epoch i are the expert actions at the expert-controlled states §II eq. 2.

\mathcal{D}_i = \{ (\mathcal{O}(x_t), \pi_H(x_t)) \mid g(x_t) = 1,\ x_t \in \xi_i \} eq. 2

The novice is an ensemble of neural networks. Its doubt at observation o_t is the l2-norm of the diagonal of the output covariance matrix C_t §II eq. 3.

d_N(o_t) = \lVert \operatorname{diag}(C_t) \rVert_2 eq. 3

During training, the doubt at each takeover moment is logged in I. The threshold is the mean of the final quarter of I §II eq. 4.

\tau = \frac{1}{\operatorname{len}(\mathcal{I})/4} \sum_{i=\lfloor 0.75 N \rfloor}^{N} \mathcal{I}[i] eq. 4

The paper says the final quarter is chosen so that the threshold uses many interventions but mainly from novices trained on more data §II. At test time the trained policy runs without the expert §II. The threshold is used only to evaluate the policy, by estimating a permitted set from it §IV-B eq. 5. The paper proposes but does not implement using the doubt as a control gate at test time §II §V.

\hat{\mathcal{P}} = \{ x_t \mid d_N(\mathcal{O}(x_t)) \leq \tau \} eq. 5

The full loop is given as Algorithm 1 alg. 1. Each block below restates it in the paper's notation.

# HG-DAgger  [alg. 1]
# Input:  expert pi_H, initial novice, behavioral-cloning labels D_BC
# Output: final novice policy and doubt threshold tau

def hg_dagger(expert, novice, bc_labels, epochs, rollouts):
    # Aggregate dataset starts from the behavioral-cloning labels  [alg. 1]
    dataset = list(bc_labels)
    # Doubt is logged at each takeover, across all epochs  [alg. 1] [§II]
    doubt_log = []
    for epoch in range(epochs):  # K epochs  [alg. 1]
        for rollout in range(rollouts):  # M rollouts per epoch  [alg. 1]
            for step in rollout_steps:  # T timesteps  [alg. 1]
                state = environment_state()  # x_t  [§II]
                observation = observe(state)  # o_t = O(x_t)  [eq. 1]
                # Gate g = 1 when the state is outside the permitted set
                # [eq. 1]
                expert_control = not in_permitted_set(state)
                if expert_control:
                    # Labels only while the expert has uninterrupted control
                    # [eq. 2]
                    dataset.append((observation, expert(state)))
                if takeover_starts_now(state):
                    # Log the novice's doubt at the moment of takeover  [§II]
                    doubt_log.append(doubt(novice, observation))
        # Retrain the novice on all data gathered so far  [alg. 1]
        novice = train(novice, dataset)
    # Threshold from the logged takeover doubts  [alg. 1] [eq. 4]
    tau = learn_threshold(doubt_log)
    return novice, tau

def doubt(novice, observation):
    # Covariance of the ensemble members' outputs at this observation  [§II]
    covariance = ensemble_output_covariance(novice, observation)
    # l2-norm of the diagonal of the covariance matrix  [eq. 3]
    return l2_norm(diagonal(covariance))


def learn_threshold(doubt_log):
    # Mean of the final quarter of the logged doubts, as written in  [eq. 4]
    start = floor(0.75 * len(doubt_log))
    final_quarter = doubt_log[start:]
    return sum(final_quarter) / (len(doubt_log) / 4)

All three methods share one behavioral-cloning initialisation and then run their own refinement epochs §III-A Training. DAgger also uses a β schedule (listed under Hyperparameters). The paper gives no pseudocode for DAgger itself §III-A Training.

Hyperparameters

Results

Author-reported. Setup: a simulated two-lane road with stationary obstacle cars, a bicycle model and a real MG-GS test vehicle, compared against DAgger and behavioral cloning §III-A §IV.

Simulation learning curves are against expert labels, with error bars for the standard deviation fig. 3 fig. 4. The paper reports HG-DAgger learning faster and more stably than DAgger and behavioral cloning in both road-departure and collision rates §IV-A fig. 3 fig. 4. Simulation risk-split results are in tab. 1.

On the test vehicle, all three policies were run on five fixed obstacle configurations §IV-C. The paper's summary table counts the first 5,000 samples tab. 2.

The paper reports that HG-DAgger's steering-angle distribution is 21.1% closer to the human driving data than DAgger's, by Bhattacharyya distance, and calls the test data too limited for statistical significance §IV-C tab. 2.

For the learned threshold, the paper shows risk maps for three thresholds, with the learned τ in the centre fig. 6. Across 40 obstacle configurations, the learned threshold is reported as near-optimal on every pixel-wise free-versus- occupied metric shown fig. 7 §IV-D.

Headline numbers (author-reported): in simulation, collisions are about 12 times lower and road departures about 20 times lower inside P̂ than outside it, and departures inside P̂ last less than half as long §IV-B tab. 1.

Paper vs code

No official code. The paper gives no repository or project page. Searches for "HG-DAgger" and "human gated dagger" on GitHub returned only third-party reimplementations, and an org search of the authors' lab organisation found nothing relevant. No repository was cloned.

Pitfalls

• Who controls what. The gate is a human judgment, so nothing in the paper automates it. The paper's test-vehicle setup has a human who takes control with a steering wheel and hands control back with a button [§III-A Vehicle Experiments]. How expert labels were generated in simulation is not stated.
• Doubt definition. The doubt is the l2-norm of the diagonal of C_t eq. 3, but the paper does not say which ensemble outputs form C_t (steering, speed, or both), how the covariance is estimated, or how many members the ensemble has.
• Threshold formula. As written, the sum in eq. 4 runs over indices from ⌊0.75N⌋ to N but is divided by len(I)/4. Those two counts differ by about one, and N and the index base are not defined. The paper also gives no minimum number of takeovers. Pick one convention and state it.
• Threshold is not a control gate here. τ is used only to draw the permitted set for evaluation §IV-B eq. 5. Using it at test time is future work §V.
• Takeover timing matters. Doubt is logged when the expert takes over, so I mixes takeovers from all epochs. The paper argues that the final quarter reflects later, better novices §II.
• Rates and denominators. Rates are "per meter" §IV-A, but the distance used for each method is not reported. Table 2 gives counts and rates for "the first 5,000 samples", and the rates do not equal count divided by 5,000. As a derived check (not stated in the paper), the counts and rates imply about 1,028 m of driving for behavioral cloning and about 980 m for DAgger, so the two rates are not on the same length basis. Treat the on-vehicle rates with that in mind.
• Small on-vehicle test. The on-vehicle evaluation uses five fixed scenarios, and the paper calls it limited. No variance is reported for the vehicle results §IV-C tab. 2.
• Label budgets differ. The final HG-DAgger epoch uses fewer labels than the other epochs §III-A Training footnote, and the exact number is not stated, so the total label count per method is not stated either. Learning curves are plotted against labels fig. 3.
• DAgger instability. The paper attributes DAgger's late-epoch instability to perceived actuator lag, but calls this a hypothesis §IV-A.
• Expert-state versus novice-state. eq. 1 and eq. 2 give the expert the full state x_t and the novice the observation o_t. Labels are stored as (O(x_t), π_H(x_t)), so the novice trains on observations, while the expert acts on full state.
• Unspecified training details. Architecture, optimiser, learning rate, batch size, training length and control rate are not stated. The paper gives no hyperparameters for the ensemble itself.
• Unsafe for slow humans. The method depends on a human who can spot and react to danger during training. The paper says it is not suitable where that is not possible §II footnote.

Lineage

• Builds on: DAgger, as the baseline and the source of the β-mixing idea (cited in the paper as Ross et al. §I); and EnsembleDAgger, which uses ensemble doubt for a robot-centric gate and sets its threshold ad hoc (arXiv:1807.08364, cited in the paper as [menda2018ensembledagger]) §II.
• Related: Confidence-Based Autonomy, where the novice can also request demonstrations, and whose threshold is set ad hoc (Chernova and Veloso, JAIR 2009, [chernova2009]) §II.
• Compared against: DAgger and behavioral cloning, both experimentally §IV.
• Superseded by: not known from this wiki.