wikillm

Paper · 2021-03-12 · cs.RO

Goal-Driven Autonomous Exploration Through Deep Reinforcement Learning

Goal-driven exploration for a mobile robot with a laser. A global layer scores laser-derived points of interest by distance to the goal and a map term; a TD3 policy trained in simulation drives the robot between waypoints. Author-reported: 5 of 5 goals in both indoor sets, less time than NF and LP-AE, but longer travel distance than NF.

Actor-criticFrontier explorationReward designmobile-robotnavigationexplorationreinforcement-learningtd3waypoints

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A robot that only reacts to what its sensors see can get stuck. In a dead end or a corner it has no idea that the way out lies somewhere else, so it keeps pushing against the same wall.

This work combines two layers. The first one keeps a map of the surroundings while the robot moves and looks at its laser for spots that might lead onward. It rates those spots by how well they seem to lead toward the goal and picks one as the next waypoint. The second layer is a small learned driver. It was trained in a simulated room with deep reinforcement learning, and it steers the robot to the waypoint while avoiding obstacles, using only laser readings. The driver needs no map, and the planning layer helps it escape dead ends.

The authors report trials in indoor settings against planners and against a learning-only version. Their system reached the goal in every run and took less time than the other methods, except a path planner that already knows the map. It covered more ground than the nearest-frontier baseline.

TL;DR

  • A mobile robot (Pioneer P3-DX base) explores an unknown space toward a global goal, without prior map information §1 §4.1.
  • A global layer picks waypoints from points of interest in the laser data, scored by distance to the goal, distance from the robot and a map term §3.1 eq. 1.
  • A TD3 actor-critic policy, trained in a 10 × 10 m simulator, drives between waypoints from 21 laser groups and the waypoint's polar coordinates §3.2 §4.1.
  • Author-reported: 5/5 goals in both indoor sets, less time than NF and LP-AE, but longer distance than NF tab. 1 tab. 2.
  • The public code scores waypoints and builds its state differently from the paper; read "Paper vs code" before implementing.

Problem

Planning-based navigation needs a map or a prior plan §2, and it must re-plan when a new waypoint is chosen, so the robot waits for the new path §4.2. Reactive learned policies need no plan, but they only see the local surroundings, so they fall into local optima on large navigation tasks §1 §2. Goal-driven exploration needs both: the robot must choose where to explore and must steer locally without a map §1.

Method

The system has two layers. The global layer extracts points of interest (POI) from the laser, deletes stale ones and selects one waypoint with the IDLE score §3.1 alg. 1. The local layer is an actor-critic policy (TD3) trained separately in simulation. It receives the waypoint in polar form and a laser summary, and outputs a linear and an angular velocity §3.2. Mapping runs alongside, building an occupancy grid from laser and odometry §3.3.

  1. POI extraction §3.1 fig. 2. Two sources are used. A POI is added at a gap where two neighbouring laser readings differ by more than a threshold. Readings beyond the sensor range are treated as free space, and a POI is placed when sequential readings are out of range; where in the run it goes is not stated §3.1. No POI is created in places the robot has already visited §3.1.
  2. POI pruning §3.1. A POI that a later scan finds near an obstacle is deleted. A selected waypoint that cannot be reached for a number of steps is deleted, and a new waypoint is chosen §3.1.
  3. Waypoint choice §3.1 eq. 1 eq. 2 eq. 3 eq. 4 alg. 1. Each candidate POI gets an IDLE score: a saturating distance term from the robot, the distance from the candidate to the global goal, and an exponential map term. The lowest score is the waypoint. When the robot is within the goal threshold and within δ of the global goal, the goal itself becomes the waypoint alg. 1.
  4. Local state §3.2 §4.1. The front 180° of the laser is split into 21 groups; each group contributes its minimum range. The waypoint is added as distance and bearing relative to the robot's heading §3.2.
  5. Actor §3.2 fig. 3. Two fully connected layers, each followed by ReLU, and a tanh output layer with two values. The layer widths are not given in the text (only in fig. 3). Backward motion is not modelled, so the linear velocity is kept non-negative §3.2.
  6. Action scaling eq. 5. The two tanh outputs are scaled to the maximum linear and angular velocities.
  7. Critics §3.2 eq. 6 fig. 3. Two critics with the same structure and delayed parameter updates. The state passes through one fully connected ReLU layer. Its output and the action go through two separate transform layers, are combined, and pass through a ReLU and an output layer that gives Q. The smaller of the two Q values is used.
  8. Reward §3.2 eq. 7. A goal reward when the distance to the goal is below a threshold, a collision reward on contact, and otherwise the forward speed minus the absolute turn rate.
  9. Delayed goal credit §3.2 eq. 8 §4.1. When the goal is reached, the previous n steps receive a share of the goal reward that shrinks with how far back they are. n = 10 in the experiments §4.1.
  10. Loop alg. 1 §3.3. Read the sensors and update the map. Add POIs. If the robot is within the goal threshold, reach the goal or change the waypoint. Otherwise keep the current waypoint. Then get an action from the policy and execute it. The waypoint is re-chosen whenever it is reached.

Equations, as printed in the paper:

h(c_i) = \tanh\left(\frac{e^{\left(\frac{d(p_t,c_i)}{l_2-l_1}\right)^2}}{e^{\left(\frac{l_2}{l_2-l_1}\right)^2}}\right) l_2 + d(c_i,g) + e^{I_{i,t}} eq. 1

\tanh\left(\frac{e^{\left(\frac{d(p_t,c_i)}{l_2-l_1}\right)^2}}{e^{\left(\frac{l_2}{l_2-l_1}\right)^2}}\right) l_2 eq. 2

e^{I_{i,t}} eq. 3

I_{i,t} = \frac{\sum_{w=-k/2}^{k/2}\sum_{h=-k/2}^{k/2} c_{(x+w)(y+h)}}{k^2} eq. 4

a = \left[v_{max}\left(\frac{a_1+1}{2}\right),\ \omega_{max}\,a_2\right] eq. 5

L_c = L_s W_{\tau_1} + a W_{\tau_2} + b_{\tau_2} eq. 6

r(s_t,a_t) = \begin{cases} r_g & \text{if } D_t < \eta_D \\ r_c & \text{if collision} \\ v - |\omega| & \text{otherwise} \end{cases} eq. 7

r_{t-i} = r(s_{t-i},a_{t-i}) + \frac{r_g}{i}, \quad \forall i = 1,\dots,n eq. 8

Pseudocode of the global loop, following alg. 1:

Pseudocode
# Goal-driven exploration loop  [alg. 1] [§3.3]
# Input:  global goal g, thresholds delta and eta_d
# Output: one velocity command per control step

def explore_step(scan, odometry, goal):
    # Update the map from the laser and odometry  [§3.3]
    update_map(scan, odometry)
    # Add new points of interest from this scan  [§3.1]
    pois = add_new_pois(pois, scan, odometry)
    # Reached the goal threshold around the waypoint (D_t < eta_D)  [alg. 1]
    if distance(robot, waypoint) < eta_d:
        # The global goal itself was the waypoint: finished  [alg. 1]
        if waypoint == goal:
            return REACHED
        # Close enough to the goal: aim straight at it  [alg. 1]
        if distance(robot, goal) < delta:
            waypoint = goal
        # Otherwise take the candidate with the lowest IDLE score  [eq. 1]
        else:
            waypoint = best_poi(pois, goal)
    # The local policy turns the waypoint into a velocity command  [§3.2]
    state = make_state(scan, polar(robot, waypoint))
    return to_velocity(actor(state), v_max, w_max)

Pseudocode of the IDLE score, following eq. 1–eq. 4:

Pseudocode
# IDLE score of one candidate point; lower is better  [eq. 1]
# Units: the first two terms are metres; the unit of the map term is not stated.

def idle_score(robot, candidate, goal, cell_values):
    # Saturating distance term, limits l1 and l2  [eq. 2]
    ratio = distance(robot, candidate) / (l2 - l1)
    scaled = exp(ratio ** 2) / exp((l2 / (l2 - l1)) ** 2)
    distance_term = tanh(scaled) * l2
    # Straight-line distance from the candidate to the goal  [eq. 1]
    goal_term = distance(candidate, goal)
    # Mean cell value in a kernel of size k around the candidate  [eq. 4]
    info = mean_over_kernel(cell_values, candidate, k)
    # Exponential of that mean  [eq. 3]
    map_term = exp(info)
    return distance_term + goal_term + map_term

Pseudocode of the local input, actor and critics:

Pseudocode
# Local policy input: 21 laser groups plus the waypoint  [§4.1] [§3.2]
# Shape: (23,)

def make_state(scan, waypoint_polar):
    # Minimum range in each of 21 groups over the front 180 degrees  [§4.1]
    groups = min_per_group(scan, count=21)
    # Waypoint distance and bearing relative to the robot  [§3.2]
    return concat(groups, waypoint_polar)

# Actor: two fully connected layers with ReLU, tanh output  [§3.2]

def actor(state):
    # Hidden widths are not stated in the text  [fig. 3]
    hidden_1 = relu(dense_1(state))
    hidden_2 = relu(dense_2(hidden_1))
    # Two raw outputs in (-1, 1)  [§3.2]
    return tanh(dense_out(hidden_2))

# Scale raw outputs to velocities  [eq. 5]

def to_velocity(raw, v_max, w_max):
    # Linear velocity is never negative: no backward motion  [§3.2]
    linear = v_max * (raw[0] + 1) / 2
    angular = w_max * raw[1]
    return linear, angular

# Twin critics: state and action are merged  [§3.2] [eq. 6]

def critic(state, action):
    # State branch: fully connected layer with ReLU  [§3.2]
    state_layer = relu(dense_state(state))
    # Separate transforms of state and action, summed, then ReLU  [eq. 6]
    combined = relu(
        state_transform(state_layer)
        + action_transform(action)
        + action_bias)
    return dense_q(combined)

# The smaller of the two critic values is used  [§3.2]

def q_value(state, action):
    return min(critic_1(state, action), critic_2(state, action))

Hyperparameters

Values are as stated in the paper. "Not stated" means the paper does not give the value; the code values are in "Paper vs code".

NameValueWhereNotes
Training episodes800§4.1About 8 hours on the training computer §4.1
Steps per episode (cap)500§4.1Episode also ends at goal or collision §4.1
Max linear velocity v_max0.5 m/s§4.1 eq. 5Same value on the real robot §4.1
Max angular velocity ω_max1 rad/s§4.1 eq. 5Same value on the real robot §4.1
Delayed reward window n10 steps§4.1 eq. 8
Parameter update delay2 episodes§4.1Paper's wording; see "Paper vs code"
Training environment10 × 10 m simulated room, 4 box obstacles§4.1 fig. 4Obstacles, start and goal randomised every episode §4.1
Sensor and action noiseGaussian, σ not stated§4.1Added to sensor and action values §4.1
Laser input21 groups, minimum of each, front 180°§4.1 §3.2Two RpLidar sensors, range 10 m §4.1
Kernel size k1.5 m§4.1 eq. 4
Distance limits l1, l25 m, 10 m§4.1 eq. 2
Waypoint and global goal reachedwithin 1 m§4.1
Goal threshold η_Dnot statedeq. 7 alg. 1Value of the distance threshold in the reward
Goal-near threshold δnot statedalg. 1
Goal reward r_g, collision reward r_cnot statedeq. 7
POI gap thresholdnot stated§3.1
Map cell values cnot statedeq. 4Scale and source of c not defined
Actor/critic hidden widthsnot stated in textfig. 3Shown in the figure only
Learning rate, optimiser, batch size, discount, target-update rate, target-noisenot statedNot given anywhere in the paper
Training computerNVIDIA GTX 1080, 32 GB RAM, Intel Core i7-6800K§4.1
Real-robot computerIntel NUC i7-8559U, 16 GB RAM§4.1Robot base Pioneer P3-DX §4.1
Qualitative-run computerLaptop, RTX 2070M, i9-10980HK, Ubuntu 18.04, ROS Melodic§4.1
MappingSLAM Toolbox, top laser and odometry§4.1Global map built from the top sensor only

Results

Author-reported. Setup: simulation training in a 10 × 10 m room §4.1; quantitative runs on the real robot in two indoor environments, five runs per method §4.2. Environment 1 has smooth walls and several local optima; the goal is at (-12, 15) §4.2. Environment 2 has furniture, chairs, tables and glass walls, and its goal is not stated §4.2. Baselines: NF, the nearest frontier method, adapted to include goal distance §4.2; LP-AE, the same system with a ROS local planner instead of the learned layer §4.2; GD-RL, the learned policy without the global layer §4.2; PP, a Dijkstra path on a known map §4.2. Statistics for each method are computed over successful runs only §4.2. Map size is measured from known cells only §4.2.

Table I, environment 1 (5 runs per method) tab. 1

MethodAv. Dist. (m)Av. T. (s)Min. Dist. (m)Min. T. (s)Max. Dist. (m)Max. T. (s)σ(Dist.)σ(T)Av. Map (m²)Goals
GD-RL77.02171.8270.17147.0779.49184.983.8614.96580.635/5
NF40.74109.1134.6774.650.5150.256.7329.31475.755/5
LP-AE40.02123.5635.4396.4846.02155.44.9525.49491.575/5
GDAE (this paper)41.4288.0334.7768.2454.06118.058.3320.42492.015/5
PP32.2661.5332.0359.5232.3263.880.221.89-5/5

Table II, environment 2 (5 runs per method) tab. 2

MethodAv. Dist. (m)Av. T. (s)Min. Dist. (m)Min. T. (s)Max. Dist. (m)Max. T. (s)σ(Dist.)σ(T)Av. Map (m²)Goals
GD-RL73.12206.0268.97172.9577.37234.124.230.88986.923*/5
NF56.44188.0252.81178.963.13205.785.7915.371014.873/5
LP-AE82.59296.6856.13187.16103.89431.2719.73100.761031.254/5
GDAE (this paper)67.79156.5454.31124.379.69197.09.6627.64791.65/5
PP50.5381.1648.779.3453.3283.152.041.6-5/5

* Successful only after human intervention tab. 2.

Reading the tables against the text: in both environments GDAE needed less time than NF, LP-AE and GD-RL, and only PP was faster tab. 1 tab. 2 §4.2. Its travel distance was longer than NF's in both environments tab. 1 tab. 2. In environment 2, GDAE's map was smaller than the other learned or planned methods' maps tab. 2.

Qualitative runs have no metrics. They show a goal at (22, 12) past a room corner, a goal at (23, 2) from a start inside a local optimum, and a goal at (10, 10) through clutter §4.3 fig. 7. Longer runs reach goals at (12, -90) and (60, -7), and a goal 100 m away across an underground parking lot at (100, 0) §4.3 fig. 8.

Paper vs code

The official repository is https://github.com/reiniscimurs/GDAE (owner: an author), at commit fc793eda8de23bed98ba3acd32908843c535510f, linked from the paper's footnote §4.3. It has no LICENSE file, so no code is quoted here; citations below are paraphrases.

  • Scope. Code/ holds only the deployment loop and the environment wrapper GDAM.py GDAM_env.py. There is no TD3 training code, no critic and no reward function in this repository. The README points to a training example in another repository (see below), which the paper does not name as its training code.
  • Launch files, world and checkpoint are missing. The ROS launch file is looked up in an assets/launch folder that is not in the repository GDAM_env.py. The trained checkpoint path is a hard-coded personal directory GDAM.py.
  • Waypoint score. The paper's IDLE score has a saturating distance term, a goal term and an exponential map term eq. 1. The code uses only a different branch: distance to the robot is clipped piecewise (0 below 5 m, 5 between 5 and 10 m) and the distance to the goal is added; there is no tanh, no l1/l2 and no map term GDAM_env.py. The flags that would enable the other branches are fixed to off [code: Code/GDAM_env.py#L553-L554]. The initial best value is the raw Euclidean distance to the first node, not that node's score, so the comparison mixes two scales GDAM_env.py.
  • Goal switching. Algorithm 1 switches to the global goal inside a distance δ alg. 1. The code has no δ; the goal is just another node scored by the same heuristic GDAM_env.py [code: Code/GDAM_env.py#L599-L609].
  • Arrival thresholds. The paper says waypoints and the global goal are reached at 1 m §4.1. The code uses 1.0 m for waypoints [code: Code/GDAM_env.py#L436-L437] and 1.5 m for the global goal [code: Code/GDAM_env.py#L237-L241]. At 1.5 m the code prints a message and then loops forever, so the process never returns.
  • Local state size. The paper says the front laser is bagged into 21 groups §4.1. The code takes the minimum of two lidars per beam and then forms 19 groups, with widths that vary and one beam skipped at most boundaries GDAM_env.py. Readings are capped at 10 m and divided by 10 GDAM_env.py [code: Code/GDAM_env.py#L212]. The distance to the waypoint is capped at 5 m and divided by 10, and the bearing is mapped to [0, 1] [code: Code/GDAM_env.py#L498-L499].
  • Actor input. The paper's state is laser plus waypoint §3.2. The deployment loop feeds 19 laser values, the previous two actions and the two waypoint values, which is also 23 values GDAM.py. The first input of each run is ordered differently (waypoint, then two zeros), so only the first action differs GDAM.py [code: Code/GDAM.py#L48-L50].
  • Actor network. Hidden widths are 800 and 600, with ReLU and a tanh output GDAM.py. The paper gives these only in its figure fig. 3.
  • Velocity mapping. Linear velocity is (a1 + 1)/4, which equals a maximum of 0.5 m/s, matching eq. 5 with v_max = 0.5 GDAM.py. Angular velocity is taken as is, which matches ω_max = 1 GDAM.py.
  • Behaviour not in the paper. Acceleration and deceleration limits per step GDAM_env.py GDAM_args.py. Angular commands below 0.3 are squared and divided by 0.3 [code: Code/GDAM_env.py#L317-L325]. Collision recovery with a turning lock GDAM_env.py. A stuck-robot recovery when the standard deviation of the last 400 positions is below 0.16 [code: Code/GDAM_env.py#L519-L528] GDAM_env.py.
  • Node rules not in the paper. Suppression radius 2.5 m, deleted-node radius 0.75 m, deletion distance 0.3 m, memory caps of 400, 500 and 500, waypoint removal after 700 steps, and waypoint re-selection every 50 steps GDAM_args.py GDAM_env.py.
  • POI sources. Gap threshold 1.5 m, applied only when both readings are below 8.5 m GDAM_env.py. Paper's second source (out-of-range readings) is implemented as runs of more than 35 readings between 4.5 and 9.9 m, with POIs placed at a fixed 4 m GDAM_env.py. Readings at the 10 m cap do not count. A third source, for gaps between close readings GDAM_env.py, is not in the paper.
  • Global plan. The code subscribes to the ROS move_base global plan and adds points from it when the current waypoint is more than 5 m away [code: Code/GDAM_env.py#L147-L148] GDAM_env.py. It also sends each chosen waypoint to move_base GDAM_env.py. The paper describes the learned layer only.
  • Other repository (DRL-robot-navigation, MIT, commit 1adfe511d6f3ff5d7d3bdc357914501c29646c39). Linked from the GDAE README as an example of training. It uses a 3D Velodyne simulator and PyTorch, and it is not verified to be the paper's training code. Its reward is +100 on goal and -100 on collision, with a shaping term based on speed, turning and obstacle distance velodyne_env.py. It has no delayed credit as in eq. 8. Its discount is 0.99999 [code: drl-robot-navigation/TD3/train_velodyne_td3.py#L232], its batch 40 [code: drl-robot-navigation/TD3/train_velodyne_td3.py#L231], and its policy update every second gradient iteration, not every second episode [code: drl-robot-navigation/TD3/train_velodyne_td3.py#L173] [code: drl-robot-navigation/TD3/train_velodyne_td3.py#L236]. It maps the actor's linear output to [0, 1] in its own units, while the paper uses 0.5 m/s [code: drl-robot-navigation/TD3/train_velodyne_td3.py#L24]. Its hidden widths are also 800 and 600 train_velodyne_td3.py.

Pitfalls

  • Missing values. The paper does not state the goal reward, collision reward, goal threshold η_D, δ, the POI gap threshold, the scale of cell values c, the optimiser, learning rate, batch size, discount, target-update rate, or noise scales. An implementer has to choose these, and the code and the other repository use different values.
  • Map term units. With l1 = 5 m and l2 = 10 m, eq. 2 gives about 0.18 m at 0 m, about 0.50 m at 5 m and about 7.6 m at 10 m, and it never exceeds l2 = 10 m. These numbers are computed from the printed formula. The map term e^{I} is added to metres, and its scale depends on c, which is not defined.
  • Which goal D_t measures. Eq. 7 and Algorithm 1 use D_t without saying whether it is the distance to the waypoint or to the global goal. The code uses the waypoint GDAM_env.py.
  • Delayed credit edge cases. Eq. 8 does not say what happens when fewer than n earlier steps exist, or whether credit goes to every step or only to steps before a goal. An implementation that indexes back from the goal step must guard against negative indices.
  • No reversing. The linear velocity is non-negative §3.2. The local policy cannot back out of a dead end; the global layer has to choose a new waypoint for that.
  • Sensors. Mapping uses only the top laser, and the local policy uses both §4.3. Low obstacles such as parking bumpers (about 0.2 m high) are therefore seen by the policy but not by the map, so a planner-based route cannot avoid them §4.3.
  • Failure modes reported. A glass wall at lidar height was tried through by NF in two of five runs; a chair's legs were a path for LP-AE; obstacles looked like sensor noise until several readings confirmed them §4.2.
  • Results are author-reported. Five runs per method in two environments §4.2. GD-RL needed a human in Table II tab. 2. Statistics use successful runs only §4.2. PP uses a known map, so it is an upper bound rather than a learned rival §4.2. NF was modified to include the goal distance §4.2.
  • Generalisation. Training used the same robot model that was deployed, and the authors name other robots and robot dynamics as future work §5.
  • Training cost. 800 episodes took about 8 hours on the stated GPU and CPU §4.1.

Lineage

  • Builds on: the IDLE evaluation method [cimurs2021information], cited in §3.1; TD3 [fujimoto2018addressing], cited in §3.2; and the bagged 180° laser input [choi2020fast], cited in §3.2.
  • Compared against: NF, the nearest-frontier method [da2020novel]; LP-AE and GD-RL, variants built by the authors §4.2; PP, a planner on a known map §4.2.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “Goal-Driven Autonomous Exploration Through Deep Reinforcement Learning” by Reinis Cimurs, Il Hong Suh, Jin Han Lee (arXiv:2103.07119), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/reiniscimurs/GDAE at commit fc793eda8de2 (none (no LICENSE file in repository)). Any quoted code is shown under that license.