A Survey on Reinforcement Learning Applications in SLAM

In plain words

A robot that explores an unknown room has to build a map of it while working out where it is inside that map. This is called SLAM. Reinforcement learning is a way for a program to learn good decisions by trying actions and keeping the ones that lead to rewards. This survey asks where that kind of learning has been used to help SLAM. The answer it gives is mostly about choosing where the robot goes next, how it finds its way around obstacles, and how it recognises a place it has seen before.

The authors first explain the two kinds of SLAM: passive, where something else moves the robot, and active, where the SLAM system chooses the motion. They then list the sensors that feed SLAM and the main families of reinforcement learning. Thirteen published studies are sorted into three groups and compared. Most of those studies were evaluated mainly in simulation, and the DQN-style methods, which learn a value for each action, appear more often than any other kind. The survey closes with the open problems, such as compute cost, safety, generalisation and delays.

TL;DR

• A survey, not a method: it reviews reinforcement learning (RL) in simultaneous localisation and mapping (SLAM) for mobile robots and autonomous vehicles §I.
• It sorts RL uses in SLAM into path planning (with exploration and obstacle avoidance), loop closure detection, and active SLAM §V.
• It compares 13 studies by simulator, deep-learning method, SLAM back end and RL algorithm tab. 1. Nine of the 13 use DQN-family methods (survey text and tab. 1).
• Simulation is the main testbed for the reviewed work, and Gazebo is the simulator used most often §V.C.
• It reports no benchmark, hyperparameters or code of its own. Use it for orientation and for the list of prior work, not as an implementation reference.

Problem

The survey addresses a gap in the literature: SLAM systems struggle with sensor noise, outliers, loop-closure ambiguity, dynamic scenes and real-time limits §I. It asks whether RL can improve exploration, localisation and map building, and where it has been applied §I. It does not state a formal problem definition of its own beyond the standard SLAM formulation.

Method

This is a literature review. Its structure is:

1. Background on SLAM as graph optimisation, passive versus active SLAM §II eq. 1.
2. Sensor inputs: wheel odometry, LiDAR, cameras, GNSS/INS, IMU and radar §III.
3. RL basics: return, Bellman equations, Q-learning, value-based, policy-based, actor-critic and deep RL §IV eq. 2 eq. 3 eq. 4 eq. 5 eq. 6 fig. 1.
4. Applications in robotics, autonomous vehicles and drones §IV.D.
5. A taxonomy of 13 studies §V fig. 2 tab. 1.
6. Challenges, future directions and conclusion §VI §VII §VIII.

The survey states the standard graph-SLAM objective: the configuration that minimises the sum of weighted constraint errors eq. 1. The RL definitions follow standard texts, with the return as a discounted sum over an episode eq. 2. The Bellman equations for the state value and action value are given in eq. 3 and eq. 4, and the state value as the policy-weighted average of action values in eq. 5.

The tabular Q-learning update is the only learning rule the survey writes out in full eq. 6:

# Tabular Q-learning update  [eq. 6]
# Input:  Q table (S, A); transition (s, a, r, s_next); alpha in (0, 1]
# Output: updated Q table
def q_update(q_table, s, a, r, s_next, alpha, gamma):
    # Best value available from the next state  [eq. 6]
    best_next = max(q_table[s_next])
    # Temporal-difference error from the observed reward  [eq. 6]
    td_error = r + gamma * best_next - q_table[s][a]
    # Move the stored value by a step of size alpha  [eq. 6]
    q_table[s][a] = q_table[s][a] + alpha * td_error
    return q_table

The active SLAM process is described as three steps. The survey attributes this to Makarenko et al. §II.B:

# Active SLAM decision cycle  [§II.B]
# Input:  current pose and map estimate, sensor readings
# Output: the action sent to the robot
def active_slam_step(pose, map_estimate, sensors):
    # 1. Identify candidate exploration locations  [§II.B]
    candidates = exploration_locations(map_estimate)
    # 2. Estimate the benefit of moving to each candidate  [§II.B]
    benefits = [benefit(pose, c, map_estimate) for c in candidates]
    # 3. Choose and execute the most advantageous action  [§II.B]
    return best_action(candidates, benefits)

The survey does not define how benefit should be computed, nor which candidate set to use. The text names information theory, optimal control and RL as tools for this step §II.B.

The taxonomy is:

• Path planning, with two subcategories: environment exploration and obstacle avoidance §V.A §V.A.a §V.A.b.
• Loop closure detection §V.B.
• Active SLAM, which includes the active-SLAM studies in tab. 1 §V.C.

Hyperparameters

The survey gives no training hyperparameters of its own, and the reviewed studies' values are not summarised. The only parameters stated in the text are the RL symbols below.

Results

Not applicable in the usual sense: the survey does not run experiments. The compiled findings are the descriptive summary of the 13 studies in tab. 1, plus the few numbers quoted in the text. All figures below are as reported by the cited works, not measured by this survey.

Numbers quoted in the text, as reported by the cited works:

• Chen et al. report zero collisions and almost no repeated exploration after 300,000 training rounds, with explored area within 5% of ROS slam-gmapping §V.A.a.
• SLAMuZero [Fang et al.] reached coverage 0.961, against 0.948 for Active Neural SLAM, and needed 1 million training frames where the baselines used 10 million §V.C.
• In Wang's comparison, DQN converged fastest and performed best among Q-learning, DQN and SARSA, especially in complex, high-dimensional environments §V.A.
• Kourtzanidis et al. report an RL agent for planar pose-graph optimisation that outperformed the g2o solver on challenging instances §III.
• Alcalde et al. report that both agents completed maps without collisions, the uncertainty-based one with longer paths and better maps §V.C.
• Most other entries are qualitative. Liu et al. report "significantly improved" path planning and "reduced collision rates" with no figures §IV.D.

Paper vs code

No official code. The survey has no implementation of its own. I checked the arXiv metadata, the abstract and the LaTeX source for code or project links and found none. A search of GitHub for a survey would return only awesome-list repositories, which are not code for this paper, so none was run.

Pitfalls

• Not a benchmark. The "results" are descriptions of other studies in different simulators, with different robots, metrics and budgets. The table cannot be used to rank the methods.
• Selection is not stated. The survey calls its 13 studies "representative rather than exhaustive" §VIII, but gives no inclusion criteria. Several studies discussed in the text are not in tab. 1, for example Wong et al., Su et al. and Liu et al. §IV.D, Chaplot et al. §IV.B and Kourtzanidis et al. §III.
• Table and text disagree for Alcalde et al. tab. 1 lists PPO as the RL algorithm. The text §V.C describes two agents, a completeness-based one and an uncertainty-based one, and names no algorithm. Check the original paper before you rely on either.
• Simulation dominates. The survey itself says most studies are evaluated mainly in simulation §V.C §VIII. Real-world transfer is not shown by most of the listed work.
• Unsourced strengths. Some sentences state RL benefits without a study behind them, for example that RL is "essential" for real-time SLAM in cars §IV.D and that actor-critic and PPO "further improve stability" in urban driving §IV.D. Treat these as the authors' view.
• Horizon and discount are underspecified. The return is a finite sum to T eq. 2, but the Bellman equations use the same symbols without restating T or a terminal condition eq. 3 eq. 4. The discount range [0, 1] includes 1, which matters for an unbounded horizon.
• Exploration is not defined. The survey says Q-learning is off-policy and learns from an exploratory policy, but names no exploration scheme §IV.A. Any implementation has to choose one.
• The active SLAM loop is not specified. The three-step process §II.B names no benefit measure and no candidate-generation method. The sensor-delay and computation challenges §VI are discussed in general terms with no quantities.
• Section numbers are from the class layout. The source uses the IEEEtran journal class, where sections are numbered in Roman numerals and subsections with letters. Citations use that layout, for example §IV.D for "RL-Based SLAM Applications". I did not compile the PDF to confirm the rendered numbers.
• Sensor section is high-level. Sensor descriptions in §III give no noise models, rates or calibration values.

Lineage

• Builds on: standard RL texts and the SLAM tutorials cited in §II and §IV. It does not extend a specific method.
• Revised version: the title block says this is a revised version of a paper in the Journal of Machine Learning and Deep Learning, vol. 1, no. 1, 2024 p. 1.
• Compared against: nothing. The survey compares the 13 reviewed studies with each other, not with a baseline.
• Superseded by: not stated in this wiki.