DRAL: Deep Reinforcement Adaptive Learning for Multi-UAVs Navigation in Unknown Indoor Environment

In plain words

GPS is imprecise indoors, and a drone that cannot carry heavy or power-hungry sensors is hard to fly autonomously there. This paper asks a single camera on a drone to guide it through an unknown indoor space toward a target, such as a package.

The authors train a control policy with deep reinforcement learning, which means the policy learns by trying actions in a simulator and keeping what works. During training, the learner can see the full simulated state. When the drone flies, it only has what its sensors show. The paper then extends the idea to several drones that lift one unknown load together on cables.

The reported results come from simulated indoor rooms. In those runs, the learned approach reaches the target faster and more often than three other learning methods. Real-world flight is claimed in the text but no real-world results are given.

TL;DR

• Deep RL policy (PPO with an asymmetric actor-critic) for three cable-carrying hex-rotor drones that navigate an unknown indoor room and transport an unknown payload, using a single camera §1 §3.1 §4.
• Author-reported: in Gazebo, success rate 0.824, 0.857 and 0.903 for three payload classes, against 0.502 to 0.786 for Q-learning, DQN and Sarsa tab. 1.
• The method is under-specified: no reward, observation or action definition, network, optimizer, or training budget is given §3.2.
• Several claims (real-world tests, ablations, visualizations, a multi-drone adaptive controller) are in the abstract or text but have no method or result in the source §1 §4 §5.
• No official code was found.

Problem

GPS is imprecise indoors, and SLAM is computationally expensive and weak where walls have few features §1 §2. Stereo depth is poor on texture-less regions §1. Greedy exploration methods produce suboptimal tours and overly cautious manoeuvres §2. Carrying an unknown payload adds dynamics uncertainty (mass, inertia, grasp point) that changes between tasks §1 §3.1. The paper assumes the drones fly in a simulated room with obstacles such as walls, and that the sensor has a 50 × 50 degree field of view and a range of 3 m §4.

Method

Overview. A team of hex-rotors is attached by cables to a rigid cylindrical payload §3.1 fig. 1. The paper writes the translational dynamics of the lumped drone-payload system eq. 1 eq. 2. A reinforcement learning formulation is defined as a tuple of states, actions, transitions, observations, reward and discount factor §3.2. The training objective is the expected discounted return §3.2. The policy maps observation histories to action distributions §3.2.

Steps.

1. Model the system. Each hex-rotor produces thrust from six motors, with thrust proportional to the squared motor speeds §3.1 eq. 1 eq. 2. The payload's drag and the wind terms enter the translational equations eq. 1 eq. 2. The paper states the total mass as the payload mass plus the drone masses §3.1.
2. Train in simulation with full state, then deploy with partial observations §3.2. The critic gets privileged information during training; the actor gets only observations at deployment §3.2.
3. Optimise the policy with PPO. The paper writes the clipped-ratio-style policy objective as one minimum over a single term, with the old policy as the denominator and an advantage from the old policy eq. 3 §3.2.
4. Estimate advantages with Generalized Advantage Estimation and fit the value function by regressing on return §3.2 eq. 4.
5. Coordinate the drones via adaptive control and plan global tours and minimum-time trajectories. These are named in the introduction, related work and results, but no equations or procedure for them appear in the method §1 §2 §4.

Pseudocode. The paper has no algorithm box. This is a reconstruction of the training loop from the text in §3.2, not the authors' algorithm.

# Reconstructed from the text, not an algorithm in the paper  [§3.2]
# Privileged state for the critic, partial observation for the actor
def train_step(policy, value_fn, envs):
    # Roll out the current policy in simulation  [§3.2]
    observations, privileged, actions, rewards = rollout(policy, envs)
    # Advantage from Generalized Advantage Estimation  [§3.2]
    advantages = gae(rewards, value_fn(privileged))
    # PPO objective; the paper's form has no clip term  [eq. 3]
    policy_loss = ppo_objective(policy, observations, actions, advantages)
    # Regress the value function on the return R_t  [eq. 4]
    value_loss = return_error(value_fn(privileged), rewards)
    update(policy, policy_loss)
    update(value_fn, value_loss)

Hyperparameters

Results

Author-reported. Setup: a Gazebo simulation of an indoor room, three drones carrying an unidentified payload, three runs per method, and three payload classes (Box, Package, Bucket) §4 tab. 1 fig. 2 fig. 3. Baselines are tabular Q-learning, Deep Q-learning and Sarsa §4. Hardware is not stated. The paper does not define "reach time" and does not state how many episodes make up one success rate.

Exploration progress over time for the four methods is shown in fig. 4 and described only qualitatively: shorter exploration duration, lower temporal variance, a more concise trajectory, and smoother flight §4. No variance or confidence intervals are reported.

Paper vs code

No official code.

Searched: the LaTeX source has no code link [raw/src/2409.03930/main.tex]; the arXiv record has no comments field with a link; there is no project page. GitHub repository searches for "DRAL drone", "DRAL uav" and "multi-UAV indoor navigation Kangtong Mo" returned no repository attributable to the authors. A GitHub user search for "kangtong" returned profiles, and the one with the most repositories (kangtong) lists unrelated projects; it cannot be tied to the authors, so it was not used.

Pitfalls

• No reward, no observation or action definition. The policy cannot be re-implemented from this paper alone. R(s, a), O and A are only symbols §3.2. Any reward and observation design would be an invention and must be taken from a later source.
• The policy loss is incomplete. eq. 3 is a minimum over a single term: the ratio times an advantage, with no clipping range and no expectation. The advantage is written as A with the old policy as superscript, but the text says GAE is used §3.2. Standard PPO clipping is not written; do not assume it is present without another source.
• The value loss is a norm, not a squared error. eq. 4 writes an L2 norm of R_t − V(s_t) with no expectation over a batch §3.2. Whether the training uses squared error, Huber or a norm is not stated.
• Discount factor and GAE parameters are missing. γ is defined as lying in [0, 1] §3.2; no value is given. The return R_t is defined only as a discounted sum §3.2. λ for GAE is not stated.
• Number of drones is inconsistent. eq. 1 sums over three drones, eq. 2 sums over two, and the text speaks of "dual-UAV-payload" dynamics §3.1. The experiment uses three drones §4. Which dynamics model the trained policy used is not stated.
• Rotor type is inconsistent. The text and eq. 1 use hex-rotors with six motors §3.1; fig. 3 caption says "three quadrotors" fig. 3.
• Dynamics equations have apparent typos. In eq. 2, the z-axis drag term uses V_wy(z) rather than V_wz(z), and the drag term for z uses φ_i. The thrust constant k_b is not defined §3.1 eq. 1 eq. 2. The load orientation is defined by Euler angles α, β, γ, with α named as the main tilt angle §3.1, but eq. 1 and eq. 2 use the drones' angles φ_i, θ_i, ψ_i. How the load angles enter the dynamics, and the "load distribution strategy" the text mentions, are not described §3.1.
• Sensing model is ambiguous. Each drone "perceives information within a 3-meter radius" §4, while the sensor is specified as a 50 × 50 degree field of view with 3 m range §4. Which one the simulation implements is not stated.
• Real-world claims have no results. The paper says the same setup parameters apply to its simulated and real-world experiments §4, and the conclusion says the method was shown "in both simulation and real-world settings" §5. Only simulation results appear in the source.
• Abstract and conclusion claim content that is not in the text. Visualization techniques for the trained network, a multi-drone adaptive control algorithm for cooperatively lifting an object, and ablation experiments are named §1 §5, but no method or results section covers them.
• Global tours and minimum-time trajectories are not defined. §2 and §4 credit these for the speed and coverage gains, but §3 does not describe them. Do not attribute the reported speed-up to a specific mechanism without more information.
• Domain randomization is named but not described. The introduction says the method uses domain randomization and reward structures for multiple drones §1; neither is described in the method §3.
• Success rate definition is incomplete. Success is described as transporting the payload from start to target without collision or "crush" §4. The number of trials behind each rate is not stated, and "reach time" is not defined tab. 1. The table header is also inconsistent with the text: the text describes "both scenarios" §4 while the table lists three payload classes.
• Baselines are not described. Q-learning, Deep-Q learning and Sarsa are compared §4, but their state, action, reward and training budgets are not given. Whether they used the same payload dynamics and sensing is not stated.
• Off-topic introduction text. One introduction paragraph discusses humanoid robots and quadrupeds §1, which does not match the drone setting. Treat its citations with caution.
• Sim-to-real is claimed, not shown. The introduction frames the work around zero-shot transfer §1; no transfer experiment is reported.
• No variance is reported. The text claims "diminished temporal variance" §4 with no numbers.

Lineage

• Builds on: Proximal Policy Optimization, cited in §3.2 (no card in this wiki yet); asymmetric actor-critic, named in §3.2 without a citation.
• Related work cited: decentralized adaptive control for cooperative payload transport §1; frontier-based exploration §2.
• Compared against: Q-learning, Deep-Q learning and Sarsa §4 tab. 1.
• Superseded by: not stated.