A Review on Deep Learning in UAV Remote Sensing

In plain words

Drones now collect large numbers of aerial photographs, and checking them by hand is slow and inconsistent. Deep learning, the family of neural networks that learns visual patterns from labelled examples, is a common way to automate that work. This paper asks what has been done so far when the two are combined.

The authors searched the scientific literature and sorted the hits by the kind of network used, the kind of sensor used, the job the method did, and the setting it was used in: the environment, cities or farms. They summarise what worked and list some public drone datasets. They also point to open problems: processing images fast enough for real-time use, keeping fine detail in deep layers, learning from few labels, and handling images taken under different conditions. The paper is a map of the field. It does not add a new method.

TL;DR

A review of deep learning applied to UAV imagery, built from a systematic search of Web of Science and Google Scholar for 2016 to 2021 §3. The authors code 232 articles and proceedings by network, learning task, sensor and application context, and report the share of each §3. They also list six public UAV datasets §4 tab. 1 and discuss open research directions §5. The paper gives no model, no code and no new measurement, so every number is the authors' own tally.

Problem

• Visual inspection of UAV imagery is time-consuming, biased and inaccurate, and automatic methods that are fast and accurate are the aim §1.
• At the time of writing, no review combined deep learning with UAV-based remote sensing. Earlier surveys covered deep learning in remote sensing in general, in hyperspectral imaging, or machine learning for UAV images, but not deep learning for UAV images §1.
• UAV images are high resolution and vary in altitude, viewpoint, illumination and sensor. The same object therefore looks different across flights, which makes it hard to detect across domains §1.
• Deep networks usually need many labelled examples. The authors cite a rule of thumb of at least 5,000 labelled examples per category §2, and note that annotating object masks for segmentation is more expensive than annotating class labels or boxes §2.2.1.
• Standard architectures often reduce the input resolution. Keeping high resolution in the deeper layers, which UAV data could benefit from, is described as an open challenge §5 intro.
• Real-time use on a UAV is limited by compute, since models often run on data centres or GPUs §5.1.
• Each open direction the paper lists is in its own subsection of §5: real-time processing, dimensionality reduction, domain adaptation, attention, few-shot, semi-supervised and unsupervised learning, multitask, open-set, and photogrammetry.

Method

The paper proposes no model and has no algorithm to implement. Its method is a literature review with three parts: background on deep networks and tasks §2; a systematic search with coding of UAV-based deep learning papers §3; and a narrative of applications §3.2 to §3.4, public datasets §4, and perspectives §5.

Step 1. Search §3. The authors ran a topic search in Web of Science (WoS). The query combines deep-learning terms (deep learning, CNN, convolutional neural network), UAV terms (UAV, unmanned aerial vehicle, drone, RPAS) and remote-sensing terms (remote sensing, photogrammetry). It is limited to English and to articles, books, book chapters, book reviews, letters, proceedings and reviews, and to a set of WoS indexes. The query gives no date range; the included papers fall in 2016 to 2021 §3. WoS returned 190 records §3. The authors then searched Google Scholar with the same terms and added records that were not in WoS and that they judged to come from respected journals §3.

Step 2. Filter §3. Only papers that implemented approaches on UAV-based systems were kept. The paper gives no count of records removed at this step.

Step 3. Separate reviews §3 fig. 1. Review articles were set aside and used only for a word cloud of their vocabulary. The word-cloud text had 283.785 words as printed; words in fewer than 5% of documents and in more than 95% were dropped §3.

Step 4. Code each study §3 fig. 7. The 232 articles and proceedings were coded on four axes:

• network architecture: CNN or RNN (including CNN-LSTM), with other families such as GANs noted §2.1 §3;
• learning task: scene-wise classification, object detection, segmentation, or regression §2.2 §3;
• sensor: RGB, multispectral, hyperspectral or LiDAR §3;
• mapping context: environmental, urban or agricultural §3.

Each study was also tagged with its evaluation metric approach and objective §3. Journal, citation count, year and country were counted for the 232 papers §3 fig. 8.

Step 5. Cross-tabulate §3.1 fig. 9. The authors combine the four axes in one diagram and draw four observations from it: CNNs dominate; object detection is the largest task but segmentation has grown; RGB is the most used sensor; and the environmental context has the most papers, with forest applications the most common within it §3.1.

Step 6. Narrate §3.2 §3.3 §3.4. For the environmental, urban and agricultural contexts, the authors describe selected papers and the problems they solve. The narrative is qualitative, and the choice of papers is not stated.

Background used for the coding §2:

• A DNN is a network with two or more hidden layers that learns progressively more abstract features. The authors describe ReLU as the most popular activation at the time of writing, and mention Mish §2.
• CNNs and RNNs (LSTM) are the main supervised families. GANs are described as a generative family with a generator and a discriminator §2.1 fig. 3 fig. 4.
• Classification and regression differ in the output layer. A regression network replaces the last classification layer with one that estimates continuous values §2.2.
• Detection networks have a backbone (feature extraction), a neck (for example FPN, PAFPN or NAS-FPN, which fuse scales), and a head that classifies and regresses boxes §2.2.1.
• Two-stage detectors propose regions first (Faster R-CNN with an RPN is the classic example). One-stage detectors predict directly, are faster, and tend to be less accurate §2.2.1.
• Foreground and background boxes are very unbalanced during training, and non-maximum suppression is used to remove overlapping candidates §2.2.1.
• Instance segmentation separates objects of the same class, semantic segmentation does not, and panoptic segmentation combines both §2.2.1.

Open directions the paper lists §5. These are the authors' proposals; none is evaluated in the paper.

• Real-time processing §5.1: quantization (16, 8 or 1 bit weights), knowledge distillation from a large teacher to a small student, efficient layers such as MobileNets, and embedded platforms such as the NVIDIA Jetson.
• Dimensionality reduction §5.2: hyperspectral bands are often highly correlated. PCA is the classic remedy, usually run before the network. The paper points to band selection built into an end-to-end network.
• Domain adaptation and transfer §5.3: GAN-based image translation between domains, and category-aware adaptation.
• Attention §5.4: attention for high-resolution inputs. Vision Transformer, DeiT and DETR are named as not yet evaluated in remote sensing.
• Few-shot learning §5.5: one cited study fine-tunes a maize detector with 150 images from a new area, against 600 in the original training.
• Semi-supervised, unsupervised and contrastive learning §5.6: contrastive losses and clustering with augmented image pairs.
• Multitask learning §5.7: joint tasks such as segmentation with height estimation, or plant detection with plantation-line detection.
• Open-set recognition §5.8: a softmax threshold marks unknown pixels, then morphological filters clean them up.
• Photogrammetry §5.9: deep learning for feature matching in Structure-from-Motion, for filtering the DSM, and for colour balancing orthomosaics.

The coding procedure, as the paper describes it, is summarised here. It is not an algorithm from the paper and has no code behind it.

# Literature screening and coding  [§3] [fig. 7]
# Input:  records from Web of Science and Google Scholar
# Output: included studies, each coded by four axes

def screen_and_code(records):
    # Keep records that match the topic query  [§3]
    matches = [r for r in records if matches_query(r)]

    # Keep only studies whose system is UAV-based  [§3]
    included = [r for r in matches if uses_uav_system(r)]

    # Reviews go to the word cloud, not to the coding  [§3] [fig. 1]
    studies = [r for r in included if not is_review(r)]

    # Code each study on four axes  [§3] [fig. 7]
    codes = []
    for study in studies:
        network = code_network(study)    # CNN, RNN, GAN, ...
        task = code_task(study)          # classif., detection, segmentation
        sensor = code_sensor(study)      # RGB, multispectral, LiDAR, ...
        context = code_context(study)    # environmental, urban, agri.
        codes.append((network, task, sensor, context))

    return codes

Hyperparameters

The paper trains nothing, so it has no model hyperparameters. The parameters of the review's procedure are below.

Results

Author-reported tallies over the coded studies, not independent measurements. Setup: 232 articles and proceedings from WoS and Google Scholar, published 2016 to 2021 §3. The paper does not state the year-by-year split or the denominator for each percentage. Shares are as printed.

Cross-tabulated observations §3.1 §3.2 §3.3 §3.4: LiDAR studies are all environmental; RGB studies are mostly urban, then agricultural; object detection studies use mainly RGB, while segmentation studies use all four sensors. Forest applications are the most common environmental topic.

Public UAV datasets §4 tab. 1. The paper lists six datasets with their best reported method and score, as the authors give them:

The reference column uses the paper's citation keys, not arXiv identifiers. The paper also names other public aerial datasets for pre-training or benchmarking: DOTA, UAVDT, VisDrone, WHU-RS19, RSSCN7, RSC11 and the Brazilian Coffee Scene dataset §4. It links a dataset web page, which this card could not read (see Paper vs code).

Paper vs code

No official code.

The paper's abstract and text contain no code link, and the arXiv metadata has no comment field. The only online resource is the dataset page in §4. That page redirects to a Google sign-in, so its content was not read; it is a dataset page, not a code repository. A GitHub repository search for "uav deep learning review remote sensing" returned no results. No code fields are set on this card.

Pitfalls

• Counts do not add up. The WoS breakdown 136 articles, 46 proceedings and 10 reviews sums to 192, but the text says 190 §3. The later totals (232 articles and proceedings, 18 reviews) are consistent with the per-type counts, so the 190 figure is the one that is off. Use the paper's totals, and note the discrepancy.
• "Classification" has two meanings. In the evaluation-approach count it covers every task that outputs a class (97.7%) §3. As a learning task it means scene-wise classification only (5.4%) §3. The paper also says that scene-wise classification is not commonly used in remote sensing §2.2.1. Do not mix the two senses when comparing numbers.
• Percentages do not all sum to 100. The mapping-context shares total 100.2% as printed §3. This is probably rounding; the other sets sum to 100%.
• The search is not reproducible past WoS. Google Scholar records were added by the authors' judgement of which journals are respected §3. The inclusion rules and the number of records excluded are not stated §3.
• Word-cloud details are unclear. The count "283.785 words" is probably 283,785 with a European decimal mark. The filter wording "less than 5% occurrences" does not say whether it means document frequency or word frequency §3 fig. 1. The word cloud is only a rough indicator, and the paper notes that the word "UAV" barely appears in it §1.
• Figures are not checked against the text. The shares are given in the text §3. The diagram fig. 9 and the paper chart fig. 8 were not checked against these numbers in this card.
• Table 1 mixes metrics and datasets. The rows report AP for one and F1 for the others, on different datasets and at different GSDs, so the "Result" column is not a like-for-like ranking tab. 1. Three rows name the authors' own methods as "best".
• Several claims are not backed by a measurement in the paper. Examples: studies are said to show that deep learning is the most prominent solution for UAV disadvantages §1; CNN-LSTM and GANs are called emerging §6; and the paper calls the CGnet and DLNet segmentation models state of the art §2.2.1. Treat these as the authors' opinion.
• Quantization figures are quoted, not verified. The paper gives ResNet-18 at 89.2% top-5 on ImageNet and an XNOR-Net version at 73.2% top-5 §5.1. These were not checked against the cited sources here, and the paper gives no setup for them beyond the names.
• Missing data. The paper gives no year-by-year breakdown, no per-study table and no per-study citations for the shares §3. The application narrative cites examples, but the selection rule is not stated §3.2 to §3.4.

Lineage

• Builds on: none recorded as a method. The paper positions itself against earlier surveys: Zhang et al. (classification), Cheng and Han (object detection), Ball et al. (theory and challenges), Signoroni et al. (hyperspectral and enhancement), Ma et al. (seven remote-sensing subfields), and Bithas et al. (machine learning on UAV images) §1.
• Compared against: no quantitative comparison; the paper compares coverage with the surveys listed above §1.
• Superseded by: not known from content already in this wiki.