Recognition of Urbanized Areas in UAV-Derived Very-High-Resolution Visible-Light Imagery

In plain words

A drone photographs a village from low altitude, and each pixel of the resulting map is either built-up ground (roofs, roads, yards) or everything else (bare soil, grass, trees). The authors need that split to know which parts of the ground can be used to measure movement. The built-up parts keep their look and stay fixed to the ground, which is why the authors need to find them.

The authors compared two cheap ways to make the split using only the normal red, green and blue camera channels. One way is a simple colour formula per pixel, followed by a cut-off value. The other is a small neural network that looks at a pixel together with its neighbours. The network was more accurate, but the simple formula Excess Blue held up well on a second site photographed with a different drone, camera and season.

Each method is checked on two sites, one used to choose the settings and one used only for testing. Accuracy is judged with the Matthews correlation coefficient (MCC), because most pixels are non-built-up.

TL;DR

• Task: classify UAV RGB pixels as urbanized or non-urbanized, for a displacement-monitoring workflow §1.
• Compares 16 vegetation indices with a threshold each, and linear, MLP and CNN classifiers on pixel patches §2.2.
• Settings are chosen on the JERZ site; WIEL, a second site with a different UAV and season, is the test set §2.1, §2.2.1.
• Author-reported: the best network reaches MCC about 0.82 and accuracy about 0.96 on WIEL; the best index, Excess Blue (ExB), reaches MCC 0.61 and accuracy 0.87 there tab. 2 tab. 3.
• No official code found; the data are in a repository [see Paper vs code].

Problem

UAV photogrammetry can measure ground displacement from repeated surveys. The authors need a map of a monitored region with two classes: urbanized areas, whose objects keep their appearance and are fixed to the ground, and non-urbanized areas, which are mostly bare soil and vegetation. Horizontal and vertical displacement can be computed on urbanized areas, but on non-urbanized areas only vertical displacement can be computed, with lower accuracy §1.

Building such a map from high-resolution imagery needs manually made reference data, which is slow to produce, so the training set is usually small. The authors look for a method that uses only the existing UAV RGB orthomosaics and is simple to apply §1.

Earlier RGB-UAV studies mostly trained and tested on subsets of the same flight over the same area. The authors say this does not show how well a method holds up across flights, resolutions, seasons or areas §1.

Method

The paper compares two families on the same two datasets §2:

1. Vegetation-index (VI) thresholding. Compute a per-pixel index, then label pixels by comparing it to a single threshold §2.2.1.
2. Neural networks (NNs). Classify each pixel from an RGB patch centred on it, with a linear network, an MLP or a CNN §2.2.2.

Both are judged with the confusion-matrix metrics in §2.3.

Datasets

• JERZ (calibration and training) §2.1.1 fig. 1. UAV photogrammetry over about 1.5 km² of central Jerzmanowice village, Lesser Poland. Platform DJI Matrice M300 with a DJI Zenmuse P1 camera (35 mm F2.8 lens, full-frame sensor, 8192 × 5460 px). Ground sampling distance (GSD) 10 mm/px. Eleven monthly series from September 2021 to August 2022; January 2022 was excluded because of snow. Ground control and check points were measured with GNSS RTK. Processed in Agisoft Metashape Professional 1.6.2 (build 10247), reported accuracy ±1–2 cm horizontal and ±2 cm vertical.
• WIEL (test only) §2.1.2 fig. 2. UAV photogrammetry over about 1.0 km² south of Wieliczka. Platform DJI Phantom 4 RTK with a "generic camera" (1-inch CMOS, 5472 × 3648 px, 8.8 mm lens). GSD 15 mm/px. Four quarterly series, November 2021, February, May and August 2022. Processed in the same software, reported accuracy ±1.5–3 cm horizontal and ±4–5 cm vertical.
• Samples. In each dataset, 54 square sample areas of 4 m² were selected on a uniform grid, and every pixel in them was labelled by hand §2.1.1 §2.1.2. JERZ has about 24 million labelled pixels, about 12.9% urbanized. WIEL has about 3.8 million, about 13.3% urbanized §2.1.1 §2.1.2.
• Data availability: the data are in the ROD-BUK repository, DOI 10.58032/AGH/9TJA3T (stated in the paper's Data Availability Statement).

Vegetation indices and thresholding

The paper tests 16 indices tab. 1. Thirteen use RGB and three use CIELab. Before computing RGB indices, the channels are standardized to the range 0 to 1, and the chromatic coordinates are r = R/(R+G+B), g = G/(R+G+B), b = B/(R+G+B) tab. 1 note.

tab. 1

Steps, as the paper runs them §2.2.1 §3.1 fig. 3:

1. Compute all 16 indices for every JERZ pixel.
2. Cluster the indices with δ = 1 − |r| (Pearson r between index values) using UPGMA with arithmetic-mean linkage, and cut the dendrogram at δ = 0.1 eq. 8 fig. 5. This gives six clusters: (1) ExG, CIVE, GLI, ExGR, AL; (2) GBdiff, ExB, ExGB, RGBVI; (3) ExR, NGRDI, MGRVI; (4) MExG; (5) TGI, AI; (6) AB.
3. From each cluster, keep the index whose best threshold gives the highest MCC on JERZ. The result: ExG, ExB, NGRDI, MExG, AI and AB §3.1.
4. For each kept index, find the threshold that maximizes MCC on JERZ tab. 2. WIEL is only used to assess accuracy with those thresholds §2.2.1.
5. Direction: for ExG, NGRDI, MExG and AI, a value below the threshold means urbanized. For ExB and AB the rule is reversed §3.1 fig. 6.

Neural networks

• Input. An RGB patch centred on the pixel to classify. Patch sizes 1, 3, 5, 7 and 9 px for the linear network and MLP; 5, 7 and 9 px for the CNN §2.2.2. Only the centre pixel is classified §2.2.2.
• Output. A fully connected layer with two linear units and a softmax. A pixel is urbanized if its membership probability exceeds 50% §2.2.2.
• Architectures §2.2.2.
  • Linear: a network with no hidden layer and the two-unit softmax output. The paper does not describe it further, so "linear network" is read as softmax regression on the patch values.
  • MLP: one hidden layer with five units and tanh. Deeper networks and more units gave no gain in preliminary tests.
  • CNN: one convolution layer with five filters of size 3 and ReLU, followed by max pooling with pool size 3 and stride 2.
• Loss. Cross-entropy with C = 2 eq. 1.
• Training data. 80% of JERZ for training, 20% for validation, and all of WIEL for testing. No augmentation and no regularization §2.2.2 fig. 4.
• Optimizers. Limited-memory BFGS for the linear network §2.2.2 (refs. [50,51]); scaled conjugate gradient for the MLP; adaptive moment estimation for the CNN §2.2.2.
• Early stopping. Stop when the validation cross-entropy is greater than or equal to the lowest value seen so far, six times in a row §2.2.2.
• Model selection. Three networks of each structure are trained. The one with the highest validation MCC is used for testing §2.2.2.

# Early stopping on validation cross-entropy  [§2.2.2] [eq. 1]
# Input:  network, train_set, val_set
# Output: trained network

def train_until_stop(network, train_set, val_set):
    # A pass counts as a miss when its validation loss is not below the
    # lowest loss so far; stop after six misses in a row  [§2.2.2]
    best_loss = infinity
    misses = 0
    while misses < 6:
        train_one_pass(network, train_set)       # pass size not stated
        val_loss = cross_entropy(network, val_set)
        if val_loss < best_loss:
            best_loss = val_loss
            misses = 0
        else:
            misses = misses + 1
    return network

# Pixel classifier with best-of-three selection  [§2.2.2] [fig. 4]
# Input:  jerz_patches (N, P, P, 3); jerz_labels (N,) in {urbanized, not}
# Output: one trained network per (structure, patch size)

def select_network(structure, patch_size, jerz_patches, jerz_labels):
    # Split JERZ 80/20; validation is the 20 % part  [§2.2.2]
    train_set, val_set = split_80_20(jerz_patches, jerz_labels)
    best_net = None
    best_mcc = -infinity
    # Train three networks of the same structure  [§2.2.2]
    for run in range(3):
        net = new_network(structure, patch_size)
        net = train_until_stop(net, train_set, val_set)
        score = mcc(net, val_set)                # [eq. 7]
        if score > best_mcc:
            best_mcc = score
            best_net = net
    return best_net

Metrics

With the confusion matrix M = [[TP, FN], [FP, TN]] eq. 2:

\text{accuracy} = \frac{TP + TN}{TP + TN + FP + FN} \quad [\text{eq. 3}]

\text{recall} = \frac{TP}{TP + FN} \quad [\text{eq. 4}] \qquad \text{precision} = \frac{TP}{TP + FP} \quad [\text{eq. 5}]

\text{IoU} = \frac{TP}{TP + FP + FN} \quad [\text{eq. 6}]

\text{MCC} = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} \quad [\text{eq. 7}]

MCC is the paper's main criterion, for both threshold selection and evaluation, because accuracy is misleading on imbalanced data §2.3. F1 is not used §2.3.

Hyperparameters

Results

Author-reported. Setup: two sites §2.1. Thresholds are optimized on JERZ and the networks trained on JERZ, so WIEL is an independent test set §2.2.1 §2.2.2. The abstract's headline figures (Excess Blue about 87%, network about 96%) are WIEL accuracies tab. 2 tab. 3.

VI thresholding tab. 2. Columns: JERZ and WIEL for each metric.

tab. 2 (J = JERZ, W = WIEL)

• ExB is the best index on both sites by MCC tab. 2 §3.1.
• Season: optimized per series, ExB's threshold ranges from 0.100 (Sep 2021) to 0.131 (Dec 2021). Using the single value 0.108 lowers MCC by at most 0.04, against about 0.44 in the December 2021 series §3.1 fig. 7.
• AI's per-series thresholds vary much more, and the choice of dataset matters more for AI than for ExG §3.1 fig. 7 fig. 8.

Neural networks tab. 3. JERZ* is the validation part of JERZ; WIEL is the test set.

tab. 3

• On JERZ*, the CNN is best for MCC, IoU and accuracy; linear and MLP are slightly worse §3.2.
• On WIEL, linear and MLP are slightly better than CNN; the highest WIEL MCC is about 0.82, IoU about 0.73 and accuracy about 0.96 §3.2 tab. 3.
• Training-part results were similar to the JERZ* validation results, so no over- or underfitting was detected tab. 3 note.
• Patch sets that use the WIEL orthomosaics subsampled to the same physical patch size were tested and gave similar results, slightly worse than the unsubsampled set used in Table 3 §4.
• Feeding RGB plus the ExB value as a fourth channel, or brightness plus ExB in two channels, did not beat RGB-only inputs §4.

Otsu baseline, JERZ only tab. 4 §4. Otsu's method on each index gives much lower scores than the optimized thresholds, for ExB especially: MCC drops by 0.4308, IoU by 0.3652 and accuracy by 0.5100 §4.

tab. 4

Paper vs code

No official code. The paper names no code repository, and its data-availability statement names a data repository (ROD-BUK, DOI 10.58032/AGH/9TJA3T), not code. The arXiv metadata has no code link. No project page is linked. A GitHub repository search for "urbanized UAV excess blue", "Gruszczynski UAV", "urbanized areas classification UAV", "AGH photogrammetry" and "Puniach" returned no results.

Pitfalls

• Sample arithmetic does not add up. A 4 m² square at 10 mm/px is 200 × 200 = 40,000 px, but the paper gives 40,401 px (201²) per JERZ sample §2.1.1. At 15 mm/px a 4 m² square is about 133 px per side, but WIEL samples are given as 17,956 px (134²) §2.1.2. Both totals are consistent with the stated overall pixel counts for JERZ (54 × 40,401 × 11 ≈ 24.0 M) but not for WIEL (54 × 17,956 × 4 ≈ 3.9 M, versus the stated 3.8 M). Use the stated values, and note the discrepancy.
• Sampling procedure is unclear. The sample areas are described as "systematically selected in a uniform grid (random sample)" §2.1.1 §2.1.2. It is not clear whether this is a grid or a random draw.
• Train/validation split is not described. The 80/20 split of JERZ is not said to be random, by sample area, or by series §2.2.2. If it is random per pixel, neighbouring pixels from the same area can sit on both sides, which would inflate JERZ* validation scores. WIEL is independent, which limits this concern for the test numbers.
• Threshold search is not described. The grid, step size, search range and whether the comparison is strict (< vs ≤) are not stated §2.2.1.
• Accuracy is misleading here. About 87% of JERZ pixels are non-urbanized §2.1.1, so a classifier that labels everything non-urbanized scores about 0.87 accuracy. The paper uses MCC for this reason §2.3. For some indices, accuracy and MCC peak at different thresholds §3.1 fig. 8. AB reaches accuracy 0.86 on JERZ with MCC 0.23 tab. 2.
• Positive class is not stated. Recall, precision and IoU are defined in §2.3, but the text does not say which class is positive. Read them as urbanized-positive only if the figures confirm it.
• Early stopping. The paper does not say whether the returned network is the last one or the one with the lowest validation loss §2.2.2.
• CNN geometry depends on padding. A 5 × 5 patch through a 3 × 3 convolution and then a pool of size 3 with stride 2 gives a 1 × 1 output if there is no padding. The padding is not stated, so the layer sizes cannot be confirmed §2.2.2. Derived from the stated sizes, not from the paper.
• Missing training details. Learning rate, batch size, epoch limits, weight initialization, input normalization and seed handling are not stated §2.2.2. The three networks per structure may differ only by initialization. Their variation is not reported, so the Table 3 values are single numbers without spread.
• Input size, derived. A 9 px patch of RGB is 9 × 9 × 3 = 243 input values, so the largest network input is about 243 values per pixel. The inference cost per pixel grows with the patch area. This is arithmetic from the stated sizes.
• CIVE constant. Table 1 gives CIVE with a constant of 18.78745, but the text standardizes RGB to 0–1 tab. 1 note. With 0–1 channels, that constant dominates the rest of the formula. Whether the authors intended this is not stated; implement it as printed, and check it against the cited source.
• Dark and shadowed areas fail. Both ExB and NNs misclassify shadows. Bare soil and dead matter in shadow are read as vegetation §4 fig. 9. ExB calls dark pixels urbanized, while the NNs call them non-urbanized §4. The paper suggests adding a brightness rule before ExB, to label pixels below a brightness value as non-urbanized §4. This is a suggestion, not tested.
• Two sites only. Results come from one site each, with one UAV per site. The paper states that generalization needs larger datasets §4.
• Reference labels are manual. Pixel-level labels were made by hand. Their error rate is not estimated §2.1.1 §2.1.2.
• Season effects. The per-series threshold analysis uses JERZ only. WIEL covers four series, so the season effect on the WIEL thresholds is not reported in the paper §3.1 fig. 7.

Lineage

• Builds on: vegetation-index thresholding and the MCC criterion, with references [9], [37], [52]; the Otsu threshold [54]; UPGMA clustering [53]. The paper reuses these methods without modification.
• Compared against: other UAV land-cover studies cited in §1 and §4. The paper says their accuracies are not directly comparable because the datasets differ. It quotes up to 98% accuracy for a CNN on UAV RGB images with a DSM [27] §4.
• Superseded by: not stated.