Crop mapping from image time series: deep learning with multi-scale label hierarchies

In plain words

Farms grow many kinds of crops, and the satellites that photograph them pass over every few days. This paper uses a year of those pictures to label each small patch of land with its crop. The crop names form a family tree. "Orchard" splits into apple, pear and vine, and "grassland" splits into meadow and pasture. Rare crops such as apples and pears have too few examples to learn well on their own, but their parent group has many.

The model predicts three levels of this tree at once, from coarse to fine. Because all levels share one network, a rare fine-grained crop can benefit from what the model learns about its broader group. A small final network then checks the three answers against each other. Where the fine label is uncertain, the model can fall back to a coarser label. That matters when a map has to be complete rather than detailed.

TL;DR

The authors build a three-level expert crop-label tree into a convolutional recurrent network, so each pixel gets a coarse, a mid and a fine label. A small CNN then refines the fine label using all three levels §3.2, §3.3. On their ZueriCrop dataset of Sentinel-2 series (48 crop classes), it reports 52.4% mean per-class F1, 9.9 points above the best compared method tab. 1, tab. 2. These are author-reported numbers from five geographic folds. Use the paper when your classes form a hierarchy and rare classes dominate. The released code's loss weights and checkpoint selection differ from the paper; see Paper vs code.

Problem

Crop maps support subsidies, policy and summary statistics. Collecting them through farmer self-reports and field checks is slow, costly and error-prone §1. Satellite images give a spectral signal that changes with plant development, so a time series carries more evidence than one image §1. Sentinel-2 offers 10 m ground sampling and a revisit every 3 to 5 days §1.

Most public crop datasets have few classes that are fairly balanced. Field-level data covering a whole region has a long tail of rare classes §1. Balancing tricks such as oversampling or a class-balanced loss give small gains, and they tend to lower overall accuracy §1, §6.1. The authors' idea is to use the expert label hierarchy as a prior. Rare fine classes then share features with their coarser parent, and a coarse label stays useful when the fine one is uncertain §1, §3.

Method

Label levels and data §3, §4, §4.1, fig. 1. The paper uses N = 3 levels. Level 1 is the coarsest and level 3 the finest, and the authors predict all three per pixel §3. The 48 classes are grouped by domain experts. Level 1 has field crops, grassland, orchards, special crops and forest. Level 2 groups crops by plant family and agronomic use. Level 3 gives species. Where a crop has no level-3 label, the level-2 label is copied, and for forest the level-1 label is copied to both lower levels §4.1. Grassland (with meadow and pasture among its subclasses) is the largest class, at about 56% of pixels in the class-frequency table app. C.

Input and data preparation §3.4, §4. Each sample is a 24 × 24 px patch with 4 bands: red, green, blue and near-infrared at 10 m §4. The sequence covers 71 Sentinel-2 Level-2A images from 2019 §3.4, §4. Atmospheric correction uses Sen2Cor v2.8 §4. Patches with no ground truth are discarded, and about 48% of the pixels in the remaining patches have no reference label §4. No cloud filter is applied, because the authors argue that recurrent networks tolerate uninformative inputs §4, §6.4. Labels are one crop per field polygon, and the dataset covers 116,000 fields over 50 km × 48 km §4.

convSTAR cell §3.1, eqs. 1–3. The cell is a convolutional version of a gated recurrent unit. A gate decides how much of the candidate update flows into the state.

K_t^l = \sigma( W_x * H_t^{l-1} + W_h * H_{t-1}^l + B_K ) eq. 1

Z_t^l = \tanh( W_z * H_t^{l-1} + B_z ) eq. 2

H_t^l = \tanh\big( H_{t-1}^l + K_t^l \circ (Z_t^l - H_{t-1}^l) \big) eq. 3

The cell is the one from the authors' earlier convSTAR work, which they cite as §3.1. Here is the same cell as a sketch:

# convSTAR cell, layer l, time t  [§3.1] [eq. 1-3]
# Paper form. The code differs; see Paper vs code.

def convstar_cell(layer_input, prev_state):
    # Gate K from the layer input and the previous state  [eq. 1]
    gate = sigmoid(conv_x(layer_input) + conv_h(prev_state) + bias_gate)
    # Candidate Z from the layer input only  [eq. 2]
    candidate = tanh(conv_z(layer_input) + bias_candidate)
    # Move the state towards the candidate, then squash  [eq. 3]
    return tanh(prev_state + gate * (candidate - prev_state))

Hierarchical stack §3.2. The network has three stages, and each stage is a two-layer convSTAR stack with 64 channels per layer, so six layers in total §3.2. Each stage's final hidden state goes to a one-layer 3 × 3 CNN that outputs scores for that level §3.2. The training loss is a weighted sum of per-level cross-entropy terms, with weights λ_n that sum to one eq. 4.

L = \sum_{n=1}^{N} \lambda_n \, CE(Y^n, \hat{Y}^n) eq. 4

The paper adds up the per-level losses so that the coarse levels shape the shared features. The label hierarchy then works as a soft prior §3.2.

Label refinement §3.3, eqs. 5–6. The refinement CNN takes the three stages' predictions, concatenated along the channel axis. It adds a correction to the finest prediction §3.3. The paper says the inputs are probability volumes. The code feeds logits; see Paper vs code. Refinement is applied only to the finest level, because the authors found it did not change the coarser levels' results §3.3.

\hat{Y}^N_{\text{ref}} = \hat{Y}^N + F(\text{concat}[\hat{Y}^1, \ldots, \hat{Y}^N]) eq. 5

The total loss adds the refinement term, weighted by γ §3.3, eq. 6.

L = \sum_{n=1}^{N} \lambda_n \, CE(Y^n, \hat{Y}^n) + \gamma \, CE(Y^N, \hat{Y}^N_{\text{ref}}) eq. 6

Training step (sketch). The block below follows the paper's equations. The weights and the handling of unlabelled pixels are marked where the code differs.

# One training step of ms-convSTAR (paper form)
# Input:  patches (B, T, 4, 24, 24), with T = 71 time steps
# Targets: one label map per level, each (B, 24, 24); 0 = unlabelled
# Output: scalar loss

def training_step(patches, targets_1, targets_2, targets_3):
    # Run the six-layer convSTAR stack over time  [§3.2] [eq. 1-3]
    hidden = zeros_for_each_layer(B, 64, 24, 24)
    for step in range(T):
        hidden = convstar_stack(patches[:, step], hidden)
    # Each stage reads its final hidden state with a 3x3 conv  [§3.2]
    # Stage 1 = layer 2, stage 2 = layer 4, stage 3 = layer 6
    scores_1 = head_1(hidden[1])             # (B, C1, 24, 24)
    scores_2 = head_2(hidden[3])             # (B, C2, 24, 24)
    scores_3 = head_3(hidden[5])             # (B, C3, 24, 24)
    # Refine the finest scores with a CNN over all levels  [§3.3] [eq. 5]
    # Paper: probabilities. Code: logits.
    stacked = concat_channels(scores_1, scores_2, scores_3)
    scores_refined = scores_3 + refine_cnn(stacked)
    # Weighted cross-entropy per level, with the paper's weights  [eq. 4]
    loss = 0.1 * cross_entropy(scores_1, targets_1)
    loss += 0.3 * cross_entropy(scores_2, targets_2)
    loss += 0.6 * cross_entropy(scores_3, targets_3)
    # Refinement loss with weight gamma  [§3.4] [eq. 6]
    loss += 0.6 * cross_entropy(scores_refined, targets_3)
    return loss

Augmentation and evaluation §3.4, §5. Patches are flipped at random with probability 0.66 during training §3.4. Evaluation uses five cross-validation folds. Each fold is one of five geographically disjoint strips, and the remaining four form the training set §5. Each field polygon takes the majority prediction over its pixels §5, app. A. Scores are pixel-wise: overall accuracy, and precision, recall and F1 averaged over classes §5. Overall accuracy is dominated by frequent crops, so the averaged per-class metrics are the ones the authors rely on §5.

Baselines §5, tab. 1, tab. 2, app. D. The same convSTAR backbone is used for a single-level baseline. Variants add data augmentation, a class-balanced loss by inverse frequency, a second class-balanced variant that matches the median effective learning rate, and the effective-number loss of Cui et al. with β ∈ {0.99, 0.999, 0.9999} §5. External methods are a Random Forest, an LSTM, a temporal CNN, a Transformer, a U-Net, a U-Net with convLSTM and a bidirectional convGRU §5, tab. 2, app. D.

Hyperparameters

Results

Author-reported. Setup: ZueriCrop, 5-fold geographic cross-validation, a convSTAR backbone with 6 layers and 64 channels §5, §6. Metrics are mean precision, recall and per-class F1 over classes, plus pixel accuracy §5.

By level, ms-convSTAR gives mean F1 of 57.0 at level 1, 54.5 at level 2 and 52.4 at level 3, with pixel accuracy of 96.3, 89.2 and 88.0 tab. 4. Coarser labels are more accurate. The paper also reports coverage. At a confidence threshold p = 0.9, adding hierarchy levels raises the share of confidently labelled pixels by about 0.20 to 0.25, with accuracy on covered pixels unchanged. The comparison is against models with fewer levels, and some of the gain comes from coarse labels §6.3, fig. 11, fig. 12. Per-class results show that several rare classes are still at or near 0% accuracy app. C §6.1.

Paper vs code

The official repository is github.com/0zgur0/ms-convSTAR, which the paper links in its implementation details §3.4. The README cites this paper and its owner handle matches the first author's name. Commit d0b764ff5109bca0d013784369b00332ea5e5a44 was cloned at depth 1. The repository has no LICENSE file, so no code is quoted here. Code paths are cited only. Five trained fold checkpoints are shipped (trained_models/fold1.pth to fold5.pth); they were not loaded.

• Loss weights. Paper: λ = 0.1, 0.3, 0.6 and γ = 0.6, summing to one §3.4 eq. 4 eq. 6. Code defaults: λ1 = 0.1 and λ2 = 0.5, with weight 1 for the global level and 1 for the refinement loss. train.py train.py train.py. The README gives no flags, so these defaults are what a reader gets.
• Refinement input. Paper: "3-dimensional probability volumes" §3.3. Code: the networks are built with wo_softmax=True, so the refinement CNN gets raw logits train.py.
• Refinement width and dropout. Paper: not stated. Code: 128 channels by default, dropout 0.5 networkConvRef.py train.py train.py.
• Refinement gradients. Training does not detach the stage outputs before refinement, so the refinement loss reaches the recurrent stack train.py. At test time they are detached eval.py. The paper says nothing about this.
• convSTAR cell. Paper: an outer tanh, and the sigmoid gate weights the candidate eq. 3. Code: no outer tanh, and the gate weights the previous state, so new = g·prev + (1−g)·candidate convstar.py.
• Gradient clipping. Paper: "magnitudes clipped to 5" §3.4. Code: norm clipping at 5, applied to each of the two networks separately train.py.
• Flip direction. Paper: flip with probability 0.66 §3.4. Code: after the 0.66 draw, one of vertical, horizontal or both flips, uniformly dataset.py. Image and target are flipped together.
• Layers, hidden size, batch, epochs, learning rate, weight decay. Paper and code agree: 6 layers, 64 channels, batch 4, 30 epochs, Adam, learning rate 0.001 with a step of 10 epochs and factor 0.1, weight decay 0.0001 §3.4 train.py train.py.
• Unlabelled pixels. Paper: silent on the loss. Code: class 0 gets weight zero at all three levels, and evaluation drops class 0 train.py eval.py.
• Patch filtering. Paper: patches without ground truth are discarded §4. Code: the default threshold is 0, so no patch is dropped at load time train.py dataset.py. Filtering may have happened when the HDF5 file was built, but that file is not in the repository.
• Time steps. Paper: T = 71 images §3.4 §4. Code: the loader hard-codes 142 stored steps (max_obs = int(142/2)) and keeps every second one with X[0::2] dataset.py dataset.py. The paper does not mention this subsampling, and the HDF5 file is not in the repository.
• Input scaling and bands. Code: values are multiplied by 1e-4 dataset.py. The first four stored channels are used dataset.py. The channel order is not documented in the code. The paper names red, green, blue and NIR §4.
• Cloud masking. Paper: the mask is added as an input channel §6.4. Code: the input is multiplied by the mask, pixels with cloud cover of 0.1 or more are zeroed, and no channel is added dataset.py.
• Folds. Paper: five geographic strips §5. Code: five contiguous blocks of the sample index, which are geographic strips only if the HDF5 file is ordered that way dataset.py. The repository does not show the file layout.
• Checkpoint selection. Paper: not stated. Code: after the third epoch, each epoch is evaluated on the test fold, and the checkpoint with the best test accuracy is kept train.py.
• Coverage procedure. The confidence-based fallback of §6.3 does not appear in the released evaluation code eval.py.
• Table 1 and the channel table. The same four-band model appears with F1 52.4 in tab. 1 and F1 52.1 in app. B. The paper does not explain the difference.
• Baseline code. train_baseline.py was not compared against the paper's baseline settings.

Pitfalls

• Loss weights. The paper's weights sum to one and include γ = 0.6. The code's defaults differ, as listed above. It is not stated which weights produced Table 1. Choose one set explicitly and record it.
• Unlabelled pixels. About 48% of pixels in the kept patches are unlabelled §4. The paper does not say they are excluded from the loss. The code excludes them, and this changes the class balance the model sees. Treat exclusion as a choice you must make and state.
• Checkpoint selection on the test fold. The released training code keeps the epoch with the best test-fold accuracy. The reported fold results could be optimistic. Select the epoch on a validation split taken from the training strips, and report the last epoch as well.
• Logits or probabilities. The paper's refinement takes probabilities. The code takes logits. Both are plausible, and the paper does not say which was used for Table 1.
• Refinement gradients. Whether the refinement loss flows into the recurrent stack changes what each stage learns. The code lets it flow during training. The paper does not say.
• Metrics. Precision, recall and F1 are averaged over classes that appear in the test labels, and class 0 is excluded eval.py eval.py eval.py. Classes that are only predicted do not count. Overall accuracy is not class-balanced §5.
• Rare classes. Classes with fewer than about 1,000 pixels do not improve, and some are absent from some folds §6.1. Several rare classes have 0% accuracy app. C.
• Class-balance-2. The paper's description of the variant (the median effective learning rate is set to match the default) is not precise enough to reimplement §5.
• Cell implementation. The code's cell differs from the paper's equation, as listed above. Verify the cell before reusing it. Two smaller issues in models/convstar.py: it calls init.orthogonal and init.constant without the underscore, which are deprecated names convstar.py. It also uses Variable without importing it, on the path where the previous state is None convstar.py. The encoder always passes zero states, so that path does not run.
• Unstated details. Weight initialisation, warm-up, random seeds beyond the code's default of 0, and the refinement CNN's width and dropout are not stated §3.3 §3.4.
• Hidden costs. The paper reports no runtime or memory figures. The stack is unrolled over all 71 time steps §3.4 multi_stage_sequenceencoder.py. Since training backpropagates through every step, activation memory grows with T (inference from the architecture, not stated in the paper).

Lineage

• Builds on: convSTAR, the single-level gated convolutional recurrent cell the authors stack here §3.1. The bibliography cites it as "Gating revisited: Deep multi-layer RNNs that can be trained" (IEEE TPAMI, 2021). No arXiv id is given, so it is not linked. The multi-stage design follows hierarchical CNN classifiers for single images §3.2.
• Compared against: a single-level convSTAR, its augmentation and class-balanced variants, the effective-number loss of Cui et al. (2019) §5, tab. 1. External methods are a Random Forest, an LSTM, a temporal CNN, a Transformer, a U-Net with and without convLSTM, and a bidirectional convGRU tab. 2, app. D.
• Superseded by: not known from the content in this wiki.