Apple Flower Detection using Deep Convolutional Networks

In plain words

Apple growers thin their trees by removing some flowers early in the season, and how many to remove depends on how many flowers are in bloom. Counting them by hand is slow, and checking a few trees and guessing for the rest is unreliable. Earlier camera systems mostly looked for flower-coloured pixels, so they failed when the light changed or when leaves and branches covered the flowers.

This paper takes a different route. It first splits a photo into small patches of similar colour, called superpixels, and treats each patch as a candidate. Each candidate is cut out as a square picture, and a neural network that was already trained to find important regions in images is retrained to recognise flowers. The network turns each picture into a long list of numbers. A simpler classifier then decides whether the patch is a flower.

The authors tested the method on their own apple photos and on three other sets: two more apple sets and one set of peach flowers. Those sets were not used for training. The method was clearly more accurate than colour-based methods in these tests, and it still did reasonably well on the unseen sets, as the authors report.

TL;DR

• Splits each image into SLIC superpixels, makes a mean-padded square portrait per superpixel, and classifies it as flower or not alg. 1 §3.
• Features come from the fine-tuned Clarifai CNN's first fully connected layer (FC6, 4,096 values), are reduced to 69 dimensions by PCA, and go to an RBF SVM §3 §3.1 tab. 2 tab. 3.
• Author-reported: CNN+SVM reaches F1 93.4% and AUC-PR 97.7% in 10-fold cross-validation on AppleA, against 87.1% and 92.9% for the best colour baseline tab. 4.
• Author-reported: with the same settings, F1 is 80.2%, 82.2% and 79.9% on AppleB, AppleC and Peach, against 70.7%, 48.6% and 49.0% for the baseline tab. 5.
• No official code is known. The paper gives no numbered equations, and several training settings (learning rate, batch size, superpixel count) are not stated.

Problem

Bloom intensity decides how many flowers and fruitlets to thin, and it affects fruit size, colour, taste and firmness §1. Estimation is usually done by hand on a sample of trees and extrapolated to the orchard §1. The paper reports that labour is more than half of apple production costs, citing another source §1. Colour-threshold methods depend on lighting, camera position and flower density, and they confuse leaves, stems and other flowers with flowers §1 §2.1.

The aim is a flower detector that is robust to clutter and lighting, and works across flower species without retuning §1 §2.2 §4.3. The evaluation is at the level of superpixels, not a count of flowers per tree, so the card does not show that bloom intensity itself was estimated §4 §4.2.

Assumptions: the training data are labelled by hand, by selecting superpixels with flower parts over roughly half their area §3.1. The exact rule is not stated. The unseen sets are handled by background removal and colour adjustment, not by retraining §4.3.

Method

Data §3.1 fig. 3. AppleA: 147 photos taken with a Canon EOS 60D in daylight, 5184 x 3456 pixels, from several angles and distances. The user labelled superpixels in a MATLAB GUI. The images were split randomly into 100 training and 47 validation images tab. 1.

Step 1: region proposals §3 fig. 2 alg. 1. The image is divided into SLIC superpixels. Each superpixel gets the smallest square portrait that encloses it. Pixels outside the superpixel are replaced with the training-set mean colour. The portrait is resized to 227 x 227.

Step 2: features §3 fig. 2. The portrait is mean-centred by subtracting the same mean colour and passed through the fine-tuned CNN. The output is the ReLU output of FC6, a 4,096-dimensional vector.

Step 3: classification §3 §4.1.1 tab. 2. PCA reduces the vector to 69 dimensions, which keeps about 94% of the variance. An RBF SVM labels the superpixel as flower or not flower.

Fine-tuning §3.1 fig. 2. The starting point is the Clarifai network that Zhao et al. fine-tuned for saliency detection §3.1. Training uses superpixel portraits from the training images, with 10,000 backpropagation iterations. The final layers FC7 and FC8 are used during training only fig. 2. After fine-tuning, the features of the training portraits are reduced to 69 dimensions to train the SVM §3.1.

Positive augmentation §3.1 fig. 4. Positive portraits are mirrored vertically, horizontally and on both axes, so positives are quadrupled. Positives were about 4% of the data before augmentation tab. 1.

Design choices tested §4.1. Three choices were compared: the feature dimensionality, the layer where features are read (FC6 or FC7, or the softmax output), and the background of the portrait (original, blurred, or mean-padded) §4.1.1 §4.1.2 §4.1.3 fig. 8.

Baselines §3.2.

• HSV: replicates the thresholding method of Hocevar et al. Pixels are binarised by HSV colour, then filtered by cluster size §3.2.
• HSV+Bh: superpixels with a 100-bin HSV histogram, compared to positive samples by Bhattacharyya distance with a Gaussian kernel. The mean distance is the score §3.2.
• HSV+SVM: the same HSV histograms fed to an SVM §3.2.

Background removal for the unseen sets §4.3. Local entropy is computed for each image and binarised with Otsu's threshold to find low-texture regions. Morphological size filtering is applied. The background is modelled by the RGB means of the largest low-texture regions. Each other region is assigned to background if its RGB mean is close to a background mode. The AppleB and Peach sets use two modes (blue panel or grey sky, and trunk or branches). AppleC uses three modes, the third for reflective shadows. The saturation channel is then histogram-equalised and histogram-matched to the training distribution. The mean of the value channel is shifted to match the training set.

The pipeline as the paper describes it, from the detection steps alg. 1:

# Flower detection for one image  [alg. 1] [§3]
# Input:  image (H, W, 3) RGB; fitted cnn, pca, svm; mean_rgb
# Output: one flower / no-flower decision per superpixel

def detect_flowers(image, cnn, pca, svm, mean_rgb):
    # Step 1: divide the image into SLIC superpixels  [§3] [alg. 1]
    superpixels = slic_superpixels(image)
    decisions = []
    for superpixel in superpixels:
        # Step 1: smallest square crop that encloses the superpixel  [§3]
        crop = smallest_square_crop(image, superpixel)
        # Step 1: replace pixels outside the superpixel with mean_rgb  [§3]
        portrait = pad_background(crop, superpixel, mean_rgb)
        # Step 1: resize to 227 x 227  [§3]
        portrait = resize(portrait, (227, 227))
        # Step 2: subtract the same mean_rgb  [§3]
        centred = portrait - mean_rgb
        # Step 2: ReLU output of FC6, 4096 values  [§3]
        feature = cnn_fc6_relu(centred)
        # Step 3: project onto the first 69 principal components  [§3]
        reduced = pca.transform(feature)
        # Step 3: RBF-SVM decision  [§3] [§4.2]
        decisions.append(svm.predict(reduced))
    return decisions

The training procedure, as the paper describes it §3.1 §4.2:

# Fine-tuning, PCA and SVM training  [§3.1] [§4.2]
# Input:  labelled training images, saliency-pretrained Clarifai CNN
# Output: fine-tuned cnn, pca, svm

def train_flower_model(training_images, cnn):
    # Build labelled superpixel portraits as in detection  [§3.1]
    portraits, labels = labelled_portraits(training_images)
    # Mirror each positive vertically, horizontally and on both axes  [§3.1]
    portraits, labels = mirror_positives(portraits, labels)
    # Fine-tune for 10,000 backpropagation iterations  [§3.1]
    cnn = fine_tune(cnn, portraits, labels, iterations=10000)
    # Fit PCA on FC6 features; fit data not stated, training assumed  [§3.1]
    features = cnn_fc6_relu(portraits)
    pca = fit_pca(features, components=69)
    # Train an RBF-SVM with the CNN+SVM settings  [§4.2]
    svm = fit_rbf_svm(pca.transform(features), labels, C=30, gamma=1e-4)
    return cnn, pca, svm

Hyperparameters

Results

Author-reported. Three evaluation settings appear in the paper.

• Design choices §4.1: scored on the 47 validation images (42,430 portraits) with an SVM trained on the training images §3.1 §4.1.
• Comparison with baselines §4.2: 10-fold cross-validation over all 133,918 AppleA portraits §4.2 tab. 4.
• Unseen sets §4.3: three sets of 20 images each, using the AppleA settings unchanged §4.3 tab. 5.

Design choices: dimensionality of the FC6 features, validation set tab. 2. Bold in the source marks the best F1 (23 dimensions) and the best AUC-PR (69 dimensions).

Source: tab. 2. The dimension scan uses the validation split.

Layer and classifier choice, validation split tab. 3:

Source: tab. 3 fig. 7. The paper reports that the SVM variants are slightly better than the softmax output, and that B and C are very similar.

Portrait background, validation split: mean-padded portraits give the best PR curve, and the paper ties this to false positives on superpixels that sit inside flowers §4.1.3 fig. 9 fig. 10. The source gives no table of values for this comparison.

Main comparison, AppleA, 10-fold cross-validation tab. 4:

Source: tab. 4 fig. 11. See Pitfalls for three F1 values in this table that do not match their recall and precision.

Unseen sets, same settings tab. 5: the paper states that AUC-PR is above 85% for all three sets for the CNN+SVM method §4.3. The AUC-PR values are only printed on the curves in fig. 14 and are not repeated here.

Source: tab. 5 fig. 13 fig. 14.

Paper vs code

No official code.

The paper links no code, and the arXiv page names no code repository (its comment field reads "14 pages"). The searches were:

• GitHub repository search for "apple flower detection", "apple flower SLIC CNN fine-tuned Clarifai", "flower detection CNN SVM" and "apple flower detection deep convolutional SVM". None returned a relevant repository.
• Repositories of the first author (Philipedias and phiLdias25, two accounts whose names match the author). Neither has a repository on flowers or detection.
• Repositories of the second author (amy-tabb, all 59 repositories). None is about apples, flowers or fruit.
• A GitHub user search for the third author's name with "marquette". It returned no accounts.

Pitfalls

• Same data for design and reporting. The design choices (dimensions, layer, background) are scored on the 47 validation images §4.1, and the feature and SVM settings used for the unseen sets were chosen on AppleA §4.3. No separate test split exists for the design choices §3.1. The 10-fold result is the only protocol that covers all AppleA portraits §4.2.
• Cross-validation folds may share images. Samples are superpixel portraits. The paper gives 133,918 samples in 10 folds of 13,391 §4.2. Whether folds were split by image or by superpixel is not stated. Superpixels from one photo in both training and test folds would inflate the result.
• Fold sizes do not add up. Ten folds of 13,391 make 133,910, not 133,918. Eight samples are unaccounted for §4.2.
• Three F1 values do not match their own recall and precision. Table 4's F1 is the harmonic mean of recall and precision, 2PR/(P+R). Recomputed from the printed values, HSV+Bh gives 58.6% (printed 64.6%), HSV+SVM gives 88.1% (printed 87.1%), and CNN+SVM gives 92.4% (printed 93.4%) tab. 4. HSV and all other tables agree to 0.1 point. The paper does not say whether recall and precision come from the same threshold as the F1 maximum, so the source of the gap is unknown.
• Two protocols give different numbers for the same method. The design-choice tables (validation split) give CNN+SVM AUC-PR 97.2-97.3% tab. 2 tab. 3. The cross-validation table gives 97.7% tab. 4. Quote the protocol with the number.
• Positive labelling rule. Users selected superpixels holding flower parts over roughly half their area §3.1. The paper then calls superpixels with less than 50% flower area false positives §4.1.3 §4.3. The labelling rule is described only loosely, so the exact threshold for a positive label is not stated, and label noise near 50% is likely.
• Mean-pad source. The text says the padding is "the training set mean" and also "the average RGB color of all images composing the dataset" §3. The two differ if the mean uses 147 images instead of 100. Which one was used is not stated.
• Pixel scale and channel order. The mean-centring step is defined in RGB §3. Scale (0-255 or 0-1) and channel order are not stated. A Caffe-trained network depends on these, so check them against the weights used.
• Hyperparameters missing for the fine-tuning. Learning rate, batch size, momentum, weight decay, dropout, which layers were updated, and the number of positive versus negative samples per batch are not stated §3.1. "10,000 iterations" is the only training setting given.
• Augmentation and counts. The paper says positives were quadrupled §3.1. Table 1 gives 3,691 positives in training. Whether the augmented count is used in Table 1, and whether augmentation was also applied to the validation or CV data, is not stated tab. 1.
• PCA fit data. The text does not say which portraits PCA was fitted on. The card's pseudocode assumes the training portraits [inferred].
• Grid search details. The C and gamma values are given §4.2, but the grid range, step size and whether the search used cross-validation are not stated. The HSV thresholds were also set by grid search on training data §4.2.
• Superpixel count not stated. The number of SLIC superpixels per image, and the compactness parameter, are not given §3. Both change the portraits the CNN sees.
• Cost scales with superpixels. Each superpixel needs its own CNN forward pass on a 227 x 227 crop alg. 1. The paper reports no runtime or hardware figures.
• Per-dataset background mode count. The number of background modes is set per dataset (2 for AppleB and Peach, 3 for AppleC) §4.3. This is a dataset-specific choice, although the paper describes the adjustments as generic.
• Histogram matching target not described. The paper says matching uses the "characteristic form" of the training saturation distribution §4.3. How the target histogram was built is not stated.
• Background parameters not stated. Entropy window size, morphological filter size and the number of low-texture clusters used for the background model are not given §4.3.
• Percentage-point wording. The text says that moving from HSV+Bh to HSV+SVM improves F1 and AUC-PR "around 20% and 30%" §4.2. The table values differ by about 22.5 and 31.3 percentage points tab. 4. The text does not say whether it means points or relative gains (the relative gains would be about 35% and 51%).
• Results on unseen sets not fully tabulated. AUC-PR for the unseen sets is only in fig. 14. The text states the threshold ("above 85%") §4.3.
• Dataset and code availability not stated. The paper does not say whether the AppleA images or labels are public, and it mentions supplementary material that is not in the source §3 fig. 3. Neither is available to this card.
• Per-superpixel metrics only. All metrics are per superpixel on a class-imbalanced set (about 4% positives) §4 tab. 1. The paper argues that AUC-PR and F1 suit this imbalance better than accuracy §4. It does not evaluate flower counts or bloom intensity itself.

Lineage

• Builds on: the Clarifai network §2.2 §3 (cited in the paper as Zeiler2014), and its saliency fine-tuning by Zhao et al. §3.1.
• Also builds on: fine-tuning of CNNs for detection by Girshick et al. §3.1, and SLIC superpixels by Achanta et al. §3.
• Faster R-CNN was considered and not used, because it has more parameters to learn §3.
• Compared against: HSV colour thresholding, which replicates Hocevar et al. §3.2; HSV+Bh, the authors' superpixel histogram baseline §3.2; and HSV+SVM, which adds an SVM to the same HSV histograms §3.2.
• Journal version: Computers in Industry, vol. 99, pp. 17-28 (Aug 2018), DOI 10.1016/j.compind.2018.03.010, as named in the abstract's footnote.
• Superseded by: not stated in this card.