DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning

In plain words

Weed control on big farms and rangelands is expensive, and a robot that sprays only the weeds, and not the plants around them, would save labour and herbicide. Such a robot first has to recognise weeds from camera images. Most plant photo collections were taken in labs or on crop fields, not in the rough, cluttered scrub where the robot would work.

This paper releases DeepWeeds, a set of field photos of eight weed species from northern Australia. Every photo is labelled by people. Anything that is not one of the eight weeds, including other plants and bare ground, goes into a "Negatives" class. The photos are split into five train, validation and test folds. The authors then train two standard image classifiers, ResNet-50 and Inception-v3, that start from weights learned on ImageNet. To keep the models from memorising the photos, the training images are heavily altered at random (rotated, scaled, recoloured, flipped).

Both classifiers reach about 95 percent accuracy in cross-validation. The ResNet-50 model, once converted with NVIDIA's TensorRT for an embedded GPU board, is fast enough for the robot's planned driving speed. These are the authors' results, so treat them as a baseline to beat, not a final answer.

TL;DR

• DeepWeeds is a public image dataset of 17,509 in-field photos of eight weed species plus a Negatives class, collected at eight sites in northern Australia § The DeepWeeds Dataset tab. 1.
• Baselines are ImageNet-pretrained ResNet-50 and Inception-v3 with 9 sigmoid outputs, 5-fold cross-validation, and heavy augmentation § Deep Learning.
• Author-reported: ResNet-50 reaches 95.7% and Inception-v3 95.1% weighted top-1 accuracy tab. 2.
• Author-reported: ResNet-50 runs at 53.4 ms per image on a Jetson TX2 with TensorRT, but 180 ms with TensorFlow tab. 4.
• The authors' code is public (Apache-2.0). It differs from the paper in several augmentation, validation and preprocessing details; see Paper vs code.

Problem

Robotic weed control needs a classifier that works on images from a moving robot in rangeland conditions, where lighting, scale, orientation, occlusion and cluttered backgrounds vary widely § Introduction fig. 1. Earlier plant datasets were mostly taken under controlled lab conditions or were site-specific and crop-focused § Introduction. The paper's stated gaps are the lack of a large, public, field-collected, multiclass rangeland weed dataset and of a baseline on it § Introduction.

Method

Dataset construction

1. Capture rig. A custom logger, the WeedLogger, was built around a FLIR Blackfly 23S6C camera with a 1920 × 1200 px sensor, a Fujinon CF25HA-1 25 mm lens, and a GPS receiver § Data Collection Platform fig. 3. The authors chose the optics to match their prototype spray robot: a lens 1 m above the ground, a 450 × 280 mm field of view, aperture f/8, and a shutter faster than 0.05 ms § Data Collection Platform.
2. Collection. Images were taken from June 2017 to March 2018 at infestation sites across northern Australia § The DeepWeeds Dataset.
3. Classes. Eight weed species plus a Negatives class. Negatives are all other plant life and background imagery § Dataset Collection. The class indices used in the code are 0 to 7 for the weeds and 8 for Negatives deepweeds.py.
4. Balance goals. The authors aimed for at least 1,000 images per species and an even positive/negative split per location § Dataset Collection. Table 1 shows the goals were met approximately, not exactly (see Pitfalls).
5. Labels. Expert analysis labelled each image for whether it contains a target species § Dataset Collection. The label file gives one class per row labels.csv. The paper also says an image may contain several weed species § Deep Learning, which the label file does not show (see Pitfalls).
6. Folds. Stratified random splits of 60% train, 20% validation and 20% test, repeated for five folds (k = 5). The negative class is not stratified because it is much larger § Deep Learning. The released fold files are train_subsetK.csv, val_subsetK.csv and test_subsetK.csv for K = 0..4 deepweeds.py.

Table 1 gives the image count per species and location tab. 1:

The dataset is also listed in the TensorFlow Datasets catalogue README.md. The images (about 468 MB) and trained models (about 477 MB) are on Google Drive, and the download script fetches them from there README.md deepweeds.py.

Preprocessing and augmentation

Training and validation images share the same augmentation chain § Deep Learning deepweeds.py. Test images are not augmented deepweeds.py. Pixel values are 8-bit (0 to 255) in the paper's ranges § Deep Learning.

# Training and validation preprocessing  [§ Deep Learning]
# Input:  image, 8-bit RGB, any size
# Output: float image (224, 224, 3), values in [0, 1]

def preprocess_train(image):
    # Resize to 256 x 256 before augmenting  [code: deepweeds.py#L35]
    image = resize(image, (256, 256))
    # Random rotation in [-360, +360] degrees  [§ Deep Learning]
    image = rotate(image, uniform(-360, 360))
    # Random zoom with factor in [0.5, 1]  [code: deepweeds.py#L178]
    image = zoom(image, uniform(0.5, 1))
    # Shear with shear_range 0.2; the paper says perspective instead
    # [code: deepweeds.py#L177]
    image = shear(image, shear_range=0.2)
    # Random shift per colour channel in [-25, +25]  [code: deepweeds.py#L181]
    image = channel_shift(image, uniform(-25, 25))
    # Brightness factor in [0.75, 1.25]  [code: deepweeds.py#L182]
    image = image * uniform(0.75, 1.25)
    # Horizontal flip with probability 0.5  [code: deepweeds.py#L179]
    if random() < 0.5:
        image = flip_horizontal(image)
    # Crop to 224 x 224; the code takes the centre
    # [code: deepweeds.py#L120-L130]
    image = center_crop(image, (224, 224))
    # Scale to [0, 1]; the order relative to the steps above is not verified
    # [code: deepweeds.py#L175]
    return image / 255

Test images are resized directly to 224 × 224 and scaled to [0, 1], with no 256 step and no crop deepweeds.py deepweeds.py. The paper does not state the test preprocessing.

Model

Two ImageNet-pretrained Keras networks, ResNet-50 and Inception-v3, each with its 1000-way classification head removed § Deep Learning deepweeds.py. The head is replaced by global average pooling and a 9-unit dense layer with sigmoid activation. The sigmoid is used so that each class gets its own score, which the authors chose because an image may contain more than one species § Deep Learning deepweeds.py. The paper gives the parameter counts as 23.5 million for ResNet-50 and 21.8 million for Inception-v3 § Deep Learning.

# Classifier and decision rule  [§ Deep Learning]
# Input:  preprocessed image (224, 224, 3)
# Output: class index 0..8, where 8 is Negatives  [code: deepweeds.py#L44]

def classify(image):
    # Pretrained backbone without its 1000-way head  [§ Deep Learning]
    features = backbone(image)              # (h, w, f) feature map
    # Average over the spatial grid  [code: deepweeds.py#L246]
    pooled = global_average_pool(features)  # (f,)
    # One sigmoid score per class, not a softmax  [code: deepweeds.py#L248]
    scores = sigmoid(dense(pooled, units=9))  # (9,)
    best = argmax(scores)
    # A best score below 1/9 is treated as Negatives
    # [§ Deep Learning] [code: deepweeds.py#L306-L307]
    if scores[best] < 1 / 9:
        best = 8
    return best

Training

The loss is binary cross-entropy, optimised with Adam at an initial learning rate of 1e-4 and batches of 32 § Deep Learning deepweeds.py. The learning rate is halved whenever the validation loss has not improved for 16 epochs. Training stops when it has not improved for 32 epochs. The model then restarts from its best weights with a lower initial rate § Deep Learning deepweeds.py deepweeds.py.

# Training of one fold  [§ Deep Learning]  [code: deepweeds.py#L261-L301]
# Adam with binary cross-entropy, batches of 32

def train_fold(model, train_set, val_set):
    # Start learning rate  [§ Deep Learning] [code: deepweeds.py#L43]
    learning_rate = 1e-4
    restarts = 0
    epochs_run = 0
    # 200-epoch cap: in the code only, not stated in the paper [code#L38]
    while epochs_run < 200:
        # Train until 32 epochs pass without a new lowest validation loss.
        # Halve the rate after each 16 epochs without a new minimum.
        # Keep the best weights.  [code: deepweeds.py#L253-L256]
        epochs_run, stopped_early = train_until_stalled(
            model, learning_rate, train_set, val_set, epochs_run
        )
        if not stopped_early:
            break
        # Restart from the best weights with the rate halved again
        # [code: deepweeds.py#L287]
        restarts = restarts + 1
        learning_rate = 1e-4 / 2 ** restarts
    # Keep the best checkpoint over all restarts  [code: deepweeds.py#L300-L301]
    return best_checkpoint(model, restarts)

Reported training details: about 13 hours per model on an NVIDIA GTX 1080Ti. The validation accuracy plateaued after roughly 100 epochs § Deep Learning fig. 5.

Augmentation details

• Paper: rotation over the full circle, scale in [0.5, 1] on both axes, a channel shift of ±25 on 8-bit colours, an intensity shift of ±25 applied to all channels, an intensity scale in [0.75, 1.25], random perspective transforms, and a 50% horizontal flip § Deep Learning.
• The code implements rotation, zoom, shear, channel shift, brightness scale and flip deepweeds.py. It has no separate intensity shift and no perspective transform (see Paper vs code).
• The paper says augmentation without it caused strong over-fitting § Deep Learning.

Real-time inference

ResNet-50 is deployed on an NVIDIA Jetson TX2 (a 15 W embedded GPU module). It is timed in two ways: TensorFlow, which was also used for training, and TensorRT § Real Time Inference. Each timing includes loading and resizing the image § Real Time Inference. The target is 10 frames per second, which corresponds to about 100 ms per image given the camera's field of view and a spray speed of 10 km/h § Real Time Inference § Data Collection Platform.

# Per-image timing, batch size 1  [§ Real Time Inference]
# [code: deepweeds.py#L340-L366]

def time_image(model, path):
    # Preprocessing: read, resize directly to 224 x 224, then scale
    # [code: deepweeds.py#L343-L349]
    start = now()
    image = resize(read_image(path), (224, 224)) / 255
    preprocess_ms = (now() - start) * 1000
    # Forward pass and decision  [code: deepweeds.py#L351-L355]
    start = now()
    scores = model.predict(image[None])
    inference_ms = (now() - start) * 1000
    return preprocess_ms, inference_ms

Hyperparameters

Results

Author-reported. Setup: DeepWeeds, five-fold cross-validation, test subsets, ImageNet-pretrained ResNet-50 and Inception-v3 trained with Keras on an NVIDIA GTX 1080Ti § Deep Learning § Deep Learning Performance. The paper calls the 95.7% and 95.1% numbers the class-weighted top-1 accuracy § Deep Learning Performance. The "top-1 accuracy" column is averaged over the five folds § Deep Learning Performance tab. 2.

Per-species results tab. 2:

ResNet-50 confusion matrix, combined over the five test folds, in % tab. 3. Rows are the true class, since each row sums to about 100 and the diagonal equals the ResNet-50 top-1 column in tab. 2 (inferred). Columns and rows use the class indices of the code: 0 Chinee apple, 1 Lantana, 2 Parkinsonia, 3 Parthenium, 4 Prickly acacia, 5 Rubber vine, 6 Siam weed, 7 Snake weed, 8 Negatives deepweeds.py.

Real-time inference of the ResNet-50 model on a Jetson TX2 tab. 4, in milliseconds:

Only TensorRT meets the 10 fps target § Real Time Inference tab. 4.

The learning-curve figure shows one fold's validation accuracy: 96.7% for Inception-v3 at epoch 193 and 97.6% for ResNet-50 at epoch 155. These are validation numbers from a single fold, not test results fig. 5.

Paper vs code

Code: the authors' repository, github.com/AlexOlsen/DeepWeeds, at da084f62bcd1a2b0afeb8b81f1a27be3186391a1. The paper links it in its Data Availability statement, and the first author owns it § Data Availability. Checked against the training script deepweeds.py, the label files, the README and the TensorRT C++ code.

Agreements: the image size of 256 and crop of 224, batch size 32, 5 folds, the 1/9 decision rule, rotation, zoom and brightness ranges, the sigmoid output, binary cross-entropy, the learning rate of 1e-4, the 16 and 32 epoch rules, the restart at half the rate, and selection of the lowest-validation-loss checkpoint all match the paper deepweeds.py deepweeds.py deepweeds.py.

Discrepancies and gaps, each cited:

1. Validation step count. The validation image count is taken from the training dataframe deepweeds.py. This count is used as validation_steps deepweeds.py. Each epoch therefore runs about 328 validation batches (training count divided by 32) instead of about 109 (validation count divided by 32). The generator repeats, so each epoch sees about three times as many augmented validation images as the validation set has. The paper does not state validation steps.
2. Augmentation. The paper describes random perspective transforms and a separate intensity shift of ±25 applied to all channels § Deep Learning. The code uses shear with range 0.2, a channel shift of ±25 and a brightness factor deepweeds.py. No perspective transform and no separate intensity shift appear in the code.
3. Scaling. The paper says the images are scaled vertically and horizontally in [0.5, 1] § Deep Learning. The code uses zoom_range=(0.5, 1) deepweeds.py. The code does not show whether the scale is per axis.
4. Crop. The docstring of crop_generator says "random crops" deepweeds.py. The crop function takes a centred window deepweeds.py. The code therefore uses a centre crop. The paper only says "cropped" § Deep Learning.
5. Test preprocessing. Test images go straight to 224 × 224 with no 256 step and no crop deepweeds.py. The paper does not state it.
6. Inference preprocessing. The Python timing function resizes directly to 224 × 224 deepweeds.py. The TensorRT path resizes with nearest-neighbour interpolation resnetInference.cpp. Neither matches the 256 → 224 training path. The paper says only "resizing it for input" § Real Time Inference.
7. Double scaling in the Python timing function. skimage.transform.resize returns floats in [0, 1] by default. The code then divides by 255 again deepweeds.py. This does not change timing, but predictions made with this function would be wrong, so do not reuse it for accuracy.
8. Epoch cap and learning-rate floor. The code caps training at 200 epochs deepweeds.py and sets a learning-rate floor of 3.125e-6 deepweeds.py. Neither appears in the paper.
9. Restarts. The paper gives only the first restart rate § Deep Learning. The code sets the rate to 1e-4 / 2^restarts and keeps restarting until the cap deepweeds.py.
10. Freezing and initialisation. The code trains all layers of both networks deepweeds.py. The paper says nothing about freezing. The paper says the output layer uses Glorot uniform initialisation § Deep Learning. The code sets no initialiser deepweeds.py.
11. Splits. The paper describes stratified random splitting with a seed § Deep Learning. The code loads fixed fold files deepweeds.py and does not contain the splitting procedure or the seed.
12. Metric definitions. The per-species "top-1 accuracy" is not defined in the paper. The code prints a scikit-learn classification report deepweeds.py. That report's per-class recall is the likely source of the column (inferred). The false positive rate is not computed in the code. Confusion matrices are saved per fold deepweeds.py, and the paper's cross-fold aggregation is not in the repository.
13. Inference precision. The paper does not state the precision used on the Jetson. The TensorRT engine is built in FP32; the FP16 branch is disabled by #if 1 resnetInference.cpp.
14. Dataset licence. The README says the images, annotations and source code are CC BY 4.0, and the repository contents are Apache 2 README.md. The code is under Apache-2.0 LICENSE. The card's code_license is Apache-2.0.

No discrepancies were found in the classifier head, the sigmoid rule or the threshold.

Pitfalls

• Multi-species images. The paper says an image may contain several weed species § Deep Learning. Each row of the label file has one class labels.csv, and the paper does not say how multi-species images were labelled.
• Balance. The goal was a 50:50 split per location § Dataset Collection. Table 1 shows the split is close but not exact. For example, Charters Towers has 434 positives and 605 negatives tab. 1.
• Text vs table. The text says the lowest precision values are "91.0% to 93.0%" for four species § Deep Learning Performance. Table 2 lists Snake weed at 90.9% tab. 2.
• Equations missing. The caption of tab. 2 says the metric equations are "provided below" tab. 2. The paper has no such equations.
• Unnumbered sections. The paper's section headings are unnumbered, so citations use the heading name in square brackets, e.g. § Deep Learning.
• Input scale. The paper's ±25 shifts are on the 8-bit scale § Deep Learning. If your pipeline applies them after dividing by 255, they are about 255 times too strong. Check the order against your framework.
• Test vs validation numbers. The 97.6% and 96.7% in the learning-curve figure are single-fold validation figures fig. 5. Do not compare them with the 95.7% and 95.1% test numbers.
• Data downloads. The images and models come from Google Drive links, which the download code follows deepweeds.py. The paper gives no direct URL for them.
• Timing scope. The paper's TensorFlow and TensorRT timings are on a Jetson TX2 § Real Time Inference. The Python timing script runs on whatever machine it is started on deepweeds.py.

Lineage

• Builds on: ResNet-50 and Inception-v3, used through Keras with ImageNet weights § Deep Learning deepweeds.py. Other models were considered and not used: VGGNets (144 million parameters), GBD-Net (needs per-pixel labels) and SE networks (not in the Keras and TensorFlow backend) § Deep Learning.
• Compared against: Inception-v3 within the paper; ResNet-50 performs slightly better on both accuracy and false positive rate § Deep Learning Performance. Inference is compared across TensorFlow and TensorRT only § Real Time Inference.
• Superseded by: not stated.