Comparing YOLOv8 and Mask R-CNN for instance segmentation in complex orchard environments

In plain words

Farm robots that pick or thin fruit need to know exactly where each branch, trunk or fruit is in a camera image, not just a rough box around it. This paper trains two common vision models on its own photos of apple trees and compares them. One is a fast single-pass model (YOLOv8). The other is an older two-pass model (Mask R-CNN) that first proposes candidate regions and then refines each one.

The photos come from two seasons. In winter, the trees are bare, and the images are labelled for trunks and branches. In early summer, the leaves are green and the images are labelled for small unripe apples. Across both sets, the fast single-pass model finds the objects more accurately and runs faster. The authors report these results themselves, and some of their numbers do not agree with each other, so read the figures with care.

TL;DR

• The paper compares a one-stage detector (YOLOv8) and a two-stage one (Mask R-CNN) for instance segmentation in apple orchards. It is a benchmark study, not a new method.
• Dormant-season trunks and branches (multi-class) and early-season green fruitlets (single-class) were labelled by hand, in 1553 images in total §3.2.
• Author-reported, YOLOv8 has higher precision and recall, and faster inference, in both datasets tab. 3.
• The paper does not release code, and it reports several mAP values that disagree between text, figures and table (see Pitfalls).

Problem

Instance segmentation in orchards supports selective harvesting, precision pruning and fruit thinning, all of which need per-object masks §1. The authors cite earlier traditional image methods as requiring manual tuning and being less reliable across conditions §1. Learned models, such as Mask R-CNN and YOLO, are widely used in agriculture §1. The paper asks which of the two is better suited to orchard automation under changing light and foliage §1 §2, using two datasets that differ in season, object type and class count §3.2.

Method

This is an empirical comparison, so the "method" is the experimental pipeline §3 fig. 3a.

1. Acquisition. RGB images were captured with an Intel RealSense 435i camera in a commercial apple orchard in Prosser, Washington, in dormant season (Nov 2022) and in early growing season (Jun 2023) §3.1 fig. 3.
2. Annotation. Objects were outlined as polygons in Labelbox §3.2 fig. 4. Dataset 1 has two classes, trunk and branch. Dataset 2 has one class, green fruit (fruitlets).
3. Preprocessing. Both datasets were resized to 640 x 640 pixels. Each class was split randomly 8:1:1 into training, validation and test subsets §3.2.
4. Training. Both models were trained in PyTorch on one workstation §3.3. Hyperparameters are in the table below.
5. Evaluation. Precision, recall, mAP at IoU 0.5, AUC and inference time were computed §3.4 (see the pseudocode).

Datasets §3.2 fig. 4

Both datasets contain 1553 images in total §3.2. Dataset 1 is the dormant apple trees, and Dataset 2 is the canopy with immature green fruits fig. 4.

Mask R-CNN §2.1 fig. 1. It is a two-stage model. A backbone (ResNet101 with a feature pyramid, per the architecture diagram fig. 1) feeds a region proposal network. Each proposal is cropped by RoIAlign and passed to three parallel branches: class and box, and a mask fig. 1. The paper cites He et al. (2017) for the architecture §2.1.

YOLOv8 §2.2 fig. 2. It is a one-stage model that predicts boxes and classes directly. The paper describes an anchor-free head, a C2f block in place of C3, a 3 x 3 stem convolution, and a decoupled head with class, box and objectness losses §2.2 fig. 2. The paper says YOLOv8 supports instance segmentation §2.2. The variant size used (Tiny, Small, Standard or Large) is not stated §2.2.

Augmentation and training schedule §3.3 tab. 2

• A warm-up phase runs during the first three epochs, with momentum 0.8 and a bias learning rate of 0.1 §3.3.
• Augmentations are listed in the table below tab. 2. The text says they were applied to enhance robustness, but it does not say whether Mask R-CNN received the same set §3.3.
• Training stops early if validation performance does not improve for 20 consecutive epochs §3.3.

# Per-class evaluation at one confidence threshold  [§3.4] [eq. 1] [eq. 2]
# Input:  predictions: list of (instance, class, score)
#         ground_truth: list of (instance, class)
#         iou_threshold = 0.5           [§3.4] [fig. 7]
# Output: precision, recall for the class

def match_predictions(predictions, ground_truth, iou_threshold):
    # Pair each prediction with an unused ground-truth instance of the
    # same class when their overlap reaches the threshold. Greedy order
    # (by score) is not stated [§3.4].
    ...
    return true_positives, false_positives, false_negatives

def precision_and_recall(true_positives, false_positives, false_negatives):
    # Precision: correct share of predictions  [eq. 1]
    precision = true_positives / (true_positives + false_positives)
    # Recall: found share of ground-truth objects  [eq. 2]
    recall = true_positives / (true_positives + false_negatives)
    return precision, recall

# Mean average precision over classes  [eq. 4] [§3.4]
# Input:  per-class precision-recall curves at IoU 0.5
# Output: mAP@0.5

def mean_average_precision(class_curves):
    # Average precision (AP) of each class is the area under its
    # precision-recall curve. The interpolation and integration rule
    # is not stated [§3.4].
    average_precisions = [area_under(curve) for curve in class_curves]
    # mAP is the mean of AP over the k classes  [eq. 4]
    return sum(average_precisions) / len(average_precisions)

Hyperparameters

Results

Author-reported. Setup: orchard images from Datasets 1 and 2, the test splits, mAP at IoU 0.5 §4 tab. 3.

Per-class curve readings, from the figures:

AUC is shown as ROC curves for both datasets fig. 15. No numeric AUC is given in the text or table §4 fig. 15.

Speed comparison, from the text: YOLOv8 runs at about 128 FPS for single-class and 92 FPS for multi-class, against 78 and 64 FPS for Mask R-CNN §4.2. The text calls YOLOv8 about 1.65 times faster on single-class and about 1.43 times faster on multi-class §4.3.

Qualitative examples: fig. 8 and fig. 9 show green-fruit failures and successes; fig. 13 and fig. 14 show trunk and branch results.

Paper vs code

No official code. The paper does not contain a code, repository or project-page link. The abstract, the arXiv page and the journal record name no code either. A GitHub repository search for "yolov8 mask r-cnn" and for "yolov8 mask rcnn apple orchard" returned no relevant results on the first page. The first author's GitHub account (ranzosap, display name "Ranjan Sapkota") lists a repository named ComparingYOLOv8withMaskR-CNN. The paper does not link it, so its ownership is not confirmed by the paper. Its README contains only the title, with no code, no LICENSE file and no configs (commit 6618c93f3b8d94b0f8e3031db93b56c3dad53298). It is therefore not used.

Pitfalls

• Conflicting learning rate. One paragraph gives the learning rate as 0.001 §3.3. The next gives the initial rate as 0.01 for both models §3.3. The card records both values.
• Conflicting training length. The paper says training stops before "1000 epochs" §3.3 and, in another sentence, that training ran for "1000 iterations" §3.3. Which is meant is not stated.
• Conflicting mAP values for the same models. Section 4.1 gives YOLOv8 single-class mAP 0.939 and Mask R-CNN 0.902 §4.1 fig. 7. Table 3 gives YOLOv8 0.902 and Mask R-CNN 0.85 tab. 3. For multi-class, the text gives YOLOv8 0.845 and Mask R-CNN 0.748 §4.2 fig. 12, while Table 3 gives 0.74 and 0.700 tab. 3. Use the figure values only with the text in mind.
• Conflicting recall and precision values. The abstract gives Mask R-CNN multi-class precision and recall as 0.81 and 0.81 [abstract]. Section 5 gives 0.81 and 0.83 §5. Table 3 gives 81.3 and 83.7 tab. 3. The abstract gives YOLOv8 single-class precision as 0.93, section 5 gives 0.92, and Table 3 gives 92.9 [abstract] §5 tab. 3.
• Confidence threshold. The abstract says the precision and recall values are at confidence 0.5 [abstract]. The body does not state the threshold used for Table 3 §4. The per-curve readings in the table above point to different thresholds (for example 0.929 and 0.847 in fig. 5).
• Timing hardware. Section 4.2 gives the Xeon W-2155 and TITAN Xp for timing §4.2. Section 4.3 says Mask R-CNN ran on a "System76 workstation" §4.3. Whether timing included pre- and post-processing, and the batch size at test, are not stated.
• Matching rule. The paper says a prediction counts as correct at 50% overlap §3.4, but does not say whether overlap is measured on masks or boxes, nor how ties and duplicates are handled. The pseudocode leaves this open.
• Equation notes. The IoU formula eq. 3 is written as area overlap over area union, but its right-hand side uses TP, FP and FN, which only matches under one-to-one matching. The AUC integral eq. 5 uses an inverse-function notation that is hard to read, and the text beneath it describes time per image, which is unrelated. Use standard ROC AUC and confirm the unit (pixels or instances) with the authors.
• Single runs, unequal datasets. No variance is reported, and the number of runs is not stated §3.3. Dataset 1 has two classes and Dataset 2 has one, so the datasets do not give like-for-like comparisons.
• Test set sizes. The number of test images and instances per class is not stated.
• Data release. The paper does not state that the images or annotations are public.
• Implementation details. The YOLOv8 library version, the exact model config and the pretrained weights are not stated, so the YOLOv8 results cannot be reproduced from the text alone.

Lineage

• Builds on: Mask R-CNN, as described in the paper with a citation to He et al. (2017) §2.1. It also builds on the YOLO family, with YOLOv8 described as following YOLOv3 and YOLOv5 §2.2.
• Compared against: Mask R-CNN only, in this paper's own experiments §3.3 tab. 3. Other models (Faster R-CNN, RetinaNet, FCOS, U-Net++) appear only as numbers quoted from other studies §4.2 §4.3, which were not reproduced here.
• Future work: the authors plan to test YOLOv9 and YOLOv10 in other agricultural settings §6. No such results are in this paper.