Deep Learning and Machine Vision for Food Processing: A Survey

In plain words

Food is often checked by eye or by hand, which is slow, tiring and uneven. A machine vision system is a camera, a light and a computer that look at each item and let a machine decide what to do with it: grade it, reject it, sort it, or pick out a foreign object. This survey collects what has been tried in that area over the last few years.

It explains the parts of such a system. It lists the kinds of imaging used, from stereo cameras and near-infrared or hyperspectral sensors to X-ray and thermal cameras. It describes how the image is cleaned up, split into the parts of interest, and finally turned into a decision. Then it compares older machine-learning methods, such as support vector machines and random forests, with deep neural networks, especially convolutional ones. For each food-industry task it lists which methods were used.

The survey's main message is that these methods work well in many studies, but robustness to real lighting and backgrounds, the cost of processing large images, and the lack of mature robot-arm actuation still limit large-scale use.

TL;DR

• A survey, not a method paper: it describes machine vision for food processing and lists applications of traditional machine learning and deep learning §1 §2 §4.
• It has no experiments of its own. Every number in the card is quoted from a cited study, with different datasets, metrics and setups, so they are not comparable tab. 4 tab. 5.
• Use it as an index of applications and as a map of machine vision building blocks. It does not give an implementable method.
• Limits named by the authors: robustness to conditions, no imaging route to smell, processing cost, and an immature decision-execution module §5.
• Two equations are misprinted (eq. 2 has a square root; eq. 4 is a chain-rule identity, not a Bayesian-network factorisation). See Pitfalls.

Problem

Food quality and safety are central to health and to the food industry. Manual inspection of grading, defects and foreign objects is labour-intensive and error-prone, and traditional lab tests are time-consuming and destructive, so they are impractical for rapid online checks §1 §3.1. Foreign objects such as bone, glass, metal or insects cause recalls and health risks, and detection by the naked eye is unreliable §3.3. The paper asks which machine vision and machine-learning methods have been applied to these tasks in food processing, and where they stand §1.

Method

This is a survey. Its structure is the method it proposes for organising the field:

1. Machine vision system (MVS). A typical MVS has an image-acquisition part (camera, illumination, a computer that processes the images) and an image-processing part. The output drives the mechanical system's next decision §2 fig. 1 tab. 2.
2. Acquisition modalities. The paper describes stereo systems, remote sensing, hyperspectral imaging, X-ray, thermal imaging, MRI, and additional methods such as a terahertz system and a liquid-crystal tunable filter §2.1.1–§2.1.7.
3. Image processing in three levels. Low-level processing enhances and denoises the image §2.2.1. Intermediate-level processing segments the target from the background, then represents and describes it, for example by shape, size or texture §2.2.2. High-level processing recognises and interprets the target so the system can decide §2.2.3 fig. 2 fig. 3.
4. Food-processing tasks. Three application areas are reviewed: food safety and quality evaluation §3.1, packaging monitoring §3.2, and foreign-object detection §3.3.
5. Traditional machine learning. Supervised, unsupervised and reinforcement learning are introduced. Algorithms covered are support vector machines, logistic regression, k-nearest neighbours, k-means, Bayesian networks, fuzzy c-means, decision trees, random forests, and dimensionality reduction with PCA and LDA §4.1 §4.1.1 fig. 5–fig. 9 eq. 1–eq. 4.
6. Deep learning. Covered are artificial neural networks and back-propagation §4.2.1 fig. 10 fig. 11, convolutional neural networks with convolution, ReLU, pooling, fully connected layers and softmax §4.2.1 eq. 5 eq. 6 fig. 12, recurrent networks, and fully convolutional networks that produce a prediction for every pixel §4.2.1.
7. Application tables. Table 3 lists non-ML classification methods, Table 4 traditional-ML applications, and Table 5 deep-learning applications, each with species, task, method, reported evaluation and reference tab. 3 tab. 4 tab. 5.

Pipeline sketch of the MVS structure described in §2 and §2.2. This is the paper's organising scheme, not a published algorithm:

# Machine vision pipeline for food inspection  [§2] [fig. 1]
# Input:  food items in view of a camera
# Output: a decision for the machine (grade, reject, sort)

def inspect(scene):
    # 1. Acquire: light the sample and capture image(s)  [§2] [§2.1]
    images = capture(scene, camera, illumination)

    # 2. Low-level: enhance, denoise and crop to the region of interest  [§2.2.1]
    images = preprocess(images)

    # 3. Intermediate: segment the target from the background  [§2.2.2] [fig. 2]
    # Segmentation may be threshold, region, gradient, classification or hybrid.
    masks = segment(images)

    # 4. Intermediate: describe the target by shape, size or texture  [§2.2.2]
    descriptors = describe(images, masks)

    # 5. High-level: classify or interpret with statistics or learning  [§2.2.3]
    labels = classify(descriptors)

    # 6. Output the decision that drives the mechanical system  [§2]
    return decide(labels)

K-nearest neighbours, the example the paper describes most fully among the traditional methods §4.1.1 fig. 6. Distances follow eq. 1 or eq. 2:

# k-nearest-neighbour classification  [§4.1.1] [fig. 6]
# Input:  training samples and labels, a query sample, an integer k
# Output: predicted class for the query

def knn_classify(samples, labels, query, k):
    # 1. Distance from the query to every training sample: Euclidean [eq. 1]
    #    or Manhattan [eq. 2] (see Pitfalls on eq. 2)
    distances = [distance(query, s) for s in samples]

    # 2. Indices of the k nearest samples  [§4.1.1]
    nearest = argsort(distances)[:k]

    # 3. Majority vote over the labels of those samples  [§4.1.1]
    return majority(labels[nearest])

Key equations, as printed in the paper:

Euclidean distance between samples x and y eq. 1:

d(x, y) = \sqrt{\sum_{k=1}^{n} (x_k - y_k)^2}

Convolution of a multi-channel input with a kernel plus bias, output at position (i, j) eq. 5:

s(i, j) = (X * W)(i, j) + b = \sum_{k=1}^{n_{in}} (X_k * W_k)(i, j) + b

Softmax over a K-dimensional vector z eq. 6:

\sigma(z)_j = \frac{e^{z_j}}{\sum_{k=1}^{K} e^{z_k}} \quad \text{for } j = 1, \dots, K

Bayesian-network factorisation over a directed acyclic graph G eq. 3:

p(x) = \prod_{i \in I} p\big(x_i \mid x_{pa(i)}\big)

Hyperparameters

The survey reports few hyperparameters, and none for training the surveyed systems in a reproducible way. Values below are what the text states.

Results

Survey: the figures below are as reported by the cited studies, not measured by the authors of this paper. Setups, datasets and metrics differ between rows, so they are not comparable to one another. The baseline column is not applicable.

Full application lists are in tab. 4 (traditional ML, 2 pages of rows) and tab. 5 (deep learning). Use the table rows as leads to the original papers, not as benchmark results.

Paper vs code

No official code. The paper is a survey with no implementation, so there is no code to compare against. No repository was searched.

Pitfalls

• Numbers are not comparable. Accuracy, R², RMSE, RMSEP, relative error and "multiple results" are mixed across rows of tab. 4 and tab. 5. Some rows give no metric at all. Quote a row only together with its study.
• No experiments. The paper does not state a common dataset, split, or evaluation protocol across the studies it reviews.
• Equation (2) is misprinted. It is written with a square root over the sum of absolute differences. The standard Manhattan distance is the plain sum of absolute differences, without a square root. Use the standard form, and note the typo.
• Equation (4) is the chain rule. It expands a joint distribution for any ordering of variables. The factorisation specific to a Bayesian network is eq. 3, where each variable is conditioned only on its parents in the graph.
• Citation problems in the bibliography. Kashyapa (2016) is cited for strawberry calyx removal, but the reference title is about automotive process automation. Pan et al. (2016a) and (2016b) share the same title, journal and page numbers. Mohd Khairi et al. (2018) appears twice. Verify these references before citing them.
• Dated coverage. The survey covers work from about the last five years before 2021 §1. Its deep-learning section covers CNN, RNN, FCN and YOLOv3 §4.2.1. The text does not discuss transformer architectures.
• Hardware and runtime claims are per study. Processing speeds, such as the conveyor and per-image time for chicken portions, depend on each setup and are not general performance figures §4.2.3.
• Open problems the authors list §5: robustness to noisy backgrounds and poor illumination; no imaging route to smell and flavour; storage and compute cost of large image volumes; an immature decision-execution module (for example a robot arm); and the need for edge processing and embedded vision.

Lineage

• Builds on: not applicable; this is a survey of existing methods.
• Compared against: not applicable; the paper compares no methods of its own.
• Superseded by: not stated.