Optimizing Neural Network for Computer Vision task in Edge Device

In plain words

Running an image classifier on a small board such as a Raspberry Pi is hard because the trained model is large and slow to compute with. This paper takes a simple task, recognising a handful of hand gestures from a camera, trains a small convolutional network for it, and then shrinks the finished model so it fits the board.

The shrinking step stores the model's numbers with less precision, which takes less memory and less arithmetic. The authors report that accuracy stayed close to the original, and that the file became roughly a third of its former size. The paper is short and practical, and it leaves many training details out.

TL;DR

A small two-convolution CNN classifies five hand gestures from images resized to 64, 96, 128 or 256 pixels. The authors train it on a GPU, then convert it with TensorFlow Lite to lower-precision weights and report size and accuracy before and after. At 96×96 the converted model is 15.9 MB and scores 96.9% on a 32-image test set, against 98.1% for the unconverted model on 100 validation images §3 tab. 2. Pruning is mentioned but not specified. Treat the numbers as a single small experiment.

Problem

The authors want an edge device to run gesture recognition itself, avoiding cloud round-trips and their delay §1. They argue that large convolutional networks such as ResNet, DenseNet and AlexNet are too big in memory and compute for such devices §1. Floating-point operations get more expensive with higher precision, so reducing precision is presented as the main lever §3. The paper does not state a memory or latency budget for the target device.

Method

The pipeline has three stages. The paper names them as dataset preparation, training, and optimisation §2.1 §2.2 §3.

1. Dataset. Images of each gesture are captured under varied lighting, backgrounds, cameras, angles and distances. Each image is resized with OpenCV to four square sizes (64, 96, 128, 256) and stored in one folder per size and class §2.1 §2.1.1.
2. Network. Each input passes through a 3×3 convolution with 32 filters and ReLU, a 2×2 max-pool and dropout. It then passes through a 3×3 convolution with 64 filters and ReLU, a 2×2 max-pool and dropout. The output is flattened, goes through dropout, then a 128-unit dense layer with ReLU, and finally a 5-unit dense layer with softmax §2.2 tab. 1. The paper does not state whether convolutions are padded and does not list batch-norm layers.
3. Training. Inputs are normalised at every iteration (method not stated). Horizontal flips are the only augmentation named. The data is split 85% for training and 15% for validation §2.2. The loss is categorical cross-entropy over a softmax output eq. 1 eq. 2. Training uses Adam with batch size 32, for at most 25 epochs, and stops when validation loss has risen for 3 epochs, keeping the best weights §2.2. The text reports that the 256×256 model stopped at epoch 14 §2 fig. 4.
4. Class weights. The loss gives slightly higher weight to the gestures "one", "three" and "four" to reduce confusions between similar gestures. The weight values are not stated §2.
5. Conversion. The trained TensorFlow model is converted with TensorFlow Lite, which reduces the precision of its parameters §3. The paper also says the model is structurally pruned using a representative dataset §3. Neither the target precision nor the pruning ratio is stated.

# Gesture model: prepare, train, convert  [§2.1] [§2.2] [§3]
# Input:  labelled gesture photos (any size)
# Output: converted model file for the edge device

def prepare_dataset(photos, sizes=[64, 96, 128, 256]):
    # Resize every photo to each square size  [§2.1]
    for size in sizes:
        resized[size] = resize_all(photos, size, size)
    return resized

def build_network(input_size, num_classes=5):
    # conv 3x3, 32 -> ReLU -> maxpool 2x2 -> dropout  [§2.2] [tab. 1]
    # conv 3x3, 64 -> ReLU -> maxpool 2x2 -> dropout  [§2.2] [tab. 1]
    # flatten -> dropout -> dense 128 (ReLU) -> dense 5 (softmax)  [§2.2]
    return model

def train(model, train_set, val_set):
    # Adam, batch size 32, cross-entropy loss  [§2.2] [eq. 1] [eq. 2]
    # At most 25 epochs; stop when validation loss rises for 3 epochs  [§2.2]
    # Keep the best weights  [§2.2]
    return best_model

def convert(model):
    # Reduce the precision of the parameters with TensorFlow Lite  [§3]
    # Pruning is described (structured, representative data) but unquantified  [§3]
    return converted_model

Hyperparameters

Results

Author-reported. Setup: a five-gesture image dataset, with models trained at four input sizes on a Tesla K80 and then converted with TensorFlow Lite §2.2 §3. "Baseline" here is the same network before conversion, not an external method. Non-optimised rows use the 100-image validation set; optimised rows use a 32-image test set §3 tab. 2.

Headline claims in the text: 96×96 gives over 98% validation accuracy and is the chosen size; its converted model is 15.9 MB and the 64×64 one 6.5 MB; the converted size is "nearly 33%" of the original §2 §3 fig. 14.

Paper vs code

No official code.

Search done: the paper has no code or project link (its only external links are TensorFlow documentation and an ORCID iD) §ref list §1. A GitHub repository search for the full title returned nothing relevant. A search for "hand gesture raspberry pi quantization" returned two repositories under third-party owners, which are not the authors' work by any evidence on the pages. A GitHub user search for "Ranjith M S" returned 16 logins, none verifiably the authors. Nothing was cloned.

Pitfalls

• Unlike sets. The unconverted models are scored on a 100-image validation set and the converted ones on a 32-image test set §3 tab. 2. Each image on the 32-image set is 3.125 points of accuracy, so 96.9% against 93.75% is one image. Loss values for the two rows also come from different sets. The paper does not show the two sets are comparable.
• Loss rises after conversion. Loss roughly triples to sextuples at each size (for example 0.0897 to 0.5665 at 96×96) while accuracy stays near the same tab. 2. The paper says the loss is "comparatively high" but gives no explanation §3 fig. 13.
• Conversion precision is not stated. The text says the precision is reduced, and that the trained model's parameters are 64-bit §3. Neither the target format nor the stored precision of the converted file is given. The 64-bit statement is not checked against the code, because the code is absent.
• Size numbers look like float32 storage, which is a derived result. I computed the parameter count of each network from the layer shapes in tab. 1. This assumes unpadded 3×3 convolutions and floor pooling, which the paper does not state. Under that assumption the 256×256 network has 31,510,213 parameters, the 128×128 network about 7.39 million, the 96×96 about 3.99 million and the 64×64 about 1.63 million. The unconverted files come to about 12 bytes per parameter at every size. The converted files come to about 4 bytes per parameter, which is the storage size of 32-bit floats. So the reported roughly 3× reduction is consistent with dropping training state, not necessarily with lower-precision weights. The paper does not say what "model size" measures. Verify before relying on it.
• Pruning is not quantified. §3 says the parameters are pruned in a "structured" way on a representative dataset §3. It gives no ratio, no criterion and no before-and-after parameter count. The size arithmetic above shows no sign of pruning at 256×256.
• Almost all parameters sit in one layer. The first dense layer has 31,490,176 of the 31,510,213 parameters, about 99.9% tab. 1. Any size reduction is therefore driven by that layer. The paper does not discuss this.
• Batch norm and initialisation are mentioned but not in the architecture. §1 says Batch Normalization is used to speed training §1. Neither tab. 1 nor the text in §2.2 lists a batch-norm layer. §1 also prefers He initialisation and mentions LSUV, but does not say which one the trained model used §1.
• Stopping rule and reported epoch. The rule is patience 3 on validation loss, and the 256×256 model is said to stop at epoch 14 §2.2 §2. The text does not say whether the restored weights come from the best epoch.
• Dataset not described. Images per class, number of subjects, and how the 32-image test set was drawn are not stated §2.1 §3. Results are therefore hard to reproduce or transfer.
• Accuracy trend is small. At 96×96 and 64×64 the unconverted accuracies differ by about 0.4 points (0.9811 vs 0.9773) on 100 validation images tab. 2. The text's choice of 96×96 as best rests on this difference and on a single run; no variance is reported.
• No latency or memory measured. The text claims "good accuracy and performance during inference" §3 but gives no timing, throughput or RAM figure for the Raspberry Pi.
• Table captions. Both Table 1 and Table 2 are captioned "Network structure". Table 2 holds the optimisation results.
• Figure numbering. The numbers used here follow the order of captions in the LaTeX source, because the PDF was not available. The label names in the source do not match these numbers (for example the label "fig3" sits on the loss figure, which is fig. 4 in source order). Check against the compiled PDF before citing numbers.

Lineage

• Builds on: standard CNN training with Adam and cross-entropy, He or LSUV initialisation, dropout, and batch normalisation, all cited in the paper's introduction §1 §1.1. The conversion step uses TensorFlow Lite post-training precision reduction, cited to TensorFlow documentation only.
• Compared against: the same network before conversion, at four input sizes tab. 2. No external method is compared.
• Superseded by: not stated.