Advanced Steel Microstructural Classification by Deep Learning Methods

In plain words

The inside of a metal, its microstructure, decides most of its mechanical properties. Engineers identify the parts of that microstructure by looking at microscope images, and today experts still do this by hand, so two experts can disagree. Earlier automatic methods relied on shape measurements designed by people and reached 48.89% accuracy on the same dataset.

This paper trains a neural network to colour every pixel of a steel image with the kind of structure it belongs to. The network sees the texture itself rather than a list of hand-made measurements. Each separate grain is then given the label that most of its pixels received. The method reaches 93.94% accuracy on the authors' dataset, far above the earlier 48.89%. The result comes from a small dataset of 21 images, so it is a proof of concept rather than a general tool.

TL;DR

Steel microstructure constituents (martensite, tempered martensite, bainite, pearlite, with ferrite as matrix) are classified from SEM images. The paper compares object-based CNN classification of cropped, resized grains with a pixel-based pipeline called MVFCNN: a fully convolutional network labels every pixel of 1000x1000 px patches, the patches are stitched back together, and each object gets the majority label of its pixels. On the authors' 21-image dataset the best MVFCNN (fine-tuned, class-balanced, augmented) reaches 93.94% against 48.89% for the earlier method, all author-reported on one train/test split. Read this if you work on pixel-level classification of large images where objects must not be resized. The source contains no code, and several numbers in it do not reconcile (see Pitfalls).

Problem

Microstructure classification is done manually by experts, which brings subjectivity §1. Automatic classification is hard because a microstructure can mix several phases with complex sub-structure §1. Earlier automatic work used hand-designed features with SVMs; on the same dataset, that approach reached 48.89% §2, tab. 1. Resizing each cropped object to a fixed CNN input can destroy the texture that distinguishes classes §4.1, §6. Full SEM images (about 7000x8000 px) cannot be fed to a network as one input because of GPU memory §4.2.

Method

The paper's main method is the pixel-based MVFCNN §4.2, fig. 3. The object-based CNN baselines §4.1, fig. 2 are compared against it. The FCN architecture is that of Long et al., almost the same as VGG16 with the fully connected layers converted to convolutions, plus up-sampling and skip layers §4.2, §3.1.

1. Object locations from a binary LOM mask. A user-defined intensity threshold on the LOM image gives a binary mask. The SEM image, already registered to the LOM image, is masked with it. This localises each constituent §4.1. The threshold value is not stated.
2. Patch cropping with a sliding window. The full image is cut into HxW windows that slide with a horizontal and a vertical stride. Patches of 1000x1000 px were used in training §4.2, §4.3.3. Patch size is limited by GPU memory §4.2. Inference stride is not stated §4.2.
3. Pixel-wise segmentation. Each patch goes through the FCN. The output has one channel per class and holds a softmax posterior per pixel §4.2, eq. 3. The pixel label is the class with the highest posterior §4.2. The matrix (ferrite) is a class as well and is treated as background §5.3, tab. 6.
4. Stitching. Segmented patches are placed back at their positions to form a full-size label map §4.2, fig. 3.
5. Max-voting per object. For each object from step 1, the class held by the most of its pixels is assigned to the whole object §4.2, §5.2.
6. Class balancing by stride. Stride is set per class so that every class yields about the same number of patches. The horizontal and vertical strides are equal. Rarer classes get smaller strides §4.3.4. Exact per-class values are not stated. The paper's worked example is a 7000x8000 px image with a 100 px stride, giving 61*71 = 4331 patches of 1000x1000 px §4.3.4.
7. Augmentation by rotation. Patches are rotated by 90, 180 and 270 degrees §4.3.4.
8. Training schedule. Start with FCN-32s, then add skip layer A (FCN-16s) and fine-tune, then add skip layer B (FCN-8s) and fine-tune. Direct training of FCN-8s gave worse results §4.3.3. Details are in the schedule below and the Hyperparameters table.

The object-based path, for comparison §4.1, §4.3.1, fig. 2: each object is cropped from the masked SEM image, resized to 224x224 px, and classified with one of three techniques. (I) CIFARNet trained from scratch. (II) VGG16 fine-tuned by transfer learning. (III) Features from the fully connected layer before the classifier of a pre-trained VGG19, fed to an SVM with RBF kernel §4.1, §4.3.2. Softmax and cross-entropy are standard §3, eq. 3, eq. 4. Gradient descent with learning rate \eta is standard §3, eq. 5.

Inference (sketch, our own pseudocode, based on steps 1-5 above):

# MVFCNN inference for one steel image  [§4.2] [§5.2]
# Input:  image (H, W) from SEM or LOM; object_mask (H, W) from binary LOM
# Output: one class label per object

def classify_objects(image, object_mask, fcnn, window, stride):
    # 1. Cut the image into windows of size `window`, moved by `stride`
    #    [§4.2] [§4.3.4]
    patches = sliding_windows(image, window, stride)
    # 2. Label each patch pixel-wise: argmax of the FCN's softmax over
    #    classes, matrix (ferrite) included  [§4.2] [eq. 3]
    patch_labels = [argmax_over_classes(fcnn(patch)) for patch in patches]
    # 3. Place the labelled patches back into one full-size map  [§4.2]
    #    Overlap handling for overlapping windows is not stated.
    label_map = stitch(patch_labels, image.shape)
    # 4. Max-voting: each object takes the majority pixel label  [§4.2] [§5.2]
    object_labels = {}
    for object_id in objects_in(object_mask):
        # pixels that belong to this object  [§4.1]
        votes = label_map[object_mask == object_id]
        object_labels[object_id] = most_common(votes)  # ties not stated
    return object_labels

Training schedule for MVFCNN, as stated §4.3.3. Each stage starts from the weights of the previous one:

# Staged fine-tuning of MVFCNN  [§4.3.3]
# Momentum 0.9, weight decay 5e-4, batch size 1 in every stage.
# Iterations: 7000 is stated once; per-stage count is not stated.

def train_mvfcnn(patches, pretrained_fcn32):
    # 1. Train FCN-32s (no skip layers) with learning rate 1e-10  [§4.3.3]
    fcn32 = train(pretrained_fcn32, skips=[], learning_rate=1e-10)
    # 2. Add skip layer A and fine-tune FCN-16s, learning rate 1e-11 [§4.3.3]
    fcn16 = train(fcn32, skips=["A"], learning_rate=1e-11)
    # 3. Add skip layer B and fine-tune FCN-8s, learning rate 3e-12 [§4.3.3]
    fcn8 = train(fcn16, skips=["A", "B"], learning_rate=3e-12)
    return fcn8

The metrics used for pixel-wise segmentation are §5.2. n_{ij} is the number of pixels of class i predicted as class j, n_{cl} the number of classes, and t_i = \sum_j n_{ij} the number of pixels of class i.

\text{pixel acc} = \frac{\sum_{i} n_{ii}}{\sum_{i} t_{i}} eq. 7

\text{mean acc} = \frac{1}{n_{cl}} \sum_{i} \frac{n_{ii}}{t_{i}} eq. 8

\text{mean IU} = \frac{1}{n_{cl}} \sum_{i} \frac{n_{ii}}{t_{i} + \sum_{j} n_{ji} - n_{ii}} eq. 9

\text{f.w. IU} = \Big(\sum_{k} t_{k}\Big)^{-1} \sum_{i} \frac{t_{i}\, n_{ii}}{t_{i} + \sum_{j} n_{ji} - n_{ii}} eq. 10

The paper also gives the up-sampling weight in the form below eq. 6. It is reproduced as typed in the source. See Pitfalls.

y_{ij} = \sum_{\alpha,\beta=0}^{1} |1-\alpha-\{i/f\}|\;\;|1-\beta-\{i/j\}|\;\;x_{\lfloor i/f\rfloor+\alpha,\,\lfloor j/f\rfloor+\beta} eq. 6

Equation numbers were counted from the order of equation environments in the LaTeX source, since the rendered PDF was not inspected.

Hyperparameters

Results

Author-reported. Setup: 21 registered LOM and SEM image pairs from the MECS dataset, one image-level train/test split (11/10 images), Caffe, one K40m GPU §5.1, §4.3. The baseline is Pauly et al., which the paper reports as 48.89% on the same dataset with four classes §2, §5.3, tab. 1. No repeated runs or variance are reported.

Main comparison, SEM images, object-level accuracy §5.3, tab. 1:

Effect of data and fine-tuning, MVFCNN, SEM §5.3, tab. 2:

LOM versus SEM, object-level accuracy §5.3, tab. 3:

Best MVFCNN confusion matrix, SEM, object-level §5.3, tab. 4. Columns are the actual class, rows the predicted class. Recall is per column and precision per row. The 48 objects missed by segmentation are excluded from this table.

Pixel-wise segmentation metrics, SEM and LOM §5.3, tab. 5:

Per-class pixel accuracy, same configurations §5.3, tab. 6:

Further results, from the text §5.3, §6: the best segmentation reports 93.92% pixel accuracy. On the 10 test images, the image-level composition is correct for all ten after max-voting §5.3, fig. 7. The system is reported to give similar responses to rotated patches, and inputs under 100x100 px give worse results than 1000x1000 px for the same area §6. Inference is about 600 ms per 1000x1000 px patch §4.3.3.

Figures, described and cited, not reproduced: fig. 1 shows SEM (top) and LOM (bottom) examples of martensite, tempered martensite, bainite and pearlite. fig. 2 is the object-based CNN workflow. fig. 3 is the MVFCNN pipeline with cropping, segmentation, stitching and max-voting. fig. 4 shows the dataset classes. fig. 5 explains the object definition. fig. 6 shows success and failure cases of the best SEM segmentation, with bainite as the main failure. fig. 7 shows max-voting examples. fig. 8 compares LOM and SEM segmentation and tests robustness to noise patches §5.3.

Paper vs code

No official code. The arXiv abstract page links no code, project page or repository. A GitHub repository search for the title words and for "microstructure FCNN max-voting" found none. The paper names Caffe §4.3 but does not release code or the dataset (see Pitfalls). Source version: LaTeX of arXiv v2 (updated 2018-02-15) as fetched.

Pitfalls

• Object counts do not reconcile. §5.1 gives 2262 test objects. Table 4 columns sum to 2205 objects, plus 48 unsegmented, makes 2253. The headline 93.94% would be 2125 of 2262, but Table 4 has 2100 correct objects. The denominator of the headline figure is therefore not stated and cannot be reproduced from the tables.
• Table 4 totals. Recomputed from the counts, the total accuracy is 2100/2205 = 95.24% (printed 95.23%), bainite precision 325/386 = 84.20% (printed 84.19%), pearlite recall 317/333 = 95.20% (printed 95.19% in the recall row). These differences are rounding-level but should not be copied as exact figures.
• Augmentation gain. The text says augmentation improves performance by 2 percentage points §6. Table 2 gives 90.97% to 93.94%, which is 2.97 points.
• Learning rate for the object-based CNNs. One sentence fixes 0.001. A later sentence gives 0.0001 "chosen on validation set" for CIFARNet and VGG16 "respectively", with a single value. The validation split is not described.
• Input size. §4.3.1 says CIFARNet can take any input size, then resizes the objects to 224x224 for training from scratch as well.
• Pre-training for the FCN. The FCN-32s starting point is "trained with an ImageNet dataset" §4.3.3. Whether the VGG16 weights are the standard ImageNet release is not stated.
• Iterations and stopping. For the MVFCNN, 7000 iterations is stated once, and no stopping or validation rule is given. For the object CNNs, training stops at a loss plateau with no criterion.
• Test-time stitching. How overlapping windows are merged, the inference stride, and how max-vote ties break are not stated.
• Object masks depend on LOM. The method needs a registered LOM image and a threshold chosen by the user §4.1, §4.2. The threshold is not stated. For SEM work, the cost of the registration step is not discussed.
• Unsegmented objects. 48 objects are not segmented and are left out of the confusion matrix tab. 4. The source does not say whether they count as errors in the headline figure.
• Baseline re-run. §5.3 says the Pauly et al. accuracy of 48.89% is "shown" on the same dataset. It does not say whether that number was re-measured or copied.
• Small test set. There are 10 test images, one of them a transformed bainite sample absent from training §5.1, §6. Bainite pixel accuracy is 37.32% even with the best setup tab. 6. Results come from one split and one run.
• Equation 3. The softmax denominator sums over k but uses W_j. The standard form uses W_k.
• Equation 6. The second fractional term is written as \{i/j\} where the horizontal position would be \{j/f\}. The equation is reproduced as typed. Implement standard bilinear up-sampling and check it against the PDF.
• Class balancing. Per-class strides are described only qualitatively. The balanced patch counts in the text are for one example image.
• Rotation claim. The paper says the system is rotation invariant and cites matching responses for rotated patches §6. This is an observation, not a tested property of the architecture.
• Dataset availability. The sentence on public availability is commented out in the source. Availability is not stated.
• Caffe version. No version number is given. Layer definitions, such as the skip-layer wiring, are not provided, so the FCN-16s and FCN-8s fusion must be rebuilt from the description.

Lineage

• Builds on: fully convolutional networks for semantic segmentation (Long et al., cited as [fcnlong] in the paper §3.1, §4.2), VGG (Simonyan et al., [vgg], used as the backbone §4.3.1), and CIFARNet, a LeNet-based network §3, §4.3.1. These are not in this wiki and no arXiv id is given in the source.
• Compared against: Pauly et al. (SVM with morphological features, 48.89% on this dataset) §2, §5.3; object-based CNNs (CIFARNet, VGG16 and VGG19 DeCAF features) §5.3.
• Dataset and ground-truth procedure: Britz et al. §5.1 and Aarnts et al. §5.1.
• Superseded by: not stated in this source.