wikillm

Paper · 2018-05-04 · cs.CV

High throughput quantitative metallography for complex microstructures using deep learning: A case study in ultrahigh carbon steel

Applies PixelNet, a VGG-16 hypercolumn network with a per-pixel MLP, to segment ultrahigh-carbon-steel micrographs into microconstituents and spheroidite particles, trained with focal or cross-entropy loss on 24 micrographs per task. Six-fold cross-validation gives 86.5% overall pixel accuracy (focal model); particle mean IU is 72.4 versus 69.9 for Otsu. Author-reported.

Semantic segmentationFocal lossSegmentation evaluation (scoring foreground, saliency and segmentation masks)Transfer learningsegmentationmaterials-sciencemicroscopy

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Microscopy labs photograph metal under a microscope, and the picture shows different regions: hard particles, a network of carbide along the grain boundaries, the soft metal in between, and needle-like plates. Measuring how much of each there is, and how big the particles are, has usually meant either hand-tracing the regions or running image filters that were tuned for one kind of sample. Both are slow or brittle.

This paper trains a neural network to label every pixel of such a picture. A person first draws the correct labels on a set of photos. The network learns to reproduce them, and then it can label new photos in about a second each. Once every pixel has a label, the usual measurements (particle sizes, the width of the soft band next to the network) can be computed automatically.

The authors are candid about the limits. The training set is small, the smallest particles and the thinnest plates are hard to see at the image resolution used, and the human labels themselves are subjective. The paper presents its results as a case study rather than a finished tool.

TL;DR

  • Task: per-pixel segmentation of ultrahigh-carbon-steel micrographs, in two settings: four microconstituents, and spheroidite particles. §2.2
  • Model: PixelNet, a VGG-16 backbone whose intermediate feature maps are upsampled and concatenated, then classified per pixel by a small MLP. §2.1.1
  • Training: two stages with AdamW and a focal loss, followed by evaluation with six-fold cross-validation. §2.1.2 §2.3
  • Headline results are author-reported. Focal model: 86.5% overall pixel accuracy tab. 1. Particles: mean IU 72.4 versus 69.9 for Otsu tab. 4.
  • Code: the official TensorFlow/Keras code has no LICENSE file, so it can be read but not reused.

Problem

Quantitative microstructure analysis needs region-level labels. Conventional segmentation relies on low-level intensity and connectivity rules that often need expert tuning for each microstructure system, so the microstructures that can be analysed are limited. §1 Manual annotation is the alternative, but it is laborious: the authors report 20 to 30 minutes per micrograph after an initial learning curve. §3.3

Method

Overview

The network maps a grayscale micrograph to a per-pixel class. Feature maps from six layers of an ImageNet-pretrained VGG-16 are bilinearly upsampled to the input resolution and concatenated into a hypercolumn vector for each pixel. An MLP maps that vector to class scores. Following PixelNet, the MLP is only evaluated at a sample of pixels during training. §2.1.1

Pixel representation and head §2.1.1 fig. 1

  1. The backbone is VGG-16: thirteen convolutional layers, followed by two dense layers. The second dense layer (layer 7) is turned into a 7x7 convolution filter. §2.1.1
  2. Hypercolumn layers are the last ReLU output of blocks 1 to 5 and the second fully-connected layer (layer 7). Each is batch-normalised and bilinearly upsampled, then concatenated into h(x). §2.1.1
  3. The MLP has 1024-unit hidden layers with ReLU and batch normalisation. The number of hidden layers is not stated in the text; the code uses two. §2.1.1 pixelnet.py
  4. A softmax output layer gives the class probabilities. [code: pixelnet/pixelnet.py#L103]
Pseudocode
# PixelNet forward pass at sampled pixels  [§2.1.1]
# Input:  image (H, W, 3); pixel coordinates (P, 2), in [0, 1]
# Output: class probabilities (P, 4)

def predict_pixels(image, coords):
    # Feature maps after ReLU in blocks 1-5 and fc7  [§2.1.1]
    maps = [vgg_feature(image, layer) for layer in HYPERCOLUMN_LAYERS]
    # Batch-normalise each map (no learned scale)  [§2.1.1]
    maps = [batch_norm(m, scale=False) for m in maps]
    # Bilinear sample at each pixel coordinate, then concatenate  [§2.1.1]
    hypercolumns = concat([sample_bilinear(m, coords) for m in maps])
    # Two hidden layers of 1024 units  [§2.1.1]
    hidden = mlp_layer(hypercolumns, 1024)
    hidden = mlp_layer(hidden, 1024)
    # Softmax over the four classes  [§2.1.1]
    return softmax(dense(hidden, 4))

Loss §2.1.2 eq. 1 eq. 2

The focal loss is used to counter class imbalance: widmanstatten cementite is about 3% of pixels. §2.1.2 With the true-class probability p_t,

p_t(p, y) = \begin{cases} p & \mathrm{if } y = 1 \\ 1-p & \mathrm{if } y = 0 \end{cases} eq. 1

\mathrm{FocalLoss}(p_t) = - \alpha_t (1-p_t)^{\gamma} \log(p_t) eq. 2

The focusing parameter is gamma = 2. The class weights alpha_t are set from class frequencies; the exact formula is discussed under Paper vs code. §2.1.2 A standard cross-entropy model is the comparison baseline. §3.1 tab. 2

Training §2.1.2 fig. 1

The training procedure has two stages on a fixed training split:

Pseudocode
# Two-stage PixelNet training  [§2.1.2]
# Input:  micrographs (N, H, W); pixel labels (N, H, W), four classes
# Output: trained network (VGG-16 backbone and pixel MLP)

def train_pixelnet(images, labels):
    # Local histogram equalisation  [§2.1.2]
    inputs = equalise_local_histogram(images)
    # Freeze the VGG-16 feature layers  [§2.1.2]
    freeze(vgg16_layers)
    # Stage 1: 125 updates at learning rate 1e-3  [§2.1.2]
    for update in range(125):
        # 4 augmented images, 2048 random pixels from each  [§2.1.2]
        batch = sample_augmented_pixels(inputs, labels, 4, 2048)
        loss = focal_loss(predict_pixels(batch), batch.labels)
        adamw_step(learning_rate=1e-3, weight_decay=5e-4)
    # Stage 2: unfreeze and fine-tune for 125 updates at 1e-5  [§2.1.2]
    unfreeze(vgg16_layers)
    for update in range(125):
        batch = sample_augmented_pixels(inputs, labels, 4, 2048)
        loss = focal_loss(predict_pixels(batch), batch.labels)
        adamw_step(learning_rate=1e-5, weight_decay=5e-4)
    return network

Each update is computed from 2048 pixels sampled from each of four augmented training images. §2.1.2 Augmentation is random rotation over the full circle, horizontal and vertical mirroring, random zoom, and a random intensity shift of plus or minus 5%. §2.1.2 The augmentation is applied jointly to the image and its label map; nearest-neighbour interpolation is used for labels. §2.1.2 utils.py

Denuded-zone width §2.4 fig. 5

The denuded zone is the particle-free band of metal next to the carbide network. Its width is measured after cleaning the predicted label map.

Pseudocode
# Denuded-zone widths from a predicted label map  [§2.4]
# Input:  label map (H, W): matrix, network, spheroidite, widmanstatten
# Output: widths in pixels along the matrix-particle interface

def denuded_zone_widths(labels):
    # Fill holes inside the network phase  [§2.4]
    labels = fill_network_holes(labels)
    # Close matrix pixels; keep only matrix touching the network  [§2.4]
    labels = drop_matrix_disconnected_from_network(labels)
    # Matrix nearer to widmanstatten than to network becomes particle  [§2.4]
    labels = relabel_nearby_matrix(labels)
    # Widmanstatten regions are removed (relabelled as particle matrix)  [§2.4]
    labels = relabel_widmanstatten(labels)
    # Euclidean distance from each pixel to the nearest network pixel  [§2.4]
    distance = distance_to_network(labels)
    # Pixels on the matrix-particle interface  [§2.4]
    interface = matrix_particle_boundary(labels)
    return distance[interface]

Particle size distributions are compared with a two-sample KS test. §2.3 The code's size measure is the square root of each region's pixel area perf.py.

Evaluation metrics §2.3

Pixel accuracy, per-class precision and recall, and per-class intersection over union (IU) are reported. §2.3 eq. 3 eq. 4 eq. 5

\mathrm{Precision}(c) = \frac{ \sum_i [\hat{y}_i = c \wedge y_i = c] }{ \sum_i [\hat{y}_i = c] } eq. 3

\mathrm{Recall}(c) = \frac{ \sum_i [\hat{y}_i = c \wedge y_i = c] }{ \sum_i [y_i = c] } eq. 4

IU(c) = \frac{ \sum_i [\hat{y}_i = c \wedge y_i = c] }{ \sum_i [\hat{y}_i = c \vee y_i = c] } eq. 5

The paper states that overall accuracy equals the mean of the per-class recall (or precision). §2.3 The overall IU in the tables is the unweighted mean over the classes. tab. 1 perf.py

Hyperparameters

NameValueWhereNotes
BackboneVGG-16, ImageNet-pretrained§2.1.2weights loaded from a hard-coded local path in the code vgg.py
Hypercolumn layersblock 1 conv 2, block 2 conv 2, block 3 conv 3, block 4 conv 3, block 5 conv 3, fc7 (7x7 conv)§2.1.1code uses the ReLU output of each named layer transfer_pix.py
Hypercolumn normalisationbatch norm on each map, before upsampling§2.1.1code uses batch norm with no learned scale hypercolumn.py
MLP width1024§2.1.1code transfer_pix.py
MLP hidden layersnot stated§2.1.1code uses 2 pixelnet.py
Dropout0.1§2.1.2paper: after final MLP layer; code: after each MLP layer pixelnet.py
OptimiserAdamW with the recommended default parameters (not further specified)§2.1.2code sets AMSGrad on transfer_pix.py
Weight decay5e-4§2.1.2AdamW-style decoupled decay adamw.py
Stage 1 learning rate1e-3, VGG frozen§2.1.2125 updates transfer_pix.py
Stage 1 length20 epochs (125 updates)§2.1.2code: 25 epochs of 5 steps; same 125 updates transfer_pix.py
Stage 2 learning rate1e-5, all layers§2.1.2125 updates transfer_pix.py
Batch4 images x 2048 pixels per update§2.1.2transfer_pix.py
Initialisation (new layers)He-normal, sigma = sqrt(2/c)§2.1.2code's MLP uses the Keras default; see Paper vs code
Focal gamma2§2.1.2transfer_pix.py
Class weights alpha_tproportional to inverse class frequency§2.1.2code uses sqrt of inverse frequency, L1-normalised transfer_pix.py
Input equalisationlocal histogram equalisation§2.1.2code: scikit-image adaptive equalisation with default parameters data.py
Rotationfull circle, [0, 2pi)§2.1.2code: 360 degrees transfer_pix.py
Mirroringhorizontal and vertical§2.1.2transfer_pix.py
Scalingrange [1, 2]§2.1.2code: zoom factor 1 + 0.5 x U, i.e. [1, 1.5) utils.py
Intensity shiftplus or minus 5%§2.1.2transfer_pix.py
Interpolationbilinear for inputs, nearest for labels§2.1.2code uses scipy's default spline order for inputs utils.py
Cross-validation6 folds, 4 micrographs each§2.3uhcs-validation-sets.json
Bottom crop (UHCS)not statednot in papercode removes the bottom 38 rows (micron bar) transfer_pix.py
Denuded zone: hole fillnot statednot in papercode: holes up to 128 px measure.py

Results

Author-reported. Setup: micrographs of 645 x 484 pixels, 24 annotated micrographs for the four-class microconstituent task and 24 for the particle task, six-fold cross-validation with four micrographs per validation set. §2.2 §2.3 Baselines: a cross-entropy model §3.1 and Otsu thresholding, particles only §3.2. The authors report that a commodity GPU segments about one micrograph per second after 2 to 4 hours of training. §3.3

Four-class microconstituents tab. 1 tab. 2

Focal-loss model tab. 1; values are mean plus or minus standard error over validation images.

ClassIUPrecisionRecall
matrix49.1 ± 3.460.3 ± 4.472.3 ± 3.7
network72.9 ± 5.385.5 ± 4.080.7 ± 5.9
spheroidite85.7 ± 1.895.1 ± 1.289.8 ± 1.7
widmanstatten42.7 ± 2.950.2 ± 3.673.5 ± 3.9
overall62.6 ± 2.586.5 ± 1.686.5 ± 1.6

Cross-entropy model, trained with hyperparameters the paper calls revised (values not stated) §3.1 tab. 2:

ClassIUPrecisionRecall
matrix67.3 ± 6.281.8 ± 8.079.4 ± 4.0
network89.4 ± 3.696.1 ± 0.692.8 ± 3.5
spheroidite91.9 ± 3.594.9 ± 2.696.6 ± 1.7
widmanstatten52.9 ± 6.472.2 ± 7.166.8 ± 8.1
overall75.4 ± 3.792.6 ± 2.592.6 ± 2.5

The paper reports the cross-entropy model as better on most classes and overall, and uses the focal model for the rest of the paper because of its particle results. §3.1

Spheroidite particles tab. 3 tab. 4

ModelIU matrixIU spheroiditeIU meanPixel accuracyPSD KS
Otsu tab. 486.2 ± 7.253.7 ± 12.169.9 ± 9.388.1 ± 6.1not applicable
PixelNet, focal tab. 490.0 ± 1.054.8 ± 3.472.4 ± 3.191.1 ± 0.90.042

Per-class precision and recall for PixelNet are in tab. 3: matrix precision 95.0 ± 0.6 and recall 94.5 ± 1.1; spheroidite precision 74.6 ± 2.8 and recall 70.3 ± 4.3. The KS value of 0.042 is read by the authors as the fraction of micrographs in which the predicted and annotated size distributions are not distinguishable (one of 24). §3.2 tab. 4

Derived quantities §3.3 fig. 4 fig. 5

The authors demonstrate, without tabulated numbers, particle size distributions conditioned on spheroidite regions and denuded-zone width distributions. They say the input resolution is insufficient for quantitatively accurate particle size distributions. §3.3

Paper vs code

Two repositories are relevant, both under the GitHub owner bdecost, which matches the first author's name. The paper's footnote links bdecost/pixelnet as the authors' implementation, and that repository's README links bdecost/uhcs-segment as the training scripts for this paper. The training code is uhcs-segment at code_commit. The network code is pixelnet at fae2dd53c71291f26f2611ecd4417ded1ed581e4. Paths beginning pixelnet/ refer to the second repository; all other paths refer to the first. Neither repository has a LICENSE file, so no verbatim code is quoted here. No other official code was searched for, because both links were given by the authors.

  1. Class weights. Paper: alpha_t proportional to inverse class frequency §2.1.2. Code: square root of inverse frequency, then L1-normalised transfer_pix.py. A comment in the same script says the alpha-balanced variant is not used, yet the weights are still passed to the loss transfer_pix.py transfer_pix.py.
  2. Loss reduction. The paper does not say how the loss is reduced over a batch. The code sums it, without averaging losses.py. The default alpha of 0.5 is ignored once class weights are given losses.py.
  3. Epochs. Paper: 20 epochs for stage 1, with 125 updates §2.1.2. Code: 25 epochs of 5 steps each (int of 20 training images over batch 4), which also gives 125 updates transfer_pix.py. The update count matches; the epoch count does not.
  4. Optimiser variant. Paper: AdamW with "recommended default parameters" §2.1.2. Code: AMSGrad enabled in both stages transfer_pix.py transfer_pix.py adamw.py. The decay term is scaled by the bias-corrected learning rate adamw.py.
  5. Scaling range. Paper: scale in [1, 2] §2.1.2. Code: zoom factor 1 + 0.5 x U, i.e. [1, 1.5) utils.py transfer_pix.py.
  6. Interpolation order. Paper: bilinear for input images §2.1.2. Code passes no interpolation order to scipy's rotate and zoom, so scipy's default spline order applies to images utils.py utils.py. Labels use order 0, which matches the paper utils.py.
  7. Dropout placement. Paper: 10% dropout after the final MLP layer §2.1.2. Code: applied after each of the two MLP layers pixelnet.py.
  8. MLP order. Paper: ReLU followed by batch normalisation §2.1.1. Code: a bias-free dense layer, then batch normalisation, then ReLU pixelnet.py.
  9. Initialisation. Paper: He-normal with sigma = sqrt(2/c) §2.1.2. Code: the MLP's dense layers use the Keras default initialiser. The variance-scaling initialiser (scale 0.5, fan-in) is used only in the SELU path, which the training script does not select pixelnet.py pixelnet.py transfer_pix.py.
  10. Preprocessing. The code also replicates the grayscale input to three channels and applies VGG's standard input preprocessing; the paper does not mention either transfer_pix.py. The equalisation uses scikit-image defaults data.py.
  11. Crop. The code removes the bottom 38 rows of UHCS micrographs (the micron bar) before training transfer_pix.py data.py. The paper gives the size 645 x 484 and does not mention a crop §2.2.
  12. Pixel sampling. Paper: a random sample of 2048 pixels per image §2.1.2. Code draws pixel coordinates uniformly and excludes the last row and column utils.py. A class-stratified sampler exists in the same module but the training script does not use it utils.py transfer_pix.py.
  13. Widmanstatten removal. Paper: widmanstatten regions are removed §2.4. Code relabels them as particle matrix, so they still count toward the matrix-particle interface measure.py. The code also fills network holes up to 128 pixels, which the paper does not state measure.py.
  14. Size unit. The paper speaks of particle "radii" and a 5-pixel size threshold §3.2. The code's particle size is the square root of the region's area in pixels, not a radius perf.py. Distances in the denuded-zone code are in pixels measure.py; no conversion to micrometres is stated in the paper.
  15. KS score. Paper: the KS score is the fraction of micrographs in which the test does not reject equality §2.3. The code's KS helper returns a per-micrograph p-value perf.py. I found no code for the fraction in the modules read.
  16. Cross-entropy model. The paper says the cross-entropy model used revised hyperparameters §3.1 without listing them. The training script has the cross-entropy compile line commented out transfer_pix.py. bin/train_pixelnet.py uses Adam with learning rate 1e-3, categorical cross-entropy, at most 10 epochs and early stopping train_pixelnet.py train_pixelnet.py train_pixelnet.py. It imports pixelnet_model, which pixelnet/pixelnet.py does not define train_pixelnet.py. I could not establish from the repository which configuration produced Table 2.
  17. Other code. bin/crossval.py implements leave-one-out cross-validation with a linear SVM on reduced hypercolumn features. That is a different method from the CNN evaluation in the paper crossval.py.
  18. Data. The scripts read data/<dataset>.h5 transfer_pix.py. Only TIFF labels and a SQLite database are in the repository. I did not read bin/mitk_to_hdf5.py, which may build the HDF5 files.
  19. Dataset size wording. The paper says "24 annotated micrographs total" in one place §2.3 and 24 per task in another §2.2. The repository has 24 label files for each task, with disjoint names, so there are 48 in total. uhcs-validation-sets.json spheroidite-validation-sets.json

Pitfalls

  • Unclear numbers in the text. The text describes the cross-entropy improvement as roughly ten IU points, but the two overall IU values in the tables differ by 12.8 (75.4 minus 62.6). §3.1 tab. 1 tab. 2
  • Unclear sample size for error bars. The standard errors are described as computed over "the six validation images" §2.3, while the table captions say "across validation images" tab. 1. With 24 micrographs, n could be 6 or 24.
  • The zoom augmentation may fail on three-channel input. The code zooms the whole (H, W, 3) array with a single scalar factor utils.py. scipy's zoom resamples every axis, including the channel axis of size 3. A factor above about 1.17 rounds the channel count to 4, and the crop assignment on the next line would then fail utils.py. I read this from the code and did not run it; check before reuse.
  • Small particles are missed. The model misses most particles of one to five pixels, and many of the rest are partially labelled. §3.2 The annotations also omit particles that touch the image border, and watershed splitting is applied to annotations but not predictions. §3.2 These biases affect the particle size distributions. §3.3
  • Labelling is subjective. The authors note labelling bias, for instance where the matrix around Widmanstatten lath is unlabelled. §3.1 §3.3
  • Resolution limits thin features. Many Widmanstatten lath are only a few pixels wide, so one pixel of dilation or erosion changes the scores noticeably. §3.1
  • Small training sets. The authors frame the study as data-limited and do not provide learning curves. §3.3
  • Legacy code. The repositories use TensorFlow 1-era and Keras 2-era APIs (for example tf.placeholder, fit_generator, and scipy.misc.bytescale) and hard-coded paths. I did not run them.
  • No seeds. The training scripts set no random seed, so exact results will not reproduce.
  • Licence. Neither repository has a LICENSE file. The pseudocode above is written from reading the code; do not copy the code into a project without permission from the authors.
  • Cost. Training takes 2 to 4 hours on a commodity GPU, and manual annotation took 20 to 30 minutes per micrograph. §3.3

Lineage

  • Builds on: PixelNet (arXiv:1702.06506), for the hypercolumn MLP head §2.1.1; VGG-16 (arXiv:1409.1556) for the backbone §2.1.2; focal loss (arXiv:1708.02002) §2.1.2.
  • Compared against: Otsu thresholding on the particle task §3.2; U-Net, which the authors say underperformed PixelNet, with no numbers given §3.1.
  • Superseded by: not applicable from this card's content.

Source and license

Adapted from “High throughput quantitative metallography for complex microstructures using deep learning: A case study in ultrahigh carbon steel” by Brian L. DeCost, Bo Lei, Toby Francis, Elizabeth A. Holm (arXiv:1805.08693), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0. Code citations refer to https://github.com/bdecost/uhcs-segment at commit 7aea6f37f524 (none (no LICENSE file)). Any quoted code is shown under that license.