FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics

In plain words

Connectomics tries to map the wiring of a brain. Scientists slice tissue into thousands of very thin sections and photograph each one with an electron microscope. Someone then has to mark which pixels belong to which nerve cell or cell membrane, and doing that by hand does not scale. This paper trains a neural network to draw those outlines automatically.

The network is a U-shaped design: it shrinks the picture step by step to capture context, then grows it back to full size. Each step is built from small residual blocks, which pass the input along a shortcut so very deep stacks remain trainable. The "fusion" part is that the decoder adds, rather than concatenates, the matching encoder features back in. The authors also show each training image in eight orientations and warp it slightly, so the network sees more variety from a small labelled set. On two electron-microscopy datasets it outlines membranes and cell bodies more accurately than the original U-Net.

TL;DR

• Segmentation network for electron-microscopy images of neural tissue; it predicts a per-pixel map at the same 640x640 size as its input.
• Differences from U-Net: residual blocks inside each level and summation (not concatenation) on the long skip connections §3.1 tab. 1.
• Training uses 8-fold orientation enrichment, elastic deformation and Gaussian noise; deployment averages predictions over the 8 orientations §3.2.
• Author-reported: beats U-Net on both test sets; V_rand 0.97804 vs 0.97276 on Drosophila tab. 2; Dice 0.9461 vs 0.9085 on zebrafish tab. 3.
• No official code found; training hyperparameters are largely not stated.

Problem

Manual segmentation of serial-section electron microscopy does not scale to the terabyte-to-petabyte volumes that connectomics produces. The authors argue that fully convolutional networks with encoder-decoder layouts can do the job, but that the existing U-Net is limited in depth because of vanishing gradients §1 §2. They also note that per-pixel CNN classification, a common alternative, is expensive at that data scale §1.

Method

The network is an encoder-decoder with four down levels, a bridge and four up levels. It takes a 640x640 single-channel tile and outputs a 640x640 map §3.1 tab. 1 fig. 2.

Building blocks §3.1 fig. 2:

• Regular block: a convolution followed by ReLU and batch normalisation. Figure 2 notes that ReLU and batch norm are omitted from the drawing.
• Residual block: three convolutional blocks with one skip connection inside the block. A convolution sits before and after it so that channel counts match and the whole network stays symmetric §3.1.
• Downsampling: 2x max pooling after each encoder level tab. 1 fig. 2.
• Upsampling: a learnable deconvolution §3.1 fig. 2.

Steps:

1. Encoder: at each of the four levels, apply conv, residual block, conv, then store that map for the skip connection and max-pool it by 2. Feature counts are 64, 128, 256 and 512, so resolution goes 640, 320, 160, 80, 40 tab. 1.
2. Bridge: at 40x40, apply conv, residual block, conv with 1024 maps tab. 1.
3. Decoder: at each level, deconvolve by 2x, add the stored encoder map of the same level pixel-wise, then apply conv, residual block, conv §3.1 tab. 1. This addition replaces the concatenation used in U-Net §3.1 fig. 3.
4. Output: one convolution to 1 channel at 640x640 tab. 1.
5. Training: the network sees pairs of EM tile and label tile and is trained with mean-square error §3.1.

# Forward pass of FusionNet  [fig. 2] [tab. 1] [§3.1]
# Input:  tile (640, 640, 1), mirror-padded EM image  [§3.2]
# Output: prediction (640, 640, 1)

def fusion_net(tile):
    # Encoder: keep each level's map for the skip, then halve  [§3.1]
    filters = [64, 128, 256, 512]
    skips = []
    x = tile
    for f in filters:
        x = conv_res_conv(x, f)              # (H, W, f)
        skips.append(x)
        x = max_pool(x, 2)                   # (H/2, W/2, f)
    # Bridge at 40x40 with 1024 maps  [tab. 1]
    x = conv_res_conv(x, 1024)               # (40, 40, 1024)
    # Decoder: deconvolve, add the same-level skip  [§3.1] [tab. 1]
    for f, skip in zip(reversed(filters), reversed(skips)):
        x = deconvolve(x, f)                 # 2x upsample to f maps
        x = x + skip                         # pixel-wise addition
        x = conv_res_conv(x, f)
    # Single-channel output layer  [tab. 1]
    return conv(x, 1)                        # (640, 640, 1)

# Training step  [§3.1] [§3.2]
# Input:  EM tile and label tile, both (640, 640, 1)
# Output: scalar mean-square-error loss

def training_step(tile, label):
    # Orientation, elastic warp and Gaussian noise  [§3.2]
    tile, label = augment(tile, label)
    prediction = fusion_net(tile)
    # Mean-square error between prediction and label  [§3.1]
    return mean((prediction - label) ** 2)

# Deployment on one tile  [§3.2]
# Input:  tile (640, 640, 1) after mirror padding
# Output: averaged map cropped to the original size

def deploy(tile):
    # Run the 8 orientations used in training  [§3.2]
    outputs = []
    for k in range(8):
        # Map the prediction back to the original orientation (implied)
        outputs.append(invert(fusion_net(orient(tile, k)), k))
    # Average the 8 results  [§3.2]
    return crop(mean(outputs), 512)

Data augmentation §3.2:

• Orientation enrichment: rotations of 90, 180 and 270 degrees plus reflections give 8 orientations of every image and label pair §3.2 fig. 4.
• Elastic deformation: a random sparse vector field on a 12x12 grid is interpolated to image size and used to warp image and label together. Its amplitude is zero at the boundaries. A new field is drawn every epoch §3.2 fig. 5.
• Noise and padding: Gaussian noise with mean 0 and variance 0.1 is added to the raw image §3.2. Mirror-reflected borders of 64 px are added around the original 512-pixel image, giving 640x640 inputs; convolutions use "same" padding and the prediction is cropped back §3.2.
• Shuffle and validation: the enriched data is shuffled and a three-fold cross-validation is run §3.2.

Experimental setup §3.3: Keras, with Theano or TensorFlow as backend, on an Intel i7 CPU with 32 GB RAM and an NVIDIA GTX 1080 GPU.

Hyperparameters

Results

Author-reported. Setups: Drosophila ventral nerve cord (ISBI 2012 EM segmentation challenge data) and larval zebrafish nuclei §4.1 §4.2. Drosophila training uses a 2x2x1.5 um^3 volume from 30 sections, with test labels held out by the challenge §4.1. Zebrafish uses two manually segmented training sub-volumes (512x512x512 and 512x512x256) and a 512^3 test sub-volume §4.2.

Drosophila, test set tab. 2. Metrics are foreground-restricted V_rand and V_info after border thinning. Table values:

Zebrafish, one test volume tab. 3:

Qualitative results: fewer false positives than U-Net on the zebrafish test slice, marked by arrows fig. 7 fig. 8. The authors also apply the trained network to the full zebrafish volume, count about 180,000 cell bodies, and report volume, surface area and sphericity clusters §4.2 fig. 9 fig. 10. These are descriptive; no ground truth is given for that analysis.

Paper vs code

No official code. Searched: the paper and the arXiv metadata in raw/arxiv/1612.05360.json contain no code link; no project page is linked. A GitHub repository search for "fusionnet connectomics" found two third-party repositories (GunhoChoi/FusionNet-Pytorch, mental689/connectomics), which are not used. A search restricted to the corresponding author's handle (wkjeong, from the paper's e-mail address) returned nothing.

Pitfalls

• Level count is inconsistent. Table 2 calls the Nov 2016 model "five-level", while the text calls the same model "four downsampling and upsampling blocks" and the May 2016 one "three downsampling" §4.1 tab. 2. Table 1 shows four down blocks plus a bridge, which matches the text. Follow Table 1.
• Noise is specified as a variance. "variance sigma = 0.1" §3.2 means a standard deviation of about 0.32. The paper does not say whether noise is added before or after normalisation, nor the pixel value range.
• Output activation and label encoding are not stated. MSE is named §3.1, but the output activation and whether labels are 0/1 are not. The membrane/cell-body label convention is also not stated.
• Batch norm and ReLU are not in the body text. They appear only in the caption of Figure 2 fig. 2.
• Deployment averaging is only half specified. The paper says the 8 orientations are averaged §3.2. It does not say how each prediction is mapped back to the original orientation, though an inverse transform is implied. The paper also says the output "can be a combination of 8 probability values" §1, which is not the same thing.
• Elastic deformation is under-specified. The grid is 12x12 §3.2, but the interpolation method, the amplitude scale and the boundary taper are not stated beyond "amplitudes at the boundaries have vanished" fig. 5.
• Training is not reproducible from the text. The optimiser, learning rate, batch size, epoch count and weight initialisation are all absent §3.3. Any reimplementation must choose them itself and report that.
• Memory cost. Processing 640x640 tiles with a 1024-channel bridge and 8x enrichment is heavy. The paper names the GPU but gives no memory or runtime figures §3.3.
• Post-processing changes the rank. The top-three claim assumes a 2D median filter with radius 2 px; without it the ranking is not claimed §4.1.
• Scores are on one volume. The zebrafish comparison uses one 512^3 test volume §4.2 tab. 3 with no variance. The Drosophila scores come from the challenge's private test set and cannot be re-checked here.
• "Fully residual" is a naming choice. The decoder uses addition on the long skips, which the authors contrast with concatenation in U-Net §3.1 fig. 3. Whether this helps is shown only by the overall comparison, with no ablation of the addition alone.

Lineage

• Builds on: U-Net (Ronneberger et al., MICCAI 2015, ref. [25]) for the encoder-decoder layout and skip connections; residual blocks from ref. [14]. The changes are the residual blocks inside each level and summation skips instead of concatenation §2 §3.1.
• Compared against: U-Net [25]; CUMedVision (deep contextual networks) [8]; LSTM multi-dimensional segmentation [30]; IAL IC [22]; Masters [34]; and the paper's own May 2016 submission tab. 2.
• Superseded by: not known from content already in this wiki.