Understanding the Mechanisms of Deep Transfer Learning for Medical Images

In plain words

Finding an organ in an ultrasound image is hard. The images are noisy, soft tissue looks alike, and the kidney's shape, size and angle vary from patient to patient. Learning a detector from scratch needs a lot of labelled scans, and this study had only 90.

The authors start from a neural network trained on ordinary photographs (ImageNet) and reuse it for kidney detection. They test three ways of adapting it: leave it unchanged, adjust most of it, or adjust all of it. The network's internal features then feed a classic boosted-tree classifier, which scores candidate rectangles in the image.

The best result came from combining this score with the score of an older, hand-designed texture method. The authors also compare the adapted network's filter outputs with well-known hand-made image filters, to see what the network has learned.

TL;DR

• Transfers an ImageNet CaffeNet to kidney detection in ultrasound with 90 annotated images §3.1.
• Compares no, partial (conv1 and conv2 frozen) and full fine-tuning; full fine-tuning gives the best single-network result §3.2 tab. 1.
• Averaging the likelihood maps with a Haar-feature gradient-boosting model gives the best reported result: mean Dice 0.857 and 3/45 failures tab. 1.
• All numbers come from one split of 45 validation images; no variance is reported.

Problem

Kidney detection in B-mode ultrasound is hard because the kidney varies in shape, size and orientation, and because the renal sinus, diaphragm and liver boundaries can look alike §1. Labelled data is scarce, and the authors note that feature engineering is needed for each problem §1. Prior kidney detectors used texture models and machine-learning classifiers built on kidney texture §2. The paper asks how a CNN trained on camera images performs on this task and how the adaptation depth affects it §1 §4.

Method

The detector is a sliding-window search with a binary classifier §3. A classifier labels every candidate region of interest (ROI) S as kidney (Y = 1) or not, and returns a likelihood L. The detected kidney is the positive candidate with the highest likelihood eq. 2 eq. 3. Candidate generation at test time is not described in the text.

\{Y, L\} = \text{ML Classifier}(S), \quad S^{+} = \{S \in \mathcal{S} \mid Y = 1\} eq. 2

S^{*} = \arg\max L(S^{+}) eq. 3

Training data §3.1. The authors use 90 long-axis kidney images from a GE LOGIQ E9 scanner. They divide them into two separate groups of 45, one for training and one for validation §3.1 §4. Ground truth is a rectangular ROI marked by a clinical expert. Patches are sampled by sweeping the field of view with overlapping boxes of several sizes, restricted to sizes whose length, width and proportions are plausible for an adult kidney §3.1 fig. 1. Each patch is downsampled to a common size. Its Dice overlap with the ground truth decides its class: positive at or above a threshold of 0.8, otherwise negative §3.1. The exact patch sizes and the common size are not stated.

Feature extractors §3.2. The CaffeNet architecture §3.2 fig. 2 is initialised from ImageNet weights and used as a feature extractor. Features are read after the fc7 layer, giving 4096 values per patch §3.2. Three variants are compared:

1. CaffeNet_FA (full adaptation): all weights are updated on the kidney patches §3.2.
2. CaffeNet_PA (partial adaptation): conv1 and conv2 are frozen, and the other layers are updated with the same settings §3.2.
3. CaffeNet_NA (no adaptation): the original ImageNet weights are used unchanged §3.2.

The network is a generic CNN layer stack, where each layer computes

G_j = \mathcal{S}_{\downarrow}\left(\sigma\left(\sum_k P_k * v^j_k + b^j\right)\right) eq. 1

with M(NS+1) parameters per layer §2.

Baseline features §3.3. Haar features are extracted similarly to Carneiro et al., giving about 2000 values per patch §3.3. The paper also names GLCM and HoG as texture features used elsewhere, but only Haar features are used in the experiments §3.3.

Classifier §3.4. Each feature set (Haar and the three CaffeNet variants) gets its own gradient-boosting classifier. The implementation is "inspired by" Becker et al. §3.4. Parameters are shrinkage 0.5, sampling factor 0.5, tree depth capped at 2, and 200 iterations §3.4.

Hybrid §3.5. The authors average the spatial likelihood maps of the Haar-GBM and the CaffeNet_FA-GBM, and use the averaged map in the detector §3.5 eq. 2. The paper says "averaging"; any weighting is not stated. The authors motivate this by noting that the two methods fail on different images §3.5 §4.

# Build labelled patches for training  [§3.1] [fig. 1]
# Input:  90 annotated images (45 train, 45 validation), ground-truth boxes
# Output: list of (patch, label), label 1 = kidney-like

def make_training_patches(images, gt_boxes):
    samples = []
    for image, gt_box in zip(images, gt_boxes):
        # Overlapping boxes of several sizes, kept only if plausible for an  [§3.1]
        # adult kidney (size range not stated)
        for box in sweep_overlapping_boxes(image):
            # Downsample to one common size (value not stated)  [§3.1]
            patch = resize(crop(image, box), common_size)
            # Dice overlap with the expert ground-truth box  [§3.1]
            overlap = dice(box, gt_box)
            # Threshold 0.8; boundary handling at exactly 0.8 not stated  [§3.1]
            label = positive_if(overlap, threshold=0.8)
            samples.append((patch, label))
    return samples

# CaffeNet fc7 features in three adaptation modes  [§3.2] [fig. 2]
# Modes: "FA" full, "PA" conv1 and conv2 frozen, "NA" none

def caffenet_fc7_features(patches, mode):
    # Start from ImageNet weights  [§3.2]
    net = load_caffenet(weights="imagenet")
    if mode == "PA":
        # Freeze the first two conv layers  [§3.2]
        freeze(net.conv1, net.conv2)
    if mode in ("FA", "PA"):
        # SGD, batch 100, momentum 0.5, weight decay 5e-4  [§3.2]
        # Learning rate, epochs and loss are not stated  [§3.2]
        fine_tune(net, patches, batch_size=100,
                  momentum=0.5, weight_decay=5e-4)
    # Read the fc7 layer: 4096 values per patch  [§3.2]
    return net.fc7_output(patches)  # (N, 4096)

# Gradient-boosting classifier per feature set  [§3.4]
# Input: features (N, D), labels (N,)

def train_gbm(features, labels):
    # Shrinkage 0.5, sampling 0.5, max depth 2, 200 iterations  [§3.4]
    model = gbm(shrinkage=0.5, sample_fraction=0.5,
                max_depth=2, n_iterations=200)
    model.fit(features, labels)
    return model

# Hybrid likelihood: average two spatial likelihood maps  [§3.5]
# Input: haar_map, cnn_fa_map with the same spatial shape

def hybrid_likelihood(haar_map, cnn_fa_map):
    # Plain average is implied by "averaging"; weights not stated  [§3.5]
    return (haar_map + cnn_fa_map) / 2

# Detect the kidney ROI in one test image  [§3.5] [eq. 2] [eq. 3]
# Input: candidate ROIs S, classifier returning (Y, L)
# Output: best ROI, or nothing if no candidate is positive (not stated)

def detect_kidney(candidates, classifier):
    positives = []
    for roi in candidates:
        # Keep only ROIs labelled kidney (Y = 1)  [eq. 2]
        label, likelihood = classifier(roi)
        if label == 1:
            positives.append((roi, likelihood))
    # Return the positive ROI with the highest likelihood  [eq. 3]
    best_roi, best_score = max(positives, key=lambda p: p[1])
    return best_roi

Hyperparameters

Results

Author-reported. Setup: 45 validation images from 90 annotated kidney ultrasound scans; a kidney detection counts as a failure when the Dice overlap with the ground truth is below 0.80 §4. Baseline: Haar features with GBM §4.

Qualitative results: the Haar baseline is distracted by the diaphragm, kidney and liver boundaries, which create a texture like the renal sinus, while CaffeNet_FA localises correctly fig. 3. In another case CaffeNet_FA over-segments and includes the diaphragm fig. 3. The authors read this as a sign that domain knowledge still matters with limited data §4.

Filter-response analysis §5: in a per-layer count of filters whose weights moved by over 40% in ℓ2 norm, conv1 has 0, conv2 has 5, conv3 has 125, conv4 has 22 and conv5 has 62 tab. 2. Visually, some layer-1 and layer-2 responses of CaffeNet_FA resemble phase congruency and Frangi vesselness outputs for one sample patch fig. 4 §5.

Paper vs code

No official code.

No code link appears in the paper, the TeX sources or the arXiv metadata. Three searches were run: the paper's own TeX (no URL to a repository), a GitHub repository search for "kidney ultrasound transfer learning" (repositories found belong to unrelated owners), and a GitHub search for "Ravishankar ultrasound kidney" (no repositories found). None of these is attributable to the authors, so none is recorded.

Pitfalls

• Single split, small test set. Results come from one 45/45 split. One image is 1/45, about 2.2 percentage points of failure rate. No variance, repeated splits or confidence intervals are reported tab. 1.
• Text and table disagree on margins. The text says CaffeNet_NA beats the baseline by "2%" §4; Table 1 gives 0.825 vs 0.793, about 3.2 points. The text says CaffeNet_FA gives a "4%" boost §4; Table 1 gives 0.842 vs 0.793, about 4.9 points. Use the table values.
• "20%" is undefined. The abstract says the hybrid gives "20% higher performance" §1 and the results say 3/45 failures is "20% better than either method" §4. Table 1 gives Dice 0.857 vs 0.842 for the best single method and 3/45 vs 10/45 failures. The paper does not define the percentage.
• Training recipe is incomplete. Learning rate, number of epochs, loss, input size, and patch sizes are not stated §3.1 §3.2. An implementer will have to choose these; the paper gives no code to check them against.
• Threshold boundary. The 0.8 Dice threshold is given, but whether a patch with exactly 0.8 is positive is not stated §3.1.
• Hybrid weights. The hybrid is described as averaging §3.5; equal weights are the plain reading, but weights are never given.
• Test-time candidates. How candidate ROIs are generated at inference, and what happens when no candidate is positive, is not described §3.5.
• Filter table labels. Table 2's column header says "≥ 40%" while its caption says "more than 40%" tab. 2.
• Missing comparison in the figure. The text compares CaffeNet_FA with "CaffeNet_PA L1_1" for speckle reduction §5, but Figure 4 has no PA panel: it shows only NA and FA responses fig. 4.
• Causal claims are conjecture. The explanation that ReLU avoids vanishing gradients and so causes the uneven weight changes across layers is the authors' interpretation and is not tested §5.
• Filter similarity is qualitative. The resemblance to phase congruency and Frangi filters is judged visually on one sample patch fig. 4 §5.
• Data availability. The dataset is not described as public. The patient count is not stated; "90 images" is the only size given §3.1.
• Fine-tuning head. The text says the GBM replaces the CNN's softmax at inference §3. Whether the fine-tuning phase used a softmax head is not stated.

Lineage

• Builds on: CaffeNet [Jia et al. 2014] §3.2; Haar features as in Carneiro et al. 2008 §3.3; GBM implementation inspired by Becker et al. 2013 §3.4.
• Prior work by the same group: Ravishankar and Annangi 2016, kidney texture and edge analysis, where Haar features with GBM performed best §2 §3.3.
• Related: Shie et al. 2015 (CaffeNet features for otitis media) and Chen et al. 2015 (CNN features for ultrasound standard planes) §2. The authors compare the network's responses with hand-designed phase-congruency and Frangi-vesselness filters §5 fig. 4.
• Superseded by: not known from content on this wiki.