An Approach for Thyroid Nodule Analysis Using Thermographic Images

In plain words

Doctors can look for thyroid problems with an infrared camera, which records how warm each patch of skin is. A single picture is easy to misread, so the authors take a series of pictures over a few minutes while the skin cools down. People move a little during that time, so the pictures first have to be lined up with each other. The paper describes a way to do this, including an automatic method that finds the strip of neck that matters and ignores the shoulders and the rest of the frame.

Once the neck region is found, the paper measures a few numbers from it: how warm the hot spots are, how spread out the temperatures are, and how lopsided the left and right sides are. It then compares each patient's numbers with the others' to guess whether a patient looks healthy or unhealthy.

This is a small pilot with four patients. It shows the steps are workable, but it does not show that the approach can diagnose anyone.

TL;DR

A pilot pipeline for thyroid thermography: a dynamic acquisition protocol, rigid registration of the image sequence (manual or autonomous), a Sobel-like shadow filter that locates the neck region of interest, four features (mean intensity, standard deviation, maximum temperature, asymmetry) and a nearest-neighbour comparison. It is tested on four patients only, with no accuracy or validation metric reported §4. Not a diagnostic tool as published.

Problem

Thermal images of the neck show temperature patterns that may relate to thyroid nodules. Two obstacles are named. First, the patient moves during the acquisition (breathing, head tilts), so the images must be registered before any pixel-wise comparison §3.1. Second, the number of candidate points for registration is large and automatic keypoint detectors often produce false correspondences §2.2. The paper also points out that thyroid cancer is projected to become more common and that early detection helps §2.3. The abstract adds that physicians have too much information to review in the time available.

Method

The pipeline has these stages. Each one is discussed in the sections below.

1. Acquire a dynamic sequence of thermal images while the patient's skin recovers after a thermal stress §3, item 1 fig. 6.
2. Register the images to remove patient movement, either semi-manually with user-chosen points or autonomously with the Sobel-like filter §3.1 §3.2 §3.2.1.
3. Find the neck region of interest (ROI) with a fixed-size rectangle that maximises the number of filter responses inside it §3.2.1 fig. 14 fig. 16.
4. Threshold the ROI at a fixed value to isolate hot regions §3.3 fig. 17.
5. Compute four features per patient from the hot region and the ROI §3.3 eq. 7.
6. Compare each patient with the other patients feature by feature and assign a class from the nearest patients §3.3 tab. 1 tab. 2.

Acquisition

The paper uses a dynamic protocol, modified from earlier work because it has a stopping condition for the thermal stress §3, item 1. The patient sits with the head tilted slightly back, and a camera records one image every 15 s for 5 min §3, item 1. The sequence used for the figures has 20 images fig. 6. The stress itself is described only as something that is stopped when the mean skin temperature reaches 29 °C §3, item 1. Section 2.1 describes cooling with an air stream in general terms, but §3 does not say which stress was applied in this study.

Preparation rules are given in §3 as numbered items. Patients avoid alcohol, caffeine, physical exercise, nicotine, and creams, oil or deodorant for at least 2 h before the exam §3, item 2. The room is held at 22 to 25 °C, without windows, openings, airflow directed at the patient, or light other than fluorescent bulbs §3, item 3. The patient removes earrings and necklaces, has a central temperature check, and waits 20 min in the room before the exam §3, item 4. A blue square physical marker is placed on the chin as a registration and recognition aid §3. Images were taken with a FLIR ThermaCam S45 camera at 0.5 m from the patient, with relative humidity and room temperature entered as camera parameters §3, item 1 fig. 1.

Rigid registration

Only rigid transformations are used: rotation θ and translation (t_x, t_y), written in homogeneous coordinates eq. 1. The paper argues that more general transformations such as perspective or warping cost more and are unnecessary if the movement is only a shift and rotation §3.1 §2.2. Movements are classed as full-body (lateral, up/down, or combinations) and local (head tilt or chin displacement) §3.1 fig. 9 fig. 10. Only full-body movements are said to be corrected by rigid transformation; local ones would need elastic transformations §3.1.

Semi-manual registration uses user-selected common points in both images. The authors state that such registrations are usually very accurate but time consuming §3.2. Once a few landmarks are chosen on a subsequent image, the sequence can be aligned by matching the points, and the whole image is registered at once §3.2 fig. 11 fig. 12. Errors remain in the result §3.2.

Autonomous registration and ROI

The autonomous path applies a shadow variant of the Sobel operator, which the authors say is a modification of a previously published filter with kernels of variable size §3.2.1. The shadow filter marks a pixel when its two neighbours along one axis, each at distance d, are both more than a threshold t above the pixel's own value eq. 2 eq. 3 eq. 6. The paper also defines the same test along the other axis eq. 3 and a light variant that looks for valleys eq. 4 eq. 5. The text states that only the shadow form is used, with t = −40 §3.2.1.

# Shadow filter as printed in eq. 6  [§3.2.1] [eq. 6]
# Input:  image (H, W), grey levels 0-255; d; t = -40
# Output: mask (H, W) with values 0 or 1

def shadow_mask(image, d, t):
    # Start with nothing marked  [§3.2.1]
    mask = zeros_like(image)
    # Test each pixel that has a neighbour d rows away on both sides
    # (edge handling is not stated)  [eq. 6]
    for i in range(d, H - d):
        for j in range(W):
            # Compare the neighbour above and below with the pixel  [eq. 6]
            above = image[i - d, j] - image[i, j] > t
            below = image[i + d, j] - image[i, j] > t
            if above and below:
                mask[i, j] = 1
    return mask

The mask is then searched for the ROI. A fixed rectangle of 330 wide by 310 high is placed at a few candidate positions, and the position with the most marked pixels is kept §3.2.1 fig. 14. The authors then shrink the rectangle to 110 by 110 to find the two central residual responses, which are caused by a noise source that pulls the rectangle upwards §3.2.1 fig. 15. They erase these residuals, then repeat the search from bottom to top along the vertical axis §3.2.1 fig. 16.

# ROI search with a fixed rectangle  [§3.2.1] [fig. 14]
# Input:  mask (H, W); rectangle height and width (330 x 310 in the paper)
# Output: top-left corner of the best rectangle

def best_rectangle(mask, rect_h, rect_w, candidates):
    # Score each candidate by the number of marked pixels inside it
    best_score = -1
    best_corner = None
    for corner in candidates:
        # Candidate set and stride are not stated; "a few" positions
        score = count_ones(mask, corner, rect_h, rect_w)
        if score > best_score:
            best_score = score
            best_corner = corner
    return best_corner

The ROI can be taken before or after registration §3.2.1. Before registration, only the neck is compared, which the authors expect to be more accurate but slower to match §3.2.1. After registration, the shoulders and other structures are still present, which makes landmarks easier to find §3.2.1. The authors say a pairwise comparison of ROIs takes only a few milliseconds §3.2.1.

Features and classification

The image is thresholded at 209: pixels at or above this value become white and the rest become black §3.3 fig. 17. Four features are computed from the ROI §3.3:

1. mean intensity of the hot region, normalised §3.3 tab. 1 tab. 2;
2. standard deviation of intensities in the hot region §3.3;
3. maximum intensity (called temperature in the text) in the hot region, normalised §3.3 tab. 1 tab. 2;
4. asymmetry of the ROI about its vertical axis §3.3 eq. 7.

Asymmetry is the average, over all pixels of the ROI, of the absolute intensity difference between each pixel and its best match in a 3 by 3 neighbourhood of the mirror position, divided by 255 eq. 7 §3.3. The two halves are matched in the paper's index scheme, with the right half searched in the left half §3.3. Eq. 7 as typeset has an index for the second half that needs checking; see Pitfalls.

# Asymmetry of an ROI, left-half term  [§3.3] [eq. 7]
# Input:  roi (m, n), grey levels 0-255; 1-based indices as in eq. 7
# Output: scalar asymmetry

def asymmetry(roi, m, n):
    # Sum the best mirror match over the left half  [eq. 7]
    total = 0.0
    for i in range(1, m + 1):
        for j in range(1, n // 2 + 1):
            # Best match in the 3 x 3 window around the mirror column  [eq. 7]
            best = min(abs(roi[i, j] - roi[i + p, n - j + q])
                       for p in (-1, 0, 1) for q in (-1, 0, 1))
            total += best / 255.0
    # The right half is summed the same way, with the printed index in
    # question (see Pitfalls)  [eq. 7]
    # Normalise by the ROI area  [eq. 7]
    return total / (m * n)

Classification is 1-nearest-neighbour on each feature separately, with Euclidean distance (for a single feature this is an absolute difference) §3.3. For each feature, the nearest patient is found; the class that appears most often among the four features is assigned §3.3 tab. 1 tab. 2. The paper does not give a tie-breaking rule.

# Per-feature nearest patient and vote  [§3.3] [tab. 1] [tab. 2]
# Input:  features of the patient and of the other patients (4 values each)
# Output: predicted class

def classify(patient, others, labels):
    votes = []
    for f in range(4):
        # Absolute difference on feature f (the paper calls it Euclidean)
        dists = [abs(patient[f] - o[f]) for o in others]
        nearest = others[argmin(dists)]
        votes.append(labels[nearest])
    # Majority vote; ties are not resolved in the paper  [§3.3]
    return most_common(votes)

Hyperparameters

Results

Author-reported. The setup is a single university hospital, four patients, and an unnamed number of acquisitions per patient. No baseline, accuracy, sensitivity or specificity is reported §4. The paper calls the study a "small scale preliminary study" §4.

The authors conclude that the features "may be sufficient" to predict sick patients, but that more data is needed §4. They also note a lack of thermographic thyroid images with abnormalities in their repository §4.

Paper vs code

No official code.

Searches run: the paper and abstract have no code link; the paper cites github.com/Oyatsumi/Uacari as [35], the source of its Sobel filter. That repo is a general Java image library. Its owner shows only a handle, its README does not mention thyroid, thermography or the filter, and it has no LICENSE file, so it is not recorded as code. A GitHub repository search for "thyroid thermography" returned two unrelated repositories (one without a license, one a WordPress theme). No project page was linked.

Pitfalls

• Four patients. The sample has four patients: two with abnormality and two healthy §3. There is no accuracy, no cross-validation and no held-out set, and the authors say the result may not apply generally §4. Do not use the classification step as a validated method.
• Labels. The "sick" and "healthy" labels are the authors' own. The nodule location is said to be confirmed by a specialist "on the available images" only §3.3. The paper says that the project is still to be submitted to the ethics committee after "future adjustments" [abstract]. Treat it as a pilot without clinical approval.
• Sign of the shadow threshold. Eq. 6 uses t = −40 §3.2.1. With this sign, the test passes unless a neighbour is more than 40 levels darker than the pixel. The text describes "shadow valleys", which suggests the intended sign may be the opposite. The paper does not discuss this. Choose the sign by testing on the example images fig. 13.
• Which axis. The text says the term s1 was dropped and that the filter was applied on the x-axis §3.2.1. Eq. 6 contains the terms for neighbours at i − d and i + d, which is the s1 form. Clarify which axis and which terms you implement before testing.
• Distance d. Not stated. The paper says d would be close to 1 in classical Sobel operators §3.2.1. Pick it and record it.
• ROI search. The stride, the candidate set, boundary handling and tie-breaking are not stated §3.2.1. The residual-erasing step is described as "we could just erase them", so the exact method is not specified §3.2.1 fig. 15.
• Eq. 7 second term. As typeset in the PDF, the second half of the sum indexes the mirror column as n − (n − j) + q, which reduces to j + q. That would match each pixel against its own neighbourhood, and the second half would then be trivially near zero. The text says the right half is matched against the left, so the mirror index is probably intended to be n − j + q. Confirm this before implementing. The paper gives no code to check it against.
• Eq. 7 boundaries. Indices i + p and mirror + q go outside the ROI at its edges. Boundary handling is not stated.
• Rotation sign in eq. 1. As typeset, the upper-right entry is +sin θ, the same as the lower-left. A rotation matrix normally has −sin θ in that position. This may be a typesetting slip. Check before implementing.
• Normalisation. The features "normalised mean intensity" and "normalised maximum temperature" are not defined beyond the 255 division in eq. 7 tab. 1 tab. 2.
• Units. The threshold of 209 is applied to "pixel or temperature value" §3.3. Eq. 7 divides by 255, which implies 8-bit grey levels. The unit is not settled in the text.
• Classifier details. The paper says "Euclidean distance" but the procedure compares one feature at a time, and the vote has no tie rule. Patient O2 is a two-to-two split in the text §3.3.
• Registration limits. Rigid registration only. The authors say local deformations such as head tilt would need elastic methods, which are not implemented §3.1. Errors remain after semi-manual registration fig. 12.
• Figure 17h. A small hot spot appears for O4, which is labelled healthy. The text does not discuss it fig. 17 §3.3.
• Cost. The ROI step takes a few milliseconds per pair in the authors' words §3.2.1. The acquisition needs about 5 min per patient plus the preparation time of 20 min and 10 min §3.

Lineage

• Builds on: the Sobel filter from earlier work by the same group, applied to finger-knuckle images §3.2.1 (Rodrigues et al., 2016, ref. 34). The filter's code is referenced in the paper through the Uacari repository [35] as discussed under Paper vs code.
• Builds on the acquisition protocol of earlier dynamic thermography work §3, item 1 §2.1 (cited as refs. 1 and 3).
• Compared against: classical keypoint detectors SIFT, ASIFT, SURF and Harris are named as alternatives for registration, but not compared §2.2.
• Superseded by: not known.