Medical Text Classification using Convolutional Neural Networks

In plain words

Health notes are full of sentences that say what kind of problem a patient has or how they are doing. Sorting those sentences into topics by hand takes a lot of time for doctors and nurses.

This paper trains a neural network to read a single sentence and decide which of 26 medical topics it belongs to. Each word is first turned into a list of numbers that places it near words used in similar contexts. The network then scans the sentence with small sliding windows, looking for patterns of words, and finally picks the most likely topic.

The authors compare this approach with simpler ways of turning a sentence into numbers and with simpler classifiers. Their network scores clearly higher in their tests. The training data comes from encyclopedia-style medical articles rather than real patient notes, which the authors list as future work.

TL;DR

• A CNN classifies one sentence into one of 26 medical categories, using 100-dimensional word2vec vectors for up to 50 tokens §3.
• Architecture: two groups of two conv layers (256 filters, size 5, ReLU), each group followed by max pooling; dropout 0.5; dense 128 with ReLU; dropout 0.5; dense 26 with softmax §3 fig. 1.
• Author-reported accuracy is 0.68, against 0.28 to 0.51 for four baselines fig. 2.
• The training and validation data (Merck Manual articles) is not released, and several training details are not stated.
• No official code is linked.

Problem

Clinical notes hold information that is needed for care management, but reading them takes time and health information is easy to miss §1. The authors argue that dictionary-based representations of meaning limit how well social factors of health can be captured, and that a learned, context-aware representation is needed §1.

The paper assumes that sentences can be labelled with a broad health category, and that a model trained on labelled medical text transfers to new sentences of similar kind §3. The paper does not test transfer to patient notes; the authors name this as future work §5.

Method

The pipeline has three stages §3.

1. Word vectors. A word2vec model is trained on a corpus of 15k clinical research papers from PubMed §3. Its hidden layer has 100 units, so each word becomes a 100-dimensional vector §3.
2. Sentence matrix. Following Kim [8], each sentence becomes a matrix with one row per word vector. The fixed input size is 50 rows. Shorter sentences are padded with a stop word; longer sentences keep only their first 50 words §3. Stop words are kept and no stemming is used §4.
3. CNN classifier. Two groups of convolutional layers each contain two layers with 256 filters of width 5. Each group ends with max pooling. A dropout of 0.5 follows the second pooling. A dense layer with 128 units follows, then a second dropout of 0.5, then a dense layer with 26 units and softmax §3 fig. 1.

The authors ran a grid search over the number of filters and the filter sizes, and tried 2, 4 and 6 convolutional layers. The configuration above performed best §3. The grid ranges are not stated.

The training data was built from Merck Manual articles, each filed under one parent category. The paper chose 4000 sentences at random from each of the 26 categories, so every sentence takes the label of its article's category §3. The validation set has 1000 sentences per category §3. The paper gives no separate test split §3 §4.

# Sentence classifier  [§3] [fig. 1]
# Input:  sentence as a list of words
# Output: probabilities over the 26 categories (sums to 1)

def classify(sentence, word_vectors, weights, training):
    # Pad with a stop word or keep the first 50 words  [§3]
    tokens = fix_length(sentence, length=50, pad=STOP_WORD)
    # One 100-dim word2vec vector per token  [§3]
    features = stack([word_vectors[t] for t in tokens])  # (50, 100)
    # Group 1: two conv layers, then max pooling  [§3] [fig. 1]
    hidden = conv_relu(features, filters=256, width=5)
    hidden = conv_relu(hidden, filters=256, width=5)
    hidden = max_pool(hidden)
    # Group 2: same pattern  [§3] [fig. 1]
    hidden = conv_relu(hidden, filters=256, width=5)
    hidden = conv_relu(hidden, filters=256, width=5)
    hidden = max_pool(hidden)
    # Dropout after the second pooling  [§3]
    hidden = dropout(hidden, rate=0.5, training=training)
    # Dense layer with 128 units and ReLU  [§3] [fig. 1]
    hidden = relu(dense(flatten(hidden), units=128))
    # Second dropout  [§3]
    hidden = dropout(hidden, rate=0.5, training=training)
    # Dense layer to 26 classes, then softmax  [§3] [fig. 1]
    logits = dense(hidden, units=26)
    return softmax(logits)

Hyperparameters

Results

Author-reported, on the paper's 26-category sentence task. The metric is accuracy, meaning the percentage of sentences classified correctly §4. The figure gives values only as bar labels, to two decimals. The text does not say which split these numbers come from fig. 2.

All five methods, as labelled in the figure fig. 2:

The paper states that its approach is better than the others "by at least 15%" in accuracy §1 and "about 15%" in the abstract. Figure 2 gives the gap to the best baseline as 0.17 absolute points.

Paper vs code

No official code. The paper and abstract contain no code link. A GitHub repository search for "medical text classification cnn" and for "clinical sentence classification cnn" within the IBM organisation returned no repository owned by the authors or their lab. User searches for the author surnames returned accounts whose ownership I could not verify, so none were used.

Pitfalls

• Training text is not clinical notes. The sentence labels come from Merck Manual encyclopedia articles, while word2vec was trained on PubMed research papers §3. The abstract says "clinical text", and the authors list clinical notes as future work §5. Expect the model to transfer poorly to patient notes; the paper gives no test of this.
• Labels are article-level. Every sentence inherits its article's category, so some sentences will not fit their label. The paper does not measure label noise §3.
• Dataset not released. The sentence set is built from Merck Manual articles and is not published. The validation and evaluation sets cannot be reproduced exactly from the paper §3.
• Input layout is ambiguous. The text says the input is 100 × 50, while figure 1 labels it "100 x 50". Each row is a word vector §3 fig. 1. Implement 50 rows of 100 values, which matches the text description.
• Stop-word padding. The padding word is not named §3, and the paper does not say whether padding participates in training or pooling.
• Layers not fully specified. Pooling window and stride, the reshape before the 128-unit layer, the filter width along the embedding dimension, and the weight initialisation are not stated §3 fig. 1.
• Figure 1 is hard to match to the text. At print resolution, the number of convolutional blocks drawn in figure 1 is hard to count against the text. The text is followed here §3.
• Training recipe absent. Optimizer, learning rate, batch size, epochs, and loss are not stated for the CNN §3. §4 states 60 epochs "for all the experiments", but does not say how that applies to the CNN.
• "About 15%" is ambiguous. The gap is 0.17 absolute accuracy to the best baseline, or about 33% relative to it, based on the figure values fig. 2 §1.
• Single run, no variance. No seeds, repeated runs, or confidence intervals are reported §4.
• Baseline details are thin. Doc2vec and bag-of-words are described briefly §4. The soft-assignment weighting uses a rank-based weight 1/R, and the exact formula is only given in words §4.

Lineage

• Builds on: Kim's sentence-level CNN for matrix input, arXiv:1408.5882 §3 [ref. 8]; word2vec, Mikolov et al. §2 [ref. 7].
• Compared against: Doc2vec, Le & Mikolov §2 §4 [ref. 4]; mean word embeddings; bag-of-words with k-means §4.
• Superseded by: not stated.