Sentiment Analysis Based on Deep Learning: A Comparative Study

In plain words

Reading a tweet or a product review and deciding whether the writer is positive or negative is called sentiment analysis. It is hard because people use negation, sarcasm and domain-specific wording, and computers have to turn text into numbers before any model can learn from it.

This paper takes three neural network designs that are popular for the task and runs each one on eight datasets of tweets and reviews. Every text is turned into numbers in one of two ways: learned word vectors, or TF-IDF, a score that rewards words frequent in one document but rare across the collection.

The main finding is that word vectors help the convolutional network and the LSTM on every dataset. The LSTM with word vectors was the most accurate on the largest tweet set, but it took far longer to train than the convolutional network. The authors conclude that the convolutional network gives the best balance of accuracy and training time. These scores are the authors' own.

TL;DR

The paper compares a DNN, a CNN and an LSTM (labelled "RNN") for binary sentiment polarity on eight tweet and review datasets §4.1. Each model is fed either learned word embeddings or TF-IDF vectors §4.3 §4.4. Word embeddings give higher accuracy than TF-IDF for the CNN and the RNN on all eight datasets, while for the DNN TF-IDF is higher on five of eight tab. 2. Evaluation uses 10-fold cross-validation, and many training details are not stated, so reproducing the numbers needs care.

Problem

Earlier comparison studies mostly report one reliability measure, such as accuracy or F-score, and usually on one model and a few datasets. They leave out processing time §1. The authors also note that the models in the literature have not been compared against each other on the same data §1 §4.2. Accuracy is limited by known difficulties: negation and domain dependence §1.

Method

The paper is an experimental comparison, not a new model. Its pipeline has four parts.

1. Datasets. Eight labelled datasets: Sentiment140 (1.6 million tweets), Tweets Airline (14,640), Tweets SemEval (17,750), IMDB Movie Reviews in two versions (25,000 and 50,000), Cornell Movie Reviews (10,662), and Book and Music Reviews (2,000 each) §4.1. Sentiment140 labels 0 and 4 are mapped to 0 (negative) and 1 (positive). For Tweets Airline and Tweets SemEval, neutral samples are removed §4.3. Only the text and polarity fields are used for Sentiment140 §4.1.
2. Cleaning. Remove Twitter RTs, @ mentions, # signs and links; stem or lemmatise (the paper names both); lower-case; keep letters only; remove English stop words and punctuation; collapse extra spaces; decode HTML §4.3.
3. Features. Either learned word embeddings (vocabulary 15,000, width 300, maximum length 40) §4.4 fig. 9, or TF-IDF from the scikit-learn vectorizer class §4.3.
4. Models. DNN, CNN and RNN. The CNN is described layer by layer in §4.4 and fig. 9. The RNN is described as an LSTM with 200 cells, then a 128-unit dense layer and a sigmoid output §2.1.3 fig. 4. The DNN is drawn with two hidden layers of 128 units fig. 2. Two models are trained per algorithm, one per feature type §4.3.

# Study pipeline  [§4.1] [§4.3] [§4.4] [§5]
# Input:  texts, polarity labels; Output: per-fold scores
def run_study(texts, labels, model_name, feature_kind):
    # Map labels to 1 = positive, 0 = negative; drop neutral  [§4.1]
    texts, labels = normalise_labels(texts, labels)
    # Clean, stem or lemmatise, remove stop words  [§4.3]
    cleaned = [preprocess(text) for text in texts]
    # Build the feature input for the model  [§4.3] [§4.4]
    if feature_kind == "word_embedding":
        features = to_token_ids(cleaned, vocab=15000, max_len=40)
    else:
        # TF-IDF settings (n-grams, max features) are not stated
        features = tfidf(cleaned)
    # 10-fold cross-validation  [§4.2] [§5]
    for fold in kfold(features, labels, k=10):
        # Batch size 4096; "echoes" = 5 as written in the paper  [§5]
        train_model(model_name, fold.train, batch_size=4096, echoes=5)
        # Score on the held-out fold  [§4.2]
        scores.append(evaluate(model_name, fold.test))
    # Aggregation across folds is not stated; scores are per fold
    return scores

# CNN classifier  [§4.4] [fig. 9]
# Input: token ids (B, 40); Output: probability positive (B, 1)
def cnn_forward(token_ids):
    # Learned word vectors, 15,000 words, width 300  [§4.4]
    x = embedding(token_ids)                  # (B, 40, 300)
    # First conv: 64 filters, kernel 3  [fig. 9]
    x = conv1d(x, filters=64, kernel=3)       # (B, 40, 64)
    # Second conv: 32 filters, kernel 3  [fig. 9]
    x = conv1d(x, filters=32, kernel=3)       # (B, 40, 32)
    # Max pooling of size 3 cuts 40 steps to 13  [§4.4] [fig. 9]
    x = max_pool1d(x, size=3)                 # (B, 13, 32)
    # Third conv: 16 filters, kernel 3  [fig. 9]
    x = conv1d(x, filters=16, kernel=3)       # (B, 13, 16)
    # Fourth conv: 8 filters; kernel 2 inferred from parameter count [fig. 9]
    x = conv1d(x, filters=8, kernel=2)        # (B, 13, 8)
    # Average over the sequence, not max  [§4.4]
    x = global_average_pool(x)                # (B, 8)
    # One sigmoid unit for positive vs negative  [§4.4]
    return sigmoid(dense(x, units=1))         # (B, 1)

# LSTM classifier, labelled RNN  [§2.1.3] [fig. 4]
# Input: token ids (B, T); Output: probability positive (B, 1)
def rnn_forward(token_ids):
    # Same embedding as the CNN  [§2.1.3]
    x = embedding(token_ids)                  # (B, T, 300)
    # LSTM with 200 cells  [§2.1.3] [fig. 4]
    states = lstm(x, units=200)               # (B, T, 200)
    # Which LSTM output feeds the dense layer is not stated
    # Fully connected layer with 128 cells  [§2.1.3] [fig. 4]
    h = dense(states, units=128)              # activation not stated
    # Sigmoid output  [§2.1.3]
    return sigmoid(dense(h, units=1))         # (B, 1)

# DNN classifier  [§2.1.1] [fig. 2]
# Input: feature vector (B, F); F is not stated; Output: (B, 1)
def dnn_forward(features):
    # Two hidden layers of 128 units, as drawn  [fig. 2]
    h = dense(features, units=128)            # activation not stated
    h = dense(h, units=128)                   # activation not stated
    # Output activation not stated
    return dense(h, units=1)                  # (B, 1)

Hyperparameters

Results

Author-reported. Setup: eight datasets §4.1; 10-fold cross-validation; the three models with two feature types; no classical machine learning baseline §4.2. Accuracy from tab. 2, rounded to four decimals from the paper's eight-digit values:

WE = word embedding. Recall, precision, F-score and AUC per dataset are in tab. 3 to tab. 6 and are not transcribed here. Sentiment140 results across 10–100% subsets are plotted in fig. 10 to fig. 15; the word-embedding curves are flat, and TF-IDF with RNN fluctuates. The paper's text names the RNN with word embeddings as the best combination in most cases, and names book reviews as an exception where CNN with word embeddings is highest §5. tab. 2 shows the same for Music Reviews.

Processing time on the GPU for Sentiment140 at 100% of the data tab. 7 tab. 8:

Paper vs code

No official code. The paper has no code link. Searches on GitHub by title and by the first author's name found no repository from the authors; the title matches belong to third parties and were not used.

Pitfalls

• Dataset total. The text says the five non-tweet datasets total 125,000 comments §4.1. The listed sizes (25,000 + 50,000 + 10,662 + 2,000 + 2,000) sum to 89,662. Use the listed sizes; the total is unexplained.
• Embedding initialisation conflicts. One passage says the word-vector step uses Word2Vec to train initial vectors §4.4. Another says the embedding layer starts from random weights and learns during training §4.4. Figure 9 lists the embedding parameters (4,500,300) as non-trainable fig. 9. The paper does not say which setup produced the reported results.
• Pooling conflicts. The architecture overview shows global max pooling §2.1.2 fig. 3. The experimental CNN uses global average pooling, which the authors say gives better results, without numbers §4.4.
• Inferred CNN details. Kernel 2 in the last conv, "same" padding, and pool stride 3 are inferred from parameter counts and output shapes in fig. 9. Activation functions are not stated for any layer except the sigmoid output.
• Parameter count. 4,500,300 is 300 more than 15,000 × 300 fig. 9. The paper does not explain the difference.
• Cross-validation leakage risk. Not stated whether cleaning, vocabulary, Word2Vec and TF-IDF are fitted inside each fold or on all data. Tables give one number per cell, not a mean with spread, and do not say whether it is a fold average.
• AUC looks like accuracy. For several rows AUC equals accuracy to every printed digit, for example IMDB (1) TF-IDF DNN at 0.85232 in tab. 2 and tab. 6. That suggests AUC was computed from hard 0/1 predictions, but the paper does not say so. Treat the AUC column as unverified.
• Neutral removal. Tweets Airline and Tweets SemEval lose their neutral samples, leaving two classes §4.3. The resulting sizes are not stated.
• "RNN" is an LSTM. Results labelled RNN use the LSTM described in §2.1.3 and fig. 4. Whether the last state or all states feed the dense layer is not stated.
• "Ten times" claim. The conclusion says the RNN takes about ten times as long as the CNN §6. From tab. 7, Sentiment140 with word embeddings gives 3,856 s / 423 s ≈ 9.1, and with TF-IDF 6,532 s / 2,297 s ≈ 2.8.
• "CNN outperforms" claim. The conclusion says the CNN "outperforms other models" §6. tab. 2 shows the RNN with word embeddings higher on six of eight datasets; Book and Music Reviews are the exceptions. The claim is the authors' balance judgement of accuracy against time, not a ranking by accuracy.
• "Echoes" and batch size. The paper's "echoes = 5" with batch size 4096 §5 is the only training schedule given. Whether it means epochs, and what the learning rate and optimizer are, are not stated.
• No variance. There are no seeds, repeat runs or confidence intervals. Differences of a few tenths of a point, such as the Tweets SemEval DNN pair in tab. 2, may be noise.

Lineage

• Builds on: not stated. The paper uses Word2Vec for word vectors §4.2 and scikit-learn's TF-IDF vectorizer §4.3, and cites classical DNN, CNN and LSTM papers in its background §2.1. None of these have cards in this wiki.
• Compared against: only its own three models with two feature types. It reviews 32 earlier sentiment studies §3 tab. 1 but runs no baseline from them §4.2.
• Superseded by: not stated in the paper.