A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks

In plain words

Sarcasm is hard for text classifiers. A tweet like "I love the pain of breakups" uses positive words to say something negative, so counting words or spotting a few phrases often misses it. The paper's approach is to first train small neural networks on other labelled text to recognise sentiment (positive, negative or neutral), emotion (six basic emotions) and a personality trait. Those networks are then used as readers: each tweet is turned into a list of numbers that summarise these clues, and a separate classifier uses those numbers, together with what a sarcasm-trained network learns, to decide whether the tweet is sarcastic.

The authors find that the extra clues improve on the baseline on all three datasets, and that the combined model beats the earlier word-based methods they compare against. They also report that one direction of cross-dataset testing performs poorly.

TL;DR

Sarcasm detection is framed as text classification over tweets. Three convolutional networks are pre-trained on sentiment, emotion and personality corpora and used as feature extractors (the paper does not state whether their weights are updated during sarcasm training; see Pitfalls). Their output features are concatenated with a baseline sarcasm CNN's features and classified by an SVM. The authors report F1 above the re-implemented baselines on all three datasets, but the model transfers poorly across datasets.

Problem

Sarcasm can flip the polarity of an apparently positive sentence, which hurts sentiment analysis §1. Most earlier methods treat sarcasm detection as text categorisation over unigrams, n-grams, emoticons, or hand-made lexical and incongruity features §1, §2. The authors argue that these features need a deeper understanding of the sentence and do not capture the contradiction between a stated feeling and the situation it refers to §1, §3. They also note that sarcasm depends on the topic and time of the tweet, so n-gram models trained on older tweets see many unseen words on newer ones §5.6.

Method

The approach has three stages: pre-trained feature extractors, a baseline sarcasm CNN, and a merge of their outputs. All CNNs share one architecture, described in §4.1 and configured per model in tab. 1.

1. Sentence model. Each sentence is a sequence of word vectors. A convolution with kernel vector w and window of k words produces a feature for each position eq. 1. Max pooling over each feature map keeps one value per kernel §4.1. Two convolution layers are stacked, followed by a fully connected layer and a softmax tab. 1. The network uses ReLU §4.1.

2. Word vectors. Words are represented by 300-dimensional word2vec vectors (continuous bag-of-words, Google News) §4.1. Words missing from word2vec get random vectors. The vectors are trained along with the network ("non-static"). The authors give two reasons: informal tweet vocabulary is often missing from word2vec, and sarcasm shifts polarity in ways word2vec was not trained to capture §4.1.

3. Sentiment model §4.2. A CNN of the type in step 1 is trained on the SemEval 2014 Twitter sentiment data (9,497 tweets) with three output classes (positive, negative, neutral). Its fully connected layer (100 units) is then used as a feature vector.

4. Emotion model §4.3. The same CNN type is trained on sentence-level blog data labelled with six emotions (anger, disgust, surprise, sadness, joy, fear; 5,205 sentences) §4.3. Its fully connected layer (150 units) is the feature vector.

5. Personality model §4.4. Five personality traits (OCEAN) are modelled. For each trait a separate CNN is trained as a binary classifier on 2,400 essays §4.4. Each produces a 150-unit fully connected output; the five are concatenated into 750 features.

6. Baseline sarcasm CNN §4.5. The same CNN type is trained directly on sarcastic versus non-sarcastic labels. Its 100-unit fully connected layer gives the baseline features.

7. Merging §5.2. Two settings are described:

   • (a) Standard merge. Features from all models are concatenated and an SVM is trained on the result. The paper calls this CNN-SVM §4.1. With all four feature groups the vector has 1,100 dimensions (100 baseline + 100 sentiment + 150 emotion + 750 personality) §5.3.
   • (b) Append to the baseline CNN. The pre-trained features are fed as extra static inputs to the hidden layer of the baseline CNN, just before the output softmax §5.2.

8. Evaluation. Experiments use 5-fold cross-validation tab. 2 caption, and for CNN-SVM the fully connected output is extracted in each fold and passed to the SVM §4.1.

Preprocessing §5.1:

• Remove "user", "URL" and "hashtag" references with regular expressions.
• Tokenise with the NLTK Twitter tokeniser, keeping emoticons as vocabulary items.

The core loop, for one tweet set, in Python-like pseudocode. Source: §4 §5.2 tab. 1:

# Pre-trained feature extractors and sarcasm classifier  [§4] [§5.2]
# Input:  tweets (list of token lists); one labelled corpus per extractor
# Output: sarcasm predictions for each tweet

def train_cnn(sentences, labels, config):
    # Word vectors start from word2vec; unknown words are random  [§4.1]
    embeddings = init_embeddings(vocabulary, word2vec_300d)
    # Embeddings are updated during training (non-static)  [§4.1]
    model = build_cnn(embeddings, config)  # config from [tab. 1]
    model.fit(sentences, labels)
    return model

def fc_features(model, sentences):
    # The fully connected layer output is the feature vector  [§4.1]
    return model.fc_output(sentences)  # (N, 100) or (N, 150)

def merged_features(tweets):
    # Pre-trained extractors applied to tweets; not retrained on them
    # [§4.2] [§4.3] [§4.4]
    sentiment = fc_features(sentiment_cnn, tweets)           # (N, 100)
    emotion = fc_features(emotion_cnn, tweets)               # (N, 150)
    personality = concat([fc_features(p, tweets)
                          for p in personality_cnns])        # (N, 750)
    # Baseline CNN, trained on the sarcasm labels of the current fold  [§4.5]
    baseline = fc_features(baseline_cnn, tweets)             # (N, 100)
    # Standard merge: concatenate, then SVM  [§5.2] [§5.3]
    return concat([baseline, sentiment, emotion, personality])  # (N, 1100)

Hyperparameters

Training settings for the four CNNs tab. 1. Legend from the table: S = sentiment, E = emotion, P = personality (one network per trait), B = baseline. "Kernel size" lists the window sizes used in each convolution layer. The table does not say whether "feature map" counts are per kernel size or in total tab. 1.

Other values:

Results

Author-reported. Setup: three Twitter sarcasm datasets §5.1; the proposed model vs. the authors' re-implementations of Ptacek et al. (2014) and Joshi et al. (2015) §5.2, with 5-fold cross-validation tab. 2.

Full results tab. 2 (F1, in percent). The "B" column is the baseline CNN features; S, E, P are the sentiment, emotion and personality features:

The paper's comparison with prior methods tab. 3 (F1, percent; prior methods are the authors' re-implementations §5.2):

"D3 → D1" means trained on Dataset 3 and tested on Dataset 1 tab. 3.

Other numbers, stated in the text only:

• Training on Dataset 1 and testing on Dataset 3 drops F1 to 33.05% §5.6.
• Random instead of pre-trained word vectors, with all features, give 86.23% on Dataset 1 §5.3.
• With pre-trained features plus baseline features, the D3 → D1 test improves by 4.19 points over the baseline features; without the baseline features the pre-trained features alone give 64.25% §5.6.
• Adding sentiment features to baseline features on Dataset 1 changes F1 by 0.07 points (97.67 vs 97.60) tab. 2; the all-feature gain over baseline is 0.11 points on Dataset 1 §5.3.
• Spearman correlations between sentiment, happy, fear, openness and conscientiousness features are in tab. 4 (0.59 and 0.83 between sentiment and openness and conscientiousness respectively).

Setup for the baseline comparison: the paper contrasts a feature vector of about 1,100 dimensions with more than 500,000 dimensions used by Ptacek et al. §5.3.

Paper vs code

No official code.

Searched: the paper text and the arXiv metadata (no code link, no comment field); no project page is linked. GitHub search for "sarcasm cnn" returned only third-party repositories (none by the authors); a search for "sarcasm" in the authors' lab organisation returned one unrelated project. Neither result is used.

Pitfalls

• Text and tables disagree in three places. (a) §5.4 says the sentiment model is best among pre-trained models on Dataset 2 at 87.00%; tab. 2 gives 86.50% for Dataset 2, and 87.00% is the Dataset 1 value. (b) §5.5 says the sentiment model is the best pre-trained model on Dataset 3; tab. 2 shows personality at 74.41% above sentiment at 73.50%. (c) §5.5 gives the Ptacek et al. Dataset 3 F1 as 62.37%; tab. 3 gives 63.37%. Use the tables when implementing and compare against them, not the prose.
• Merge path for CNN-only rows is not stated. tab. 2 lists CNN results for feature combinations, but the paper does not say which merge produced them. Merge (b) is the only one described for a CNN-only classifier with pre-trained features §5.2. The headline numbers in tab. 3 are the CNN-SVM rows with all features, i.e. merge (a) §5.4 §4.1.
• Pre-trained extractors: frozen or fine-tuned is not stated. The paper says the pre-trained CNN is used "to extract features" §4.2 but does not say whether its weights are updated during sarcasm training.
• Personality negatives undefined. Each trait network is binary, with 2,400 essays labelled by one of five traits §4.4. How non-trait examples were constructed is not described.
• Macro-F1 versus F1. The preamble names macro-F1 §5, but the results are reported as "F1-score" and the positive class is not named. Imbalanced Dataset 2 makes these differ more.
• Random seeds and fold splits. Dataset 3 sampling and CV fold assignment are not reproducible from the text §5.1, tab. 2.
• Re-implemented baselines. The comparison numbers are the authors' re-implementations, not values copied from prior papers §5.2. SenticNet was substituted for the lexicon of Joshi et al. because the original lexicon was not stated §5.2.
• Generalisation is weak in one direction. Dataset 1 → Dataset 3 drops to 33.05% §5.6, but Dataset 3 → Dataset 1 reaches 76.78% tab. 3. The authors attribute the gap to Dataset 1 being mostly linearly separable and Dataset 3 not §5.6, fig. 3. The linear vs. RBF kernel claim is in the text only, without a table.
• Dataset 3 role. It is called the "test dataset" §5.1, but it is the training set in the D3 → D1 experiment §5.6. It comes from a third-party website, and this card does not redistribute it.
• Pooling values in tab. 1 are ambiguous. The value 2 is listed for each pooling layer. Whether it is window size or stride is not stated.
• Ablation not shown. The table footnote says changing kernel sizes or adding/removing layers does not improve results tab. 1 fn. 1 in §4.1, but no ablation numbers are given.
• Costs. The network training itself is not costed. The CNN-SVM route requires one extraction and SVM fit per fold §4.1, and the personality route needs five extra CNNs §4.4.

Lineage

• Builds on: CNN-SVM scheme from Poria et al. §4.1; word2vec vectors from Mikolov et al. §4.1; sentiment CNN from SemEval 2014 data §4.2.
• Compared against: Tsur et al. 2010 (cited, not in the comparison table) §2; Riloff et al. 2013 §2; Ptacek et al. 2014 §2, §5.2; Joshi et al. 2015 §2, §5.2.
• Sentic patterns (Poria et al. 2015) are discussed as a possible future addition, not used §3.