CASCADE: Contextual Sarcasm Detection in Online Discussion Forums

In plain words

Sarcasm is hard for software because the words alone often look sincere. A sentence like "Whew, I feel much better now!" only reads as sarcastic if you know what was said just before it, or if you know who usually writes like that. Earlier detectors mostly looked at the sentence on its own.

CASCADE keeps a standard text classifier for the comment itself and adds two kinds of context. The first is a profile of the author, built from their past comments: how they write, and a rough guess at their personality. The second is a summary of the forum (subreddit) the comment was posted in, so that topics where sarcasm is common can push the decision. The three pieces are joined and a final layer decides "sarcastic" or "not sarcastic".

The authors report that the context pieces give a clear gain over a text-only network and over an earlier user-profiling method on Reddit data.

TL;DR

• Hybrid sarcasm classifier for Reddit comments: a CNN on the comment text, concatenated with an author vector and a forum vector §3.2 §3.6.
• Author vector = CCA fusion of a stylometric ParagraphVector view and a personality view from a CNN trained on essays §3.4.
• Forum vector = ParagraphVector over the concatenated comments of each subreddit §3.5.
• Author-reported: 0.77 accuracy and F1 on SARC main balanced, against 0.70/0.69 for CUE-CNN tab. 2.
• Code is public at declare-lab/CASCADE but has no license, and it differs from the paper in several hyperparameters and in the output architecture (see Paper vs code).

Problem

Sarcasm often depends on context outside the sentence: the previous comment, the topic, or what the author usually says §1. Text-only classifiers, including CNNs, do not see this context. Earlier user-modeling work on sarcasm learned user embeddings that capture homophily §2. The paper argues that author and forum context improve detection on top of the text §1.

Method

The method has three inputs per comment, each computed in advance except the comment itself.

1. Content. A single-layer CNN with word embeddings reads the comment. It has filters of three heights, a fixed number of filters per height, ReLU, and max-pooling over time §3.4.2 "CNN". The paper says the heights are F_{[1,2,3]} but does not give their values §3.4.2.
2. Author vector. Two views are built per user.
   • Stylometric view §3.4.1: all comments of a user are joined into one document with a <END> delimiter. ParagraphVector, which the paper describes as a CBOW-style method, learns one vector per user document eq. 1 eq. 2 eq. 3.
   • Personality view §3.4.2: a CNN pre-trained for multi-label Big-Five prediction on essays is run on each comment. The hidden vector of each comment is averaged over the user's comments eq. 4.
   • Fusion §3.4.3: the two views are fused with canonical correlation analysis into a K-dimensional user vector eq. 7 eq. 8 eq. 9 eq. 10 eq. 11.
3. Forum vector §3.5. All comments of a subreddit are joined into one document and embedded with ParagraphVector.

Final prediction §3.6: the CNN vector, the author vector and the forum vector are concatenated, projected to two output units with softmax, and trained with cross-entropy eq. 12. The paper does not say whether the author and forum vectors are updated during classifier training (see Pitfalls).

# CASCADE prediction for one comment  [§3.6] [§3.2]
# Input:  tokens (up to 100 word ids), author id, forum id
# Output: probabilities for {not sarcastic, sarcastic}

def cascade_predict(tokens, author_id, forum_id):
    # Content: CNN over the comment's word embeddings  [§3.4.2 "CNN"]
    pooled = max_over_time(conv_relu(embed(tokens)))  # (3 * M,), M = 128
    content = linear(pooled)                          # (d_c,), size not stated

    # Author: fused user vector, looked up by author id  [§3.4.3]
    author_vec = user_embedding[author_id]            # (K,), K = 100

    # Forum: ParagraphVector vector of the subreddit  [§3.5]
    forum_vec = forum_embedding[forum_id]             # (d_t,), d_t = 100

    # Join the three vectors and classify  [§3.6] [eq. 12]
    joined = concat(content, author_vec, forum_vec)
    return softmax(output_layer(joined))              # (2,)

# User embedding  [§3.4]
# Input:  all comments of one user; the user's stylometric document
# Output: user vector of size K

def user_embedding(user_comments):
    # Stylometry: ParagraphVector on the user's joined comments  [§3.4.1]
    doc_vec = paragraph_vector(join(user_comments, sep="<END>"))  # (d_s,)

    # Personality: mean of per-comment hidden vectors  [§3.4.2] [eq. 4]
    hidden = [personality_cnn(c) for c in user_comments]  # (d_p,) each
    pers_vec = mean(hidden)                               # (d_p,)

    # Fusion: CCA projections of both views, then combined  [§3.4.3]
    # The paper sums the projections (eq. 11); the code averages them.
    return cca_project(doc_vec, pers_vec)                 # (K,)

Hyperparameters

Results

Author-reported. Setup: SARC corpus §4.1; 10 runs with different initialisations, paired t-test at p < 0.05 against CUE-CNN tab. 2. The paper does not state whether the table shows means over runs, a single run, or the best run. Hardware is not stated.

Dataset sizes tab. 1:

Main comparison tab. 2 (accuracy / F1):

Ablation tab. 3. All rows use the content CNN. "cca" and "concat." are the two ways of forming the user vector; "discourse" is the forum vector.

Row 6 is the full model. Its main-balanced accuracy is 0.78 in tab. 3 but 0.77 in tab. 2, and row 5 has a higher imbalanced accuracy (0.80) than the full model (0.79). The paper does not explain these differences.

Other reported results: the user-embedding t-SNE plot shows the sarcastic users with a wider spread than non-sarcastic users, with reported variances of 10.92 and 5.20 §4.6 fig. 3. Case studies on Pol show two comments that are sarcastic only in light of the previous comment §4.7.

Paper vs code

Code: https://github.com/declare-lab/CASCADE at commit f1b14f1ad093074ffbe12bfac73da531ec718b02. The paper does not link the code; it was found by GitHub search and the owner is the declare-lab organisation, which the README ties to this paper. The repository has no LICENSE file, so it is cited only, with no code copied.

Checked: the README, the classifier and training scripts (src/), the stylometric, personality, discourse and fusion scripts (users/, discourse/). The balanced training and test CSVs are under data/; the imbalanced and Pol variants are not in the repository. The comments.json corpus, the personality weights and the GCCA embeddings must be downloaded separately README.md.

• Learning rate. Paper: 1e^{-4} §4.2. Code: Adam at 1e-3 train_cascade.py.
• Early stopping. Paper: patience 12 §4.2. Code: no patience. It trains for a fixed 100 epochs train_cascade.py and saves the checkpoint with the best validation accuracy train_cascade.py.
• Output architecture. Paper: CNN vector, author vector and forum vector are concatenated and fed to a 2-unit softmax §3.6. Code: the CNN output goes through a 100-unit linear layer text_cnn.py; the three vectors are concatenated (300 units) and passed through a 400-unit ReLU layer text_cnn.py, then dropout text_cnn.py, then a 2-unit output text_cnn.py.
• Dropout. Not stated in the paper. Code keep-probability 0.5, applied after the 400-unit layer text_cnn.py train_cascade.py.
• L2. Not stated in the paper. Code adds 0.5 times the L2 norm of the output-layer weights and bias only train_cascade.py text_cnn.py.
• Loss base. Paper writes the cross-entropy with \log_2 eq. 12. Code uses the natural log via softmax cross-entropy text_cnn.py. The difference is a constant factor of \ln 2.
• Trainable embeddings. Not stated in the paper. Code makes the word, user and forum embedding matrices trainable variables text_cnn.py.
• Batch size and epochs. Not stated in the paper. Code uses batch 4096 train_cascade.py and 100 epochs train_cascade.py.
• Stylometric document vectors. Paper: after training, the document vectors learned during training are used §3.4.1. Code: vectors are re-inferred with infer_vector for each user after training generate_stylometric.py. Inference is not seeded in the script.
• Stylometric window. Paper: window k_s = 2 §4.2. Code: window 10 train_stylometric.py.
• Stylometric softmax. Paper: hierarchical softmax §3.4.1. Code does not set the hs option, so the gensim default applies train_stylometric.py.
• Stylometric epochs. Not stated in the paper. Code: 50 epochs, learning rate reduced by 0.002 per epoch train_stylometric.py.
• Stylometric documents. Code joins comments with " <END> " create_per_user_paragraph.py, matching the delimiter in the paper.
• Forum documents. Code groups the rows of train-balanced.csv by the subreddit column and joins comments with <END> create_per_discourse_paragraph.py create_per_discourse_paragraph.py. Because the training file is used, the forum document includes training comments; the paper does not say which comments make up a forum document.
• Forum epochs and window. Not stated in the paper. Code: window 10, 100 epochs train_discourse.py.
• Personality CNN. Paper: a dense layer with d_p units and a nonlinearity, then a 5-unit sigmoid output §3.4.2 eq. 6. Code: pooled features go to a 100-unit linear layer with no activation text_cnn.py, then a 5-unit sigmoid output text_cnn.py. The personality vector is therefore linear in the pooled features.
• Personality training. Not stated in the paper. Code: sequence length 1000 train_personality.py, batch 64 train_personality.py, Adam at 1e-3 train_personality.py, 100 epochs, dropout 0.5 train_personality.py, no L2 train_personality.py.
• CCA normalisation and fusion. Paper: the views are mean-centred and standardised, then combined as a sum of projections §3.4.3 eq. 11. Code: no centring or standardisation was found in the merge script merge_user_views.py or the GCCA script user_wgcca.py; the merge script only concatenates the two views. The embedding is a weighted mean of the projections, dividing by the sum of view weights user_wgcca.py, so it is half the paper's sum when both views are present. The code is a weighted, SVD-based GCCA with regularisation 10^{-8} user_wgcca.py, not the closed-form expression written in the paper. The paper points to a supplement for the generalised CCA, which is not in the source.
• Unknown users and forums. Code maps unknown authors and forums to row 0, a random vector train_cascade.py generate_discourse.py. The paper does not say how unseen users are handled.
• Imbalanced and Pol variants. Paper reports both tab. 2. Only the balanced data and training path are in the repository my_train_balanced.csv.
• F1. The paper reports F1 tab. 2. The training and evaluation scripts print accuracy and confusion matrices only; the F1 computation was not found in the code read.

Pitfalls

• Results need the external corpus. The paper's comments come from SARC, which the README says must be downloaded separately (comments.json, about 2.5 GB). The personality weights and GCCA embeddings are also external downloads README.md.
• Author and forum vectors may or may not be trained. The paper does not say. In the code they are trainable variables, so their values change during classifier training text_cnn.py. Decide which is intended before comparing numbers.
• Unknown-author row. All unseen authors share row 0, a random vector that is also trained, so it absorbs every unseen author in the training run. Test authors missing from the table get this shared row train_cascade.py.
• Word cut-off. Comments are cut to their first 100 words after cleaning process_data.py. Longer comments lose their final words; the paper notes failures on very long contextual comments §4.7.
• Checkpoint selection. The model is selected by validation accuracy, and test accuracy is printed at every evaluation train_cascade.py. Only validation is used for selection in the code. The paper's test numbers are therefore only safe if reported from the selected checkpoint, which the paper does not state.
• Training cost. Under the batch size and balanced sizes above, one epoch is roughly 34 steps at batch 4096 after the 10% hold-out. This is an estimate from tab. 1 and the code's batch size, not a figure the paper reports.
• Doc2vec randomness. Re-inferring stylometric vectors at inference is not seeded in the script generate_stylometric.py, and ParagraphVector training is multi-threaded (workers=11). Results may not reproduce exactly.
• Personality vector has no nonlinearity in the code. The paper describes a nonlinear hidden layer §3.4.2; the released code uses a linear one text_cnn.py. Follow the code if matching the released weights, the paper if matching the text.
• The paper's own tables disagree. The full model's balanced accuracy is 0.77 in tab. 2 and 0.78 in tab. 3, and row 5 of tab. 3 beats the full model on imbalanced accuracy (0.80 vs 0.79).
• Content vector size. The paper never states the dimension of the content vector c. The code uses 100 text_cnn.py.
• GCCA reproduction. The GCCA run is only reproducible with the same user set and view file. The README gives the command with --k 100 --no_of_views 2 README.md.
• Significance. The paper marks significance only against CUE-CNN tab. 2.

Lineage

• Builds on: ParagraphVector (cited as Le et al., 2014 §3.4.1) and the CNN text classifier of Kim et al., 2014 §3.4.2. Their arXiv ids are not in this wiki.
• Compared against: CUE-CNN (Amir et al., 2016), which uses user embeddings from a ParagraphVector-like model §4.3; CNN-SVM (Poria et al., 2016), which combines a CNN with sentiment and personality features in an SVM §4.3; a bag-of-words baseline and a plain CNN §4.3.
• Superseded by: not known from this wiki.