All-in-one: Multi-task Learning for Rumour Verification

In plain words

A rumour is a claim that spreads on social media before anyone has checked it. Deciding whether a rumour is true, false or still unknown is hard. People usually split the job into steps: notice that a post is a rumour, follow the replies, and work out whether each reply backs the claim, questions it or denies it. The last step, the final verdict, depends on the others.

Most earlier systems solved each step separately, and the output of one fed the next. This paper trains one neural network for the verdict and for the helper steps at the same time. Both kinds of task share the same core of the network, so the side tasks can give the verdict useful hints. This idea is called multi-task learning.

The authors compare several combinations of tasks against a single-task model and a strong earlier system. The best model uses all three tasks, and on PHEME with five events it scores higher on the verdict than the single-task model and the earlier system on the PHEME dataset with five events. They also look at how the results relate to the statistics of each dataset.

TL;DR

• Rumour verification is the final step of a four-step pipeline. The paper treats veracity (true, false, unverified) as the main task and stance and rumour detection as auxiliary tasks §1 §2.1.
• A shared LSTM stack reads each reply branch, with separate dense and output layers per task. Training uses the sum of task losses §4.1 §4.2.
• Author-reported macro-F on veracity: PHEME 5 events 0.396 (three tasks) against 0.339 (NileTMRG*); PHEME 9 events 0.405 against 0.297; RumourEval 0.558 (veracity + stance) against 0.539 for NileTMRG* tab. 3.
• Official code (MIT) is public. It shows the two-task and three-task models but not the single-task or NileTMRG* baselines, and it differs from the paper in loss weights, activations and the search space outer.py.
• Read "Paper vs code" and "Pitfalls" before reimplementing.

Problem

Rumour resolution is a pipeline of four sub-tasks: rumour detection, rumour tracking, stance classification and rumour verification. Earlier work studied each sub-task separately, and the authors state that they know of no earlier work covering the whole pipeline §1 §2.1.

Earlier models that use stance as a feature assume stance and veracity labels for the same data. The authors state that this does not apply to their setting, because not every thread has stance labels §2.2.

Verification is hard for people too. For the PHEME rumours, a non-expert annotator agreed with the journalist labels on 60 to 65 percent of the stories from the five largest events §3.2. The veracity classes are imbalanced, so a majority-class baseline already scores well on accuracy §6.

Method

Data. The paper uses two Twitter datasets. RumourEval (SemEval-2017 Task 8) has 325 threads, with stance labels and veracity labels only, since every thread is a rumour §3.1 tab. 1. PHEME has nine events with rumour detection, veracity and, for a subset, stance labels §3.2 tab. 2. A conversation is split into linear branches, each running from a leaf reply up to the source tweet. Each branch is one training instance §3 fig. 2.

Features. Text is lower-cased, non-alphabetic characters are removed and tokens are split with NLTK. Each word gets a 300-dimensional word2vec vector trained on Google News, and a tweet is the average of its word vectors §4.4. The official repository does not compute these features. It loads preprocessed arrays from a figshare archive README.md MTL3_CV_VeracityStanceDetection.py.

Sequential model (single task). An LSTM layer reads the tweets of a branch in order, followed by dense ReLU layers and a softmax §4.1. Stance uses the output at every tweet. Detection and veracity use only the output at the last tweet. Per-thread predictions are majority votes over the branches of the thread §4.1. The single-task baseline is named branchLSTM §4.1 tab. 3.

Multi-task models. All tasks share the LSTM layers (hard parameter sharing), and each task then has its own dense layers and output §2.2 §4.2 fig. 3. The paper tests three settings: veracity with stance (MTL2), veracity with detection (MTL2), and all three tasks (MTL3) §4.2. In the code, the veracity and detection heads each have their own LSTM layer on top of the shared stack, while the stance head reads the shared states directly MTL3_CV_VeracityStanceDetection.py. The paper does not describe these private LSTM layers.

Loss. The paper uses a sum of the task losses. When a branch has no label for a task, that task adds nothing to the loss §4.2. The code implements this with per-sample weights close to zero MTL3_CV_VeracityStanceDetection.py. For MTL3 the code uses fixed loss weights of 0.3 for stance, 0.4 for veracity and 0.3 for detection MTL3_CV_VeracityStanceDetection.py.

Training. Categorical cross-entropy is used for stance and veracity, and the code uses binary cross-entropy for detection §4.1 MTL3_CV_VeracityStanceDetection.py. Dropout of 0.5 is applied before each output layer §5.1 MTL3_CV_VeracityStanceDetection.py. Hyperparameters are chosen by Tree-structured Parzen Estimator search with hyperopt on the development set, minimising a loss built from macro-F §5.1 parameter_search.py.

Evaluation. RumourEval: tune on dev, retrain on train plus dev, test on the test split §5.2 MTL2_RumEval_VeracityStance.py. PHEME: leave-one-event-out cross-validation over five or nine events. Charlie Hebdo is the development event §5.2 MTL3_CV_VeracityStanceDetection.py.

# Multi-task branch model, MTL3 variant  [§4.2] [fig. 3]
# Input:  one branch, zero-padded to 25 tweets
# [code: MTL3_CV_VeracityStanceDetection.py#L116]
# Output: stance per tweet, veracity and detection per branch

def forward(branch):
    # Shared LSTM stack; masked (padded) steps are skipped  [§4.2]
    shared_states = shared_lstm_stack(mask(branch))  # (T, H)

    # Stance head: dense layers applied to every tweet  [§4.2] [fig. 3]
    stance_hidden = dense_layers_per_tweet(shared_states)  # (T, D)
    stance = softmax(linear(dropout(stance_hidden)))       # (T, 4)

    # Veracity head: own LSTM, read at the last unmasked step  [§4.1]
    veracity_state = lstm_veracity(shared_states)  # (H,)
    veracity = softmax(linear(dropout(dense(veracity_state))))  # (3,)

    # Detection head: own LSTM, read at the last unmasked step  [§4.1]
    detection_state = lstm_detection(shared_states)  # (H,)
    detection = softmax(linear(dropout(dense(detection_state))))  # (2,)

    return stance, veracity, detection

# Training loss for one batch of branches  [§4.2]
# [code: MTL3_CV_VeracityStanceDetection.py#L88-L104]
# Labels that are missing for a task contribute (almost) nothing

def training_loss(branches, labels):
    # Stance is labelled per tweet; padded or unlabelled tweets get weight
    # 1e-5 in the code, not exactly 0
    # [code: MTL3_CV_VeracityStanceDetection.py#L92]
    stance_weight = any_label_per_tweet(labels.stance) + 1e-5
    # Veracity weight is 1e-5 when a branch has no veracity label
    # [code: MTL3_CV_VeracityStanceDetection.py#L93]
    veracity_weight = any_label_per_branch(labels.veracity) + 1e-5
    # Per-task cross-entropy, weighted by the masks above
    stance_loss = weighted_cross_entropy(stance, labels.stance, stance_weight)
    veracity_loss = weighted_cross_entropy(
        veracity, labels.veracity, veracity_weight
    )
    detection_loss = cross_entropy(detection, labels.detection)
    # Code: fixed weights 0.3, 0.4, 0.3
    # [code: MTL3_CV_VeracityStanceDetection.py#L77-L79]
    # Paper text: a plain sum of the task losses  [§4.2]
    return 0.3 * stance_loss + 0.4 * veracity_loss + 0.3 * detection_loss

# Leave-one-event-out evaluation on PHEME  [§5.2]
# [code: MTL3_CV_VeracityStanceDetection.py#L336-L346]
# Thread-level veracity is a majority vote over branch predictions  [§4.1]

def evaluate_leave_one_event_out(events):
    # Hold out each event once; train on all the others
    for test_event in events:
        train_events = [e for e in events if e != test_event]
        model = train(train_events)  # Adam, 50 epochs, batch 32  [§5.1]
        branch_predictions = model.predict(test_event.branches)
        # Ties go to the smallest class index
        # [code: branch2treelabels.py#L22-L24]
        thread_predictions = majority_vote(branch_predictions)
    # Macro-F over the pooled predictions of all folds
    # [code: MTL3_CV_VeracityStanceDetection.py#L491-L526]
    return macro_f1(pooled_labels, pooled_predictions)

Key quantities, in the paper's notation (no numbered equations are given):

• Macro-F on each task, computed on the test set §5.2.
• Hyperparameter objective for multi-task models: the paper writes the product of (1 − macro-F) over the tasks, and the code uses the sum §5.1 MTL3.

Hyperparameters

Results

Author-reported. Setup: RumourEval (train/dev/test split) and PHEME (leave-one-event-out, five or nine events), with word2vec features, LSTM models and macro-F and accuracy as metrics §5.2 §6. The paper gives one value per cell and no variance or seed information tab. 3. The MTL2 detection and MTL3 models were run only on PHEME, since RumourEval has no detection labels §6.

Per-event results for MTL3 on the five PHEME events tab. 4:

The per-event figure for the single-task and two-task models is described in words only fig. 4 §6.1.

Dataset statistics for RumourEval tab. 1, for stance classes S, D, Q and C (support, deny, query, comment; the abbreviations are inferred from the text §3.1):

Dataset statistics for PHEME tab. 2:

Label-distribution statistics per event for stance (S), veracity (V) and detection (D): kurtosis, entropy and token-type ratio (TTR). A dash means the paper reports no value tab. 5 §6.2:

The authors report that multi-task models outperform single-task models for each of the five largest events §6.1, and that all models predict the majority class (true) best §6.1.

Paper vs code

Official repository: https://github.com/kochkinaelena/Multitask4Veracity at commit 56d9aed33d15d58fae5d6049db67622e6d07a4a4, MIT licence. The repository belongs to the first author, and the paper's footnote links to it §5.1. It contains the MTL2 (RumourEval, PHEME 5, PHEME 9) and MTL3 models, the hyperparameter search and the evaluation code. The single-task branchLSTM baseline, the NileTMRG* baseline and the majority baseline are not in the repository, and neither are the preprocessed data (figshare) or the saved_data/ folder that the code loads README.md.

Differences between paper and code:

• Search objective. The paper writes the multi-task objective as a product of (1 − macro-F) terms §5.1. The code sums them MTL3_CV_VeracityStanceDetection.py MTL3_CV_VeracityStanceDetection.py MTL2_CV_VeracityStance.py.
• Loss weights. The paper uses a plain sum of task losses §4.2. The code uses weights 0.3, 0.4, 0.3 for MTL3 MTL3_CV_VeracityStanceDetection.py and 0.5, 0.5 for MTL2 MTL2_CV_VeracityStance.py MTL2_CV_VeracityDetection.py. The RumourEval MTL2 model uses no weights MTL2_RumEval_VeracityStance.py.
• Activation. The paper says the dense layers are ReLU §4.1 §4.2. Every Dense in the code is built without an activation, so it is linear MTL3_CV_VeracityStanceDetection.py. A search for relu in the repository finds nothing.
• Dense layers. The paper's search space allows one to four dense layers §5.1. The code searches one or two parameter_search.py.
• L2 strength. The paper searches {1e-4, 1e-3} §5.1. The code fixes 1e-3 parameter_search.py.
• Learning rate. The code searches {1e-4, 1e-3} parameter_search.py, but build_model never reads that parameter and compiles with Adam's default MTL3_CV_VeracityStanceDetection.py. The paper does not state an optimizer or a learning rate §5.1.
• Number of trials. The paper reports 30 trials §5.1. The code's default is 10 outer.py, and the README lists 50 README.md.
• Veracity label encoding. The MTL3 code passes veracity labels without to_categorical, while detection labels are one-hot encoded MTL3_CV_VeracityStanceDetection.py MTL3_CV_VeracityStanceDetection.py. The MTL2 stance code converts veracity labels to one-hot MTL2_CV_VeracityStance.py. The data files are not in the repository, so the expected encoding cannot be checked here.
• Averaging. The paper says it micro-averages across events §3.2. The code pools predictions across folds and computes macro-F over the pool MTL3_CV_VeracityStanceDetection.py. The reported numbers use macro-F, so the code matches Table 3, but the paper's wording is ambiguous.
• Model shape. The paper draws the shared LSTM followed by task-specific layers fig. 3 §4.2. The code gives the veracity and detection heads their own LSTM layers MTL3_CV_VeracityStanceDetection.py MTL3_CV_VeracityStanceDetection.py. The paper does not describe this.
• Python version. The paper says Python 3 §5.1. Several files still have a Python 2 shebang, such as parameter_search.py and branch2treelabels.py, and they compare byte-string IDs MTL3_CV_VeracityStanceDetection.py.
• Title. The repository README gives a different title, "All-in-one: Multi-task Learning for Rumour Stance classification, Detection and Verification" README.md.

No discrepancies were found in the RumourEval protocol: the code retrains on train plus dev before testing MTL2_RumEval_VeracityStance.py, as §5.2 describes.

Pitfalls

• Objective and loss disagree. Tuning uses a product (paper) or sum (code) of (1 − F) terms, and training uses a weighted sum (code) or a plain sum (paper). Implement the code's choices if you want to match the reported numbers, and say so in your write-up.
• Dense activations are unclear. The paper says ReLU, the code uses none. Whether the reported numbers came from a ReLU or a linear model cannot be verified from the repository, so test both before concluding.
• Learning rate is effectively not searched. The searched value is never used, so the released code runs with Adam's default. Whether the paper's results used the searched value cannot be verified; the paper does not state an optimizer or learning rate.
• Detection loss. The detection head has two softmax outputs but is trained with binary cross-entropy MTL3_CV_VeracityStanceDetection.py. Which class is "1" is not stated in the paper or the code; F1 is taken as binary with the default positive label MTL3_CV_VeracityStanceDetection.py.
• Missing labels are down-weighted, not removed. The code uses 1e-5 for unlabelled tweets and branches MTL3_CV_VeracityStanceDetection.py. This is almost zero, not zero, and the paper says such instances "do not add anything" §4.2.
• Branch truncation. Branches are padded to a fixed length of 25 tweets and truncated at the end MTL3_CV_VeracityStanceDetection.py MTL3_CV_VeracityStanceDetection.py. Long conversations keep only their first 25 tweets. The paper gives no length limit.
• Duplicate and placeholder IDs. Stance evaluation keeps the first occurrence of each tweet ID and drops a placeholder ID b'a' MTL3_CV_VeracityStanceDetection.py. The paper does not mention deduplication.
• Tie-breaking in majority vote. The branch vote uses np.unique and argmax over counts, so a tie goes to the smallest class index branch2treelabels.py. The paper gives no tie rule.
• Thread label. A thread's label is taken from its first branch branch2treelabels.py. This is safe only if every branch of a thread carries the same label.
• Search and final training differ in size. For five-event PHEME, the search trains on three events and tests on Charlie Hebdo MTL3_CV_VeracityStanceDetection.py. Final cross-validation trains on four events. The chosen hyperparameters therefore come from a different training set size than the final models.
• Single run, no variance. Tables give one number per cell, with no seeds or standard deviations tab. 3 tab. 4. The paper gives no significance test, so small gaps between models cannot be judged from the card.
• Results depend on the split. The paper argues that PHEME with nine events is harder, because the four added events are small and contain a single rumour story §6. The per-event scores vary widely tab. 4 tab. 5.
• Baselines are not reproducible from this repository. NileTMRG* is the authors' own reimplementation, using a stance model from earlier work §4.3. The single-task branchLSTM is not in the code. Table 3 rows for these cannot be checked against the code.
• Data prerequisites. The code needs preprocessed arrays with specific names (train_array.npy, fold_stance_labels.npy, labels.npy, rnr_labels.npy, tweet_ids.npy, ids.npy), which come from a separate figshare archive README.md MTL3_CV_VeracityStanceDetection.py. Word-embedding extraction is not in the repository.
• Python 2 and 3. Some files are Python 2 scripts, so run them under the Python version the code targets, or port them with care around byte-string IDs MTL3_CV_VeracityStanceDetection.py.

Lineage

• Builds on: the branchLSTM sequential approach of Zubiaga et al. §4.1, the stance classifier of Kochkina et al. 2017 used for NileTMRG* features §4.3, and the observation of Alonso and Plank that multi-task gains depend on label distributions, which this paper extends to rumour tasks §6.2.
• Compared against: Majority (True), NileTMRG* (the best veracity system at SemEval-2017 Task 8) and the single-task branchLSTM §4.1 §4.3 tab. 3.
• Superseded by: not stated in the wiki.