Classification and Clustering of Arguments with Contextualized Word Embeddings

In plain words

Open-domain argument search means finding sentences that argue for or against a topic a user asks about, then grouping sentences that make the same point. The same sentence can mean different things under different topics, and two sentences that make the same point can share few words. Older systems used simple word features and recurrent networks, which did poorly when the topic at test time differed from the topics seen in training.

The authors fine-tune a large pretrained language model (BERT) in two ways. In the first, the topic and the sentence are fed in together, so the model can check how the sentence relates to the topic. In the second, two sentences are fed in together and the model outputs a score for how much they make the same point. A standard clustering step then merges sentences whose scores are high enough. The results are strong on the classification tasks. For clustering, the model is good at judging pairs, but turning pair scores into groups costs some accuracy, because a sentence can belong to more than one group.

TL;DR

• Fine-tuned BERT, with the topic placed before the sentence, is the strongest classifier of pro, con and no-argument sentences across topics §4.3 tab. 1.
• A BERT cross-encoder trained on pairs of argument sentences gives the best similarity scores. Its pair judgments are worse than human ones, but after average-linkage clustering it is roughly on par with the human upper bound §5.5 tab. 2.
• The published code covers only the BERT fine-tuning and evaluation. The data must be downloaded separately, and several reported settings differ from the scripts (see Paper vs code).
• Hyperparameters are short and standard (lr 2e-5, batch 16 or 32, 3 epochs), but the code has a train/eval mode bug and does not use the dev split for classification.

Problem

Open-domain argument search needs two steps: classify each sentence as pro, con or no argument for a topic, then cluster the argumentative sentences by the aspect they address §1 §2. Cross-topic generalisation is the hard part, because the test topics are new §1 §2. Earlier models used word2vec or GloVe features and LSTMs §2 §4.2, and the gap to human performance on the UKP corpus remained large §4.3.

Argument clustering is described as understudied. The authors note that no dataset supports end-to-end evaluation, so each step is evaluated on its own §3 §6. The clustering step also assumes that each sentence belongs to one cluster, which the data does not always support §5.5.

Method

Argument classification (topic-aware BERT) §4.2. For each sentence and topic, the input is the topic, a separator, then the sentence. BERT is fine-tuned with a three-way output (pro, con, no argument) using the first token's output §4.2. The paper compares this with BiLSTM and ELMo or BERT word-feature baselines, and with a model that gives only the sentence (no topic) tab. 1. The IBM dataset is handled with the same pair input §4.2.

Argument similarity (cross-encoder) §5.4. A BERT model reads both sentences together and a sigmoid unit on the first token gives a similarity between 0 and 1 §5.4. The paper also reports unsupervised baselines (Tf-IDF, InferSent, averaged GloVe, ELMo or BERT vectors) and a human upper bound §5.4 tab. 2 tab. 3.

Clustering (average linkage) §5.1. Clusters are merged greedily by average linkage, with the largest average similarity merged first. The average over all cross-cluster sentence pairs is

\mathrm{sim}(A, B) = \frac{1}{|A|\,|B|} \sum_{a \in A} \sum_{b \in B} d(a, b)

The paper does not number its equations; this formula is inline in §5.1. The merge stops when the best average falls below a threshold; the threshold is chosen on a train or dev set §5.1 §5.3. The authors also tried k-means and DBSCAN, which did worse in their preliminary runs §5.1.

# Topic-aware argument classifier  [§4.2]
# Input:  topic (string), sentence (string)
# Output: one of pro, con, no argument

def make_pair_input(topic, sentence):
    # Topic first, then the sentence, each closed by [SEP]  [§4.2]
    # [code: argument-classification/train.py#L134-L135]
    tokens_a = tokenize(topic)
    tokens_b = tokenize(sentence)
    # Shorten the longer side until the pair fits in 64 tokens
    # [code: argument-classification/train_ukp.sh#L5]
    # [code: argument-classification/train.py#L297-L311]
    tokens_a, tokens_b = truncate_pair(tokens_a, tokens_b, 64 - 3)
    return ["[CLS]"] + tokens_a + ["[SEP]"] + tokens_b + ["[SEP]"]

# Pairwise argument similarity (cross-encoder)  [§5.4]
# Input:  sentence_a, sentence_b, target in [0, 1]
# Output: scalar loss

def similarity_loss(sentence_a, sentence_b, target):
    # Both sentences in one BERT input, as a pair  [§5.4]
    # [code: argument-similarity/train.py#L281-L283]
    hidden = bert(pair_input(sentence_a, sentence_b))    # (B, L, H)
    # Use the final hidden state of [CLS], not the pooler
    # [code: argument-similarity/SigmoidBERT.py#L19-L20]
    cls_vector = hidden[:, 0, :]                         # (B, H)
    # One linear unit, then a sigmoid
    # [code: argument-similarity/SigmoidBERT.py#L22-L23]
    prediction = sigmoid(linear(cls_vector))             # (B, 1)
    # Binary cross-entropy against soft targets
    # [code: argument-similarity/SigmoidBERT.py#L26-L27]
    return binary_cross_entropy(prediction, target)

# Average-linkage clustering with a stop threshold  [§5.1]
# Input:  sentences of one topic, similarity(a, b), threshold
# Output: list of clusters (each a list of sentences)

def cluster_topic(sentences, similarity, threshold):
    # Start with one cluster per sentence
    # [code: argument-similarity/evaluation_with_clustering.py#L127]
    clusters = [[sentence] for sentence in sentences]
    while len(clusters) > 1:
        # Find the pair of clusters with the highest average similarity  [§5.1]
        # [code: argument-similarity/evaluation_with_clustering.py#L133-L139]
        best = best_cluster_pair(clusters, similarity)
        # Stop when even the best pair is at or below the threshold
        # [code: argument-similarity/evaluation_with_clustering.py#L164-L167]
        if best.score <= threshold:
            break
        # Merge the two clusters, then re-score the merged cluster
        # [code: argument-similarity/evaluation_with_clustering.py#L171-L183]
        clusters = merge(clusters, best.first, best.second)
    return clusters

Key equations: the average-linkage formula above §5.1. The classification loss is not written out in the paper.

Hyperparameters

Values stated in the paper are cited to the paper; values the paper omits are cited to the code, which is marked as code-only.

Results

Author-reported. Setup: UKP Sentential Argument Mining Corpus (eight topics, 25,492 sentences) for classification, with a 7-train/1-test topic protocol §3.1 §4.1; IBM Debater Evidence Sentences (118 topics, 83 train, 35 test) §3.1 §4.1; UKP ASPECT (28 topics, 3,595 pairs, 4-fold cross-topic) §3.2 §5.2; AFS, cross-topic with two train topics and one test topic §5.2. Hardware and training time: not stated.

Table 1, argument classification tab. 1. F1 is macro over the three labels on UKP. Accuracy on IBM.

Precision and recall per class are in the paper's table tab. 1 and are not repeated here.

Table 2, UKP ASPECT pair similarity and clustering, F-mean tab. 2.

Table 3, AFS pair similarity, Pearson r and Spearman rho, averaged over three topics tab. 3.

Table 4, UKP ASPECT F-mean for BERT when the number of fine-tuning topics grows, averaged over ten random topic samples tab. 4.

Table 5 (appendix), full AFS results by topic, with r and rho tab. 5 app. A.2.

Claims in the abstract and text, as the authors state them: +20.8 pp macro F1 on UKP over the earlier state of the art (abstract; tab. 1 gives the biCLSTM baseline as the comparison) §4.3; +7.4 pp accuracy on IBM (abstract); +7.8 pp F-mean on UKP ASPECT §5.5; and +12.3 pp on AFS in the abstract, which the text puts at about 11 pp against SVR §5.5 tab. 3.

Paper vs code

Code: the official repository at commit 72f643b06a06b9ba82a25df2c134664fc26f84f3 (Apache-2.0, UKPLab). It contains BERT fine-tuning and evaluation only. The datasets are not included. No code for the biLSTM, ELMo-feature or unsupervised similarity baselines, the human-performance estimate, or the IBM reproduction was found: a search of the repository for the baseline names returned nothing.

• Epochs, UKP classification. The paper states three epochs §4.2. The script runs 2.0 train_ukp.sh.
• Batch size, IBM classification. The paper uses 16 §4.2. The script uses 32 train_ibm.sh.
• Dev split, classification. The paper says models were tuned on the dev split (10%) of the seven training topics §4.1. The loader keeps only rows marked train or test and drops dev rows train.py. The shipped runs have no tuning step and report the final epoch. The epoch-by-epoch test outputs are written but not selected train.py train.py.
• Seeds. The paper averages over ten seeds (UKP) and five (IBM) §4.1. Each shipped script runs one seed train_ukp.sh. The README recommends several seeds README.md.
• Reproduction target. The evaluation script's comment gives an expected eight-topic average of 61.69% ukp_evaluation.py. Table 1 gives 0.6128 for BERT-base with topic. The code's README says GPU runs are non-deterministic README.md.
• BERT-large. The paper reports BERT-large results tab. 1 tab. 2. The shipped similarity and classification scripts set bert-base-uncased, and no large-model command is given. The README links a large checkpoint README.md.
• Classification head. The paper describes a softmax on the first token's output §4.2. The classifier code uses the library's sequence-classification class train.py. That class normally feeds BERT's pooled [CLS] output to the classifier; this repo does not show that step, so check it against the installed library.
• Similarity head. The paper says a sigmoid layer on the [CLS] token §5.4. The code does take the [CLS] hidden state, not the pooler SigmoidBERT.py. It defines a dropout layer but does not apply it SigmoidBERT.py.
• Target and loss, AFS. The paper does not state the loss. The code trains with binary cross-entropy on the rating divided by 5 train.py SigmoidBERT.py.
• Training pairs, UKP ASPECT. The paper does not say how pairs are ordered. The loader stores both orders of each pair for training train.py. AFS does the same train.py. Each unordered pair is evaluated once.
• Thresholds. The paper says the clustering threshold is set on the train set for unsupervised methods and on dev for supervised ones §5.3. The script selects the threshold on dev for all methods, from a grid of 20 values evaluation_with_clustering.py. No train-set threshold code is included.
• Which epoch is reported. The without-clustering script reads epoch 2 of 3 evaluation_without_clustering.py. The with-clustering script reads epoch 3 evaluation_with_clustering.py. Epoch numbers are 1-based in the output file names train.py. The paper does not say which epoch it reports.
• Clustering algorithm. Average linkage with a threshold, merging the highest pair first, matches §5.1. The priority queue is a plain list searched for the maximum on each pop evaluation_with_clustering.py.
• Per-topic averaging. The F-mean is computed per topic and then averaged over topics evaluation_without_clustering.py. The paper says "macro-average" §5.3 without naming the unit.
• Expected results. The README says results from the published code are usually a little higher than the paper's README.md.

No discrepancies were found in the AFS split (two train topics, one test topic) and the UKP ASPECT splits (17 train, 4 dev, 7 test topics per fold). Both match the paper §5.2 make_splits.py.

Pitfalls

• Dropout is off from epoch 2 on. Both trainers call model.train() once before the epoch loop, and each epoch's evaluation calls model.eval() train.py train.py train.py train.py. Nothing switches back to training mode, so every epoch after the first runs without dropout. The effect on the reported numbers is not stated. Call model.train() at the start of each epoch when you reimplement this.
• Learning-rate schedule may be applied twice. The script sets the rate each step to the base rate times a warmup-then-linear-decay factor train.py. The optimizer, BertAdam, is given the same warmup and total steps train.py. I could not check the library here (it is not vendored), so I cannot say whether BertAdam applies its own schedule on top of the script's. If it does, the effective rate is the product of the two factors. Log the optimizer's rate after a step to find out.
• No tuning on dev for classification. Results are the final-epoch test scores. The paper's text implies tuning §4.1, so the reported numbers do not reflect a dev-selected model.
• Training length is short and fixed. Two or three epochs over the training pairs, with no early stopping in the shipped scripts.
• Truncation is silent. Inputs are cut to 64 tokens. The code counts the longer inputs and logs the count, but does not store them train.py train.py. The paper does not state a length.
• Cross-encoder cost. The similarity model reads one pair per forward pass. Scoring all pairs for a topic with n sentences needs n(n-1)/2 passes, and clustering needs the full matrix for each threshold. The paper gives no timing.
• Strict partitioning loses information. The authors found that 376 of 1,714 triples (21.9%) violate transitivity on UKP ASPECT §5.5. Clustering assigns each sentence to exactly one group, so it cannot represent a sentence that fits two aspects §5.5 §6.
• Baseline numbers. The IBM baseline is read from a figure, not the text, and the authors say its exact value can only be guessed §4.3, footnote. The "about .74" in tab. 1 rests on that.
• Abstract gap on AFS. The abstract's 12.3 pp does not match a single difference in tab. 3. The text says about 11 pp against SVR §5.5.
• Internal mismatch in Table 4. Its 17-topic row gives 0.7034 with clustering, while Table 2's BERT-base row gives 0.7007 tab. 4 tab. 2. The paper does not explain the difference.
• Path mismatch for AFS. The README expects a file named ArgParis_DP.csv README.md. The loader reads ArgPairs_<topic>.csv train.py. Rename the files or fix the loader.
• Non-determinism. The code's README warns that GPU training is not bit-reproducible README.md. Compare seed averages, not single runs.
• Data is not included. The UKP classification corpus, IBM data, AFS data and UKP ASPECT data must be obtained from their owners. The repo's scripts assume a given directory layout README.md.

Lineage

• Builds on: biLSTM and biCLSTM with topic-aware cells for cross-topic argument classification §2 tab. 1; BERT and ELMo contextual embeddings §2; the IBM evidence-sentence model §2; Misra et al.'s argument-facet similarity corpus §3.2.
• Compared against: Tf-Idf, InferSent, averaged GloVe, ELMo and BERT vectors for similarity §5.4 tab. 2 tab. 3; SVR on AFS tab. 3; human performance tab. 1 tab. 2.
• Superseded by: none in this wiki.