Ex Machina: Personal Attacks Seen at Scale

In plain words

Platforms want to know how often people make personal attacks in their comments, but reading millions of comments by hand is out of reach. This paper gets around that. A small set of comments is shown to several human raters, and each rater says whether the comment is a personal attack. A simple text classifier is trained on those ratings, and then it labels the whole archive of Wikipedia discussion pages.

The useful question is whether the machine is good enough to stand in for people. The authors measure this by asking how many human raters the classifier is worth, and they find it matches the combined judgment of about three raters. Once that holds, the machine labels can answer questions that a small hand-labelled sample could not, such as whether anonymous users are more likely to write attacks, and whether attacks tend to come in clusters.

TL;DR

The paper pairs crowdsourced labels with a bag-of-n-grams classifier and uses it to label 63M English Wikipedia talk comments §3 app. A. On AUC and Spearman correlation, the classifier is compared against small crowd ensembles and matches roughly three raters §4.3 tab. 3. Use it as a template for labelling a comment property at scale when you accept crowd-label noise (Krippendorff's alpha 0.45 §3). The code is Apache-2.0, but several training choices in the code are not in the paper.

Problem

Personal attacks harm online discussion, and platforms respond with policies, but measuring how common attacks are and what they cause is hard at scale §1. Typical annotation efforts label thousands of comments, while Wikipedia alone has 63M English talk-page comments, and crowd labelling of a corpus that size is prohibitively expensive §1. The method applies to properties that crowd workers can label with high inter-annotator agreement §1.

Method

The approach has three parts: collect many crowd labels for a sample of comments, train a text classifier on them, and check the classifier against small crowd ensembles before running it on the full corpus §1 §4.3.

• Corpus. The comments are the text added to talk pages in each edit, taken from the revision history of a public English Wikipedia dump of 2016-01-13. Diffs were computed with the mwdiffs package and a longest-common-substring diff app. A. Bot and administrative comments are removed with regular expressions app. A.
• Sampling. A random set samples comments uniformly. A blocked set takes the 5 comments written by blocked users around every block event, because attacks are rare in random samples §3.
• Crowd labels. Each comment is shown the question in fig. 1 and labelled by at least 10 annotators on Crowdflower. Annotators must pass a 10-question test at 70% accuracy, and test questions are mixed into the task at 10% §3. Annotations where one worker both attacked and did not attack a comment are dropped, and comments most workers judged non-English are removed §3.
• Label targets. Two targets are compared. One-hot (OH) uses the majority class. Empirical distribution (ED) uses the fraction of annotators who said "attack" §4.1.
• Features and models. The input is the comment text only. Features are bag-of-words counts over word 1–2-grams or character 1–5-grams §4.1 §4.2 fn. 2. The architectures are logistic regression (LR) and a multi-layer perceptron (MLP) with a softmax output, trained with cross-entropy §4.1 eq. 1. The LSTM model was planned future work and is not evaluated §4.1.
• Tuning. The 8 configurations (2 architectures, 2 n-gram types, 2 label types) are split 3:1:1 into train, dev and test. Each one gets 15 random-search draws over a hyperparameter grid, trained on train and scored on dev §4.2.
• Surrogate test. For 8,000 baseline-split comments, each with 20 labels, 10 labels form the "truth" group and n_p labels form an ensemble. The ensemble's averaged prediction is compared with the model's score, for n_p from 1 to 10 §4.3 tab. 3.
• Threshold and corpus. Scores above a fixed threshold t are labelled attacks. The paper uses t = 0.425, chosen so that false positives and false negatives balance, and applies it to the full corpus §5.1.

# Train the attack classifier on crowd-labelled comments  [§3] [§4.1] [§4.2]
# Input:  comment texts; 10 crowd labels (0 or 1) per comment
# Output: the best fitted model by dev AUC

def train_attack_classifier(comments, labels):
    # Split 3:1:1 into train, dev and test  [§4.2]
    train, dev, test = split_3_1_1(comments)
    # Targets: fraction of annotators saying "attack"  [§4.1]
    targets = mean_of_labels(labels)
    # Majority class, used for the AUC score  [§4.2]
    majority = targets >= 0.5
    # Bag-of-n-grams from the comment text only  [§4.1] [§4.2 fn. 2]
    features = bag_of_ngrams(comments)
    best = None
    # 15 random-search draws over the hyperparameter grid  [§4.2]
    for config in sample_configs(grid, n_draws=15):
        # Fit on the train split only  [§4.2]
        model = fit(features[train], targets[train], config)
        # Score on dev: AUC against the majority label  [§4.2]
        score = auc(majority[dev], model.predict(features[dev]))
        if best is None or score > best.score:
            best = Candidate(model, score)
    return best.model

# Label the corpus with the fixed equal-error threshold  [§5.1] [§5.2]
# Input:  fitted model; corpus comments
# Output: each comment with an attack flag

def label_corpus(model, corpus):
    # Equal-error threshold reported by the paper  [§5.1]
    threshold = 0.425
    for comment in corpus:
        # A score above the threshold counts as an attack  [§5.1]
        score = model.predict(comment.text)
        comment.is_attack = score > threshold
    return corpus

# Annotator-ensemble baseline on the 8,000-comment baseline split  [§4.3]
# Input:  20 labels per comment; n_t truth labels; n_p ensemble labels
# Output: AUC and Spearman of the model and of the ensemble

def ensemble_baseline(labels_per_comment, n_t, n_p):
    # Random disjoint split of each comment's labels  [§4.3]
    truth, ensemble = split_disjoint(labels_per_comment, n_t, n_p)
    # Truth for AUC is majority; for Spearman it is the fraction  [§4.3]
    truth_majority = majority_of(truth)
    truth_fraction = mean_of(truth)
    # Ensemble prediction is the mean of its n_p labels  [§4.3]
    ensemble_prediction = mean_of(ensemble)
    # Repeat the whole process 25 times and average  [§4.3]
    return auc(truth_majority, ensemble_prediction), spearman(
        truth_fraction, ensemble_prediction
    )

Label formula, as written in the paper: eq. 1

H(y, \hat{y}) = -\sum_i y_i \log(\hat{y_i}) eq. 1

Hyperparameters

Values stated in the paper are cited to the paper. Values found only in the authors' code are cited to the code, and the code is the source for them. Fields not stated in the paper or code are marked "not stated".

Results

Author-reported. Setup: English Wikipedia talk pages, 2004–2015 comments, crowd labels from Crowdflower, dev split for tab. 2, baseline split for tab. 3, and the union of dev and test for the threshold metrics §4.2 §4.3 §5.1.

Annotated data: 115,737 comments in total, of which 37,611 are random (0.9% attacking) and 78,126 are blocked (16.9% attacking). The majority-vote attack rate is 11.7% tab. 1.

Model comparison on the dev split. Values are AUC and Spearman, as the paper reports them tab. 2:

The annotator-ensemble comparison on the baseline split, with standard errors in brackets, uses 10 truth labels tab. 3:

The model beats an ensemble of 3 on both metrics, which is the basis of the paper's "as good as 3 crowd-workers" claim §4.3. On AUC it sits just above n_p = 5 (97.19 against 97.13) and below n_p = 7. On Spearman it is above n_p = 3 but below n_p = 5 tab. 3.

Threshold t = 0.425 on dev plus test: precision 0.63, recall 0.63, false-positive rate 0.0034 §5.1. A bootstrap check on the test split shows the machine-estimated attack rate lies within the crowd-labelled 95% interval, also when the data are split by namespace, year, logged-in status, activity and the word "thank" §5.1 fig. 2.

Findings on the 2015 corpus, after removing bot and administrative comments §5 app. A:

• Anonymous comments have a 3.1% attack prevalence and registered comments 0.5%, which the paper calls six times as likely. Anonymous accounts contribute less than half of attacks. The difference is significant at p < 0.0001 §5.2.
• Almost half of attacks come from users with fewer than 5 comments in 2015. Users with more than 100 comments write about 30% of attacking comments §5.2.
• About 80% of attacks come from over 9,000 users with fewer than 5 attacks each §5.2.
• Within 7 days, 7.7% of attacks are followed by a warning, 7.0% by a block, and 11.3% by either. Normalised by precision 0.63, these are 12.2%, 11.1% and 17.9% §5.2.
• Attacks cluster in time. The share of attacks in the neighbourhood of an attacking comment (n = 1) is 15.6%, against 0.7% around non-attacking comments §5.2.

Paper vs code

Code: the repository ewulczyn/wiki-detox at commit 2131af65ff99ca2c32d3cd486c491ebc33c6b01f, Apache-2.0 LICENSE. Its owner handle matches paper author Ellery Wulczyn, and the paper links the repository in a footnote §4.2 fn. 1. The notebooks show Keras 1.1.0 and sklearn.grid_search cv_ngram_architectures.ipynb ngram.py, so an exact rerun needs those older versions.

Discrepancies found:

• Feature weighting. The paper describes bag-of-words representations §4.1. The code's search adds TF-IDF with sublinear_tf in {True, False} and l2 normalisation cv_ngram_architectures.ipynb. The saved best configurations use TF-IDF cv_results.csv, and the notebook's markdown says TF-IDF improved ROC. The paper does not mention TF-IDF.
• Loss. The paper uses cross-entropy eq. 1. The MLP is compiled with kullback_leibler_divergence deep_learning.py. For a fixed target, KL and cross-entropy differ only by a term that does not depend on the model, so the gradients match. This is inferred from the two formulas, not stated in either source.
• "LR" is ambiguous. The paper's LR is a logistic regression in the sklearn experiments cv_ngram_architectures.ipynb cv_ngram_architectures.ipynb. The linear rows in the saved results are Keras models with no hidden layer and an L2 penalty deep_learning.py cv_results.csv. The paper does not say which one produced Table 2.
• Table 2 numbers are not reproduced. The saved dev-split results differ from tab. 2, which the paper does not explain. Paper AUC / code AUC: LR word OH 94.62 / 94.977; LR word ED 95.55 / 95.757; LR char OH 96.18 / 96.355; LR char ED 96.24 / 96.429; MLP word OH 95.25 / 95.311; MLP word ED 96.15 / 95.354; MLP char OH 95.90 / 95.947; MLP char ED 96.59 / 95.982. The saved code gives MLP char ED a lower AUC than the paper does cv_results.csv.
• Which model is used downstream is not stated. §4.3 says the comparison uses "the best LR model" (LR char ED in tab. 2) §4.3. §5 says it uses "the best personal attack classifier" from §4.2 §5. By tab. 2 that is MLP char ED. The threshold 0.425 is not tied to a named model in the text.
• Threshold code. The threshold 0.425 is hard-coded in the plotting helper ngram.py ngram.py. A dashed marker at 0.408 is drawn in the same function, which is plot-only ngram.py.
• Split (no discrepancy). 3:1:1 is implemented as 0.6, 0.2, 0.2 with random_state 12 Split Annotations.ipynb.
• Search (no discrepancy on draws). The code uses 15 draws cv_ngram_architectures.ipynb. The random search tune trains on train and evaluates on dev as a single split, with no cross-validation ngram.py.

No discrepancies were found in the baseline split size (4,000 random and 4,000 blocked) Split Annotations.ipynb.

Pitfalls

• Label ties. With 10 annotators, a 5-to-5 split gives an attack fraction of exactly 0.5. The loader's one-hot rule is "fraction > 1/n", so both classes become 0 baselines.py. The notebook one-hot uses argmax, so ties go to class 0, "not attack" baselines.py. The paper does not state which rule it used.
• Unstated MLP details. The paper does not give the hidden size, activation, initialisation, optimiser, learning rate, regularisation, batch size or epochs. The code's choices are listed in Hyperparameters deep_learning.py cv_ngram_architectures.ipynb.
• Dev-split selection. Model selection and the reported Table 2 metrics both use the dev split, so Table 2 is optimistic. The test split is used for the threshold and the sanity checks only §4.2 §5.1.
• Threshold transfer. The 0.425 threshold depends on the model and the label type. The paper says a score from an ED model estimates the fraction of annotators who would say "attack", while an OH score estimates a majority probability §4.1. Recompute the threshold for any model you train.
• Sampling mix. The sanity checks use the test split, which combines random and blocked comments §5.1. The text does not say whether the corpus-wide 2015 prevalence is reweighted. The random sample alone had about 1% attacks §3.
• Internal inconsistency. The text says anonymous comments contribute 9.6% of comments §5.2. The anonymity table in §5.2 gives 191,460 of 2,215,019 comments, which is 8.6%.
• Ensemble baseline drops small comments. The reference implementation discards comments with fewer labels than the required total baselines.py. This matters only for rerunning the baseline in tab. 3.
• "As good as 3" is a chosen criterion. At n_p = 5 the ensemble's Spearman (68.27) is above the model's (66.02), while the two AUC values are within 0.06 (97.13 for n_p = 5, 97.19 for the model) tab. 3. The claim rests on comparing the model with n_p = 3.
• Annotator population. The labels come from 4,053 annotators whom the authors know little about, so the labels may not match the Wikipedia community's view of an attack §6. The authors also flag unintended bias §6.
• Agreement is modest. Krippendorff's alpha is 0.45, which the paper says is in line with other crowdsourced toxicity studies §3. Label noise in the training data should be expected.
• Corpus gaps. Comments on revisions removed from the public record are excluded, because the work uses only public data app. A.

Lineage

• Builds on: Nobata et al. (2016), on abusive-language detection, whose finding that character n-grams beat other features the paper confirms §1 §4.1. Warner and Hirschberg (2012) is cited for crowd-labelled abuse data §1.
• Compared against: an annotator-ensemble baseline built from the same labels §4.3. No other published classifier is compared.
• Superseded by: not known from content in this wiki.