Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

In plain words

Many datasets are labelled by several people, and they often disagree. The usual fix is to take a majority vote and train on that single label. This paper argues that the vote throws away useful information, such as which annotator tends to label what, and that the disagreement itself says something about how hard a text is to judge.

The authors instead train one model that has a separate small output for each annotator. All outputs share the same underlying text representation. Each output learns only from the texts its annotator labelled, so the shared layers still learn from the whole dataset. To get a final answer, the model takes a vote over all of its annotator outputs.

The same set of outputs also gives a rough measure of uncertainty. When the outputs split evenly, the model is unsure. The authors report that this measure tracks real annotator disagreement better than the usual confidence scores, and that it is higher on the model's wrong predictions. On seven binary tasks, the approach was as good as or better than training on majority labels, with the caveat that the gains depend on how many labels each annotator contributed.

TL;DR

• One shared BERT-base encoder with one linear output head per annotator; each head is trained only on the instances that annotator labelled §3.4 §4.2.
• The final label is the majority vote over all heads at prediction time §3.4 §4.3.1.
• Uncertainty is the variance of the head predictions, using the same formula as annotator disagreement §4.3.2 eq. 1.
• Author-reported: on Gab Hate Corpus (GHC) the multi-task model reaches majority-vote F1 59.49 against 57.32 for the majority-label baseline tab. 1.
• On GoEmotions the multi-task model is worse on four of six emotions in the full dataset, and better on four of six in a 53-annotator subset tab. 3.
• Code for the GHC experiments is in the first author's repository, which the paper does not link. It has no license file and no data file classifier.py.

Problem

Majority voting and averaging turn several annotations into one label. The authors argue that on subjective tasks such as hate speech and emotion this erases annotators' systematic differences and disagreement, and that a single label can be inconsistent across similar texts §1. Their setting assumes the dataset includes annotator-level labels for each instance, with enough labels per annotator to learn from §6.2. Majority-label training is the baseline they compare against §3.1.

Method

Setup §3: the data is a set of texts X, a set of annotators A, and an annotation matrix Y in which many entries are missing, because each annotator labels only some instances. The target is the majority vote over the labels present for each instance, which the authors take as the most common aggregation choice §3. Every variant uses BERT-base; the representation of each instance has size 768 §3. Four variants are compared:

1. Baseline (majority label) §3.1: one linear layer with a softmax, trained on the majority label.
2. Ensemble §3.2: one single-task classifier per annotator, each trained on that annotator's labels; at test time the models vote.
3. Multi-label §3.3: one linear layer with |A| sigmoid outputs; the loss uses only the labels that are present, and all outputs are aggregated at test time.
4. Multi-task §3.4: one shared representation and one softmax head per annotator. The loss is the sum over all available labels of an instance. At test time the outputs of all heads are combined into the majority label.

Uncertainty §4.3.2 eq. 1: for an instance, the authors compute the variance of the annotator labels, which is the disagreement measure. For the three multi-annotator models the same formula is applied to the predicted labels of the heads. For the baseline, the comparison uses softmax probability and Monte Carlo dropout instead §4.3.2.

\sigma^2(\bar{y}_{i,}) = \frac{\sum[y_{ij} = 1]\sum[y_{ij} = 0]}{|\bar{y}_{i,}|^2} eq. 1

Because the formula is a product of two counts over the square of their total, it lies between 0 and 0.25 §5.2.

Training loop for the multi-task model, from the paper's description §3.4 and the released GHC code classifier.py:

# Multi-task training: one shared encoder, one head per annotator  [§3.4]
# Input:  texts; labels per instance and annotator (0/1 or missing)
# Output: encoder and one classification head per annotator

def train_multi_task(texts, labels, annotators, learning_rate, epochs):
    # 1. Pretrained BERT-base encoder, and a linear head per annotator
    #    [§3] [§3.4] [code: code/classifier.py#L27-L35]
    encoder = load_bert_base()
    heads = {a: linear_layer(hidden_size, 2) for a in annotators}
    # 2. One optimizer for encoder and heads; the paper says Adam [§4.2]
    #    [code: code/classifier.py#L215]
    optimizer = adamw(encoder_and_heads(encoder, heads), lr=learning_rate)
    for epoch in range(epochs):
        # 3. Batches are fixed for the whole run
        #    [code: code/classifier.py#L397]
        for batch in fixed_batches:
            # 4. A batch with no labels at all is skipped
            #    [code: code/classifier.py#L231]
            if no_labels(batch):
                continue
            # 5. One vector per instance: the first token's last hidden state
            #    [code: code/classifier.py#L43]
            vectors = encoder(batch.tokens)[:, 0, :]
            loss = 0
            for a in annotators:
                # 6. Each head learns only from instances a labelled  [§3.4]
                #    [code: code/classifier.py#L238-L247]
                logits = heads[a](vectors[labelled_by(batch, a)])
                # 7. Class-weighted cross-entropy, summed over heads
                #    [code: code/classifier.py#L190-L208]
                #    [code: code/classifier.py#L257]
                loss = loss + weighted_cross_entropy(logits, labels_for(a))
            # 8. One optimizer step per batch
            #    [code: code/classifier.py#L259-L260]
            loss.backward()
            optimizer.step()

Prediction and uncertainty:

# Majority label and predicted uncertainty from all heads  [§3.4] [§4.3.2]
# Input:  one text, trained encoder and all annotator heads
# Output: majority label (0 or 1) and uncertainty in [0, 0.25]

def predict_with_uncertainty(text, encoder, heads, annotators):
    # 1. Shared representation of the text  [code: code/classifier.py#L43]
    vector = encoder(text.tokens)[0]
    # 2. Every annotator head gives one 0/1 label  [§3.4]
    #    [code: code/classifier.py#L59]
    head_labels = [heads[a](vector).argmax() for a in annotators]
    # 3. Majority: a share of 0.5 or more counts as positive  [§4.3.1]
    #    [code: code/classifier.py#L341]
    share = sum(head_labels) / len(head_labels)
    majority = 1 if share >= 0.5 else 0
    # 4. Uncertainty: variance formula of [eq. 1] applied to the head labels
    positives = sum(head_labels)
    negatives = len(head_labels) - positives
    uncertainty = positives * negatives / len(head_labels) ** 2
    return majority, uncertainty

Hyperparameters

Results

Author-reported. Setup: BERT-base; GHC with stratified 5-fold cross-validation repeated 5 times; GoEmotions with its pre-defined splits, repeated 5 times, one binary task per emotion, seven tasks in total §4.2 §6. Baseline is the majority-label model §3.1. Values are mean ± standard deviation across the repeats tab. 1 tab. 3.

tab. 1 GHC prediction, mean ± standard deviation over 5 iterations:

"Majority" columns compare the majority of predictions with the majority of annotations. "Individual" columns compare each predicted head label with the annotator's own label. The paper says these two tasks are not directly comparable §4.3.1 tab. 1.

tab. 2 Training time for one cross-validation fold on GHC, which is 80% of the data, in minutes: Baseline 20.5; Ensemble 158.4; Multi-label 22.8; Multi-task 22.3 tab. 2. On GoEmotions the paper reports 6.1 minutes per epoch for the multi-task model and 5.2 for the baseline §4.4.

tab. 3 GoEmotions F1, mean ± standard deviation over 5 iterations. Full dataset uses 82 annotators; subset uses 53 annotators with more than 1000 annotations:

Headline numbers, as the paper states them:

• GHC majority-vote F1: multi-task 59.49 against baseline 57.32 §4.3.1 tab. 1.
• GHC individual-label F1: multi-task 63.20, the highest of the three multi-annotator models §4.3.1 tab. 1.
• Uncertainty on GHC: the three multi-annotator methods correlate better with annotator disagreement than softmax and MC dropout. The ensemble is slightly highest. Multi-task and multi-label correlate with each other at 0.86 §4.3.2 fig. 2 fig. 3.
• Uncertainty on GoEmotions: the multi-task correlation is higher than softmax and MC dropout for all six emotions, in both the full and subset data, except joy in the full data §4.4 fig. 4.
• Errors on GHC: the multi-task and baseline models disagree on 1,945 instances. In 57.94% of these the multi-task model is right and the baseline wrong; in 22.31% the reverse §5.1 tab. 4.
• Uncertainty and error on GHC: mean uncertainty is 0.049 for correct predictions and 0.170 for incorrect ones §5.2 fig. 5.

Paper vs code

The first author's repository, aidamd/dealing-with-disagreement, is not linked anywhere in the paper's text. We use it because its owner is the first author, Aida Mostafazadeh Davani, and the repository's name matches the title. This attribution comes from the GitHub owner name, not from a link in the paper. It contains three Python files and no README, LICENSE or requirements file. Commit 2f0f3488 is the head of the shallow clone. The code covers the GHC experiments only: the GoEmotions experiments and the uncertainty analysis are not in the repository. The GHC CSV the script reads (data/GHC/ghc_multi.csv) is not included run_classifier.py.

Differences between paper and code:

• Loss. The paper sums the losses of the available labels §3.4. The code computes a class-weighted cross-entropy for each head over that head's instances in the batch, then sums the heads classifier.py classifier.py.
• Optimizer. The paper says Adam §4.2; the code uses AdamW classifier.py.
• Validation holdout. Not in the paper. The code holds out 10% of training batches for per-epoch scores; these are excluded from training and do not select the model classifier.py.
• Class weights. Not in the paper. The code computes them from the whole dataset, including the test fold classifier.py.
• Batch and sequence settings. Not in the paper. The code uses batches of four and a maximum of 128 tokens params.py params.py.
• Repetitions. The paper reports 5 iterations with different random states §4.2. The code runs one seed per invocation and has no loop over seeds params.py run_classifier.py.
• Multi-label head. The paper describes one head with |A| outputs §3.3. The code builds that head, but maps its outputs to annotators by the position labels "0" to "N-1" classifier.py. Whether this lines up with the annotator columns depends on how they are named, which the repository does not show. We did not check this path end to end.
• Monte Carlo dropout. The code runs 10 passes with dropout enabled classifier.py params.py, but only when predict is set to mc classifier.py. The default setting does not compute it. The paper gives no pass count §4.3.2.
• Predicted uncertainty. The uncertainty formula in the code is applied to the annotator labels, which is the disagreement measure classifier.py. The code that computes uncertainty from the head predictions, and the correlation analysis, are not in the repository.
• Majority vote ties. A share of exactly 0.5 counts as positive in both the label and the prediction majority classifier.py classifier.py. The paper does not state a tie rule.
• Unlicensed code. The repository has no license file. We cite its lines and paraphrase them; we do not quote them.

Pitfalls

• Ties and missing labels. An instance labelled by an even number of annotators can split exactly 50/50. The code then labels it positive. The paper does not state this rule classifier.py.
• Batches are fixed. Instances are split into batches of four in file order, once per fold. The same batches are used in every epoch. The batch order is shuffled only for the validation split classifier.py classifier.py. Reproducing the paper's numbers needs the same ordering of the input CSV.
• Training data is reduced. The 10% validation split is taken from the training folds, so the model sees less data than the fold size suggests.
• Class weights see the whole dataset. The weights are computed from all rows, not only the training fold classifier.py. This is a leak in label counts, though it is small and not quantified in the paper.
• Heads with few positives. Some GoEmotions annotator heads had no positive instances in the training set, and the paper calls learning on them "hard or even impossible" §4.4. The code gives a single-class head the weights [0.01, 1] classifier.py. A head with few labels can pull the vote, and the paper suggests choosing only top-performing heads as future work §6.2.
• Learning rate. The GHC rate is 1e-7 and the GoEmotions rate is 5e-6, a difference of fifty times §4.2. The paper does not say why they differ.
• "Significantly" without a test. The paper calls the GHC F1 gap of 59.49 against 57.32 "significantly higher" §4.3.1, and calls the ensemble result "significantly worse" §4.3.1. No statistical test is reported. The standard deviations in tab. 1 are across five iterations.
• Software versions. The paper used transformers 3.1 §4.2. The code calls DataFrame.append classifier.py and imports AdamW from transformers classifier.py. DataFrame.append was removed in pandas 2.0, and newer transformers releases deprecate that AdamW in favour of torch.optim.AdamW. Pin the older versions, or change the calls.
• Hardware is fixed to CUDA. The code sends the model to cuda classifier.py.
• Costs hidden in the ensemble. The ensemble trains one model per annotator and took 158.4 minutes per fold against 22.3 for the multi-task model tab. 2. The ensemble also gets less data per model.
• Subset selection. The GoEmotions subset (53 annotators with more than 1000 annotations) is introduced after the full-data results, to make the comparison "fairer" §4.4. Report both.
• Uncertainty scope. The authors show the uncertainty measure on GHC and on GoEmotions. The evidence for the error link comes from GHC only §5.2.
• Limits stated by the authors. Only BERT-base was tested §3. The approach needs annotator-level labels, and it becomes costly with many annotators; the paper suggests clustering annotators as a route to this §6.2.

Lineage

• Builds on: the multi-task framework, which the paper cites for the multi-task setup §1. The reference is not resolved in this card.
• Compared against: majority-label training, the ensemble of per-annotator models, multi-label classification, softmax uncertainty and MC dropout §3 §4.3.
• Named as alternatives for aggregation: MACE and Bayesian aggregation, as future work §6.1.
• Future directions named by the authors: clustering annotators to reduce the number of heads, and post-processing that selects top-performing heads §6.2.