RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models

In plain words

Chatbots learn from text written by people, and that text carries stereotypes. A model can then be more likely to write something unfair about one group than about another. Earlier checks for this mostly looked at one kind of bias, used invented sentences, and never asked whether a fix also made the bot worse at its job.

This paper builds RedditBias, a set of real Reddit comments. People read each comment and marked whether it expresses a negative stereotype about a group. The data covers religion, race, gender and queerness. The bias test takes a biased sentence, swaps the group named in it for the other group, and checks whether the language model finds one version more surprising than the other. The same model is then also run on dialog tasks, so that a fix which damages conversation shows up.

The authors apply four ways of reducing bias to DialoGPT, a chatbot model. The original model is clearly biased against one religious pairing. Two of the four fixes remove that bias, and the dialog scores mostly stay close to the original. Some fixes push other groups in the opposite direction, so the bias has to be measured again after every fix.

TL;DR

• A benchmark of Reddit comments and phrases labelled as stereotypical or not for five bias types: two religion pairings, race, gender and queerness §1 §2.1 tab. 3.
• Bias measure: a two-tailed t-test on perplexities of biased phrases against their group-swapped counterparts, after outlier removal §3.1.
• Dialog checks on DialoGPT-small: perplexity on a Reddit reference set, DST F1 on MultiWOZ 2.0 and response BLEU-4 on DSTC-7 §3.2 §5.1.
• Author-reported: DialoGPT is significantly biased on both religion dimensions. Hard debiasing (HD) and counterfactual data augmentation (CDA) remove the bias significantly for both §5.2.
• Official code: MIT (data and measurement); debiasing and dialog training are in a separate Apache-2.0 fork of Hugging Face Transformers §5.1 README.md.

Problem

Biased text in a conversational model can reach people directly, so bias in a dialog system matters more than bias in a general text model §1. Earlier work on conversational bias mostly covered one dimension, mostly gender, used artificial data, and did not measure what debiasing does to dialog quality §1. A debiasing method can also add bias: some fixes push the model toward the opposite stereotype on dimensions where it was not biased to begin with §5.2. The model studied here, DialoGPT, was pretrained on Reddit text after a filter removed offensive phrases from a blacklist; the authors expected less bias from that filtering but tested it rather than assuming it §5.

Method

The paper has three parts: a dataset (§2), an evaluation framework (§3) and four debiasing methods adapted to conversational models (§4). The experiments (§5) apply all of them to DialoGPT-small.

1. Bias specifications (§2.1)

Each bias type has an explicit specification with four term sets: two target sets, one for the minoritised group (T1) and one for the dominant group (T2), and two attribute sets, negative stereotypical attributes (A1) and their positive counterparts (A2) §2.1. The full lists are in app. A; Table 1 gives partial lists tab. 1 and Table 2 gives example comments for each type tab. 2. The five bias types are Religion #1 (Jews against Christians), Religion #2 (Muslims against Christians), Race (African against American), Gender (female against male) and Queerness (LGBTQ against straight) §1 §2.1.

Target lists are short, hand-made sets of near-synonyms that refer to one group without ambiguity. Attribute lists come from sociological literature §2.1. The authors choose small, precise term sets over large noisy ones §2.1.

2. Candidate retrieval (§2.2)

• Queries are the product T1 × A1. Target terms are combined with the correct forms of "to be", because these phrases more often introduce a biased claim §2.2.
• Retrieval uses the Pushshift API over a search period of 3.33 years §2.2.
• Cleaning removes URLs, user names and extra spaces, and lower-cases the text. Only comments shorter than 150 characters are kept §2.2.
• Each candidate also gets a context window of plus or minus seven tokens around the target term, because a window can be biased when the whole comment is not §2.2.

3. Bias annotation (§2.3)

• Three annotators with computer-science degrees annotate each candidate as biased or not, both as a whole comment and as the window phrase §2.3. The guidelines app. B say that questions are unbiased, sarcasm is biased, restatements of another person's view are unbiased, and unclear cases are skipped.
• Calibration rounds refined the guidelines before the main work started §2.3.
• Inter-annotator agreement was measured on 100 random candidates, 20 per bias type, by an extra annotator: Krippendorff's α of 0.65 at comment level and 0.67 at phrase level §2.3.
• Phrase-level labels are split into train, development and test portions §2.3 tab. 3.

4. Language model bias, LMB (§3.1)

LMB asks whether the model finds the biased phrase more likely than its counterfactual. For each biased phrase x_(t1,a1) containing a term from T1 and an attribute from A1, the counterfactual x_hat_(t2,a1) replaces the T1 term with its paired T2 term and keeps the attribute §3.1. The pairs for each bias type are in app. A. LMB is computed on the test split only §3.1 §5.1.

Pairs are removed when either side has an outlier perplexity, i.e. lies outside mean plus or minus 3 standard deviations §3.1. The bias effect is the two-tailed Student t-value over the two ordered sets of perplexities, computed after outlier removal. A negative t-value means the stereotypical direction; significance is judged at α = 0.05 §3.1.

# Language model bias (LMB)  [§3.1]
# Input:  model; biased phrases x and counterfactual phrases x_hat
# Output: t-value (negative = stereotypical bias), p-value

def language_model_bias(model, biased, counterfactual):
    # Perplexity of every phrase under the model; the text does not
    # write the formula, the code uses exp(mean token loss) [§3.1]
    biased_ppl = [perplexity(model, x) for x in biased]
    counter_ppl = [perplexity(model, x) for x in counterfactual]
    # Drop a pair when either side is outside mean +/- 3 std of its own
    # list  [§3.1]
    keep = pairs_within_three_std(biased_ppl, counter_ppl)
    biased_ppl = [biased_ppl[i] for i in keep]
    counter_ppl = [counter_ppl[i] for i in keep]
    # Two-tailed Student t-test on the ordered pairs  [§3.1]
    t_value, p_value = paired_t_test(biased_ppl, counter_ppl)
    return t_value, p_value

5. Dialog performance (§3.2)

The bias score is paired with three measures that check the model still works as a dialog system §3.2:

• Language model perplexity (LMP). Perplexity on 6K Reddit reference utterances from the original DialoGPT evaluation §3.2.
• Dialog state tracking (DST). MultiWOZ 2.0. Each slot-value pair is a binary prediction, so the model receives the previous system output, the user utterance and the domain, slot and value tokens. A feed-forward classifier reads the last input token and is trained with binary cross-entropy §3.2.
• Conversational response generation (CRG). DSTC-7. The model is fine-tuned with causal language modelling on the gold response, using only the dialog context and not the external facts. The multi-reference test set has five human responses per instance §3.2.

6. Debiasing methods (§4)

All methods start from the same term pairs P = {(t1, t2)} between T1 and T2 §4.1 §4.2 §4.3. Three of them add a loss term to the LM loss, and CDA changes the training data §4.

LMD, equalising loss (§4.1). For each position where a term from P occurs, the two terms of a pair should get equal probability. The probability is taken over the two terms only, a reduced vocabulary §4.1. The code computes the two logits as dot products with output embeddings modeling_gpt2.py.

# Equalising loss at one token position (LMD)  [§4.1] [eq. 1]
# Input:  h (d,) hidden state at the position
#         pairs_here: pairs (t1, t2) in P that contain the term here
#         E: output embedding matrix of the LM head
# Output: scalar loss term

def equalising_loss(h, pairs_here, E):
    terms = []
    for (t1, t2) in pairs_here:
        # Probability over the two terms only (reduced vocabulary) [§4.1]
        # Logits are dot products with output embeddings (the code's choice,
        # not stated in the paper)
        logit_1 = dot(h, E[t1])
        logit_2 = dot(h, E[t2])
        p_1, p_2 = softmax([logit_1, logit_2])
        # Zero when the two terms are equally likely [§4.1]
        terms.append(abs(log(p_1 / p_2)))
    # Average over the pairs containing the term [§4.1] [eq. 1]
    return sum(terms) / len(pairs_here)

ADD, attribute distance debiasing (§4.2). Each attribute occurrence should sit at the same cosine distance from both members of each pair. The attribute is represented by its transformed hidden state, and each target by its output embedding. Multi-token terms are averaged over their subwords §4.2 §4.2 fn.

# Attribute distance debiasing (ADD)  [§4.2] [eq. 3]
# Input:  h_a (d,) transformed hidden state of an attribute token a in A1
#         pairs: list of (t1, t2); E: output embedding matrix
# Output: scalar loss term for this attribute occurrence

def attribute_distance_loss(h_a, pairs, E):
    total = 0
    for (t1, t2) in pairs:
        # Gap between the two targets' cosine similarities [§4.2] [eq. 3]
        gap = abs(cosine(E[t1], h_a) - cosine(E[t2], h_a))
        total = total + gap
    return total

HD, hard debiasing (§4.3). Following Bolukbasi et al., a bias subspace is found first and attribute states are then made orthogonal to it §4.3.

# Hard debiasing, step 1: bias subspace  [§4.3]
# Input:  E output embedding matrix; pairs: list of (t1, t2)
# Output: B (d, k), the bias directions

def bias_subspace(E, pairs):
    # Half the difference of each pair's output embeddings [§4.3]
    diffs = [(E[t1] - E[t2]) / 2 for (t1, t2) in pairs]
    # Stack the vectors as rows of C and decompose C = U S V^T [§4.3]
    U, S, V = svd(stack(diffs))
    # Smallest k whose squared singular values reach 50% of the total
    # [§4.3]
    k = smallest_k_reaching(S ** 2, share=0.5)
    return V[:, :k]

# Hard debiasing, step 2: neutralise attribute states  [§4.3] [eq. 4]
# Input:  h_a (d,) hidden state of an attribute token; B (d, k)
# Output: scalar loss term for this occurrence

def neutralisation_loss(h_a, B):
    total = 0
    for j in range(B.shape[1]):
        # Absolute projection of h_a onto bias direction b_j [eq. 4]
        total = total + abs(dot(h_a, B[:, j]))
    return total

The total training loss for the three loss-based methods is a weighted sum of the LM loss and the debiasing loss §4.1 eq. 2.

# Training objective for loss-based debiasing  [§4.1] [eq. 2]
# lambda_lm weighs the causal LM loss; lambda_d weighs the debiasing loss
def total_loss(lm_loss, debias_loss, lambda_lm, lambda_d):
    return lambda_lm * lm_loss + lambda_d * debias_loss

CDA, counterfactual data augmentation (§4.4). Every training utterance that links a T1 term with an A1 attribute is duplicated, and the duplicate has the T1 term swapped for its paired T2 term. The model is then fine-tuned with the plain LM loss on both copies §4.4. Because each matching utterance is copied, the training set is effectively twice as large §5.1 fn.

# Counterfactual data augmentation (CDA)  [§4.4]
# Input:  training utterances; term pairs (t1, t2)
# Output: training set for plain causal LM fine-tuning

def augment_training_set(utterances, pairs):
    augmented = []
    for utterance in utterances:
        augmented.append(utterance)
        # Utterances linking a minoritised term with an attribute [§4.4]
        if links_term_to_attribute(utterance, pairs):
            # Copy it with the term swapped for its pair [§4.4]
            augmented.append(swap_term(utterance, pairs))
    return augmented

7. Experimental setup (§5.1)

The base model is DialoGPT-small: 12 layers, 117M parameters §5.1. The pretraining data is 147M comment-response pairs from 2005 to 2017, with offensive phrases removed by a blacklist §5. Each of the five bias types is run with the original model and with each of the four debiased variants §5.1. Debiasing runs train for two epochs with Adam §5.1. For LMD, ADD and HD, the hyperparameters are chosen on the validation split from a grid §5.1. The downstream DST and CRG models are trained for one epoch §5.1.

Hyperparameters

Results

All numbers are author-reported. Setup: DialoGPT-small; five bias types; the original model and four debiased variants; the test split for LMB §5.1.

• Figure 1 plots LMB t-values per bias type and model, and LM perplexities. The t-values are shown only in the figure, not in the text, so they are not stated here fig. 1.
• The original DialoGPT is significantly biased on Religion #1 and Religion #2. The gender effect has the stereotypical sign but is not significant. Race and queerness effects are not significant and point the other way §5.2.
• For both religion types only HD and CDA remove the bias significantly for both specifications. LMD and ADD each do so for one of the two §5.2.
• For dimensions with no original bias, all four methods tend to produce an anti-stereotypical effect, some significant §5.2.
• LMD raises perplexity on Reddit utterances a lot, but this does not hurt its DST and CRG scores §5.2.
• Two DST drops are noted as needing more study: LMD for race and ADD for gender §5.2 fn.

Table 4 (DST F1, MultiWOZ 2.0) tab. 4

Table 5 (CRG Bleu-4, DSTC-7) tab. 5

Appendix results app. C. These tables have no captions in the source. Column "SexOri" in app. C is the queerness column, labelled "Queer" above.

DST accuracy

METEOR

Entropy-4

Dist-2

The source also has a NIST-2 table, whose rows are identical to the METEOR table; see Pitfalls.

Paper vs code

Two official repositories exist. The paper links the first, umanlp/RedditBias (commit 61f9ae9, MIT), which holds the data, the LMB measurement and the data-preparation scripts §1 README.md. Its README says the debiasing and downstream code is in a second repository, umanlp/redditbias_debias_conv_ai (commit 996c6e1, Apache-2.0, a fork of Hugging Face Transformers) README.md. The second repository was cloned for this card and is cited as redditbias_debias_conv_ai/.... Paths without that prefix are in the first repository. The code_commit field holds the first repository's commit.

• Where the debiasing losses live. The loss-based methods are implemented as custom model heads in the companion fork: LMD and ADD in GPT2DoubleHeadsModelEqualisingLoss and the cosine variant, and HD in GPT2DoubleHeadsModelHardDebiasing modeling_gpt2.py modeling_gpt2.py modeling_gpt2.py.
• LMD fires on the first term only. The paper applies the loss wherever any term from P occurs §4.1. The code loops over pairs, checks only the first element of each pair, and then applies the loss to every non-padding position of that sentence, not only the term's position modeling_gpt2.py modeling_gpt2.py.
• LMD normalisation differs. The paper averages by the number of pairs containing the term at each position §4.1 eq. 1. The code divides the sum for each pair by the number of non-padding tokens, when norm_debias_loss is set, and then divides by the number of pairs modeling_gpt2.py.
• Term pairs differ from the paper. The appendix gives four pairs for Religion #1 app. A. The LMD code uses a list of six token-ID pairs for the same type, and the cosine (ADD) code uses a different list of seven pairs. The LMD comments list word pieces such as "j", "jew" and "christ" modeling_gpt2.py modeling_gpt2.py modeling_gpt2.py. The paper's footnote says multi-token terms are averaged over subwords §4.2 fn; the code uses single token IDs for these pairs.
• ADD matches the paper's formula. The loss is the absolute difference of the two targets' cosine gaps, summed over pairs and attribute occurrences eq. 3 modeling_gpt2.py. For a multi-token attribute the code uses the mean of the span only when the whole span matches, and otherwise a zero vector, which contributes nothing modeling_gpt2.py. The paper does not state the second rule.
• HD bias subspace matches the paper. The code forms (second minus first)/2 for each pair from the LM-head weights, takes an SVD, and keeps the smallest k whose squared singular values exceed 50% of the total modeling_gpt2.py. The paper says "at least" 50%; the code uses a strict comparison. Sign and orientation do not change the subspace.
• HD projection accumulates across occurrences. The running vector word_projection_total is created once, before the loop over sentences, and is never reset. Each attribute occurrence adds its projections to it, and the loss is the sum of absolute values of that running total, which is then added to hd_loss_total modeling_gpt2.py modeling_gpt2.py. The paper's equation 4 is per occurrence and does not describe this running sum. This is the most important gap for anyone reimplementing HD.
• Training defaults mostly match. Two epochs (README), learning rate 5e-5, weight decay 0 and ε 1e-8 agree with the paper §5.1 README.md training_args.py. The Adam β values were not checked. The grid in the code (batch size, accumulation, λ_LM, λ_D) matches the paper's grid exactly debias_lm_grid.py.
• LMB outlier rule. The code removes values outside mean ± 3 standard deviations of each list separately, using NumPy's default population standard deviation measure_bias.py. The paper's interval is written with the bounds reversed; the intended rule is the same.
• LMB outlier removal. Pairs are dropped when either side is an outlier, as in the paper. The code drops rows by matching the perplexity value, so rows with identical perplexities can be dropped too measure_bias.py.
• Which t-test is reported. The paper says a Student t-test on ordered sets §3.1. The code's final line computes a paired ttest_rel on the unfiltered perplexities measure_bias.py. The outlier-filtered tests (Welch unpaired and paired) are printed only in the branch that writes the reduced sets measure_bias.py. The paper does not say which one the figure uses.
• Perplexity definition. The code computes exp of the mean token loss for each phrase, with the tokeniser's tokens and no added start or end tokens helper_functions.py. The paper does not state the formula. If scoring raises an error, the code sets the perplexity to 0 instead of dropping the row measure_bias.py.
• Split sizes. The code keeps 0.75 of phrases for training for gender and 0.6 for the other types reddit_data_text_train_test.py. These fractions agree with the counts in Table 3, but the paper gives only the counts.
• DST and CRG. The DST script uses binary cross-entropy with logits, which matches the paper lm_dst_binary.py. The CRG evaluation samples responses with top-k 50 and a 128-token limit lm_dstc7.py. The paper does not state a decoding method. I did not check the DST and CRG batch sizes or epochs line by line.
• Block size. The paper does not state the block size for debiasing runs. The companion README uses 36 README.md.
• Gradient clipping and schedule. The paper states neither. The companion defaults are a clip of 1.0, zero warm-up steps and linear decay training_args.py trainer.py.
• CDA. Counterfactual training data are produced by preparation scripts rather than by the model code. I did not check them line by line.

Pitfalls

• Equation 1 has an index mismatch. The sum runs over pairs in P_i, but the normalisation is by |P_t|. Reading the text, P_t is the set of pairs containing the term at that position. The code's normalisation does not match either reading (see Paper vs code).
• Which positions get the LMD loss. Applying the loss to every position of a sentence that contains the first term of a pair gives a very different signal from applying it only at term positions. Implement the paper's version, or document which one you use.
• The weights are large. The grid pairs λ_LM of 0.001 or 0.01 with λ_D of 10 to 100 §5.1, so the debiasing term dominates the loss by a wide margin. The selected values are not stated.
• Term lists are not the same across methods. LMD, ADD and HD use different token-ID lists in the code, and none matches the appendix lists. Keep one list per run, and record it.
• Single-token IDs. Several term pairs map to word pieces, not words. Averaging over subwords, which the paper's footnote describes, is not what the code does for these pairs.
• HD running sum. See Paper vs code. Equation 4 implemented per occurrence is not what the code computes. The paper does not say which version produced its HD results.
• Pair alignment. The paired test needs each biased phrase and its counterfactual to stay in the same position. Outlier removal must drop both sides of a pair together, which the code does.
• Zero perplexity as a fallback. A failed phrase gets perplexity 0 in the code, which is never a real perplexity. It enters the t-test unless removed.
• Standard deviation convention. The outlier rule uses the population standard deviation in the code. The paper does not say which is meant.
• The t-test variant is ambiguous. Paired and Welch tests can disagree on significance. The paper's text suggests paired.
• Table 4 and Table 5 are single numbers per cell. The paper states no seed, repetition or variance, so differences of a few hundredths of an F1 point are within unknown noise.
• The NIST-2 table repeats the METEOR table row for row in app. C. This looks like a copy error in the paper. Do not use the NIST-2 values until they are confirmed.
• Anti-stereotypical effects. Debiasing can add bias where the original model had none, and the paper calls this important to measure §5.2. Measure every dimension after every fix.
• DialoGPT-small only. The paper uses one model and one size §5.1. The results say nothing about other chatbots or larger models.
• Race is African against American in the spec. The code's race data files are named for black and white groups race_black.txt. Make sure the pair list you use matches the paper's app. A.
• Gender is binary. The paper says so and names the limitation [Further Ethical Considerations]. Queerness is the paper's route for non-binary identities.
• Annotation agreement is moderate. An agreement of 0.65 to 0.67 on 100 items is a modest check §2.3. Expect label noise in the phrase-level test sets.
• Sample size. The test splits in Table 3 hold 235 to 254 phrases per bias type tab. 3. Expect wide intervals on each bias type.

Lineage

• Builds on: Qian et al. (2019) for the equalising loss §4.1; Lauscher et al. (2020, DebiasNet) for the attribute distance loss §4.2; Bolukbasi et al. for hard debiasing §4.3; Zhao et al. (2018) for counterfactual data augmentation §4.4; DialoGPT, Zhang et al. (2020), as the model under test §5.
• Compared against: the four adapted methods above, with the original DialoGPT as the baseline §5.1. The paper names Liu et al. (2020) as the one earlier study that analysed dialog performance after debiasing §1.
• Superseded by: not known from content already in this wiki.