Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do

In plain words

Language models trained on human text pick up more than facts and grammar. They also absorb a rough sense of what people tend to see as acceptable or unacceptable. The authors look for this sense inside a widely used model, BERT, and find that it is stored in a simple way. Take the model's vector for a sentence, and one direction in that vector space separates "good to do" phrases from "bad to do" ones.

Once you have that direction, you can score any phrase by where it falls along it. The score agrees reasonably well with what study participants said about the same actions. The authors then use the same score while a text generator writes, so that options the score marks as unacceptable are dropped before the next word is picked. That reduces the amount of toxic text the generator produces.

The model does not understand morality, and its scores are not advice. They reflect the text it was trained on, and the paper says so directly.

TL;DR

• BERT-large sentence embeddings contain one main direction, the first principal component (PC1) of embeddings for 64 verbs placed in question templates § Identifying the Moral Direction of Language Models [suppl. B.2].
• Projecting a phrase on that direction gives a moral score. Its correlation with human yes/no ratings is r = 0.78 and 0.79 (author-reported) § BERT's MoralDirection Strongly Correlates with Human Moral Norms.
• In GPT-2 decoding, candidates whose extended text scores at or below a threshold t are removed, with at least m kept. This lowers toxicity on RealToxicityPrompts, e.g. only 1.17% of prompts get a generation above toxicity 0.9 with t = 0 versus 9.82% for GPT-2 (author-reported) § Reducing Neural Toxic Degeneration in Language Models.
• Check before use: the paper's equations differ from the released code in centring, the sign of the score, and the default values for the number of kept tokens (see Paper vs code).

Problem

• Earlier work scored moral norms with question-answer pairs: the cosine similarity of a question to a "yes" answer minus its similarity to a "no" answer § Asking the Language Model for its Moral Score. The authors note this is applied to atomic actions and does not show how actions with added context are handled § Asking the Language Model for its Moral Score.
• Human judgements change when context is added to an action. In the regional study, a context-modified action was rated differently from its bare verb § Contextual Influence in Human Moral Judgements: A User Study. Scoring bare verbs therefore misses part of the signal.
• Large language models trained on unfiltered web text produce toxic text ("toxic degeneration"). Data-based fixes such as DAPT need further pre-training on a filtered corpus, and decoding-based PPLM needs a discriminator trained for each generator § Differences between the MoralDirection approach and related methods. The authors aim for a method that works with a fixed sentence encoder and needs only a few example verbs § Differences between the MoralDirection approach and related methods.

Method

The paper has three parts: a human user study, a method to extract a direction from BERT, and a decoding filter that uses it. The user study collected yes/no answers to moral questions from 29 students in a controlled setting, and from 234 participants on Amazon Mechanical Turk (AMT) § Contextual Influence in Human Moral Judgements: A User Study § Details on participant recruitment and study procedure.

Step 1: verbs and question frames. Choose a set of verbs with known positive or negative moral associations. Put each verb into each of 10 question frames, such as "Should I [verb]?" or "Is it ok to [verb]?" § Identifying the Moral Direction of Language Models [suppl. B.1]. The verbs are listed in [suppl. B.2] and their number is discussed in Paper vs code.

Step 2: embed and average. Encode each filled-in question with a sentence encoder. The paper uses BERT-large with the pooling mechanism of SBERT § Guiding Generative Language Models using the MoralDirection. The code loads the checkpoint bert-large-nli-mean-tokens pca.py. Average the embeddings of the 10 frames for each verb § Methods: The MoralDirection of language models.

Step 3: PCA. Fit a principal component analysis on the averaged verb embeddings. The first component is the moral direction m, a unit vector § Identifying the Moral Direction of Language Models. The authors report that PC1 explains 25.64% of the variance for BERT, against 12.11% for the first component of the Universal Sentence Encoder § Identifying the Moral Direction of Language Models [suppl. B.4].

Step 4: score a phrase. The score of a phrase is its PC1 coordinate, normalised to [-1, 1] for comparison eq. 1. A score of -1 marks a strongly non-normative phrase and +1 a strongly normative one § Moral Norms Contained in Pre-trained Language Models. In the user-study comparison, scores are divided by the largest absolute score, which belongs to "killing people" § BERT's MoralDirection Strongly Correlates with Human Moral Norms.

Step 5: validate. Correlate the scores with the mean human yes-rate for each question using Pearson's r eq. 2. The paper reports r = 0.78 and 0.79 for BERT (the text gives them as "resp.", and the regional and AMT panels are Figures 2b and 2c). With three randomly sampled verb sets, the correlation was lower in two cases (r = 0.64 and 0.60) and absent in the third (r = -0.01) [suppl. B.2].

Step 6: filter decoding. Generate with GPT-2 using top-k and then top-p sampling § Guiding Generative Language Models using the MoralDirection. For each surviving candidate token, score the whole text, the prefix plus that token § Guiding Generative Language Models using the MoralDirection. Remove candidates whose score is at or below the threshold t. If fewer than m candidates survive, keep the m best by score § Guiding Generative Language Models using the MoralDirection mcm_models.py.

The paper's question-answer baseline is the score in eq. 3, averaged over the same kind of question frames § Asking the Language Model for its Moral Score. It is used for comparison only.

\text{score}(\mathbf{u}, \mathbf{m}) = t^{(1)} = \mathbf{u} \times \mathbf{m} eq. 1

r = \frac{\sum_i (x_i - m_x)(y_i - m_y)}{\sqrt{\sum_i (x_i - m_x)^2}\,\sqrt{\sum_i (y_i - m_y)^2}} eq. 2

Eq. 2 is the standard Pearson form. The paper prints the denominator with the squares inside a single sum; implement the form above.

\text{score}(\mathbf{u}, \mathbf{a}, \mathbf{b}) = \cos(\mathbf{a}, \mathbf{u}) - \cos(\mathbf{b}, \mathbf{u}) eq. 3

Build the direction. The paper has no algorithm box, so this follows the Methods text and pca.py.

# Build the moral direction m  [§ Identifying the Moral Direction]
# Input:  verbs (list of str); frames (10 question templates)  [suppl. B.1]
# Output: m (d,) unit vector; sign (+1 or -1)

def build_moral_direction(verbs, frames):
    # Average each verb's embedding over all frames
    # [§ Methods: The MoralDirection of language models]
    # [code: MoRT/mort/pca.py#L95-L105]
    verb_embeddings = [
        mean([embed(frame.format(verb)) for frame in frames])
        for verb in verbs
    ]
    # Fit PCA on the averaged embeddings  [eq. 1]
    # [code: MoRT/mort/pca.py#L107-L110]
    pca = fit_pca(verb_embeddings, n_components=5)
    # Keep the first principal component as the direction  [eq. 1]
    m = pca.components[0]
    # Choose the sign from a two-cluster split of the verbs
    # [code: MoRT/mort/pca.py#L180-L181]
    sign = sign_of_cluster_without_kill(pca, verbs)
    return m, sign

Score a phrase and use it to filter candidates in decoding. Scores are normalised by a constant in the generation code mcm_models.py.

# Score a phrase on the moral direction  [eq. 1]
# Input:  phrase (str); pca (fitted PCA with mean and components)
# Input:  sign (+1 or -1); norm (float, a constant)
# Output: float, positive = normative

def moral_score(phrase, pca, sign, norm):
    # Embed the raw phrase, without a question frame
    # [code: MoRT/mcm_textgeneration/mcm_models.py#L277]
    embedding = embed(phrase)                   # (d,)
    # Centre on the PCA mean, then project onto PC1
    # [code: MoRT/mort/funcs_mcm.py#L267-L273]
    projected = dot(embedding - pca.mean, pca.components[0])
    # Apply the stored sign, then divide by the normalisation constant
    # [code: MoRT/mort/funcs_mcm.py#L323-L325]
    return sign * projected / norm

# Filter next-token candidates by moral score
# [§ Guiding Generative Language Models using the MoralDirection]
# Input:  logits (V,) for the next token; prefix (str)
# Input:  pca, sign, norm (as in moral_score)
# Input:  t (threshold); m (minimum number of tokens to keep)
# Output: list of token ids that remain as candidates

def filter_next_tokens(logits, prefix, t, m):
    # Apply top-k, then top-p, as in ordinary sampling
    # k = 50 from the run scripts; p = 0.9 from the code (not in the paper)
    # [code: realtoxicityprompts/generation/generation.py#L281]
    candidates = top_k_then_top_p(logits, k=50, p=0.9)
    # Score the whole text with each candidate appended
    # [code: MoRT/mcm_textgeneration/mcm_models.py#L439-L444]
    scores = [
        moral_score(prefix + text_of(c), pca, sign, norm)
        for c in candidates
    ]
    # Remove candidates at or below the threshold
    # [code: MoRT/mcm_textgeneration/mcm_models.py#L448]
    kept = [c for c, s in zip(candidates, scores) if s > t]
    # Keep the best m if too few survive
    # [code: MoRT/mcm_textgeneration/mcm_models.py#L450-L452]
    if len(kept) <= m:
        kept = top_by_score(candidates, scores, m)
    return kept

Hyperparameters

Results

Author-reported. Setup: BERT-large with SBERT pooling, scored against human yes-rates from the regional and AMT studies § BERT's MoralDirection Strongly Correlates with Human Moral Norms. GPT-2 decoding on RealToxicityPrompts, using the testbed's Perspective API toxicity scores § Testbed for evaluating the toxicity of text generations. Hardware is not stated.

Table 3: toxicity, full testbed (100,000 prompts, except PPLM with 10,000). The paper marks the best value with • and the runner-up with ∘ in each column. Columns: expected maximum toxicity (Exp. Max. Tox.) and toxicity probability (Tox. Prob.), each split into unprompted, toxic-prompted and non-toxic-prompted. Values in parentheses are standard deviations. tab. 3

† Recomputed by the authors from data provided by the testbed tab. 3.

Table 1: moral direction versus human scores (values in [-1, 1]). The LAMA top-three column in the source table is omitted here. Human columns are the average of yes/no answers rescaled to [-1, 1], from the regional and AMT studies. tab. 1

Paper vs code

Code: https://github.com/ml-research/MoRT_NMI at commit 48c2e495154ebc48abb7267a2409954ee4c93347 (MIT for the top-level repository). The repo also contains forks of LAMA (CC BY-NC) and RealToxicityPrompts (Apache-2.0); this card cites only the MoRT and RealToxicityPrompts paths and nothing from LAMA. The paper's code availability statement links this repository, so no further search was needed.

• Verb count. The main text says 54 verbs for the PCA § Methods: The MoralDirection of language models. The supplement lists 64 [suppl. B.2], and the code uses the same 64 dataMoral.py pca.py. The code agrees with the supplement.
• Centring. Eq. 1 is a plain dot product eq. 1. The code uses scikit-learn's PCA transform, which subtracts the fitted mean before projecting funcs_mcm.py funcs_mcm.py.
• Sign. Eq. 1 has no sign convention. The code multiplies the score by moral_score_sign, which defaults to -1 when the saved projection has no sign key funcs_mcm.py. When pca.py creates the projection, it takes the sign of PC1 for the verb cluster that does not contain "kill" pca.py pca.py. I did not unpickle the shipped projection file under MoRT/data/, so I could not confirm whether it carries a sign key.
• Keep-at-least. The paper uses m = 5 § Experimental setup. The GPT-2 class defaults to 30 gpt2MCM_generation.py, and the CLI default is also 30 run_prompts_experiment.py. The shipped run scripts pass --mcm-keepmin 5 run_mcm_parallel-k50-keep5.sh, so the reported runs match the paper. Anyone calling the defaults gets 30.
• Top-p and max length. The paper gives no top-p value and no maximum generation length. The code sets p = 0.9 in the wrapper generation.py and a default of 20 new tokens run_prompts_experiment.py.
• Normalisation. User-study scores are normalised by the largest absolute score § BERT's MoralDirection Strongly Correlates with Human Moral Norms. Generation divides by the fixed constant 8.946814 mcm_models.py. Neither the paper nor the code I read explains where that constant comes from.
• Scoring input in generation. The user-study scores average over the 10 frames funcs_mcm.py. The generation filter scores the raw text with no frame mcm_models.py. The supplement reports that querying raw actions did not change the correlation [suppl. B.5], so this is a choice, not a contradiction.
• Consistent with the paper. The three thresholds (t = -0.5, 0, 0.5) and n = 10 samples per prompt match the scripts run_mcm_parallel-k50-keep5-t0.5.sh run_mcm_eos_50-5.sh. The filter rates the full prefix plus candidate, as the paper describes mcm_models.py.

No discrepancies found for the threshold and sample-count settings.

Pitfalls

• Unnumbered sections. The LaTeX source uses unnumbered section headings, so citations use section titles. Equation numbers follow source order (1 = moral score, 2 = Pearson, 3 = question-answer score). Figure 1 is the first figure in the main text, Figure 2 the MC approach, and Figure 3 the toxicity curves. Table 3's label is tab:toxicity_results_1, but it is numbered third in source order. I did not check these against a rendered PDF.
• Extended Data numbering. The text refers to "Extended Data Figure 2" and "Extended Data Figure 3", which do not match the order of the figure environments in the source. The Extended Data figures are cited by section here.
• Sign and polarity. Before using any threshold, check the polarity of the scores. In Table 1, "killing people" is the most negative value (-1.0) and "helping" is positive (0.5). A sign flip reverses every decision.
• Threshold scale. The generation thresholds t are only meaningful in the scale produced by the constant 8.946814. A different encoder, a different PCA fit or different verbs would need re-tuning. The paper gives no tuning procedure.
• Cost of the filter. Per decoding step, the code decodes the full text for each candidate and embeds it with BERT-large mcm_models.py. Up to k = 50 candidates are embedded at each step, and the whole prefix is re-embedded each time. The paper reports no runtime or memory cost.
• Single direction. PC1 explains 25.64% of the variance. The authors say PC2 and PC3 do not correlate with the human scores [suppl. B.4], but they do not show the other components in full.
• Verb choice. The verbs were taken from earlier work's Dos and Don'ts lists § Methods: The MoralDirection of language models. Random verb sets gave r = -0.01, 0.60 and 0.64 [suppl. B.2], so the result depends on which verbs are used.
• Correlation sample size. The paper does not state how many items each Pearson correlation uses.
• Pearson formula. The printed Pearson formula puts the squares inside a single sum. Use the standard form given in eq. 2.
• Training data bias. The authors note that BERT was trained mainly on English books and Wikipedia, and that the direction may reflect English-speaking culture of the 21st century § Identifying the Moral Direction of Language Models. GPT-2 was trained on WebText, which is built from Reddit-linked pages [Methods: Transformer based language models]. Both inherit biases from their data, and the paper says so.
• Rater limits. The toxicity scores come from the Perspective API. The authors say such tools are imperfect and biased § Testbed for evaluating the toxicity of text generations.
• Participants. The regional study had 29 students. Of 282 AMT volunteers, 234 were kept after removing those who failed control questions or answered almost all the same way. Africa and Oceania are underrepresented § Details on participant recruitment and study procedure.
• GPT-3 result. GPT-3 was reached through a paid API only, and the result uses two example answers and binarised human scores as gold labels § Methods: GPT-3 paragraph. It is not reproducible from the paper alone.
• Table 3 caption. One caption calls the method "our MC" while the rows say "MD". They refer to the same method.
• Intended use. The paper says the model's outputs do not reflect the authors' views, that the model does not understand morality, and that its scores should not be used as advice § Pre-trained Language Models, and the Sense of Right and Wrong. Keep that limit in any product that uses the score.

Lineage

• Builds on: the question-answer moral score of Jentzsch et al. and Schramowski et al. (2020), which this paper replaces for arbitrary phrases § Asking the Language Model for its Moral Score; BERT (Devlin et al., 2018) and SBERT (Reimers and Gurevych, 2019) for sentence embeddings § Guiding Generative Language Models using the MoralDirection; LAMA (Petroni et al., 2019) for the masked-word probe § Generating (Moral) Values with LAMA; Bolukbasi et al. (2016) for the idea of a direction from a principal component § Identifying the Moral Direction of Language Models; GPT-2 (Radford et al., 2019) and RealToxicityPrompts (Gehman et al., 2020) for the generation testbed § Testbed for evaluating the toxicity of text generations.
• Compared against: DAPT (Gururangan et al., 2020), PPLM (Dathathri et al., 2020), WORD FILTER, VOCAB-SHIFT and ATCON, as reported in the testbed tab. 3.
• Superseded by: not stated.