The Moral Foundations Reddit Corpus

In plain words

People often frame arguments in moral terms: care, fairness, loyalty, respect for authority, or purity. This paper builds a labelled collection of Reddit comments that marks which of those moral ideas each comment expresses. The comments come from twelve subreddits, some about US politics, some about French politics, and some about everyday life.

Several people read each comment and marked it, and a comment counts as showing a moral idea when most of its readers agreed. The collection is meant to be used as a test set and a training set for software that detects moral language. The authors also ran standard text classifiers on it. The best ones were BERT models trained fully on the data, and they beat the language models that were only prompted or lightly tuned.

TL;DR

• A 16,123-comment Reddit dataset, labelled for the revised Moral Foundations Theory: six foundations, Thin Morality, and explicit or implicit expression.
• Labels are majority votes of at least three trained annotators; the annotator demographics are available on request.
• Author-reported baselines: fully fine-tuned BERT gives the best F1 on all seven predicted categories. LoRA-tuned Llama-3-8B comes next, and zero- or few-shot prompting is weakest.
• Use it to benchmark moral-sentiment classifiers. Do not use it to judge moral reasoning: the authors state that it does not evaluate that app. D.

Problem

Detecting moral sentiment in text is subjective, so it needs large, hand-labelled datasets §1. The existing Moral Foundations Twitter Corpus (MFTC) comes from one platform, and the authors argue that a corpus from a second platform, with a broader set of topics and a revised framework, is needed §1. The revision splits the Fairness foundation into Equality and Proportionality §1, §3.1.

Method

Corpus construction §2

• Source: English Reddit comments from 12 subreddits in three buckets §2:
  • US politics: r/politics, r/Conservative, r/antiwork; dated 1 Jan 2020 to 31 Jan 2021.
  • Everyday moral life: r/nostalgia, r/AmItheAsshole, r/confession, r/relationshipadvice; dated 1 Jan 2020 to 31 Jan 2021.
  • French politics: r/europe, r/worldnews, r/neoliberal, r/geopolitics, r/Conservative; dated 1 Jan 2017 to 30 Jun 2017.
• Filters §2.1:
  • at least 10 tokens;
  • drop posts that Reddit marks as bot posts (the text "I am a bot" at the end);
  • US and everyday buckets: comment score at least 10;
  • French bucket: the comment must mention a French presidential candidate (the list is in §2.1) and have at least 10 likes or comments §2.
• Up-sampling of moral comments §2.1: a DDR-style moral loading score is computed for each comment, using word embeddings and moral-foundation seed words. For each moral concern, the 95th percentile of the score marks the "highly moral" comments. Each bucket is half comments above the 95th percentile and half comments at or below it.
• Duplicates are removed after sampling. Counts: US politics 4,821; French politics 6,489; everyday 4,813; total 16,123 §2.1.

# Semi-supervised up-sampling of moral comments  [§2.1]
# Input: filtered comments of one bucket, with a moral loading score per concern
# Output: a sample that is half high-loading and half lower-loading

def sample_bucket(comments, loading, bucket_size):
    # Bin size is not stated in the paper  [§2.1]
    for concern in moral_concerns:
        # Mark high-loading comments above the concern's 95th percentile  [§2.1]
        threshold = percentile(loading[concern], 95)
        high = [c for c in comments if loading[concern][c] > threshold]
        low = [c for c in comments if loading[concern][c] <= threshold]
    # Half of each bucket from each group, across concerns  [§2.1]
    return sample(high, bucket_size // 2) + sample(low, bucket_size // 2)

Annotation scheme §3, app. A

• Categories. Six foundations: Care/Harm, Equality/Inequality, Proportionality/Disproportionality, Loyalty/Betrayal, Authority/Subversion and Purity/Degradation. Each foundation has a virtue and a vice pole §3.1. The paper adds Thin Morality, a moral judgement that does not clearly fit one foundation. That makes seven moral-sentiment categories §3.1. The paper also has an Implicit/Explicit label and a three-level confidence rating (not confident, somewhat confident, very confident) §3.1.
• Procedure app. A, flowchart in the appendix: annotators work through four steps in order: (1) check for thick morality, meaning a moral judgement tied to at least one foundation; (2) check for thin morality; (3) mark the comment as non-moral if neither applies; (4) rate confidence. Annotators label only the comment text, not the original post app. A.
• Annotators. A pool of 27 undergraduate research assistants trained for two months; the top five by reliability and commitment became the primary annotators §3.2. Each comment is labelled by at least three of the five §3. Labelling was done in Prodigy §3.2.
• Majority vote. A label is positive when at least 50% of the annotators who coded that comment marked it §4.

# Majority-vote label for one comment and one category  [§4]
# Input: votes, one 0/1 entry per annotator who coded the comment (>= 3)
# Output: 1 if the category is present, else 0

def majority_label(votes):
    # Positive when at least 50% of the coders marked the category  [§4]
    share = sum(votes) / len(votes)
    return 1 if share >= 0.5 else 0

Annotator metadata §3.2

Annotators reported gender, sexual orientation, age, household income, first language, political ideology, religion and moral values (Moral Foundations Questionnaire-2). The authors report that the annotators skew liberal and that family income skews above the US average. The metadata is provided on request, for privacy reasons.

Baseline models §6

• Single-label and multi-label. Single-label treats each category as independent. Multi-label predicts all categories at once.
• Zero-shot and few-shot prompting. Meta-Llama-3-8B and Ministral-8B-Instruct-2410 §6. Few-shot adds five stratified examples to the prompt. Prompt wording is not given in the main text; the definition prompt used for the personalisation study is in app. B.
• LoRA fine-tuning. LoRA parameter-efficient fine-tuning of Llama §1, §6.
• Full fine-tuning. BERT, because it is smaller §6.
• Weighted loss §6, eq. 1:

w_{i,l} \propto \frac{1}{N_l} eq. 1

where N_l is the number of samples with label l. The sample-level scope and the constant are not stated.

# Inverse-frequency label weights  [eq. 1] [§6]
# Input: labels (N, L), 0/1 per comment and category
# Output: weights (L,), defined up to a constant

def label_weights(labels):
    # N_l: number of samples carrying label l  [eq. 1]
    counts = labels.sum(axis=0)               # (L,)
    # Proportional to 1 / N_l; whether applied to positives only is not stated
    return 1.0 / counts                       # (L,)

Evaluation protocol §6

# Cross-validated baseline evaluation  [§6]
# Input: labelled comments and the category set
# Output: mean and standard deviation of F1 over five folds

def evaluate(model, comments, labels):
    # Stratified 5-fold split; multi-label runs stratify on the union of labels
    folds = stratified_folds(comments, labels, n_folds=5)
    scores = []
    for train, test in folds:
        # Hold out 10% of the training data for model selection  [§6]
        fit, validation = split(train, fraction=0.1)
        # Selection metric: binary F1 (single-label),
        # macro F1 (multi-label BERT)  [§6]
        chosen = select_best(model, fit, validation)
        scores.append(f1(predict(chosen, comments[test]), labels[test]))
    return mean(scores), std(scores)

Cross-corpus setup §8

BERT models are trained with the same settings as above. MFTC is down-sampled so that each label has the same number of samples as in MFRC §8. The MFRC split of Fairness into Proportionality and Equality is handled as follows §8:

• To score MFTC Fairness with MFRC models: the MFRC Proportionality and Equality predictions are merged with a union.
• To score MFRC with MFTC Fairness models: both Proportionality and Equality receive the single MFTC Fairness prediction.

Hyperparameters

Personalisation study app. B: 10-, 20- and 30-shot prompts, and a prompt that includes manual definitions (0-shot for GPT-OSS, 1-shot for the others). Models: Llama3-8B and Llama3-70B, Qwen3-30B and GPT-OSS-20B. Metrics: Sample Semantic F1, Jaccard, macro F1 and micro F1. Training or tuning settings for this study are not stated.

Results

Author-reported. Setup: the full MFRC corpus; stratified 5-fold cross-validation; F1 as mean (standard deviation over folds) §7, §6. The authors ran each baseline once over the whole corpus §7. Bold marks the best model per column in the paper.

Table 3 tab. 3: F1 by category, full MFRC

Headline results:

• Fully fine-tuned BERT (single-label) has the best F1 in every category tab. 3, §7.
• LoRA-tuned Llama is next; zero-shot and five-shot Llama and Ministral are lower. Llama multi-label prompting scores 0.01 on Thin in both settings tab. 3.
• The authors report that multi-label BERT is worse than single-label BERT, while multi-label Llama is comparable to or better than single-label Llama §7.

Table 1 tab. 1: Posts per subreddit by number of annotators

Table 2 tab. 2: Posts per foundation, by majority vote

Table 4 tab. 4: BERT trained and tested on MFTC (down-sampled; mean (sd); F1, precision, recall)

Table 5 tab. 5: BERT trained on MFRC, evaluated on MFTC

Table 6 tab. 6: BERT trained on MFTC, evaluated on MFRC

The authors call these cross-corpus results "encouraging" and note that the two corpora come from different time periods §8.

Table 7 tab. 7: F1 by subreddit bucket (mean (sd))

The authors report that performance differs by bucket: some foundations score higher in the everyday bucket, others in US politics §7.

Figure 1 fig. 1 shows inter-annotator agreement as two heatmaps over subreddits and foundations, one for Fleiss' kappa and one for PABAK. Darker means higher agreement. The authors note that kappa is low because positive cases are rare, and that PABAK shows medium to high agreement §4. The cell values are in the image and are not transcribed here.

Personalisation app. B: the authors describe the results as inconclusive; no single setting is best across the metrics. Tables with per-annotator scores are in app. B.

Precision and recall for the baselines, per category and with standard deviations, are in app. C. They cover BERT, multi-label BERT and the zero-, five-shot and PEFT Llama and Ministral settings.

Author-reported comparison with MFTC: the authors state that models trained and tested on MFTC score higher than the MFRC results §8. The label sets differ, so compare the rows only in direction, not in size.

Paper vs code

No official code.

Searched, in order:

1. Paper links: the only artefact is the Hugging Face dataset USC-MOLA-Lab/MFRC §2, app. A. The abstract contains no code link, and the metadata has no arXiv comments field.
2. Project page: none linked.
3. GitHub API search for "MFRC moral foundations reddit" returned one repository, kraftedcheese/nlp_assignment_2_moralfoundations. Its owner is not an author, so it is not official and was not cloned.
4. Organisation search for USC-MOLA-Lab returned HTTP 422, and github.com/USC-MOLA-Lab?tab=repositories returned 404. I also tried github.com/jacktrager?tab=repositories as a guessed handle, which returned 404.

Nothing was cloned, so code, code_commit and code_license are omitted. The dataset is on Hugging Face under the lab's name; its own license was not checked and is not needed for this card.

Pitfalls

• Eight categories versus seven predicted. The abstract speaks of eight categories, which include Implicit/Explicit. The baselines predict seven: Care, Equality, Proportionality, Loyalty, Authority, Purity and Thin §3.1, tab. 3. Implicit/Explicit is not a model target in the paper.
• "Non-Moral" is not a target. Table 2 counts it, but no model predicts it tab. 2. A comment with no moral label is a negative example.
• Subreddit overlap. r/Conservative appears in both the US and the French buckets §2. Table 1 lists two rows, but they are different samples with different date ranges and filters tab. 1, §2.
• Commented-out formula. The LaTeX source has a commented-out bin-size formula §2.1. It is not in the compiled paper, so do not implement it. The bin size is not stated.
• Scope of the loss weight. Equation 1 gives a weight proportional to the inverse label frequency. The paper does not say whether the weight applies to positive samples only, to all samples, or what constant is used eq. 1, §6.
• Bucket claim does not fully hold. The text says BERT single beats BERT multi in every category in all three buckets §7. Table 7 contradicts this for French politics Authority: single 0.20 (0.12) against multi 0.26 (0.04) tab. 7.
• Unclear bucket evaluation. The text says the best BERT models were "evaluated on the three subreddit buckets separately" §7. It does not say whether those models were retrained per bucket or only scored on each subset.
• Two statements about personalisation. The main text calls the personalisation gains "modest" and says they do not alter the overall pattern §7. The appendix calls the study "inconclusive" app. B. Trust the appendix on this point.
• "Ran once" versus cross-validation. The text says each baseline was run once over the whole corpus §7, and it reports standard deviations from a 5-fold setup §6. How the standard deviation was computed is not explicit.
• Missing training details. The BERT size, LoRA rank and alpha, optimiser, scheduler, warmup, weight decay, max length, seed and decoding settings are not stated §6. Do not reconstruct them from the other papers.
• Few-shot prompt text. The few-shot prompt wording is not in the main text. The definition prompt (with the manual's domain definitions) is in app. B; use it only for the personalisation setting.
• Fairness mapping. The cross-corpus results depend on the Fairness merge described in §8. Scores with and without the merge are not comparable.
• Cross-corpus tables are not like-for-like. Table 4 is BERT on down-sampled MFTC with mean (sd) values. The paper does not state the evaluation split for Tables 4 to 6, or whether those tables use the same 5-fold protocol as Table 3 tab. 4–6, §8.
• Annotator context. Annotators saw the comment only, not the original post app. A. Ambiguous comments are flagged through the confidence rating.
• Annotator demographics. The metadata is available on request. Plan a request before you analyse annotator effects §3.2.
• Limits stated by the authors. The theory trace card says the corpus does not evaluate moral reasoning, normative correctness, or competing moral theories, and that English Reddit data and annotator demographics limit generalisability app. D. The data disclaimer and the ethics statement say the corpus contains biases and should be used only for scientific and educational purposes §10, §11.
• Appendix letters. The appendix letters A to D are assigned in the order the sources are included. That order is the LaTeX order, not a verified printed label.

Lineage

• Builds on: the Moral Foundations Theory revision of Atari et al. (2023), which splits Fairness into Equality and Proportionality §1. Not carded.
• Builds on: the Moral Foundations Twitter Corpus (Hoover et al., 2020), the source of the cross-corpus comparison §1, §8. Not carded.
• Compared against: MFTC, in the cross-corpus tables tab. 4–6.
• Superseded by: none known from the content in this wiki.