wikillm

Paper · 2020-04-08 · cs.CL, cs.IR, cs.SI

Measuring Emotions in the COVID-19 Real World Worry Dataset

Releases the Real World Worry Dataset: 2,500 UK residents who use Twitter rated nine emotions on 9-point scales and wrote a long and a tweet-sized text in April 2020. LIWC correlations explain up to 10.6% of variance; ridge regression on TF-IDF gives worry MAE 1.26 on long texts, while short texts stay near zero R-squared.

Sentiment analysisTopic modellingRidge regressiondatasetemotioncovid-19self-report

arXivPDFCodeMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Much earlier work labels emotions in text by hand or by proxy, such as hashtags, so the label describes what the text seems to express. This paper takes a more direct route. It asks 2,500 people in the UK how worried, anxious, sad or afraid they felt at that moment, and has them write a few sentences about it. Because the people who wrote the text also said how they felt, the labels come from the authors themselves rather than from outside readers.

The authors then ask three questions of this data. Do the words people use match how they say they feel? What do people talk about when they are worried? Can a simple model guess the feeling from the text alone? The answers are modest. Word counts match the ratings only weakly, the topics are mostly about family, work and lockdown rules, and the guesses work best for the worry rating and poorly for the short texts.

TL;DR

The Real World Worry Dataset holds 2,500 UK participants' emotion ratings (April 2020) and two texts from each: a long one and a tweet-sized one §2. Word-list correlations and topic models describe the texts §3.1 §3.2. Ridge regression predicts the worry rating from long texts with a mean absolute error of 1.26 on a 9-point scale; the other emotions and the short texts predict poorly §3.3 tab. 4. All results are the authors' own.

Problem

Emotion-detection work usually labels text by hand or by proxy, such as hashtags, so the label describes what the text seems to express rather than what the writer felt §1.1. Ground-truth emotion data would be better but is costly to collect. The COVID-19 situation was new, and no dataset captured people's self-reported emotions about it while it was happening; the existing COVID-19 collections are mostly tweet streams or open diaries §1.2 §1.3.

Method

Data collection. The authors recruited 2,500 participants through the Prolific platform. They had to live in the UK and use Twitter, and each gave consent in line with their local ethics board §2. The data were collected on 6 and 7 April 2020, during the UK lockdown §1.3.

First, every person gave a 9-point score for their worry about the pandemic. Eight further feelings, the ones listed in Table 1, were scored the same way; the order of presentation is not stated §2 tab. 1. The scale runs from no feeling at all at the bottom, through a middle point, to a strong feeling at the top §2. They then picked the one emotion, other than worry, that best matched their feeling §2. Next they wrote two texts with the same instruction, to say how they feel about the Corona situation at this moment: a long text (minimum 500 characters, open-ended) and a tweet-sized text (maximum 240 characters) §2. Finally they rated how well they could express their feelings in general and in each text, how often they used Twitter, and whether English was their native language §2.

Cleaning. Nine participants were excluded for padding the long text with punctuation or repeated letters, which leaves 2,491 participants §2.2. The paper gives no other exclusion criteria.

Analysis 1: word-list correlations §3.1. Each self-reported emotion is correlated with the matching LIWC2015 category score, using Pearson's r with a 99% confidence interval. The authors also correlate worry with LIWC categories for work, money, death, family and friends. They report the share of variance explained as R-squared in percent tab. 2.

Analysis 2: topic models §3.2. The authors fit separate structural topic models (the stm R package) to the long texts and to the short texts. Text is lowercased, stripped of punctuation, stopwords and numbers, and stemmed. The topic counts, 20 for long and 15 for short, were chosen by semantic coherence and exclusivity. Only the five most prevalent topics are shown in the paper tab. 3.

Analysis 3: ridge regression §3.3. For each of four emotions (anxiety, fear, sadness, worry), a ridge regression is trained on either TF-IDF or part- of-speech features, separately for each text type. TF-IDF uses the 1,000 most frequent words of each corpus. POS features use a fixed set of 53 spaCy tags. The features are reduced with principal component analysis, and performance is measured with mean absolute error (MAE) and R-squared under five-fold cross-validation.

Pseudocode
# Correlate one self-report with one LIWC category  [§3.1] [tab. 2]
# Input:  ratings (N,) on a 9-point scale; liwc_scores (N,) for one category
# Output: Pearson r, 99% confidence interval, explained variance in percent

def correlate_emotion_with_liwc(ratings, liwc_scores):
    # Pearson correlation between the rating and the category score  [§3.1]
    r = pearson(ratings, liwc_scores)
    # 99% confidence interval for r; the method is not stated  [tab. 2]
    ci_low, ci_high = confidence_interval(r, len(ratings), level=0.99)
    # Explained variance in percent (R-squared); formula not stated  [tab. 2]
    r_squared_percent = 100 * r_squared(ratings, liwc_scores)
    return r, (ci_low, ci_high), r_squared_percent
Pseudocode
# Ridge regression for one emotion and one feature type  [§3.3] [tab. 4]
# Input:  texts (N,); ratings (N,) for one emotion, 9-point scale
# Output: mean MAE and mean R-squared over five folds

def predict_emotion(texts, ratings, feature_type):
    # TF-IDF over the 1000 most frequent words, or counts of 53 POS tags
    features = build_features(texts, feature_type)       # (N, D)
    fold_maes = []
    fold_r2s = []
    # Five-fold cross-validation; the fold split is not stated  [§3.3]
    for train_idx, test_idx in five_folds(len(ratings)):
        # PCA; number of components not stated  [§3.3]
        pca = fit_pca(features[train_idx])
        train_x = pca.transform(features[train_idx])
        test_x = pca.transform(features[test_idx])
        # Ridge penalty not stated  [§3.3]
        model = fit_ridge(train_x, ratings[train_idx])
        predicted = model.predict(test_x)
        fold_maes.append(mae(ratings[test_idx], predicted))
        fold_r2s.append(r_squared(ratings[test_idx], predicted))
    # Arithmetic mean over the five folds  [§3.3]
    return mean(fold_maes), mean(fold_r2s)

Table 1: descriptive statistics §2.2 tab. 1. Mean and standard deviation of the corpus and of the ratings. Emotions also show the share of participants who chose them as their best-fit emotion.

MeasureMeanSD
Tokens per long text127.7539.67
Tokens per short text27.7015.98
Word types per long text82.6918.24
Word types per short text23.5012.21
Type-token ratio, long text0.660.06
Type-token ratio, short text0.880.09
Characters, long text632.54197.75
Characters, short text137.2178.40
Worry (a)6.551.76
Anger, chosen 4.33% (b)3.912.24
Anxiety, chosen 55.36% (a)6.492.28
Desire, chosen 1.09% (b)2.972.04
Disgust, chosen 0.69% (b)3.232.13
Fear, chosen 9.22% (a)5.672.27
Happiness, chosen 1.58% (b)3.621.89
Relaxation, chosen 13.38% (b)3.952.13
Sadness, chosen 14.36% (a)5.592.31

(a) Above the neutral midpoint, Bayes factor above 1e+32. (b) Below the midpoint, Bayes factor above 1e+115. Source: tab. 1.

Self-ratings of expressiveness: general M = 6.88 (SD 1.69), long texts M = 7.12 (SD 1.78), short texts M = 5.91 (SD 2.12); the long-text rating is higher, with Bayes factor above 1e+96 §2.2. Participants used Twitter about weekly (M = 6.26), tweeted about monthly (M = 3.67) and took part in conversations at a similar rate (M = 3.41) §2.2.

Hyperparameters

NameValueWhereNotes
Participants2,500 recruited; 2,491 after cleaning§2 §2.2Prolific; UK residents who use Twitter
Data collection dates6 and 7 April 2020§1.3
Rating scale9-point, 1 = not at all, 9 = very much§2Worry plus eight emotions
Long textminimum 500 characters§2Open-ended
Tweet-sized textmaximum 240 characters§2README says 280; see Paper vs code
LIWC lexiconLIWC2015§3.1Matching rate 92.36% short, 90.11% long
CorrelationPearson r, 99% CItab. 2Multiple-comparison correction not stated
TF-IDF vocabulary1,000 most frequent words per corpus§3.3
POS feature set53 spaCy tags§3.3Tag names not listed
PCA componentsnot stated§3.3
Ridge penaltynot stated§3.3Regularised ridge, no value given
Cross-validationfive folds, mean over folds§3.3Fold assignment and seed not stated
Topic model packagestm (R)§3.2Other stm settings not stated
Topic count20 (long), 15 (short)§3.2Chosen by coherence and exclusivity
Stopwords, stemmingremoved; stemmed§3.2Stopword list not stated

Results

Author-reported. The setup is the RWWD (2,500 recruited, 9 excluded, UK, April 2020), with ridge regression and five-fold cross-validation §3.3. The paper reports no baseline, such as a predictor that always outputs the mean.

Prediction §3.3 tab. 4. MAE is on the 9-point scale; R-squared is averaged over folds.

ModelMAE longR² longMAE shortR² short
Anxiety, TF-IDF1.650.161.82-0.01
Anxiety, POS1.790.041.840.00
Fear, TF-IDF1.710.151.850.00
Fear, POS1.830.051.870.01
Sadness, TF-IDF1.750.121.90-0.02
Sadness, POS1.880.021.91-0.01
Worry, TF-IDF1.260.161.38-0.03
Worry, POS1.350.031.370.01

Source: tab. 4. Headline in the abstract: the worry rating is predicted "within 14%" of its actual value. The discussion gives this as an absolute error of 1.26 on the 9-point scale, shown as 14% §4, and labels it MAPE 14.00% in §4.3.

Correlations §3.1 tab. 2. Pearson r with 99% CI; R-squared in percent.

PairLong: r [99% CI] (R² %)Short: r [99% CI] (R² %)
Anger, LIWC "anger"0.28 [0.23; 0.32] (7.56%)0.09 [0.04; 0.15] (0.88%)
Sadness, LIWC "sad"0.21 [0.16; 0.26] (4.35%)0.13 [0.07; 0.18] (1.58%)
Anxiety, LIWC "anx"0.33 [0.28; 0.37] (10.63%)0.18 [0.13; 0.23] (3.38%)
Worry, LIWC "anx"0.30 [0.26; 0.35] (9.27%)0.18 [0.13; 0.23] (3.30%)
Happiness, LIWC "posemo"0.22 [0.17; 0.26] (4.64%)0.13 [0.07; 0.18] (1.56%)
Worry, LIWC "work"-0.03 [-0.08; 0.02] (0.01%)-0.03 [-0.08; 0.02] (0.10%)
Worry, LIWC "money"0.00 [-0.05; 0.05] (0.00%)-0.01 [-0.06; 0.04] (0.00%)
Worry, LIWC "death"0.05 [-0.01; 0.10] (0.26%)0.05 [0.00; 0.10] (0.29%)
Worry, LIWC "family"0.18 [0.13; 0.23] (3.12%)0.06 [0.01; 0.11] (0.40%)
Worry, LIWC "friend"0.07 [0.01; 0.12] (0.42%)-0.01 [-0.06; 0.05] (0.00%)

Source: tab. 2. The authors call the correlations weak; the largest explains 10.63% of variance (anxiety, long texts) §4.1.

Topics §3.2 tab. 3. In the long texts, the most prevalent topic (9.52% of documents) concerns following the lockdown rules. The second (8.35%) concerns work and the economy. In the short texts, the most prevalent topic (10.70%) is the government slogan to stay home and protect the NHS. The second (8.27%) calls on others to follow social-distancing rules §3.2. The full topic list is in the supplement, not the paper §3.2.

Discussion claims §4. Participants worry about family, friends, jobs and the economy. Short texts call for solidarity during lockdown, and long texts describe worries §4.2. The authors state that worry was easiest to predict, perhaps because it had the highest mean and the lowest SD, which could bias the model §4.3.

Paper vs code

The official repository is ben-aaron188/covid19worry at commit 7324d50869578589a47781bc1d26123ca48e27c8. The paper links it in its data footnote §1.3 README.md. It holds the data, a README and an appendix. It has no analysis scripts: no LIWC, topic-model, ridge-regression or cross-validation code README.md. The method above therefore rests on the paper's text alone.

Checks against the repository:

  • Counts match. Raw texts number 2,500 per length; cleaned texts number 2,491 per length, matching the 9 exclusions README.md §2.2.
  • Analysis subset. The README says LIWC was extracted only from the cleaned data README.md. The paper does not say which N each analysis uses.
  • Tweet length differs. The paper gives a maximum of 240 characters for tweet-sized texts §2. The codebook gives 280 README.md. The length of the texts was not checked against either limit.
  • Topic count differs. The paper says the supplement shows all 20 long-text topics §3.2. The appendix table lists 19 long-text topics; its prevalence values sum to 98.45%, not 100% paper_appendix.md. The short-text table lists 15 topics that sum to 100.01% paper_appendix.md. The top five long topics and the top five short topics match Table 3 §3.2 tab. 3.
  • Emotion correlations match. The appendix correlation matrix gives worry with fear 0.70 and with anxiety 0.66, anger with disgust 0.67, fear with anxiety 0.78, and happiness with relaxation 0.68, the same figures the discussion reports paper_appendix.md §4.1.
  • Licence. The repository's LICENSE is GPL-3.0 LICENSE. We cite its files by path and quote nothing from them. The licence applies to the repository; whether it covers the data files is not stated.

Pitfalls

  • No code for the analyses. The LIWC correlations, topic models and ridge regression cannot be re-run from the repository. Several settings that matter for the numbers are missing: the ridge penalty, the number of PCA components, the fold assignment, the stm settings and the LIWC matching rules §3.1 §3.2 §3.3.
  • "Within 14%" is not a per-person percentage. The paper calls 14% a MAPE §4.3. The worry MAE of 1.26 divided by the 9-point scale is 14.0%, so the figure is an error as a share of the full scale, not of each person's rating. An implementer comparing it with other MAPE values should recompute it.
  • R-squared does not match r squared. The paper gives r = 0.28 and R² = 7.56% for anger in long texts tab. 2. Squaring 0.28 gives 7.84%. The paper does not say how R² is computed.
  • Short-text models explain nothing. Every short-text R² in Table 4 is between -0.03 and 0.01. A negative R² means the model is worse than predicting the mean. The text says the models explain at most 1% §3.3.
  • Ratings are self-reports made at one moment. Worry correlates strongly with fear (0.70) and anxiety (0.66) paper_appendix.md. The authors say these may not be separate constructs in their data §4.1.
  • Correlations are not corrected for multiple tests. Many correlations are reported. The paper does not state a correction §3.1 tab. 2.
  • Tokens and types depend on the tokenizer. The paper does not state the tokenizer for the token counts §2 tab. 1.
  • Age figures are unverified. The paper reports a mean age of 33.84 years with an SD of 22.04 §2.2, which is a wide spread for adults. The demographics file was not checked against this figure. The gender share is 65.15% female §2.2.
  • The sample is narrow. It is UK-based, recruited on Prolific, limited to Twitter users, and 94.46% native English §2. The paper says results may not generalise to real Twitter data §4.4.
  • Single moment in time. Data come from one 2-day window in April 2020 §1.3. The authors suggest longer sampling as future work §4.4.
  • Topic counts were chosen by hand. The authors used coherence and exclusivity to pick 20 and 15 topics, and say there is no consensus on how to choose the number §3.2 §4.2.

Lineage

  • Builds on: none stated.
  • Related: the follow-up repeated-measures study by the same group, arXiv:2107.03466, is named in the repository README README.md. It has no card in this wiki yet.
  • Gender differences in these data are analysed in a separate paper by van der Vegt and Kleinberg §2.2.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “Measuring Emotions in the COVID-19 Real World Worry Dataset” by Bennett Kleinberg, Isabelle van der Vegt, Maximilian Mozes (arXiv:2004.04225), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ben-aaron188/covid19worry at commit 7324d5086957 (GPL-3.0). Any quoted code is shown under that license.