Paper · 2020-04-08 · cs.CL, cs.IR, cs.SI
Measuring Emotions in the COVID-19 Real World Worry Dataset
Releases the Real World Worry Dataset: 2,500 UK residents who use Twitter rated nine emotions on 9-point scales and wrote a long and a tweet-sized text in April 2020. LIWC correlations explain up to 10.6% of variance; ridge regression on TF-IDF gives worry MAE 1.26 on long texts, while short texts stay near zero R-squared.
Sentiment analysisTopic modellingRidge regressiondatasetemotioncovid-19self-report
arXivPDFCodeMarkdown11 min read
AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Much earlier work labels emotions in text by hand or by proxy, such as hashtags, so the label describes what the text seems to express. This paper takes a more direct route. It asks 2,500 people in the UK how worried, anxious, sad or afraid they felt at that moment, and has them write a few sentences about it. Because the people who wrote the text also said how they felt, the labels come from the authors themselves rather than from outside readers.
The authors then ask three questions of this data. Do the words people use match how they say they feel? What do people talk about when they are worried? Can a simple model guess the feeling from the text alone? The answers are modest. Word counts match the ratings only weakly, the topics are mostly about family, work and lockdown rules, and the guesses work best for the worry rating and poorly for the short texts.
TL;DR
The Real World Worry Dataset holds 2,500 UK participants' emotion ratings (April 2020) and two texts from each: a long one and a tweet-sized one §2. Word-list correlations and topic models describe the texts §3.1 §3.2. Ridge regression predicts the worry rating from long texts with a mean absolute error of 1.26 on a 9-point scale; the other emotions and the short texts predict poorly §3.3 tab. 4. All results are the authors' own.
Problem
Emotion-detection work usually labels text by hand or by proxy, such as hashtags, so the label describes what the text seems to express rather than what the writer felt §1.1. Ground-truth emotion data would be better but is costly to collect. The COVID-19 situation was new, and no dataset captured people's self-reported emotions about it while it was happening; the existing COVID-19 collections are mostly tweet streams or open diaries §1.2 §1.3.
Method
Data collection. The authors recruited 2,500 participants through the Prolific platform. They had to live in the UK and use Twitter, and each gave consent in line with their local ethics board §2. The data were collected on 6 and 7 April 2020, during the UK lockdown §1.3.
First, every person gave a 9-point score for their worry about the pandemic. Eight further feelings, the ones listed in Table 1, were scored the same way; the order of presentation is not stated §2 tab. 1. The scale runs from no feeling at all at the bottom, through a middle point, to a strong feeling at the top §2. They then picked the one emotion, other than worry, that best matched their feeling §2. Next they wrote two texts with the same instruction, to say how they feel about the Corona situation at this moment: a long text (minimum 500 characters, open-ended) and a tweet-sized text (maximum 240 characters) §2. Finally they rated how well they could express their feelings in general and in each text, how often they used Twitter, and whether English was their native language §2.
Cleaning. Nine participants were excluded for padding the long text with punctuation or repeated letters, which leaves 2,491 participants §2.2. The paper gives no other exclusion criteria.
Analysis 1: word-list correlations §3.1. Each self-reported emotion is correlated with the matching LIWC2015 category score, using Pearson's r with a 99% confidence interval. The authors also correlate worry with LIWC categories for work, money, death, family and friends. They report the share of variance explained as R-squared in percent tab. 2.
Analysis 2: topic models §3.2. The authors fit separate structural topic
models (the stm R package) to the long texts and to the short texts. Text is
lowercased, stripped of punctuation, stopwords and numbers, and stemmed. The
topic counts, 20 for long and 15 for short, were chosen by semantic coherence
and exclusivity. Only the five most prevalent topics are shown in the paper
tab. 3.
Analysis 3: ridge regression §3.3. For each of four emotions (anxiety, fear, sadness, worry), a ridge regression is trained on either TF-IDF or part- of-speech features, separately for each text type. TF-IDF uses the 1,000 most frequent words of each corpus. POS features use a fixed set of 53 spaCy tags. The features are reduced with principal component analysis, and performance is measured with mean absolute error (MAE) and R-squared under five-fold cross-validation.
# Correlate one self-report with one LIWC category [§3.1] [tab. 2]
# Input: ratings (N,) on a 9-point scale; liwc_scores (N,) for one category
# Output: Pearson r, 99% confidence interval, explained variance in percent
def correlate_emotion_with_liwc(ratings, liwc_scores):
# Pearson correlation between the rating and the category score [§3.1]
r = pearson(ratings, liwc_scores)
# 99% confidence interval for r; the method is not stated [tab. 2]
ci_low, ci_high = confidence_interval(r, len(ratings), level=0.99)
# Explained variance in percent (R-squared); formula not stated [tab. 2]
r_squared_percent = 100 * r_squared(ratings, liwc_scores)
return r, (ci_low, ci_high), r_squared_percent# Ridge regression for one emotion and one feature type [§3.3] [tab. 4]
# Input: texts (N,); ratings (N,) for one emotion, 9-point scale
# Output: mean MAE and mean R-squared over five folds
def predict_emotion(texts, ratings, feature_type):
# TF-IDF over the 1000 most frequent words, or counts of 53 POS tags
features = build_features(texts, feature_type) # (N, D)
fold_maes = []
fold_r2s = []
# Five-fold cross-validation; the fold split is not stated [§3.3]
for train_idx, test_idx in five_folds(len(ratings)):
# PCA; number of components not stated [§3.3]
pca = fit_pca(features[train_idx])
train_x = pca.transform(features[train_idx])
test_x = pca.transform(features[test_idx])
# Ridge penalty not stated [§3.3]
model = fit_ridge(train_x, ratings[train_idx])
predicted = model.predict(test_x)
fold_maes.append(mae(ratings[test_idx], predicted))
fold_r2s.append(r_squared(ratings[test_idx], predicted))
# Arithmetic mean over the five folds [§3.3]
return mean(fold_maes), mean(fold_r2s)Table 1: descriptive statistics §2.2 tab. 1. Mean and standard deviation of the corpus and of the ratings. Emotions also show the share of participants who chose them as their best-fit emotion.
| Measure | Mean | SD |
|---|---|---|
| Tokens per long text | 127.75 | 39.67 |
| Tokens per short text | 27.70 | 15.98 |
| Word types per long text | 82.69 | 18.24 |
| Word types per short text | 23.50 | 12.21 |
| Type-token ratio, long text | 0.66 | 0.06 |
| Type-token ratio, short text | 0.88 | 0.09 |
| Characters, long text | 632.54 | 197.75 |
| Characters, short text | 137.21 | 78.40 |
| Worry (a) | 6.55 | 1.76 |
| Anger, chosen 4.33% (b) | 3.91 | 2.24 |
| Anxiety, chosen 55.36% (a) | 6.49 | 2.28 |
| Desire, chosen 1.09% (b) | 2.97 | 2.04 |
| Disgust, chosen 0.69% (b) | 3.23 | 2.13 |
| Fear, chosen 9.22% (a) | 5.67 | 2.27 |
| Happiness, chosen 1.58% (b) | 3.62 | 1.89 |
| Relaxation, chosen 13.38% (b) | 3.95 | 2.13 |
| Sadness, chosen 14.36% (a) | 5.59 | 2.31 |
(a) Above the neutral midpoint, Bayes factor above 1e+32. (b) Below the midpoint, Bayes factor above 1e+115. Source: tab. 1.
Self-ratings of expressiveness: general M = 6.88 (SD 1.69), long texts M = 7.12 (SD 1.78), short texts M = 5.91 (SD 2.12); the long-text rating is higher, with Bayes factor above 1e+96 §2.2. Participants used Twitter about weekly (M = 6.26), tweeted about monthly (M = 3.67) and took part in conversations at a similar rate (M = 3.41) §2.2.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Participants | 2,500 recruited; 2,491 after cleaning | §2 §2.2 | Prolific; UK residents who use Twitter |
| Data collection dates | 6 and 7 April 2020 | §1.3 | |
| Rating scale | 9-point, 1 = not at all, 9 = very much | §2 | Worry plus eight emotions |
| Long text | minimum 500 characters | §2 | Open-ended |
| Tweet-sized text | maximum 240 characters | §2 | README says 280; see Paper vs code |
| LIWC lexicon | LIWC2015 | §3.1 | Matching rate 92.36% short, 90.11% long |
| Correlation | Pearson r, 99% CI | tab. 2 | Multiple-comparison correction not stated |
| TF-IDF vocabulary | 1,000 most frequent words per corpus | §3.3 | |
| POS feature set | 53 spaCy tags | §3.3 | Tag names not listed |
| PCA components | not stated | §3.3 | |
| Ridge penalty | not stated | §3.3 | Regularised ridge, no value given |
| Cross-validation | five folds, mean over folds | §3.3 | Fold assignment and seed not stated |
| Topic model package | stm (R) | §3.2 | Other stm settings not stated |
| Topic count | 20 (long), 15 (short) | §3.2 | Chosen by coherence and exclusivity |
| Stopwords, stemming | removed; stemmed | §3.2 | Stopword list not stated |
Results
Author-reported. The setup is the RWWD (2,500 recruited, 9 excluded, UK, April 2020), with ridge regression and five-fold cross-validation §3.3. The paper reports no baseline, such as a predictor that always outputs the mean.
Prediction §3.3 tab. 4. MAE is on the 9-point scale; R-squared is averaged over folds.
| Model | MAE long | R² long | MAE short | R² short |
|---|---|---|---|---|
| Anxiety, TF-IDF | 1.65 | 0.16 | 1.82 | -0.01 |
| Anxiety, POS | 1.79 | 0.04 | 1.84 | 0.00 |
| Fear, TF-IDF | 1.71 | 0.15 | 1.85 | 0.00 |
| Fear, POS | 1.83 | 0.05 | 1.87 | 0.01 |
| Sadness, TF-IDF | 1.75 | 0.12 | 1.90 | -0.02 |
| Sadness, POS | 1.88 | 0.02 | 1.91 | -0.01 |
| Worry, TF-IDF | 1.26 | 0.16 | 1.38 | -0.03 |
| Worry, POS | 1.35 | 0.03 | 1.37 | 0.01 |
Source: tab. 4. Headline in the abstract: the worry rating is predicted "within 14%" of its actual value. The discussion gives this as an absolute error of 1.26 on the 9-point scale, shown as 14% §4, and labels it MAPE 14.00% in §4.3.
Correlations §3.1 tab. 2. Pearson r with 99% CI; R-squared in percent.
| Pair | Long: r [99% CI] (R² %) | Short: r [99% CI] (R² %) |
|---|---|---|
| Anger, LIWC "anger" | 0.28 [0.23; 0.32] (7.56%) | 0.09 [0.04; 0.15] (0.88%) |
| Sadness, LIWC "sad" | 0.21 [0.16; 0.26] (4.35%) | 0.13 [0.07; 0.18] (1.58%) |
| Anxiety, LIWC "anx" | 0.33 [0.28; 0.37] (10.63%) | 0.18 [0.13; 0.23] (3.38%) |
| Worry, LIWC "anx" | 0.30 [0.26; 0.35] (9.27%) | 0.18 [0.13; 0.23] (3.30%) |
| Happiness, LIWC "posemo" | 0.22 [0.17; 0.26] (4.64%) | 0.13 [0.07; 0.18] (1.56%) |
| Worry, LIWC "work" | -0.03 [-0.08; 0.02] (0.01%) | -0.03 [-0.08; 0.02] (0.10%) |
| Worry, LIWC "money" | 0.00 [-0.05; 0.05] (0.00%) | -0.01 [-0.06; 0.04] (0.00%) |
| Worry, LIWC "death" | 0.05 [-0.01; 0.10] (0.26%) | 0.05 [0.00; 0.10] (0.29%) |
| Worry, LIWC "family" | 0.18 [0.13; 0.23] (3.12%) | 0.06 [0.01; 0.11] (0.40%) |
| Worry, LIWC "friend" | 0.07 [0.01; 0.12] (0.42%) | -0.01 [-0.06; 0.05] (0.00%) |
Source: tab. 2. The authors call the correlations weak; the largest explains 10.63% of variance (anxiety, long texts) §4.1.
Topics §3.2 tab. 3. In the long texts, the most prevalent topic (9.52% of documents) concerns following the lockdown rules. The second (8.35%) concerns work and the economy. In the short texts, the most prevalent topic (10.70%) is the government slogan to stay home and protect the NHS. The second (8.27%) calls on others to follow social-distancing rules §3.2. The full topic list is in the supplement, not the paper §3.2.
Discussion claims §4. Participants worry about family, friends, jobs and the economy. Short texts call for solidarity during lockdown, and long texts describe worries §4.2. The authors state that worry was easiest to predict, perhaps because it had the highest mean and the lowest SD, which could bias the model §4.3.
Paper vs code
The official repository is ben-aaron188/covid19worry at commit
7324d50869578589a47781bc1d26123ca48e27c8. The paper links it in its
data footnote §1.3 README.md. It holds the data, a README and an
appendix. It has no analysis scripts: no LIWC, topic-model, ridge-regression
or cross-validation code README.md. The method above therefore
rests on the paper's text alone.
Checks against the repository:
- Counts match. Raw texts number 2,500 per length; cleaned texts number 2,491 per length, matching the 9 exclusions README.md §2.2.
- Analysis subset. The README says LIWC was extracted only from the cleaned data README.md. The paper does not say which N each analysis uses.
- Tweet length differs. The paper gives a maximum of 240 characters for tweet-sized texts §2. The codebook gives 280 README.md. The length of the texts was not checked against either limit.
- Topic count differs. The paper says the supplement shows all 20 long-text topics §3.2. The appendix table lists 19 long-text topics; its prevalence values sum to 98.45%, not 100% paper_appendix.md. The short-text table lists 15 topics that sum to 100.01% paper_appendix.md. The top five long topics and the top five short topics match Table 3 §3.2 tab. 3.
- Emotion correlations match. The appendix correlation matrix gives worry with fear 0.70 and with anxiety 0.66, anger with disgust 0.67, fear with anxiety 0.78, and happiness with relaxation 0.68, the same figures the discussion reports paper_appendix.md §4.1.
- Licence. The repository's
LICENSEis GPL-3.0 LICENSE. We cite its files by path and quote nothing from them. The licence applies to the repository; whether it covers the data files is not stated.
Pitfalls
- No code for the analyses. The LIWC correlations, topic models and ridge
regression cannot be re-run from the repository. Several settings that matter
for the numbers are missing: the ridge penalty, the number of PCA components,
the fold assignment, the
stmsettings and the LIWC matching rules §3.1 §3.2 §3.3. - "Within 14%" is not a per-person percentage. The paper calls 14% a MAPE §4.3. The worry MAE of 1.26 divided by the 9-point scale is 14.0%, so the figure is an error as a share of the full scale, not of each person's rating. An implementer comparing it with other MAPE values should recompute it.
- R-squared does not match r squared. The paper gives r = 0.28 and R² = 7.56% for anger in long texts tab. 2. Squaring 0.28 gives 7.84%. The paper does not say how R² is computed.
- Short-text models explain nothing. Every short-text R² in Table 4 is between -0.03 and 0.01. A negative R² means the model is worse than predicting the mean. The text says the models explain at most 1% §3.3.
- Ratings are self-reports made at one moment. Worry correlates strongly with fear (0.70) and anxiety (0.66) paper_appendix.md. The authors say these may not be separate constructs in their data §4.1.
- Correlations are not corrected for multiple tests. Many correlations are reported. The paper does not state a correction §3.1 tab. 2.
- Tokens and types depend on the tokenizer. The paper does not state the tokenizer for the token counts §2 tab. 1.
- Age figures are unverified. The paper reports a mean age of 33.84 years with an SD of 22.04 §2.2, which is a wide spread for adults. The demographics file was not checked against this figure. The gender share is 65.15% female §2.2.
- The sample is narrow. It is UK-based, recruited on Prolific, limited to Twitter users, and 94.46% native English §2. The paper says results may not generalise to real Twitter data §4.4.
- Single moment in time. Data come from one 2-day window in April 2020 §1.3. The authors suggest longer sampling as future work §4.4.
- Topic counts were chosen by hand. The authors used coherence and exclusivity to pick 20 and 15 topics, and say there is no consensus on how to choose the number §3.2 §4.2.
Lineage
- Builds on: none stated.
- Related: the follow-up repeated-measures study by the same group, arXiv:2107.03466, is named in the repository README README.md. It has no card in this wiki yet.
- Gender differences in these data are analysed in a separate paper by van der Vegt and Kleinberg §2.2.
- Superseded by: not known from this wiki.
Source and license
Adapted from “Measuring Emotions in the COVID-19 Real World Worry Dataset” by Bennett Kleinberg, Isabelle van der Vegt, Maximilian Mozes (arXiv:2004.04225), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ben-aaron188/covid19worry at commit 7324d5086957 (GPL-3.0). Any quoted code is shown under that license.