SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions

In plain words

Mental health researchers need text written by people who have a diagnosis, and labelling that text by hand is slow and expensive. This paper finds those people without a human labeller. It searches Reddit for sentences where a person says they were diagnosed with a condition, such as depression or ADHD, and it checks that the diagnosis word sits close to a word naming that condition. For each person found this way, it picks other Reddit users who look similar but show no sign of mental health topics, so that a classifier cannot win just by spotting differences in how active people are.

The result is a large public-facing dataset (under a data-use agreement, see the pitfalls) of labelled users for nine conditions, plus a first look at how their word use differs from the control users and how well some standard text classifiers can tell them apart.

TL;DR

SMHD is a Reddit dataset of self-reported diagnoses for nine mental health conditions, with matched control users. Diagnosed users are found with high-precision diagnosis patterns, a condition keyword within 40 characters of a diagnosis match, and synonym lists. Controls have no mental health posts, at least 50 posts, subreddit overlap, and post counts within a factor of two of the diagnosed user. Author-reported baselines (bag-of-words logistic regression, linear SVM, XGBoost, FastText, CNN) peak with FastText on 7 of the 9 binary tasks. The dataset is not openly downloadable: it is shared under a data-use agreement.

Problem

Social media text is a source for mental health research, but labelled data is scarce, because annotation is costly and experts are hard to reach §1. The authors argue that earlier self-disclosure datasets were too small or too narrow: the Twitter dataset of Coppersmith et al. (2015) is limited to 140-character posts (280 from late 2017), and the Reddit depression dataset of Yates et al. (2017) covers only one condition §1 §2. The Twitter comparison in the paper gives ADHD 102 diagnosed users against 10,098 in SMHD tab. 1.

Assumptions the method relies on §3.1:

1. A self-report such as "I was diagnosed with X" is a reliable signal of a diagnosis. The authors measure this: 96.4% precision on a sample of 500 posts, before the condition step §3.2.
2. Users who share no mental health topics still differ from diagnosed users in activity. The matching is meant to remove that difference, but the matching is imperfect (see Pitfalls).

Method

The dataset is built in three steps: find diagnosed users, remove mental health content, and match controls. The paper has no equations and no formal algorithm; the pseudocode below restates its procedure in order.

1. Corpus. All Reddit posts by each user from January 2006 to December 2017, taken from a public Reddit corpus (Pushshift) §3.1.
2. Diagnosis patterns. A diagnosis pattern (a self-report such as "I was diagnosed") must occur, and a condition keyword must occur within 40 characters of it. The 40-character maximum was chosen on a development set by maximising F0.5 §3.2 fig. 2. Keywords start from DSM-5 headings and are expanded with two synonym resources, MedSyn and a behavioural synonym mapping, then vetted by hand §3.2.
3. Negative patterns. Users with a negative diagnosis match ("never diagnosed") are removed §3.1.
4. Diagnosed-user filter. A candidate needs at least 50 posts that are not mental health posts. Of the remaining candidates, 36,948 users are kept §3.1.
5. Mental health posts. A post is a mental health post if it is made in a mental health subreddit or contains a mental health term (condition names, "diagnosis", "mental illness", "suffering from" and similar). These posts are removed for diagnosed users §3.1.
6. Control pool. A control has no mental health post at all and at least 50 posts §3.1.
7. Matching. Each diagnosed user is matched to about 9 controls who posted in the same subreddit, and whose post count is between half and twice the diagnosed user's count. Controls are drawn without replacement §3.1.
8. Release split. Diagnosed users with at least 9 matched controls form SMHD (20,406 diagnosed, 335,952 controls). The 16,542 remaining diagnosed users form SMHD-RC (Relaxed Controls) §3.1 fig. 3.

# Build diagnosed users and matched controls  [§3.1] [§3.2]
# Input:  users with posts (2006-2017), diagnosis patterns, condition
#         keywords, mental health (MH) terms and subreddits
# Output: diagnosed users with condition labels, matched controls

def build_smhd(users, diagnosis_patterns, keywords, max_distance):
    # Control pool: no MH post at all, and at least 50 posts  [§3.1]
    pool = [u for u in users if no_mh_post(u) and post_count(u) >= 50]
    # Label users whose diagnosis match has a condition keyword within
    # max_distance characters (40 in the paper)  [§3.2] [fig. 2]
    labelled = label_conditions(users, diagnosis_patterns, keywords,
                                max_distance)
    # Drop users with a negative diagnosis match, then users with fewer
    # than 50 non-MH posts  [§3.1]
    diagnosed = [u for u in labelled if not has_negative(u)]
    # Remove MH posts from diagnosed users; keep 50 or more posts  [§3.1]
    diagnosed = [strip_mh_posts(u) for u in diagnosed]
    diagnosed = [u for u in diagnosed if post_count(u) >= 50]
    # Match about 9 controls per diagnosed user: shared subreddit and post
    # count within a factor of 2; controls used once  [§3.1]
    controls = match_controls(diagnosed, pool, per_user=9,
                              ratio=2.0)
    # Users with at least 9 controls form SMHD; the rest form SMHD-RC
    # [§3.1]
    return diagnosed, controls

Classification setup §5:

• Binary task per condition: the classifier is trained on the diagnosed users of that condition and their matched controls. Predictions are for one user at a time.
• Multi-label task: the classifier predicts the set of conditions a user reports, across all nine conditions §5.
• Bag-of-words models (logistic regression, linear SVM, XGBoost) use tf-idf features over all posts of a user, ℓ2-normalised, with the vocabulary cut at words seen fewer than 20 times in training §5.
• FastText is a shallow model with hidden size 100 and character n-grams of length 3 to 6, trained for 100 epochs §5.
• The CNN follows Kim (2014) with filter size 3 and pooling size 4, trained for 100 epochs, one model per condition. A user's posts are concatenated and cut to 15,000 tokens, and tokens are embedded with the FastText vectors §5.

Analysis of language differences §4 uses LIWC categories per user, Welch's t-test against controls, Bonferroni correction and Cohen's d. Most significant differences have effect sizes between 0.05 and 0.5, which the authors call small to medium §4 tab. 3.

Hyperparameters

Results

Author-reported. Setup: Reddit users and posts (2006 to 2017), binary and multi-label tasks, nine conditions, official SMHD split (sizes not stated), hardware not stated §5.

Label quality §3.2:

Dataset counts §3.1 tab. 1 tab. 2:

Post statistics, mean (standard deviation) per user and per post tab. 2:

Classification, F1 per condition on binary labels; multi-label in the last column tab. 4. Precision and recall are in tab. 4 too:

• FastText has the best F1 on 7 of 9 binary conditions. SVM is best on bipolar and CNN on eating tab. 4 §5.
• The authors note that bag-of-words models favour precision over recall, while the two neural models are more balanced §5.
• Scores are weakest for the three smallest groups (OCD, schizophrenia, eating) §5.
• The authors explain FastText beating the CNN by the fact that FastText reads all posts of a user, while the CNN samples a portion of them §5.

Paper vs code

No official code. Searched: (1) links in the paper: the only code-adjacent links are a pattern-data page (ir.cs.georgetown.edu/data/smhd/) and a data-use-agreement page; no repository is linked. A TODO comment in the TeX source says a GitHub link would be added, but none is in the compiled paper §3.1 [raw/src/1806.05258/sections/data.tex]. (2) The project page ir.cs.georgetown.edu/data/smhd/ lists pattern and subreddit text files and a LICENSE.txt; it names no code repository, and the license of those files was not shown, so this card does not use them. (3) GitHub repository search for "smhd": 62 results, none owned by an author or the Georgetown IR lab. Those repositories are third-party and were not used.

Pitfalls

• Dataset access is not open. The paper's CC BY 4.0 license covers the paper. The dataset is shared through a data-use agreement that bars publishing excerpts, contacting users, identifying them, or linking them to other data §3.4. Do not redistribute users or posts through this wiki.
• The train/dev/test split is not described. The Experiments section says all methods use "the official SMHD split introduced in Section 3", but Section 3 gives no split sizes, no procedure and no random seed §5 §3.
• Many training details are missing: optimiser, learning rate, batch size, regularisation for logistic regression and SVM, the CNN's number of filters and dropout, and XGBoost settings besides the number of trees §5. An implementer must choose these and say so.
• Dev-set tuning and ceiling scores. The 40-character distance was tuned on the same 241-post dev set that gives the 93% and 84% figures. The authors call these a ceiling §3.2.
• Label noise. Measured precision is 96.4% for the bare diagnosis pattern, 93% with the keyword step on dev and 95.8% macro on the held-out check. Known errors include negations, hypotheticals, uncertain diagnoses, and a diagnosis of one condition close to a term for another §3.2. These are self-reports, not clinical diagnoses.
• Control matching is approximate. The target is 9 controls per diagnosed user, but only 9 "on average" are reached, and controls are drawn without replacement, so the order of matching affects the result. The paper does not give the algorithm §3.1. Users with fewer than 9 controls go to SMHD-RC, which is a different population for classification.
• Activity differs anyway. Controls average about 310 posts per user, the diagnosed groups about 160. Controls' posts are shorter tab. 2. A BoW model reads all posts, while the CNN truncates each user to 15,000 tokens. The two model families therefore see different amounts of text. The authors say the sampling for the CNN was chosen for efficiency, following Yates et al. (2017) §5.
• No time split. Posts run from 2006 to 2017 and span before and after the diagnosis. The paper does not describe a temporal split §3.1.
• Mental health terms are removed, but not every related term. The authors say their method does not guarantee that all relevant terms or subreddits are excluded §3.1. Also, the term "add" is treated as a mental health term for controls, and posts containing it are removed rather than excluding the user §3.1 footnote.
• Table and figure numbers here are counted in the compiled order from the LaTeX source (arXiv v2, dated 2018-07-10): Table 1 counts, Figure 1 examples, Figures 2 to 5 in Section 3, Table 2 post stats, Table 3 LIWC, Table 4 classifiers. The compiled PDF was not checked, so if the numbering differs, the section citations are authoritative.
• Not used from the source. The Results section is commented out in the main file (% \subfile{sections/results}), so this card does not report it. A commented-out concomitance table and TODO notes in the source were ignored.
• The Twitter comparison ("up to two orders of magnitude") rests on tab. 1. The Twitter counts come from another paper and were not checked.

Lineage

• Builds on: Yates et al. (2017), the Reddit Self-reported Depression Diagnosis dataset (RSDD). SMHD extends its method with synonyms and eight more conditions §3.1. No card on this wiki for that paper.
• Synonym methods: the diagnosis-keyword expansion follows Soldaini et al. (2017) and uses MedSyn and a behavioural synonym resource §3.2.
• Compared against: Coppersmith et al. (2015), the Twitter ADHD and self-reported diagnosis dataset §1 tab. 1. No card on this wiki.
• Superseded by: not known from content already in this wiki.