Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data

In plain words

Mental-LLM asks whether general-purpose chatbot-style language models can tell, from a person's online posts, signs of stress, depression or suicide risk. The authors test three ways of getting a model to do this. In the first, the model is only given a question about the post and a label to answer with, sometimes with a hint about who wrote the post or a request to act as an expert. In the second, a few labelled example posts are placed in the prompt before the real one. In the third, open models are trained further on labelled posts from several datasets at once, so a single model learns all the tasks.

Prompting gets the models part of the way, and the larger closed models do better than the smaller open ones. The fine-tuned open models do better overall and come close to a model that was trained for one task at a time. The authors stress that these results do not mean the models are ready for real mental-health use: they can give plausible reasons that are wrong, and they note known racial and gender bias in LLMs.

TL;DR

• Six classification tasks built from four human-labelled Reddit datasets, plus three external datasets from other platforms §4.1 tab. 3.
• Zero-shot and few-shot prompting of the six models is promising but below task-specific models §5.1 §5.2.
• Instruction tuning on all six tasks gives Mental-Alpaca (7B) and Mental-FLAN-T5 (11B), which beat the best GPT-3.5 and GPT-4 prompts on average and come close to Mental-RoBERTa §5.3 tab. 4.
• The authors say the results do not show the models are deployable §5, §6.3.
• The official repo has hyperparameters and research-only model weights, but no training or inference code, and two of its fine-tuning settings disagree with the paper.

Problem

Task-specific mental-health models, such as Mental-RoBERTa, must be trained separately for each task §5.3. The paper asks how far LLMs get on these tasks with prompting alone, with a few examples, or after instruction tuning on several tasks, and whether the result carries over to datasets from other platforms §5.1–§5.3 tab. 9.

Method

Zero-shot prompt §3.1

A zero-shot prompt has four parts, concatenated in this order eq. 1: the post text (TextData), a context or expert sentence (Part1-S), the question (Part2-Q), and an output constraint such as "only return yes or no" §3.1.

• Part1-S has four variants tab. 2: blank (Basic); a sentence that the person wrote the post on social media (Context); a sentence asking the model to act as a psychologist (Mental Health); and both together.
• Part2-Q depends on the category and label type: mental state (for example stress or depression) or critical risk action (suicide), each as binary or multi-class. Each category has two or three phrasings tab. 2.
• The paper evaluates every Part1-S with every suitable Part2-Q across models and datasets §3.1 §5.1.1.

Few-shot prompt §3.2

A few-shot prompt puts M sampled pairs of (zero-shot prompt, label) before the zero-shot prompt of the query eq. 2. Model weights do not change. M is capped by the model's input limit §3.2. In the experiments M is 2 for binary tasks and 4 or 5 for multi-class tasks, with one example per class, sampled at random and repeated three times §5.2.

Instruction tuning §3.3 §4.2.1

Both single-dataset and multi-dataset tuning use the same two steps eq. 3:

1. Train on pairs of (zero-shot prompt, label) from the training split.
2. Test with zero-shot prompts only, on the test split.

Single-dataset tuning trains on one dataset and tests on all of them, to measure transfer §3.3.1 §5.3.2 tab. 8. Multi-dataset tuning trains on all six tasks in one run, each with its own instruction, to produce Mental-Alpaca and Mental-FLAN-T5 §3.3.2 §4.2.1. The prompt used for tuning is the most informative one, which the paper picks without naming the variant §5.3.

Models §4.2

The parameter counts are the paper's own figures. The paper does not say how it obtained them for the closed models.

Datasets §4.1 tab. 3

Each of the four training datasets is split 80/20 into train and test, and each user's posts are kept entirely in one split §4.1. The external datasets are used only for evaluation. Red-Sam was checked for overlapping posts with the training data §4.1.

Source of the table: tab. 3. Label percentages are as printed there.

Evaluation §4.3

The main metric is balanced accuracy, computed as the average of recall on the positive class and recall on the negative class. The authors chose it because it is less sensitive to class imbalance than accuracy or F1 §4.3.

# Balanced accuracy for a binary task  [§4.3]
# Input: gold labels and predicted labels, one per test item
# Output: one number in [0, 1]

def balanced_accuracy(gold, predicted):
    # Count true/false positives and negatives over the test items
    true_positive, false_negative = count_positive(gold, predicted)
    true_negative, false_positive = count_negative(gold, predicted)
    # Sensitivity: share of real positives that are found
    sensitivity = true_positive / (true_positive + false_negative)
    # Specificity: share of real negatives that are rejected
    specificity = true_negative / (true_negative + false_positive)
    # Mean of the two, so a majority-class guess scores 0.5
    return (sensitivity + specificity) / 2

How balanced accuracy is defined for the four- and five-level tasks is not stated (see Pitfalls).

Prompt construction

# Zero-shot prompt, same order as the paper  [§3.1] [eq. 1]
# Input: post text, a Part1-S context string, a Part2-Q question,
#        an output constraint
# Output: one prompt string

def build_zero_shot_prompt(post_text, context, question, constraint):
    # Concatenate the four parts in the order of eq. 1  [eq. 1]
    return post_text + context + question + constraint

# Few-shot prompt  [§3.2] [eq. 2] [§5.2]
# Input: a pool of labelled posts, one query post, a number of
#        demonstrations M (2 binary, 4 or 5 multi-class)
# Output: one prompt string

def build_few_shot_prompt(pool, query_post, demos_per_class, prompt_parts):
    # Draw one random example per class; the paper states this  [§5.2]
    demonstrations = sample_one_per_class(pool, demos_per_class)
    # Put each demonstration as zero-shot prompt + label, then the query
    # (the separator between demonstrations is not stated)  [eq. 2]
    prompt = ""
    for post, label in demonstrations:
        prompt = prompt + build_zero_shot_prompt(post, *prompt_parts) + label
    return prompt + build_zero_shot_prompt(query_post, *prompt_parts)

# Training pairs for instruction tuning over all six tasks  [§3.3.2] [eq. 3]
# Input: training rows, each with a post, a task definition and a label
# Output: a list of (prompt, target) pairs

def build_training_pairs(training_rows):
    pairs = []
    for row in training_rows:
        # Same zero-shot template as at test time  [eq. 3]
        prompt = build_zero_shot_prompt(
            row.post_text,
            row.context,
            row.question,
            row.constraint,
        )
        # The target is the label text  [eq. 3]
        pairs.append((prompt, row.label))
    return pairs

Hyperparameters

Values are from the paper unless the Where column says the README. Where the two disagree, both are listed.

Results

Author-reported. Setup: balanced accuracy on each test split, for zero-shot prompting (the ZS_best rows give the best of the zero-shot prompt designs for each cell), few-shot prompting and instruction tuning tab. 4. Standard deviations across prompt designs are in tab. 4 and the full per-design results in tab. 10; they are not repeated here.

Main table, six tasks, from tab. 4:

Source: tab. 4. The GPT-3.5 and GPT-4 rows use the best of zero-shot and few-shot, as in tab. 7. For GPT-4 on task #2 this is 0.724 (few-shot), while the zero-shot best alone is 0.719 tab. 4.

Key findings, with citations:

• Zero-shot. Alpaca and Alpaca-LoRA average 5.5% and 5.6% above the majority baseline. GPT-3.5 averages 12.4% above it. FLAN-T5 beats Alpaca zero-shot by 10.9% on average and beats GPT-3.5 by 4.2%. GPT-4 averages 6.4% above FLAN-T5 §5.1.
• Prompt enhancements §5.1.1 tab. 5. Context enhancement gives the most stable gains across tasks. Mental-health enhancement helps less, and the combination helps GPT-3.5 most (+3.9% on average) but not on every task. The enhancements reduce performance for Alpaca-LoRA (−2.7%) and FLAN-T5 (−1.6%). The paper reports the largest gains on Task #4 and Task #6.
• Few-shot §5.2 tab. 6. Few-shot beats zero-shot by 4.1% on average on the two datasets tested. Alpaca gains 8.1% and FLAN-T5 gains 5.9%. GPT-3.5 and GPT-4 gain 1.2% and 0.9%.
• Instruction tuning §5.3 tab. 7. Mental-Alpaca beats the unfine-tuned Alpaca by 23.4% on average (zero-shot) and Mental-FLAN-T5 beats FLAN-T5 by 14.7%. Both beat the best GPT-3.5 setting on all six tasks, by 10.1% and 11.6% on average (text). Against the best GPT-4 setting, Mental-Alpaca wins 4 of 6 tasks (+4.0% on average) and Mental-FLAN-T5 wins 5 of 6 (+5.5%) according to the text (see Pitfalls for a check of these numbers).
• Dialogue vs task-solving models §5.3.1. After tuning, Mental-Alpaca and Mental-FLAN-T5 are close (+1.4% for FLAN-T5 on average). FLAN-T5's zero-shot advantage disappears after tuning.
• Single-dataset transfer §5.3.2 tab. 8. Tuning on one dataset and testing on the others gives +4.2% on average over zero-shot Alpaca. The gain varies a lot by task: +19.2% on Task #3 and +16.4% on Task #5, but −7.6% on Task #6 and −0.4% on Task #4. Dreaddit and DepSeverity transfer well to each other, which the authors attribute to shared text.
• External datasets §5.3.2 tab. 9. Mental-Alpaca and Mental-FLAN-T5 improve on zero-shot Alpaca and FLAN-T5 on all three external sets. They rank first or second on two of the three, as the text says. On SAD, zero-shot GPT-4 is first at 0.854, ahead of Mental-Alpaca (0.819) and Mental-FLAN-T5 (0.779).
• Data size §5.3.3 fig. 1. With 1% of the training data, the tuned model beats zero-shot on five of six tasks. With 5%, it beats zero-shot on all six. The gap between 10% and 100% of the data is 5.9% on average. The authors note that a few hundred samples across datasets already give good results.
• Many datasets vs one §5.3.4. At equal total size, tuning across datasets beats tuning on one dataset by 3.8% (5%), 8.1% (10%) and 12.4% (20%) on average.
• Reasoning case study §5.4 fig. 2–fig. 4. The authors append "Provide reasons step by step" to the output constraint §5.4. GPT-4 gives the strongest reasons in the examples, followed by GPT-3.5 and Alpaca. FLAN-T5 gives short reasons. After tuning on classification only, Mental-Alpaca and Mental-FLAN-T5 no longer produce reasons §5.4 §6.1. The examples also show plausible but wrong reasoning, including over-generalising from one stressful situation §5.4.2.

Paper vs code

The official repository is github.com/neuhai/Mental-LLM at commit efe13de0748159f1f9957dcd63f503ac48145ca7 (last commit "Update README.md"). Its tracked files are README.md and three result images, and it has no LICENSE file (checked by listing the repository). Its own to-do list leaves "Organize code for prompt designing, model fine-tuning, and inference" unchecked README.md. The paper's experiments therefore cannot be checked against code.

The repo links model weights on the Hugging Face hub (NEU-HAI/mental-alpaca, NEU-HAI/mental-flan-t5-xxl). It states that both are for research use only README.md.

What the README and paper say differently:

• Mental-FLAN-T5 learning rate and epochs. The paper gives 2e-5 and 3 epochs for both tuned models §4.3. The README gives 1e-4 and 2 epochs for Mental-FLAN-T5 README.md. Which values produced the released weights is not stated.
• Mental-Alpaca settings. The README says they are "mostly" the Stanford Alpaca settings with minor changes that are not listed README.md. The paper gives the values in §4.3. Neither source states the batch size for Mental-Alpaca.
• Mental-RoBERTa baseline. The README gives training settings and says labels were converted to integers starting at 0 README.md. The paper does not give these settings in the sections read.
• Version. The README says an updated version of the paper is under review README.md. This card follows arXiv v4 (the version fetched).

No official code for prompt construction, few-shot sampling, data preprocessing, or evaluation was found. The repo does not contain the prompt templates, so tab. 2 is the only source for them.

Pitfalls

• No training or inference code. The prompt string format, label-word parsing, tokenization, loss masking, decoding parameters and the few-shot separator are not stated. Implementers must write these and expect a different result from the paper's.
• Two sets of tuning hyperparameters. Paper and README disagree on Mental-FLAN-T5 (learning rate 2e-5 vs 1e-4; 3 vs 2 epochs). Start from the paper's values and treat the README's as an alternative to test, not as fact.
• Abstract numbers are not the body numbers. The abstract says the tuned models beat GPT-3.5 by 10.9% and GPT-4 by 4.8% [abstract]. The body gives 10.1% and 11.6% against GPT-3.5, and 4.0% and 5.5% against GPT-4 §5.3. The abstract figures equal the mean of the two body figures in each pair. The paper does not state this.
• Check the body text against the table. The text says Mental-FLAN-T5 beats the best GPT-4 setting on 5 of 6 tasks, by 5.5% on average §5.3. Computed from tab. 7, the per-task differences are positive on all six, with a mean of about 6.5 points. For GPT-3.5, the same calculation gives about 12.6 points for Mental-FLAN-T5, against 11.6% in the text. These are my own calculations from the table, using a simple mean of the six per-task differences, which the paper does not define. Use the table values.
• "Best" zero-shot cells. tab. 4 reports the best of four prompt designs for each cell. The paper does not say whether that choice was made on a validation split or on the test results. Treat the ZS_best rows as optimistic unless you confirm the selection rule.
• Few-shot M. The text says M is 4 or 5 for multi-class tasks §5.2. The two datasets that were tested have four classes at most tab. 3, so the purpose of M=5 is not stated.
• Balanced accuracy for multi-class tasks. The paper defines the metric as the mean of sensitivity and specificity §4.3, which is a binary definition. The majority baselines of 0.250 (four classes) and 0.200 (five classes) suggest a macro average over classes, but the paper does not say so. Check this before you compare against the table.
• Class imbalance across tasks. Some test sets are heavily skewed: Twt-60Users is 9.3% positive and SAD is 94.0% positive tab. 3. A high balanced accuracy on one of them says little about the other.
• Closed-model snapshots. The results use gpt-3.5-turbo and gpt-4-0613. Re-running the prompts needs those snapshots, and the paper gives no other setting that would reproduce the API output.
• Parameter counts. The paper's figures of 175B and 1700B for GPT-3.5 and GPT-4 are its own statements §4.2. Do not use them as verified facts.
• Reasoning is lost after tuning. Both tuned models stop producing reasons after training on classification alone §5.4 §6.1. The reasoning results apply only to the untuned models.
• Bias and ethics. The authors name racial and gender bias in LLMs, and possible bias in the labels, and they say the results do not indicate deployability §6.3 §6.4. Fairness was not evaluated, because the datasets have no demographic information §6.4.
• Data overlap with pretraining. The posts may be in the pretraining data of the large models. The authors argue that the labels are not, but cannot measure the effect §6.4.
• Weights are for research use only. The README states this for both released models README.md. This limits how an implementer can use them, whatever the paper's license says.
• Label sources. Dreaddit labels are aggregated annotator judgements about sentence segments, and the depression and suicide labels follow annotation schemes based on DSM-5 and C-SSRS §4.1. The paper does not say how much these labels agree with clinical assessment.

Lineage

• Builds on: Alpaca (Taori et al., 2023) and LLaMA 7B §4.2; LoRA (Hu et al., 2021) for Alpaca-LoRA §4.2; FLAN-T5 (Chung et al., 2022) and T5 §4.2; the zero-shot prompting work cited in §3.1.
• Compared against: Mental-RoBERTa (Ji et al., 2021) and BERT as task-specific baselines §5.1 tab. 4; GPT-3.5 and GPT-4 as prompted closed models §4.2.
• Superseded by: not stated in the sections read.