The Opportunities and Risks of Large Language Models in Mental Health

In plain words

Mental health care is in short supply. Many people wait years for help, and there are not enough professionals. Large language models, the chatbot-style AI systems, are being tried for three jobs: explaining mental health topics, helping to judge what a person may be going through, and delivering support through chat. This paper is a review of what has been tried so far and what could go wrong.

The authors find the early results promising, especially for teaching and screening, but more cautious for direct treatment, where chatbots sometimes give poor or harmful advice. Their main warnings are about fairness, ethics, privacy, consistency, accuracy, and the lack of explanations and human oversight. They close with a list of principles for building these systems responsibly, with people who have lived experience involved from start to finish. The paper runs no experiments of its own.

TL;DR

• A narrative review, not a method: it summarises prior work on LLMs for mental health education, assessment and intervention §1, §2.
• It names eight risk areas and rates each risk for the three uses in a table §3.1, tab. 1.
• Recommendations are qualitative (for example, confidence-based withholding, human oversight, informed consent); the paper sets no numeric thresholds §3, textbox 1.
• It reports no new experiments. Quantitative results in this card are quoted from the cited studies and are author-reported by those studies §2.
• Useful as a checklist for risks and for which claims in the literature are shaky. It is not a source of implementable parameters.

Problem

The authors state that mental health concerns are rising and that care has not kept pace with demand. They cite roughly half of people experiencing a mental health disorder in their lifetime, one in eight people currently experiencing a concern, an average delay of 11 years between symptom onset and treatment in the United States, and shortages of professionals in regions with nearly half the global population §1. The question is whether LLMs can help close that gap without causing harm §1, §3.

The paper also separates two kinds of model. General-purpose LLMs are trained on broad text and can produce inaccurate, biased, stigmatising or harmful mental health content. Domain-specific LLMs are general models further fine-tuned on curated data, which reduces but does not remove that risk §2.1. The review says where a finding applies to only one of the two kinds §2.1.

Method

This is a narrative review. It contains no algorithm, model, training procedure or dataset of its own, so there are no pseudocode blocks, equations or hyperparameters to implement. The structure is:

1. Introduction and motivation §1.
2. Applications, in three groups: education §2.2, assessment §2.3 and intervention §2.4, each with the studies the authors cite.
3. Risks, with a risk-by-use table tab. 1, §3.1 and seven risk subsections: inequality, disparity and stigma §3.2; unethical provision of services §3.3; insufficient reliability §3.4; inaccuracy §3.5; lack of transparency and explainability §3.6; and not involving humans §3.7. Each subsection states a risk and a mitigation the authors recommend.
4. Conclusions §4.

The paper gives no algorithm. The block below only restates the confidence mechanism that the authors describe for Med-PaLM 2 and recommend in general, with the cutoff left as an input because the paper gives no value §3.5.

# Confidence-gated answer  [§3.5]
# Input:  question (str); cutoff (float, not stated by the paper)
# Output: answer text, or the withheld marker

def answer_or_withhold(question, cutoff):
    # Sample candidate answers with a confidence score for each  [§3.5]
    answers, confidences = model_answers_with_confidence(question)
    # Weight the candidates by their confidence scores  [§3.5]
    best = confidence_weighted_pick(answers, confidences)
    # Withhold output when confidence is below the cutoff  [§3.5]
    if best.confidence < cutoff:
        return "withheld"
    return best.text

The authors summarise their recommendations in a textbox [textbox 1]. The eight points are:

• LLMs engage in mental health tasks only when trained and shown to perform well.
• Mental health LLMs advance mental health equity.
• Privacy and confidentiality are paramount.
• Informed consent is obtained before people engage with them.
• They respond appropriately to mental health risk.
• They operate only within the bounds of their competence.
• They are transparent and able to explain themselves.
• Humans provide oversight and feedback.

Table 1 is reproduced in full from the paper. It rates each risk as Lower, Medium or Higher for each use. Indented rows are sub-risks of the row above them, and the parent row has no ratings of its own tab. 1. The ratings are the authors' qualitative judgements; the paper does not say how they were derived §3.1.

tab. 1

The paper contains four figures, each showing a stakeholder group or opportunity area. They are described here from their captions and the text; the captions are the authors' wording fig. 1 fig. 2 fig. 3 fig. 4.

• fig. 1: opportunities for LLMs in mental health education, shown with CBT and empirically supported treatments as examples.
• fig. 2: opportunities for LLMs in mental health assessment.
• fig. 3: opportunities for LLMs in mental health intervention, with CBT and evidence-based practice as examples.
• fig. 4: human involvement across development, testing, deployment and evaluation. The captioned stages are conceptualisation (co-defining the role of LLMs), development (contributing data after consent), testing (feedback on output), deployment (input on where and with whom LLMs are used, with safeguards) and evaluation (ongoing feedback after deployment).

Hyperparameters

Not applicable. The paper describes no model, training run or evaluation protocol of its own, so no values are given.

• Accuracy threshold: not stated. The paper asks for "high" standards and "prespecified" thresholds, with no values §3.5.
• Reliability threshold: not stated. It asks that LLMs not be deployed until they exceed "prespecified thresholds of adequate reliability" §3.4.
• Confidence cutoff: not stated. It recommends a cutoff but gives no value §3.5.

Results

Author-reported, and quoted from the cited studies, not from experiments in this paper. Setup and baselines are given as the review states them §2. The figures are not comparable to one another because the tasks, data and metrics differ.

Paper vs code

No official code. The paper is a review with no implementation. No project page or code link appears in the LaTeX source, the arXiv metadata or the abstract. A GitHub repository search for the title did not return any repository, so no code is cited.

Pitfalls

• No new evidence. Every number in Results comes from a cited study. The review gives only brief setups, so check the original study before relying on a figure.
• Mixed evidence. Studies differ in model generation, task and data. The early BERT-based studies are described by the authors as early generations that newer models may exceed §2.3. Do not compare numbers across rows.
• Author conflicts. The Conflicts of Interest section states that four authors are Google employees with Alphabet equity, that two are employed by Magnit and contracted to Google, and that several hold shares or consult for mental health technology companies [Conflicts of Interest]. Several systems the review discusses are Google's (Med-PaLM 2, Med-LM) §2.2, §2.3. Read the review's assessments of those systems with this in mind.
• Reference handling. In the LaTeX source every \bibitem label is duplicated (all are labelled Singhal2023 or Sezgin2023), so bibliography keys cannot be used to look up entries. The numeric citations [n] in the text were not matched to the printed reference list in this card, so cite by section where possible.
• Retracted reference. The entry for Darcy et al. (Woebot, the paper's [27]) is marked "Retracted in" Expert Rev Med Devices 2023;20(11):989. The review still cites it as a chatbot example §2.4.
• Preprints. Several cited works are preprints (arXiv, medRxiv, SSRN, Research Square). The review does not mark which findings are peer-reviewed.
• Qualitative guidance. Recommendations such as "withhold output when confidence is lacking" or "do not deploy until thresholds are exceeded" name no values. An implementer must define them.
• Section numbers. Section numbers in this card are counted from the LaTeX \section order (Introduction = §1, Applications = §2, Risks = §3, Conclusions = §4). They were not checked against the rendered PDF, so page-level citations are not given.
• Journal version. The LaTeX title block gives a journal citation (JMIR Ment Health 2024;11:e59479, doi 10.2196/59479). This card describes the arXiv version (v3, per the metadata), which may differ from the journal text.

Lineage

• Builds on: not applicable (review). It cites prior LLM and chatbot studies rather than building on a single method.
• Compared against: not applicable.
• Superseded by: not stated.