Leveraging LLMs for Translating and Classifying Mental Health Data

In plain words

Language models are being used to spot signs of depression in what people write online. Most of that work is in English, so the authors asked how well a popular chat model does on a non-English language, here Greek. Their route is indirect: they first translate English posts into Greek with the same model, then ask it to rate how severe the depression in each post is, and compare the answers with the labels that human annotators gave the English originals.

The model does poorly even in English. It finds the mildest and the most severe cases somewhat better than the middle levels, and its Greek results are worse for most levels, with one exception. The authors conclude that more research is needed, especially for languages with few resources, and that LLMs should not be relied on for translation alone in any healthcare setting.

TL;DR

The paper runs GPT-3.5-turbo, at temperature 0, as a zero-shot classifier of four depression-severity levels on the DepSeverity Reddit dataset, in English and in Greek §4, tab. 1. The Greek input is produced by the same model translating each English post §3, fig. 1. The author-reported macro F1 is 0.17 in English and 0.16 in Greek tab. 2. The paper gives no model baseline beyond its own English run, no variance across runs, and no translation quality metric.

Problem

Early detection of depression from user text needs classifiers that work outside English. Labeled mental-health data is mostly English, and Greek has no resource for this task §1, §2. The paper asks two questions: whether an LLM can predict severity from English posts, and whether the result holds when the posts are first translated by an LLM into Greek §1.

The setting assumes the LLM's translations preserve the cues that the severity label depends on. The paper's own error analysis gives a case where a panic and anxiety post was labeled minimal and predicted moderate, which the authors read as a distinction between depression and anxiety §4, Error Analysis.

Method

Pipeline. Posts are translated from English to Greek by the LLM. The resulting translations are passed to a prompt that asks the LLM for a severity level. Predicted classes are compared with the ground-truth labels in both languages §3, fig. 1. The authors state that the method is meant to apply to other language pairs, but they test only English to Greek §3, §4.

Dataset. DepSeverity, from Naseem et al. (2022), has 3553 Reddit posts in English with four labels: Minimum (2587), Mild (290), Moderate (394) and Severe (282) tab. 1 §4. The paper describes the labels as "minimal", "mild", "moderate" and "severe" depression §4. It does not state a train/test split; the classification runs are described as applied to the posts, with no training step §4.

Classification prompt. A single instruction asks the model to assign the text to one of four severity levels, coded 0 (Minimum), 1 (Mild), 2 (Moderate) and 3 (Severe) tab. 1. The paper does not state where the post goes relative to the instruction, nor whether the same instruction is used for the Greek input §4, tab. 1.

Translation. Each post is translated to Greek by GPT-3.5-turbo §4. The translation prompt is not stated.

Scoring. Per-class precision, recall and F1 are reported, with a macro average row tab. 2 §4. Text that does not contain a recognised label is counted as class 0 (Minimum) §5, Limitations.

Algorithm. The paper gives no numbered algorithm; the steps below are read from §3 and §4.

# Translate-then-classify evaluation  [§3] [§4] [fig. 1]
# Input:  English posts; gold labels in {0, 1, 2, 3}
# Output: per-class precision, recall and F1 for each language

def run_evaluation(posts, labels, llm):
    # Step 1: classify the English posts with the zero-shot prompt  [§4]
    english_predictions = []
    for post in posts:
        reply = llm(classify_prompt, post)  # temperature 0  [§4]
        english_predictions.append(parse_label(reply))  # [§5 Limitations]

    # Step 2: translate each post into Greek with the same LLM  [§3] [§4]
    greek_posts = []
    for post in posts:
        greek_posts.append(llm(translate_prompt, post))  # prompt not stated

    # Step 3: classify the Greek translations with the same prompt  [§3]
    greek_predictions = []
    for post in greek_posts:
        reply = llm(classify_prompt, post)  # Greek prompt not stated
        greek_predictions.append(parse_label(reply))

    # Step 4: score both languages against the gold labels  [§4] [tab. 2]
    english_scores = score(english_predictions, labels)
    greek_scores = score(greek_predictions, labels)
    return english_scores, greek_scores

Hyperparameters

Results

Author-reported. Setup: DepSeverity (English posts, four classes), GPT-3.5-turbo, zero-shot, temperature 0 §4. The only baseline is the same model on the untranslated English posts; there is no comparison with other models §4, tab. 2.

Text summary: the best English F1 is for Minimum (0.25), then Severe (0.22). In Greek, Mild is worst (0.06), and Moderate rises from 0.17 to 0.23 while the other classes fall §4, Classification in the target language. The authors attribute the low overall performance to the difficulty of separating the levels, which they describe as less distinct than in other conditions §4, Classification in the source language. No run-to-run variance, significance test, or confidence interval is reported.

Paper vs code

No official code.

No official repository is linked from the paper, the abstract or the arXiv listing, and the LaTeX source contains only a commented-out link to a third-party dataset repository acl_latex.tex §4. Searches for the title words and the authors' GitHub accounts returned no repository. Because no code was found, the paper's prompts and parser cannot be checked against an implementation.

Pitfalls

• Only one LLM and one prompt. The paper evaluates GPT-3.5-turbo and one zero-shot prompt. The results do not say how the model or prompt would behave on other models or prompt variants §4, tab. 1.
• Translation is part of the input, and its quality is not measured. The paper reports no translation metric. Its only check is a qualitative look at two examples in fig. 2 §4, Error Analysis. The Greek scores therefore mix classifier effects with translation effects.
• Greek prompt not stated. The authors do not say whether the classification prompt is also translated for the Greek runs. An implementer must decide this and record it.
• Single-run, temperature 0 results. Numbers come from one run each. The authors say temperature 0 was chosen for reproducibility, but the API model version is not stated, and hosted models change over time §4.
• Strong class imbalance. The Minimum class holds about 73% of the posts. Macro averages weight the rare classes equally, and the per-class recall values show the model rarely predicts Minimum (recall 0.14 in English). Any reimplementation should report per-class numbers as well as the macro average tab. 1, tab. 2.
• Rounding in the macro row. The unrounded mean of the four English per-class F1 values is about 0.178, while the table prints 0.17. The paper does not explain how the macro row was computed tab. 2.
• Unparsed output counts as Minimum. A reply with no recognisable label is scored as class 0, which inflates Minimum precision when the model produces unparseable text §5 Limitations. The label-extraction rule itself is not specified.
• Dataset name is written two ways. The table says DepSeverity, while the text says Dep-Severity. Both refer to the same dataset of Naseem et al. (2022) §4, tab. 1.
• Labels are used as given. The paper does not check the dataset's labels. Its Limitations section calls the publicly available datasets' quality a concern for this domain §5 Limitations.
• Preliminary self-description is not the evaluated prompt. Before the experiments, the model was asked how it would classify severity and gave its own four-level description §3 Preliminary Prompting. That description was not used as the evaluation prompt, which is the quoted prompt in tab. 1. An implementer should not substitute it.
• Costs and scope. The total cost was under USD 30 §4 Cost. The authors say the approach is not meant to diagnose or assist patients; it is meant to help train professionals §5 Conclusion. Use outside that scope is not supported by the paper.

Lineage

• Builds on: DepSeverity dataset from Naseem et al. (2022), cited in §4; no wiki card.
• Compared against: the same model on English posts, as the internal baseline tab. 2.
• Related to: BioBERT, ClinicalBERT, MentalBERT and DisorBERT, cited in §2 as domain models for mental-health text; no wiki cards.
• Superseded by: not stated.