Text de-identification

Overview

De-identification removes the details that identify a person from a text and leaves the rest usable. For clinical notes the target is the 18 identifier categories that HIPAA lists 2303.11032 tab. 1. Every method aims for the same end state: the identifying spans no longer appear in the output 2303.11032 §V.

Manual redaction is costly and error-prone. Rule-based tools need patterns tuned for each dataset, and learned taggers can lose accuracy when the data drifts from their training domain 2303.11032 §II.

Variants

Prompt-only redaction with a chat LLM

The model gets a prompt that states the task, the replacement command and a set of rules with examples for each identifier category. Each note is sent in a separate call, with no training. The replaced spans are marked "[redacted]" 2303.11032 §IV 2303.11032 fig. 8. Card: DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4.

Reported entity accuracy: ChatGPT 0.686 with an implicit prompt and 0.929 with an explicit one; GPT-4 0.908 and 0.99 2303.11032 tab. 2. The GPT-4 numbers come from 50 random samples run by hand in the web interface 2303.11032 §V.

Fine-tuned encoder taggers

The same paper reports fine-tuned BERT at 0.798, RoBERTa at 0.947 and ClinicalBERT at 0.974 on the same test data 2303.11032 tab. 2. The fine-tuning recipe is not stated 2303.11032 §V. These are baselines in that paper; no card exists for the models themselves.

Open prompted models

The paper also prompts mT0, Falcon-7b, Flan-T5-base and two Llama models. They score between 0.597 and 0.827 2303.11032 tab. 2. The text does not describe their setups, and the Llama rows conflict with the discussion (see Contradictions).

When to use which

The only evidence on this page is one paper on one dataset, so the choice below rests on that. Treat each row as a starting point to validate on your own data.

Implementation sketch

Prompt-only redaction of one note. The prompt has three parts: task, command and rules. The exact prompt text is only in a figure 2303.11032 fig. 8.

# Prompt-only redaction of one note  [arxiv:2303.11032 §IV]
# Input:  note (string); prompt (task, command, rules)
# Output: note with identified spans replaced by "[redacted]"

def redact(note, prompt):
    # Prompt in the system role, note in the user role  [arxiv:2303.11032 fig. 7]
    messages = [
        {"role": "system", "content": prompt},
        {"role": "user", "content": note},
    ]
    # One call per note; no examples and no weight updates
    return chat_model(messages)

Contradictions

• Llama results. The discussion says Llama failed to produce coherent output and is "not directly comparable" 2303.11032 §VI. The table still reports scores for two Llama models, and the rows are identical 2303.11032 tab. 2.
• GPT-4 wording. The text says "over 0.99" 2303.11032 §V; the table prints 0.99 2303.11032 tab. 2.
• Outperforming baselines. The text says the explicit-prompt GPT-4 beats all baselines 2303.11032 §V. The implicit-prompt GPT-4 score of 0.908 is below RoBERTa (0.947) and ClinicalBERT (0.974) 2303.11032 tab. 2.
• Accuracy definition. The paper gives accuracy as (TP+TN)/(TP+TN+FP+FN) 2303.11032 eq. 1, but the text describes the share of sensitive entities removed 2303.11032 §V, and TN is never defined. The two readings are not comparable.
• Dataset. The introduction mentions synthesized public data 2303.11032 §I; the evaluation uses the i2b2/UTHealth set, which is available on request 2303.11032 §III.

Pitfalls

• The prompt is only in a figure. Transcribe it from 2303.11032 fig. 8 and check the wording. The paper's linked repository does not contain it; see the card's Paper vs code section.
• Prompt wording is fragile. A stray period stopped the command from being read, and mixing several tasks in one prompt confused the model 2303.11032 fig. 9. Test every prompt edit against the scorer.
• The HIPAA list may be missing from the prompt. The introduction says the identifiers were built in 2303.11032 §I, but the prompt shows category rules 2303.11032 fig. 8. Decide whether to include the 18-item list.
• Algorithm 1 cannot run on a frozen API model. Its training loop has no step to run against an API model, and the reported results are zero-shot 2303.11032 alg. 1 2303.11032 §V. Use the single-call form.
• Single numbers, no variance. The paper reports no seeds, repeats or intervals 2303.11032 §V.
• One dataset. Generalisation to other data is named as future work 2303.11032 §VI.
• Privacy of the method. Any hosted model receives the PHI it is meant to remove 2303.11032 §VI.

Lineage

• Builds on prompt-based in-context learning 2303.11032 §II.
• Compared with fine-tuned BERT, RoBERTa and ClinicalBERT 2303.11032 §V.
• Follow-up named by the authors: local open-source models, fine-tuning on domain data and quantisation for hospital deployment 2303.11032 §VI.

Papers

• DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4: prompt-only redaction of clinical notes with ChatGPT and GPT-4, compared with fine-tuned encoder taggers.