AugGPT: Leveraging ChatGPT for Text Data Augmentation

In plain words

Training a text classifier needs many labelled sentences, and in medical or other private domains there are often only a handful. AugGPT gets more training sentences by asking ChatGPT to reword each of the few examples several times, keeping the same meaning and label. The classifier (BERT) is first trained on a larger base dataset, then trained on the reworded sentences, with an extra loss that pulls sentences of the same class closer together in the model's representation space.

In the authors' experiments this gave better accuracy than a long list of older, rule-based or small-model augmentation methods. The results come from the authors' own runs on three datasets, and the paper itself notes that ChatGPT can make mistakes on medical text.

TL;DR

• Few-shot text classification with BERT; ChatGPT generates six rephrasings per labelled sentence §4.2.
• Training: BERT on a base dataset with cross-entropy, then on the augmented few-shot data with cross-entropy plus a contrastive loss alg. 1 eq. 11.
• Author-reported gains over 19 augmentation baselines on Amazon, Symptoms and PubMed20K tab. 2.
• No official code found. Several details (base dataset, optimizer, ChatGPT settings) are not stated.

Problem

Few-shot text classification leaves a model with too few labelled examples per class to generalise. Existing augmentations either change words in ways that can flip the label (synonym or embedding swaps, random edits), or they stay close to the original text and add little diversity §1 §2.1. The paper's aim is augmentation that keeps labels correct and produces varied sentences §1 §2.1.

Method

Overall pipeline. alg. 1 §4.1

1. Fine-tune BERT on the base dataset D_b with cross-entropy only §4.3.
2. Generate augmented samples D_n^aug from the novel (few-shot) dataset D_n with ChatGPT §4.2.
3. Fine-tune BERT on the augmented data with the combined objective L = L_CE + λ L_CL eq. 11 §4.3.

ChatGPT augmentation. ChatGPT is applied to each input sentence and asked to rephrase it into six additional sentences that keep the label §4.2 §5. The prompt is shown as a figure only (single-turn prompt for Symptoms and PubMed20K, multi-turn prompt for Amazon) fig. 2 §4.5. The paper does not state the prompt wording in the text.

Classifier. The [CLS] vector z_c from the top BERT layer goes through a linear layer, giving the prediction ŷ eq. 7 eq. 8. Cross-entropy over the union of base and novel label spaces is eq. 9.

Contrastive loss. Computed on z_c within a batch. Same-class pairs are the positives and cosine similarity is the score eq. 10 eq. 12. The paper writes the loss without a temperature term eq. 10.

Motivation for ChatGPT. The paper argues that ChatGPT's pre-training on large corpora, its human-feedback fine-tuning and its RL training give more varied, more natural and higher-quality rewrites than earlier methods §2.3 §4.2.

# AugGPT few-shot training  [alg. 1] [§4.1]
# Input:  base data (x, y) pairs; novel few-shot data (x, y) pairs
# Output: fine-tuned BERT classifier

def train_augg(model, base_data, novel_data, epochs_base, epochs_few):
    # Stage 1: fine-tune on the base dataset, cross-entropy only  [§4.3]
    for epoch in range(epochs_base):
        train_epoch(model, base_data, loss=cross_entropy)
    # Stage 2: rephrase each novel sentence with ChatGPT  [§4.2] [alg. 1]
    augmented = augment_with_chatgpt(novel_data)
    # Stage 3: few-shot training with CE + lambda * contrastive  [eq. 11]
    for epoch in range(epochs_few):
        train_epoch(model, augmented, loss=combined_loss)
    return model

# ChatGPT augmentation  [§4.2] [fig. 2]
# Output: list of (sentence, label) pairs

def augment_with_chatgpt(novel_data):
    # Six rephrasings per input sentence, label kept  [§4.2]
    augmented = []
    for sentence, label in novel_data:
        variants = chatgpt_rephrase(sentence, label, count=6)
        augmented.extend((variant, label) for variant in variants)
    return augmented

# Combined objective  [eq. 11]
# Lambda is 1 on Symptoms and PubMed20K  [§5]

def combined_loss(batch_features, batch_labels, logits, lam):
    # Cross-entropy over all classes  [eq. 9]
    ce = cross_entropy(logits, batch_labels)
    # Contrastive term on z_c vectors  [eq. 10] [eq. 12]
    cl = contrastive_loss(batch_features, batch_labels)
    return ce + lam * cl

# Contrastive loss over a batch  [eq. 10] [eq. 12]
# Score is cosine similarity of z_c  [eq. 12]

def contrastive_loss(features, labels):
    # Pairwise cosine similarity within the batch  [eq. 12]
    sims = cosine_matrix(features)                # (B, B)
    total = 0
    for i in range(len(labels)):
        # Positives: other samples with the same class  [eq. 10]
        positive_terms = sum_exp(sims[i], same_class_mask(labels, i))
        # Negatives: samples from other classes  [eq. 10]
        negative_terms = sum_exp(sims[i], diff_class_mask(labels, i))
        total += -log(positive_terms / (positive_terms + negative_terms))
    return total  # reduction over anchors not stated  [eq. 10]

Hyperparameters

Results

Author-reported. Setup: BERT as classifier; 2-shot novel data; the "BERT" column is plain fine-tuning and "BERT + C" adds the contrastive loss. Accuracy, 0 to 1 scale tab. 2 §5.1.

The ChatGPT row is a single merged cell in the paper, so the same value appears in both columns above. Embedding cosine similarity and TransRate for augmented data are plotted, not tabled fig. 3 §5.2. The paper reports that AugGPT ranks at the top-right on Symptoms and PubMed20K for those two measures §5.2.

Paper vs code

No official code.

Search, in order: (1) the paper's text contains no code link, and the arXiv abstract page lists none; (2) the paper links no project page; (3) GitHub search for "AugGPT" returned four repositories (yhydhx/AugGPT, etherealgemini/AugGPT, KaiSheng-Zheng/AugGPT, Siamias202/SATD-Identification-and-Classification-Using-AugGPT). None is linked from the paper, and none could be shown to belong to an author: the owners are personal accounts with no author names or affiliations in their profiles, and the repositories carry no license. They were not cloned or used.

Pitfalls

• Augmentation count is inconsistent. §4.2 says each input sentence is rephrased into six additional sentences. §5 says "six augmented samples per class". Implement per sentence, as §4.2 states, unless the code shows otherwise; the code is not available.
• Training data for stage 3 is inconsistent. alg. 1 and §4.1 train only on the augmented data D_n^aug; the real novel samples are commented out of the algorithm (D' = D_n ∪ D_n^aug). §5 says the fine-tuning uses "the combination of few-shot samples and the augmented samples". Decide which one to implement and record the choice.
• Base dataset is never named. Stage 1 trains on D_b, but the paper does not say which data this is for Symptoms or PubMed20K, or how large it is. Results depend on it.
• Contrastive loss details are missing. Eq. 10 has no temperature, no stated reduction over anchors, and no stated treatment of the self-pair. The sums in the equation do not state their index ranges; the pseudocode above excludes the anchor itself, which is an assumption.
• ChatGPT is not reproducible from the text. The prompt is in a figure, the model version, decoding settings and API date are not stated, and the paper gives no cost or token count.
• Single runs. No seeds, repeats, or confidence intervals are reported. Small gaps (for example, AugGPT vs InsertCharAugmentation at 0.835 on PubMed20K, or the 0.816 tie on Amazon BERT) may not survive re-runs; the paper gives no variance to judge this (inference from the missing spread, not a reported result).
• ChatGPT row conflicts with the text. The row is labelled "2-shot" in tab. 2, but §5.3 says the classification test used a "5-shot" scheme. The row also gives 0.980 on Symptoms, above AugGPT's 0.889, which the text does not discuss.
• Medical domain. The paper says AugGPT "may produce incorrect augmentation results" on medical text because ChatGPT lacks domain knowledge §6. Two of the three datasets are medical, so review generated samples before use.
• Eq. 6 notation. The PPO-ptx objective in the RLHF background writes θ_SFT where π_SFT is meant eq. 6. Eq. 6 is background only; AugGPT itself does not use RL §2.3 §4.2.
• Supplementary samples. The sample-generation tables are in a separate supplementary source file (appendix.tex), which the main source does not include. They are not in the main paper text.
• Licence of the source. Only the arXiv licence (CC BY 4.0) is used; no code licence applies.

Lineage

• Builds on: BERT (masked language modelling for fine-tuning) §4.3; ChatGPT, InstructGPT and RLHF as the generator background §2.3 §4.2.
• Compared against: augmentation baselines from nlpaug and TextAttack, including EDA-style swaps, synonym and embedding substitution, BERT contextual insertion and substitution, back-translation, and character-level noise §4.4 tab. 2.
• Related work named by the paper: Easy Data Augmentation (EDA) §2.1 §4.4; counter-fitted embeddings §2.1; TransRate transferability metric §4.6.2.
• Superseded by: not stated in this wiki.