Efficient Few-Shot Learning Without Prompts

In plain words

Getting a text classifier to work with only a handful of labelled examples per category is hard. The two popular routes both have costs. Some methods ask a very large language model to continue a hand-written question about the text, which takes careful wording and a huge model. Others train a small model but still depend on that wording.

SetFit skips the wording. It starts from a sentence encoder that already maps similar sentences to nearby points. It then reshapes that space with a few labelled examples: sentences with the same label are pulled closer together and sentences with different labels are pushed apart. Each labelled example can be paired with many others, so a few examples produce many training pairs. After this tuning step, a simple logistic-regression classifier is trained on the new sentence embeddings.

The result is a small model, fast to train and to run, with no prompts to write. The authors report that it matches much larger prompt-based methods on several text classification tasks, and that it works across languages when the encoder is swapped for a multilingual one.

TL;DR

SetFit trains a Sentence Transformer body with a cosine-similarity loss on positive and negative sentence pairs built from the labelled examples, then fits a logistic-regression head on the body's embeddings §3.1. No prompts or verbalizers are used §1. Author-reported, with 8 examples per class, the 110M-parameter SetFit model averages 62.3 on five test datasets, against 63.4 for T-Few 3B (which needs prompts) and 58.3 for ADAPET tab. 2. The official repository (Apache-2.0) implements the method but its defaults differ from the paper in several places; see Paper vs code.

Problem

Few-shot text classification with pretrained language models usually relies on either in-context prompting of very large models, or on parameter-efficient or prompt-based fine-tuning. The paper argues that these are impractical for many researchers and practitioners because they need billion-parameter models and specialised infrastructure, and because prompt-based methods depend on manually written prompts whose quality changes the outcome §1. The assumption is that the task has a small labelled set (the paper uses N = 8 and N = 64 per class) and that a sentence encoder is available as a starting point §1 §4.4.

Method

SetFit has two stages, followed by inference §3 fig. 2.

Stage 1, contrastive fine-tuning of the Sentence Transformer body §3.1. Given K labelled examples D = {(x_i, y_i)} over C classes, the method builds two pools of pairs for each class. Positive pairs join two examples of the same class, and negative pairs join an example of class c with an example of another class §3.1. The number of pairs per class is a hyperparameter R, which the paper sets to 20 unless stated otherwise §3.1. The total is |T| = 2R|C| pairs, and the pairs are used as a single training set §3.1. The pair count is motivated by there being K(K-1)/2 possible unique pairs, far more than K §3.1. Each pair is scored by the cosine similarity of its two embeddings, and the body is fine-tuned so that this similarity matches the target label (1 for a positive pair, 0 for a negative one) §3.1 §4.4.

Stage 2, classification head §3.1. The fine-tuned body encodes the original K labelled examples, one embedding per example. A logistic-regression model is trained on these embeddings and their labels §3.1. The paper does not describe updating the body again in this stage.

Inference §3.1. An unseen sentence is embedded with the fine-tuned body and classified by the head: x_pred = CH(ST(x)).

Pseudocode. Written from the paper and cross-checked against the official code. Names and loop structure are ours. The contrastive loss is the sentence-transformers cosine-similarity loss, which is imported rather than defined in the repository training_args.py.

# Build the contrastive training pairs  [§3.1]
# Input:  texts (K strings), labels (K class ids)
# Output: list of (text_a, text_b, target), target is 1.0 or 0.0

def build_pairs(texts, labels):
    # Enumerate pairs of examples, including each example with itself,
    # in a fixed shuffled order  [code: src/setfit/sampler.py#L15-L31]
    pair_indices = shuffled_pair_indices(len(texts), seed=42)
    positives = []
    negatives = []
    for first, second in pair_indices:
        # Same class -> positive pool; different class -> negative pool
        # [§3.1] [code: src/setfit/sampler.py#L91-L101]
        if labels[first] == labels[second]:
            positives.append((texts[first], texts[second], 1.0))
        else:
            negatives.append((texts[first], texts[second], 0.0))
    # Oversampling: both pools are cycled to the larger pool's size
    # [code: src/setfit/sampler.py#L83-L85]
    pool_size = max(len(positives), len(negatives))
    return cycle_to(positives, pool_size) + cycle_to(negatives, pool_size)

# Fine-tune the Sentence Transformer body  [§3.1]
# Input:  pairs from build_pairs, body (pretrained Sentence Transformer)
# Output: fine-tuned body

def fine_tune_body(pairs, body):
    # One epoch over the pairs in batches of 16  [§4.4]
    #   [code: src/setfit/training_args.py#L170]
    #   [code: src/setfit/training_args.py#L174]
    for batch in batches(pairs, batch_size=16):
        # Embed both sides of every pair with the shared body  [§3]
        embeddings_a = body.encode([text_a for text_a, _, _ in batch])
        embeddings_b = body.encode([text_b for _, text_b, _ in batch])
        # Cosine similarity of each pair is pulled toward its 0 or 1 target
        # [§3.1] [code: src/setfit/training_args.py#L187]
        loss = cosine_similarity_loss(embeddings_a, embeddings_b, targets)
        # Update the body's weights; the optimizer and schedule are not stated
        # in the paper  [§4.4]
        body.update(loss)
    return body

# Fit the classification head on the frozen, fine-tuned body  [§3.1]
# Input:  body (fine-tuned), texts (K strings), labels (K class ids)
# Output: fitted head

def fit_head(body, texts, labels):
    # Encode every labelled example once  [§3.1]
    # [code: src/setfit/modeling.py#L316]
    embeddings = [body.encode(text) for text in texts]
    # Logistic regression; the code passes no parameters, so the library
    # defaults apply  [§3.1] [code: src/setfit/modeling.py#L849]
    head = LogisticRegression()
    head.fit(embeddings, labels)
    return head

# Predict a label for an unseen sentence  [§3.1]
# Input:  body (fine-tuned), head (fitted), text (one string)
# Output: predicted class

def predict(body, head, text):
    # x_pred = CH(ST(x))  [§3.1]
    embedding = body.encode(text)
    return head.predict(embedding)

Worked count from the code (derived here, not in the paper). With K = 16 examples, two classes of 8 each, and the default oversampling: the unordered pairs include the 16 self-pairs, so the positive pool has 72 pairs (2 × 28 same-class pairs plus 16 self-pairs) and the negative pool has 64. Both pools are then set to 72, for 144 pairs in total sampler.py sampler.py. The paper's formula gives |T| = 2 · 20 · 2 = 80 pairs for the same case §3.1. See Paper vs code.

Hyperparameters

Values are from the paper unless the "Where" column says code. Stated values are for the SetFit method; baseline settings are listed for completeness.

Results

Author-reported throughout. Setup: six English test datasets (SST-5, Amazon Counterfactual, CR, Emotion, Enron Spam, AG News) tab. 2 app. A.1; models listed in Hyperparameters; baselines fine-tuned RoBERTa-large, PERFECT, ADAPET and T-Few 3B §4.3. For each method the paper reports the mean and standard deviation over the 10 splits §4.4, except T-Few, which is the median of 5 seeds §4.3. The source prints the standard deviation as a subscript; here it follows ±. Average excludes AG News to allow a fair comparison with T-Few, whose training data includes AG News tab. 2. Bold marks in the source are omitted.

Main results, accuracy in % (Amazon Counterfactual: MCC) tab. 2

† AG News excluded from the average. ⋆ SST-5 inputs (not labels) appear in T-Few's training set via Rotten Tomatoes tab. 2. Source: tab. 2 §5.

Relative to the baselines, the paper states that SetFit (MPNet) beats fine-tuning by 19.3 points at N = 8 and by 5.6 points at N = 64; beats PERFECT by 13.6 and 2.6 points; beats ADAPET by 4.0 and 1.5 points; and beats T-Few 3B by about 5 points at N = 64 §5.

RAFT leaderboard (as of 5 Sept 2022), average accuracy tab. 3 §5

The paper reports that the RoBERTa-large SetFit beats GPT-3 by 8.6 points and PET by 1.7 points, and falls 4.5 points short of T-Few 11B §5. The RAFT numbers are leaderboard values, not the authors' own runs tab. 3.

Multilingual, MARC, mean absolute error × 100, lower is better tab. 4 §6 Languages: En, De, Ja, Zh, Fr, Es. Train settings: each (monolingual), en (English-only), all.

The paper states that at N = 8, SetFit beats fine-tuning and ADAPET in every setting, and that its best average comes from English-only training §6.

Computational cost, N = 8 tab. 5 §7.2 (estimates from FLOP formulas, not measured; T-Few 3B is an encoder-decoder, so its estimates are halved)

The paper also reports wall-clock training on AWS: about 30 seconds for SetFit (MPNet) on a p3.2xlarge at $0.025 per split, against about 700 seconds on a p4d.24xlarge at $0.7 per split for T-Few 3B §7.2. Storage is 70 MB and 420 MB for the two SetFit models, against 11.4 GB for the T0-3B checkpoint used by T-Few §7.2.

Few-shot distillation fig. 3 §7.1 Teacher: SetFit MPNet (16 labelled examples per class). Students: SetFit MiniLM (15M) versus a standard MiniLM-L3 student, both given the same labelled examples plus an unlabelled pool of size N. Values are read from the plotted data in the source:

The paper states that the gap is 24.8, 25.1 and 8.9 points at N = 8 and that the two students are on par at N = 1K §7.1. The metric is average accuracy fig. 3.

Paper vs code

Official code: https://github.com/huggingface/setfit (linked in the abstract), cloned at commit d70dad84de757d30f03465d120e0d3fad2c725b0, Apache-2.0 LICENSE. The repository is at its current HEAD, not at a release matched to the 2022 paper. I did not find a tag or commit that matches the paper's experiments, so every default below describes HEAD, and the paper numbers may come from earlier code.

Differences between the paper and HEAD:

• Pairs per class. Paper: R = 20 positive and 20 negative pairs per class, |T| = 2R|C| §3.1. Code default: num_iterations is None and the strategy is "oversampling", so each pool is sized to the larger of the two pools, cycled training_args.py sampler.py. The legacy SetFitTrainer defaults to num_iterations=20, which means 20 pairs per example (not per class) in each pool trainer.py sampler.py. With K = 16 examples and two classes, the paper gives 80 pairs, the oversampling default gives 144, and the legacy default gives 640.
• Pair construction. Paper: anchors are drawn per class, with positives from the same class and negatives from other classes §3.1. Code: all unordered pairs over the whole training set are enumerated, including each example with itself (replacement=True), then each is labelled positive or negative by its class sampler.py sampler.py. The self-pairs are not mentioned in the paper.
• Learning rate. Paper: 1e-3 §4.4. Code default for the body: 2e-5, with the head rate at 1e-2 and a second body rate of 1e-5 that applies only to the differentiable head training_args.py. The legacy SetFitTrainer defaults to 2e-5 trainer.py.
• Batch size and epochs. Paper: 16 and 1 §4.4. Code defaults match training_args.py training_args.py.
• Max sequence length. Paper: 256 tokens §4.4. Code: max_length defaults to None training_args.py. Only the classifier stage passes it on trainer.py; the body stage passes no length, so the body's own maximum is used. I did not trace the sentence-transformers code path for the body stage.
• Classifier head. Paper: logistic regression §3.1. Code default: the sklearn LogisticRegression with an empty head_params modeling.py. The code also offers a differentiable PyTorch head, off by default modeling.py modeling.py. The paper does not describe it.
• Weight decay and warmup. Not stated in the paper. Code defaults: weight decay 0.01 training_args.py, warmup 10% of steps training_args.py trainer.py.
• Loss. Paper: cosine similarity §3.1. Code: CosineSimilarityLoss from sentence-transformers training_args.py. The exact formula is not in this repository, so I cannot cite it.
• Code scope. HEAD adds components the paper does not describe, including a multi-label path, a distillation trainer and extra losses trainer_distillation.py losses.py. I did not read the distillation trainer in detail.

Pitfalls

• Pair count is ambiguous. The paper's R = 20 per class gives 80 pairs in a two-class, 16-example case; the code's default gives 144. If you reproduce the paper, set the pair count explicitly rather than relying on defaults. Derived counts above are from the code, and the paper gives no counts for specific cases.
• Oversampling repeats pairs. The smaller pool is cycled to the larger size, so many pairs are repeated, not new sampler.py. The paper's "K(K-1)/2 unique pairs" is a count of possibilities, not of unique pairs used in training §3.1.
• Learning rate is an order of magnitude apart. The paper's 1e-3 §4.4 and the code default 2e-5 training_args.py differ by 50×. The paper does not explain the choice.
• Development data set the hyperparameters. The paper says the development datasets fix the number of pairs, the loss and the epochs §4.1, and tab. 6 lists them. The paper does not say how much labelled data the development sets have, so whether the few-shot numbers reflect fully labelled tuning is not clear.
• Baseline validation. For fine-tuning, the paper says it takes validation splits from the training split, "equally sized", and searches epochs in [25, 75] §4.3. The number of examples used for validation is not stated, so the comparison's data budget is unclear.
• Standard deviations are large at N = 8. Amazon Counterfactual has 40.3 ± 11.8 for SetFit tab. 2. Compare methods with the standard deviation, not the mean alone.
• Averages use different subsets. T-Few's average excludes AG News for the reason given in the table note tab. 2, and T-Few's SST-5 result uses SST-5 inputs seen in its training data tab. 2. The comparison is weaker than the table suggests.
• T-Few runs are median-of-5 seeds, SetFit is mean-of-10 splits §4.3 §4.4. The aggregations differ.
• T-Few 11B is not in the paper's own runs. Its RAFT score comes from the leaderboard §4.3 tab. 3.
• Computational numbers are formula estimates. Inference is 2N · ℓ_seq and training is 6N · ℓ_seq · n_steps · n_batch, with n_steps = 1000 and n_batch = 8 §7.2. Two cases do not match the stated inputs. For the 15M MiniLM with ℓ_seq = 38, 2 · 15M · 38 ≈ 1.1e9, not the 1.3e9 in the table; the paper does not explain the gap tab. 5. The storage ratio also uses 26 where 11.4 GB / 420 MB ≈ 27 §7.2.
• Distillation pairs use the teacher's similarities, not the labels. The student body mimics the teacher's cosine similarities, and the student head mimics the teacher's logits §7.1. A baseline student is trained on the teacher's logits only.
• Multilingual results. The SetFit body is paraphrase-multilingual-mpnet-base-v2, while the fine-tuning and ADAPET baselines use XLM-RoBERTa-base §6. ADAPET's patterns were translated by native speakers app. A.2. The paper reports that ADAPET on non-English data was not examined before §6.
• Costs are hardware-specific. Training time and cost are for one AWS instance type, and the paper gives them for one split §7.2.
• No prompts does not mean no design choices. The paper's contribution is prompt-free, but the sentence encoder, pair count, number of epochs and classifier are still chosen by the implementer §3.1 §4.4.

Lineage

• Builds on: Sentence-BERT (siamese and triplet sentence embeddings) §2 §3; the contrastive siamese approach from image similarity §3; and the Transformer-based fine-tuning literature for few-shot text classification §2.
• Compared against: fine-tuning RoBERTa-large §4.3, PERFECT §4.3, ADAPET (an extension of PET) §4.3, T-Few 3B and 11B §4.3, GPT-3 on RAFT tab. 3.
• Superseded by: not stated in the paper, and no successor is known from this wiki.