Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks

In plain words

Speech recordings are long, dense signals, and labelling them for every new task is slow. This paper asks whether a speech model can learn useful features from unlabelled audio alone. Its answer is to train one shared feature extractor while it solves several "guess something about the audio" puzzles at the same time. Some puzzles ask the model to rebuild the raw sound or classic speech measurements such as pitch and loudness. Others ask it to tell apart short snippets that come from the same sentence and snippets that come from elsewhere, or from the past and the future of the same sentence.

The shared extractor cannot satisfy all puzzles with shortcuts, so it has to keep information that matters for speech: who is speaking, which sounds are being made, and how the speaker feels. Afterwards the extractor can be used as a fixed feature source, or fine-tuned on a labelled task. The authors show this on speaker identification, emotion recognition and speech recognition, including noisy, reverberant recordings.

TL;DR

PASE is a speech encoder that reads the raw waveform and is trained with no labels, by jointly training seven small "worker" networks on different self-supervised tasks. Its features are meant to be reused across speech problems. Using them frozen or fine-tuned beats MFCC and FBANK features on the three tasks reported, and the gain carries over to a noisy, reverberant test set. All numbers are author-reported on the authors' own setups, with a single run per setting.

Problem

Supervised speech features need large labelled corpora, and features learned for one task can be biased toward it §1. Self-supervised learning on speech is hard because speech is long, high-dimensional and hierarchical, so no single pretext task obviously yields general features §1. The authors assume a large clean corpus is available for pre-training (a LibriSpeech subset) and that the downstream tasks are speaker identification, emotion classification and phoneme or word recognition §3.

Method

PASE has two parts: a fully convolutional encoder applied to raw 16 kHz audio, and seven workers that read the encoder output §2 fig. 1. The encoder and the workers are trained together. Each worker is a very small network, so that the encoder, not the workers, has to carry the high-level structure §2.2.

Encoder §2.1. The first layer is a SincNet layer: learned band-pass filters with 64 outputs and a kernel width of 251 samples and stride 1 §2.1. Seven convolution blocks follow. Each block is a 1-D convolution, batch normalization and PReLU §2.1. The blocks use the kernel widths, filter counts and strides listed in the Hyperparameters table. A final 1×1 convolution maps to 100-dimensional embeddings, and a non-affine batch-normalization layer normalizes them §2.1. The overall stride is 160 samples, so one frame is 10 ms at 16 kHz §2.1.

Regression workers §2.2. Each predicts a target from the encoder output with one hidden layer of 256 PReLU units, except the waveform worker, which is a decoder:

• Waveform: an auto-encoder. Three transposed convolutions with strides 4, 4 and 10 upsample the encoder output back to samples, then a small MLP outputs one value per sample. Loss is L1 §2.2.
• Log power spectrum (LPS): 1025 frequency bins, 25 ms window, 10 ms step, MSE §2.2.
• MFCC: 20 coefficients, MSE §2.2.
• Prosody: four frame-level features (log F0, voiced/unvoiced probability, zero-crossing rate, energy), MSE §2.2.

Discriminative workers §2.2. Each takes an anchor chunk x_a, a positive x_p and a negative x_n, and is trained with binary cross-entropy so that positives score as matching the anchor and negatives do not §2.2. The encoder and the discriminators cooperate rather than compete §2.2. The three samplers are:

• LIM (local info max): the positive is another chunk of the anchor's sentence, and the negative is a chunk from a random other sentence §2.2.
• GIM (global info max): the anchor is the average of encoded frames over a random 1 s chunk. The positive comes from another chunk of the same sentence, and the negative from another sentence §2.2.
• SPC (sequence predicting coding): the anchor is one frame. The positive is the next 5 frames and the negative the previous 5 frames. Windows are kept outside the 150 ms receptive field, and the paper says they are drawn up to 500 ms away §2.2.

Joint training §2.3. The total loss combines the worker costs. The paper describes it as their average, so gradients from every worker reach the shared encoder §2.3. Regression targets are standardized with training-set mean and variance so that each regression loss has a comparable scale §2.3. Training uses Adam, mini-batches of 32 one-second chunks at 16 kHz, a learning rate of 5e-4 that is halved every 30 epochs, and 150 epochs §2.3.

Use downstream §2.4. Three ways to use the encoder are compared: PASE-Frozen (encoder fixed, classifier trained on its features), PASE-FineTuned (encoder and classifier trained together on the target task), and PASE-Supervised (the same architecture trained from random initialization on the target task) §2.4.

# Joint self-supervised training step for PASE  [§2.2] [§2.3]
# Input:  anchor, positive, negative: raw chunks (B, 16000) at 16 kHz
# Output: scalar total loss; encoder and all workers take one step

def training_step(anchor, positive, negative,
                  encoder, regressors, discriminators, optimizer):
    # Encode the anchor chunk, one frame per 160 samples  [§2.1]
    frames = encoder(anchor)                   # (B, N, 100), N = T / 160
    costs = []
    for worker in regressors:
        # Predict a feature or the waveform from the shared frames  [§2.2]
        prediction = worker(frames)
        # Target is standardized, except the waveform  [§2.2] [§2.3]
        costs.append(worker.loss(prediction, worker.target))
    for worker in discriminators:
        # Score anchor with positive (label 1) and negative (label 0)  [§2.2]
        scores, labels = worker.pair_scores(frames, encoder)
        costs.append(worker.loss(scores, labels))
    # Combine the worker costs as an average  [§2.3]
    total = sum(costs) / len(costs)
    optimizer.zero_grad()
    # Gradients from all workers reach the shared encoder  [§2.3]
    total.backward()
    optimizer.step()
    return total

Hyperparameters

Results

Author-reported. Setup: encoder pre-trained on LibriSpeech, downstream on the datasets in §3. Hardware and run time are not stated. Accuracies are in percent. No variance or repeated runs are reported.

Headline comparison, classifier accuracy tab. 2 and word error rate tab. 3:

Worker ablation, change in accuracy when one worker is removed from self-supervised training tab. 1. The "all workers" row is 97.5 (Speaker-ID), 88.3 (Emotion), 81.1 (ASR):

Transfer to noisy, reverberant speech (DIRHA) uses the GRU classifier throughout and the encoder trained on clean LibriSpeech §4.3.

Paper vs code

Code: the official repository santi-pdp/pase at commit 2a41e63e54fa8673efd12c16cdcdd5ad4f0f125e, MIT licence LICENSE. The owner name matches the first author, and the paper links the repository. The repository's training template is run_pase_train.sh, which points to cfg/PASE.cfg and cfg/workers.cfg; those paths do not match the repository layout, where the files are cfg/frontend/PASE.cfg and cfg/workers/workers.cfg run_pase_train.sh.

• Loss combination. Paper: average of the worker costs §2.3. Code: sums them, tot_loss += loss for each worker, then one backward pass worker_scheduler.py. Adam is largely insensitive to a uniform rescaling of gradients, but the code does not divide by the number of workers.
• Learning rate decay. Paper: halved every 30 epochs §2.3. Code: the active scheduler multiplies by 0.1 every lrdec_step epochs, lr * 0.1 ** (epoch // lr_step) lr_scheduler.py. The run script passes --lrdecay 0.5 run_pase_train.sh, but that flag is only read by the older pase/models/core.py path, which train.py does not import train.py train.py. So the factor is 0.1, not 0.5.
• Waveform decoder width. Paper: an MLP of 256 PReLU units after the decoder §2.2. Code: hidden_size: 64 for cchunk workers.cfg. The other workers use 256 workers.cfg.
• LPS window. Paper: Hamming window of 25 ms §2.2. Code: calls torch.stft with a 400-sample window length and no window argument, which gives a rectangular window transforms.py. Code also uses log-power in dB, 10·log10(|X|² + 1e-19); the paper does not state a scale.
• MFCC filter banks. Paper: 40 mel filter banks §2.2. Code: the mfcc worker uses librosa's MFCC with no n_mels set, so librosa's default applies transforms.py. A separate class MFCC_librosa defaults to n_mels=40 transforms.py, but the shipped worker config does not use it.
• SPC sampling range. Paper: up to 500 ms from the anchor §2.2. Code: keeps a margin of 16 frames (160 ms) and then draws future and past windows anywhere else inside the chunk minions.py minions.py. No 500 ms cap appears in SPCMinion.forward.
• LIM negatives. Paper: negatives "likely" come from a different speaker §2.2. Code: negative is a uniformly random other utterance, with no speaker check dataset.py.
• Standardization. Paper: all worker outputs are standardized §2.3. Code: the ZNorm transform is applied only to keys present in the stats file passed with --stats transforms.py train.py. The only stats file in the repository is data/librispeech_50h_stats.pkl, not one for the paper's 15 s-per-speaker set §3.
• Stopping rule. Paper: train until the validation losses plateau §2.3. Code: runs the fixed number of epochs passed to it (150 in the script) and saves a checkpoint per epoch trainer.py; no plateau test was found.
• Epoch length. Not in the paper. Code: batches per epoch are the total training duration divided by the chunk size, then by batch size train.py. Epoch length therefore depends on the corpus.
• Warmup. The argument parser defaults --warmup to a very large value train.py, but the trainer does not pass a warmup to the scheduler, so no warmup runs trainer.py.
• Discriminator inputs. Code forms LIM and GIM inputs by concatenating anchor and partner features along the channel axis, with labels 1 for positives and 0 for negatives cls_minions.py cls_minions.py. GIM averages the frames of each chunk over time first cls_minions.py. The paper describes this in words only §2.2.

No discrepancies were checked for the downstream classifiers in spk_id/, ASR/ and emorec/; the results in this card come from the paper.

Pitfalls

• Text and table disagree on the prosody ablation. The text reports prosody removal as "+131% in relative error" on emotion §4.1. The printed Table 1 shows −5.3 points for prosody on emotion, which is about +45% relative error. The 131% figure matches a row that is commented out in the LaTeX source (accuracy 73.0) [raw/src/1904.03416/template.tex#L253-L254]. Trust the printed table.
• Ablation baseline does not match Table 2. The "all workers" row in Table 1 is 97.5 / 88.3 / 81.1, while PASE-Frozen with an MLP in Table 2 is 97.3 / 91.5 / 81.4 tab. 1 tab. 2. The paper does not reconcile them beyond the classifier depths stated in §4.1.
• Frozen features do not always beat standard features. PASE-Frozen is below FBANK on speaker ID with an MLP (97.3 vs 98.4) and on emotion with an MLP (91.5 vs 94.1) tab. 2. The text says PASE "provides most of the times" better results §4.2.
• Equation sign. The discriminator objective is written as an expectation of log-probabilities without a minus sign, while training minimizes binary cross-entropy §2.2. Implement it as binary cross-entropy with positives labelled 1.
• Hidden decoder width. The waveform worker's hidden width is 64 in code and 256 in the text workers.cfg §2.2. The text describes 256 units; the code uses 64. The card records both and does not pick one.
• Negatives are not checked for speaker. LIM and GIM negatives are random other utterances dataset.py. Same-speaker negatives are possible, and the paper's claim is only "likely".
• Noisy transfer is one test set. DIRHA has 409 sentences from six speakers §3. No confidence intervals are given, and the differences in Table 3 are a few WER points.
• Hidden cost. Seven workers are trained in each step, so memory and compute per step are higher than for the encoder alone. Hardware and run time are not stated §2.3.
• Prosody features. The paper says features use default parameters §2.2. The code sets F0 search bounds of 60–300 Hz with SWIPE, which are the class defaults in Prosody transforms.py.
• Unverified preprocessing. The LibriSpeech subset (15 s per speaker, 2484 speakers) and the data preparation scripts were not checked against the paper.

Lineage

• Builds on: SincNet, for the first encoder layer §2.1 (not carded here); local info max, as proposed in an earlier paper §2.2 (not carded); Deep InfoMax, which GIM relates to §2.2 (not carded); CPC-style sampling, adapted for SPC §2.2 (not carded).
• Compared against: MFCC and FBANK features, with and without context windows, and the supervised-from-scratch PASE-Supervised §4.2; SincNet on DIRHA §4.3.
• Superseded by: not stated in the sources used for this card.