Keyword spotting

Overview

Keyword spotting (KWS) decides whether a short audio segment contains one of a small set of target words, and labels everything else as unknown or silence. Most of the input is silence, background noise or unrelated speech, so a detector must reject non-target audio as well as recognise targets 1804.03209 §4. The detector runs on the device, so compute, energy and latency budgets are tight 1804.03209 §4.

The unit of recognition is one word or a short phrase, not a sentence 1804.03209 §4. Sentence-aligned corpora such as Common Voice and LibriSpeech therefore do not fit the task directly 1804.03209 §3.

Coverage today. Two cards list this concept. One is the Speech Commands dataset and its evaluation protocols 1804.03209. The other converts speech datasets, one of them derived from Speech Commands, into spike trains for spiking networks 1910.07407 §2.1.2. The two cards use different settings: 12-class Top-One on audio, and 35-class top-one on spike-coded audio. Their numbers are not comparable (see Pitfalls). No card compares streaming detectors, so this page does not compare detector architectures for the streaming task.

Variants

These are four ways to use the two cards' material, not four competing model families.

Fixed-clip classification (Top-One protocol)

Each test clip is one second of 16 kHz audio. The model picks one of 12 classes: ten command words, "Unknown Word" and "Silence" [arxiv:1804.03209 §7.1]. The test set is balanced, with about 8.3% of examples per class 1804.03209 §7.1. The reference TensorFlow conv baseline reaches 88.2% when trained and tested on Speech Commands v2 1804.03209 §7.3 1804.03209 fig. 2. Card: Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.

Continuous-stream detection (streaming protocol)

The model runs over a long recording with known word times. Each detection is matched to a ground-truth word within a time tolerance, and the score counts matched, wrong-label and false-positive detections 1804.03209 §7.2. The default tolerance is 750 ms 1804.03209 §7.2. On the V2-trained baseline, the author reports 49.0% matched, 46.0% correct, 3.0% wrong and 0.0% false positives 1804.03209 §7.2. Card: Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.

Building a word corpus

The collection pipeline is a recipe for making your own labelled word corpus. Volunteers record single words through a browser on phone or laptop microphones 1804.03209 §5.3. Clips are then filtered by file size, loudness and a human check, and trimmed to the loudest second 1804.03209 §5.4–§5.6. Splits are assigned by a hash of the file name, so they stay stable across releases 1804.03209 §7. Card: Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.

Spike-coded input (Spiking Speech Commands)

The spiking card converts Speech Commands version 0.02 clips into spike trains with a fixed, cochlea-inspired model with 700 channels [arxiv:1910.07407 §2.1.2] 1910.07407 §2.2. The conversion is published, so a modeller does not choose an encoder 1910.07407 §2.2. The paper describes the resulting files as HDF5 with spike times, units and labels [arxiv:1910.07407 §2.3]. Author-reported test accuracies on this set, over 35 classes:

• Linear SVM on spike counts: 21.6% 1910.07407 tab. 1.
• LSTM on 10 ms bins: 73.0% in the table 1910.07407 tab. 1, 75.0% in the text 1910.07407 §3.1.
• CNN on 10 ms bins: 77.7% 1910.07407 tab. 1.
• Feed-forward SNNs of 1 to 3 layers: 32.5% to 41.0% 1910.07407 tab. 1.
• Recurrent SNN with max-over-time readout: 50.9% 1910.07407 tab. 1.

Card: The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks.

When to use which

Constraints the cards do not cover: model size, latency and energy figures for any detector. The paper names energy as a comparison target [arxiv:1804.03209 §2] 1804.03209 §4, but neither card reports energy or latency numbers. Do not infer them. The spiking card reports accuracy only.

Implementation sketch

The two protocols as described in the cards. Labels come from the folder or test-list entry. Thresholds and window settings are not stated in the paper; see the card for the code defaults it lists 1804.03209 §7.2.

# Top-One accuracy over the balanced 12-class test set [arxiv:1804.03209 §7.1]
for clip in test_set:
    scores = model(clip)                # one-second, 16 kHz input
    prediction = argmax(scores)         # one of 12 classes
    correct += (prediction == clip.label)
accuracy = correct / len(test_set)

# Streaming evaluation on a long recording [arxiv:1804.03209 §7.2]
detections = []
for window in sliding_windows(recording, stride):
    scores = model(window)
    if max(scores) > threshold:         # threshold is a setting, not stated
        detections.append((label, time))
detections = suppress_repeats(detections, span)   # same word within span
for word in ground_truth:
    match if a detection has the same label within the time tolerance
report matched %, wrong-label %, false-positive %

Contradictions

Within the Speech Commands card, the paper and its own tooling disagree in several places. The card lists each one and does not resolve it. Read the card's "Paper vs code" section before reproducing any result [arxiv:1804.03209 §5.5] §7.1.

• Loudness filter order. The paper measures loudness before trimming 1804.03209 §5.5. The released tool trims first, then measures as the card's "Paper vs code" section reports 1804.03209 §5.5.
• Unknown-word sampling. The paper says unknown examples come from classes "that are part of the target set" 1804.03209 §7.1, which contradicts the purpose of the class. The tutorial code uses the complement.
• Accuracy set. The text calls the 88.2% figure a training-set result 1804.03209 §7.3, while the figure labels it as the V2 test set 1804.03209 fig. 2.
• Streaming file length. The text says the result comes from a 10-minute clip 1804.03209 §7.2. The released streaming file is one hour long 1804.03209 §7.2.

Within the spiking card, the paper's table and text disagree on the LSTM result on SSC, 73.0% against 75.0% 1910.07407 tab. 1 1910.07407 §3.1. Its surrogate steepness also differs between text (40) and table (100), which affects every SNN row on SSC 1910.07407 §3.1 1910.07407 tab. 2. The card reports both and does not choose.

Across the two cards there is no direct conflict. The two cards use different class sets and inputs, so their numbers answer different questions (see Pitfalls).

Pitfalls

• Version and split. V1 and V2 differ in word count and clip count 1804.03209 §5.2 1804.03209 §7.3. Numbers from different versions are not comparable; state which one you report [arxiv:1804.03209 fig. 2]. Use the released validation and test lists, or the same hash rule 1804.03209 §7. A random split leaks speakers.
• Hashing by file name. Speaker grouping depends on the file-name prefix before the no-hash marker. Renaming files moves them between splits (code only; see the card's "Paper vs code" section).
• Which test set. The canonical Top-One number uses the released balanced 12-class test archive 1804.03209 §7.1. The training script's own final test uses a different sampling of unknown and silence, so its classes are not balanced. Say which one you used.
• Class count differs across cards. The Top-One protocol uses 12 classes 1804.03209 §7.1. The spiking card reports top-one over 35 classes 1910.07407 §2.1.2. The 88.2% and the 77.7% CNN figure are not comparable.
• Input differs across cards. The Top-One numbers use audio. The spiking numbers use spikes from one conversion chain 1910.07407 §2.2. Results depend on that chain. Its code differs from the paper in several parameters, so reproduce the chain only after reading the spiking card's "Paper vs code" section.
• Unlisted word. A word called "Visual" appears in the counts but in neither word list 1804.03209 §5.2 1804.03209 fig. 1. Build the label set from folder names, not from the paper's lists.
• Class imbalance. Command words have roughly twice as many clips as the auxiliary words and the V2 additions 1804.03209 fig. 1. Weight or sample when training on all 35 words as separate classes.
• Label noise. Each clip was checked by one crowd worker, and the alignment assumes the word is the loudest part 1804.03209 §5.5 [arxiv:1804.03209 §5.6]. Expect some mislabelled or cut-off clips.
• Generated noise. The noise generator casts to int16 without clipping, so large samples wrap around 1804.03209 §5.8. Clip before casting if you regenerate noise.
• Recording bias. No iOS devices, English only, speakers recruited through social media 1804.03209 §5.1 1804.03209 §5.3. The metadata has no demographic labels, so accent or gender bias cannot be measured from it 1804.03209 §5.1.
• Streaming settings. Report the tolerance, suppression and threshold with any streaming result 1804.03209 §7.2. Results depend on them.
• Lower bounds. The spiking paper's authors expect the datasets may not prove final and describe their accuracies as lower bounds [arxiv:1910.07407 §4] 1910.07407 §3.

Papers

• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition: a word-level dataset of one-second clips, with the Top-One and streaming evaluation protocols for small-footprint keyword spotters.
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks: Spiking Speech Commands, a spike-coded version of Speech Commands 0.02 with 35-class top-one results for spiking and non-spiking classifiers.