Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

In plain words

Phones and smart speakers listen all the time for a wake word such as "Hey Google". That detector has to run on the device itself, on little power, and must ignore silence, background noise and the many words that are not the trigger. Training and comparing such small detectors needs a public collection of short spoken words, and before this dataset most options were either licensed for research only or built for transcribing whole sentences.

Speech Commands is that collection. Volunteers recorded themselves saying single words through an ordinary laptop or phone microphone in their own surroundings, so the audio is noisy and realistic, not studio-clean. Every clip is one second long, and each word gets its own folder. Some words are meant to be detected (digits, "yes", "no", "stop", ...), and others are there only as distractors the model should reject.

The paper also says exactly how to score a model. You classify each test clip into one of ten command words, "unknown word" or "silence". A second test plays a long audio stream and counts correct detections, wrong labels and false alarms. The author's reference model from the TensorFlow tutorial gets about 88% of the test clips right, which leaves room for better models.

TL;DR

A CC BY 4.0 dataset (version 2) of 105,829 one-second, 16 kHz, 16-bit mono WAV utterances of 35 words from 2,618 anonymous speakers, plus minute-long background-noise files. Clips were filtered by file size, mean loudness and a single crowd worker's transcription, and trimmed to their loudest second. Train/validation/test membership is decided by hashing the file name, so it stays stable across releases. The standard metric is 12-class Top-One accuracy on a released balanced test set; a streaming metric on a long audio file measures matches, wrong labels and false positives. The TensorFlow conv baseline gets 88.2% (V2 train, V2 test). Use it if you build or benchmark small-footprint keyword spotters; it is not suited to full-sentence ASR.

Problem

Keyword spotting runs on-device under tight compute, energy and latency limits. Most of its input is silence, noise or unrelated speech, and the unit to recognise is a single word or short phrase, not a sentence §4. Existing corpora did not fit: Common Voice and LibriSpeech are aligned only at the sentence level, TIDIGITS needs a commercial LDC licence and uses the hard-to-decode NIST SPHERE format, and CHiME-5 is restricted and also sentence-aligned §3. The paper wants a freely redistributable (CC BY 4.0) word-level dataset with a reproducible evaluation protocol, so that model authors and chip vendors can compare accuracy (and energy) on the same task §2 §4.

Method

The "method" is a data-collection pipeline plus an evaluation protocol. Section numbers follow the compiled paper, where the abstract is §1.

Design requirements §5.1: audio comes from phone or laptop microphones, wherever the speaker is (alone in a room, door closed, for privacy); English only, with as many accents as possible; as many distinct speakers as possible (speaker independence); no personally identifiable information collected; fixed one-second duration; single isolated words.

Vocabulary §5.2:

• Core words: digits "Zero" to "Nine" plus "Yes", "No", "Up", "Down", "Left", "Right", "On", "Off", "Stop", "Go" (version 1); version 2 adds "Backward", "Forward", "Follow", "Learn".
• Auxiliary (distractor) words: "Bed", "Bird", "Cat", "Dog", "Happy", "House", "Marvin", "Sheila", "Tree", "Wow". Some, like "Tree", were chosen because they sound close to targets; the others are short words that cover many phonemes.
• The released counts table also lists "Visual", which neither word list in the text mentions fig. 1.

Recording app §5.3: an open-source web app using the WebAudioAPI (desktop Firefox/Chrome and Android; not iOS), hosted on a Google-controlled domain. Users first give formal consent (stored in a session cookie); uploads are guarded with CSRF tokens. After the user presses "Record", a random word is displayed for 1.5 s while audio is captured, followed by a 1 s pause before the next one. Core words are shown five times each, auxiliary words once, 135 prompts in total (about six minutes). Order is random so that repetition does not change pronunciation. Clips stay on the client until the user reviews them and uploads them all. Files are stored as OGG, named <session id> + requested word + unique instance id. The random session id serves as the speaker id. A cookie stops the same browser from recording a second time. Speakers were recruited through social-media appeals; some utterances came from paid crowdsourcing §5.3.

Filtering and processing pipeline §5.4–§5.8:

Algorithm: Speech Commands curation  [§5.4]-[§5.8]
Input: uploaded OGG clips (~1.5 s each), WAVs from other sources
1. Delete every OGG file smaller than 5 KB (very short or quiet audio
   compresses to tiny files)                                  [§5.4]
2. Convert OGG -> PCM WAV at 16 kHz with ffmpeg (-ar 16000); resample
   WAVs from other sources to 16 kHz the same way             [§5.4]
3. Extract Loudest Section                                    [§5.5]
   a. scale int16 samples to [-1, 1] (-32768 -> -1.0, +32767 -> 1.0)
   b. volume = mean(|x|) over the clip; drop the clip if volume < 0.004
   c. keep the contiguous 1-second window with the highest total volume
      (this roughly centres the word, assuming it is the loudest part)
4. Manual review: one commercial crowd worker per clip types the word
   they hear (given the expected word list as examples); drop the clip
   if the answer does not match its label                     [§5.6]
5. Release: one folder per word; hash 16-hex-digit speaker ids (and ids
   from other sources) into 8-hex-digit ids with a stable hash [§5.7]
6. Add minute-long 16 kHz background-noise WAVs (recorded noise such as
   running water or machinery, plus generated white and pink noise) in a
   folder named "_background_noise_"                          [§5.8]
Output: one folder of 1-second (or shorter) 16 kHz mono int16 WAVs per word

The generated-noise files are produced by drawing 60 s × 16000 samples of white or pink noise from the Python acoustics package, dividing by 3, multiplying by 32767, casting to int16 and writing at 16 kHz §5.8.

Splits §7: the archive ships validation_list.txt (for tuning during training) and testing_list.txt (only for final measurement). Membership is decided by a hash of the file name so that a file keeps its split across releases, and the Python implementation is in the TensorFlow tutorial's input_data.py §7. From that code (TensorFlow master, read 2026-10-18; not part of code_commit):

which_set(filename, validation_pct, testing_pct):       # input_data.py
  name = basename(filename) with everything from "_nohash_" on removed
  h    = int(sha1_hex(name)) mod (MAX_NUM_WAVS_PER_CLASS + 1)
         with MAX_NUM_WAVS_PER_CLASS = 2^27 - 1
  pct  = h / MAX_NUM_WAVS_PER_CLASS * 100
  if pct < validation_pct:                  return "validation"
  if pct < validation_pct + testing_pct:    return "testing"
  return "training"

Top-One protocol §7.1: 12 classes. Ten are the V1 command words (alphabetically: "Down", "Go", "Left", "No", "Off", "On", "Right", "Stop", "Up", "Yes"), plus "Unknown Word" and "Silence". The test set has equal numbers of examples per class (about 8.3% each). "Silence" clips are one-second excerpts taken at random from the background-noise files. A standard pre-built test set is released for V2 (and one for V1); score by running the model on every clip and comparing the top prediction with the label given by the clip's subfolder name §7.1.

Streaming protocol §7.2: run the model over a continuous audio file with ground-truth word times and report

• matched %: words correctly identified within a time tolerance (default 750 ms);
• wrong %: words detected as speech but given the wrong label;
• false-positive %: detections where no speech was present.

The scoring algorithm is test_streaming_accuracy.cc; repeated detections of the same word in a short time span are suppressed by default rules in recognize_commands.cc §7.2. A one-hour streaming test file with ground-truth labels is released, generated by a script in the TensorFlow tutorial §7.2.

Hyperparameters

Pipeline constants from the paper and the cloned tool:

Baseline training. The paper gives only the commands (default train.py with the V1 or V2 --data_url, checkpoint conv.ckpt-18000) §7.3. The values below are the train.py / test_streaming_accuracy.cc defaults on TensorFlow master as read on 2026-10-18, not stated in the paper and not pinned to the 2018 version used for the results:

Results

Author-reported; there is no comparison with other models. Model: the default TensorFlow tutorial conv model trained with default train.py settings §7.3.

Dataset properties, V2 §6 fig. 1:

Per-word counts, V2. The paper's text calls this "Table 1"; it is typeset as a figure, so the compiled PDF shows it as Figure 1 fig. 1:

The counts add up to 105,829.

Top-One accuracy, 12-class protocol. Typeset as Figure 2 and called "Table 2" in the text fig. 2:

Streaming, V2-trained baseline on the streaming test audio §7.2:

The author reads the V1-test gain from 85.4% to 89.7% as evidence that the V2 training data, not only an easier test set, improves accuracy §7.3. 88.2% is called reasonable but far from perfect, a baseline to beat §7.1.

Paper vs code

Code checked: petewarden/extract_loudest_section at code_commit (the tool the paper clones in §5.5). The TensorFlow tutorial files were read on master, not pinned.

• Order of the loudness filter. The paper describes measuring the mean absolute amplitude of the clip "as a first stage" and then extracting the loudest second §5.5. The code trims first and computes the mean absolute amplitude only over the trimmed 1 s window, dropping the clip if it is below 0.004 main.cc. Clips with a short loud word inside a long quiet recording therefore pass more easily than the paper suggests.
• Inconsistent window energy. The initial window sum uses |x·x| (squared amplitude) main.cc, but the sliding update adds and subtracts |x| main.cc. The running score mixes squared and absolute values, so it is neither the "highest overall volume" of §5.5 nor any consistent energy measure. Because |x|² < |x| for |x| < 1, the first window starts undervalued, and the error carries into every later window's score.
• Off-by-one window. loudest_end_index = i marks the window ending at sample i inclusive, but the copy takes [i − N, i), so the output is shifted one sample earlier than the window that was scored main.cc. Negligible acoustically, relevant if you try to reproduce the clips bit-exactly.
• Short clips. Inputs shorter than 1 s are returned unchanged main.cc, which is why released files are "one-second (or less)" §6. Their average volume is still divided by the full 1 s sample count main.cc, so short clips are more likely to be rejected as quiet.
• Stereo input is averaged down to mono before trimming main.cc; the paper does not mention it.
• Unknown-word sampling. The paper says "Unknown Word" examples are drawn "from classes that are part of the target set" §7.1, which contradicts the class's purpose. The tutorial's input_data.py puts every folder that is not in wanted_words into the unknown pool and samples from that, so the text is evidently a slip for "not part of".
• Silence in training. The paper defines test "Silence" clips as random one-second background-noise excerpts §7.1. In input_data.py the training-time silence entries all point at one training file whose samples are multiplied by zero, so what the model hears is only the mixed-in background noise (code only).

Pitfalls

• Version and split stability. V1 and V2 differ (30 vs 35 words, 64,727 vs 105,829 clips) §5.2 §6 §7.3. Always say which version and which test set you used: V1-trained numbers on the V1 test (85.4%) are not comparable with V2 numbers (88.2%) fig. 2. Use the shipped validation_list.txt / testing_list.txt or the identical hash rule; a random split leaks speakers.
• Speaker grouping depends on file names. which_set hashes only the part before _nohash_, so files sharing that prefix end up in the same split (code only, input_data.py). Renaming files, or hashing the full path, breaks speaker-disjointness and moves files between splits.
• The pre-built test set is not the same as testing_list.txt. The canonical Top-One number uses the separately released balanced 12-class test archive, with silence and unknown clips already materialised §7.1. The training script's final test accuracy uses its own seeded sampling of unknown and silence (10% each of the wanted-word count, code defaults), which does not give equal class shares. State which one you report.
• "Top-One error" means accuracy. §7.1 says the share of correct predictions gives the "Top-One error", and all reported values (88.2% etc.) are accuracies.
• Typos in the paper's text and commands. §7.3 says the V2 model scores 88.2% on the "training set"; fig. 2 shows it is the V2 test set. The evaluation command uses speech_commands_v0.0{1,1} where {1,2} is evidently meant §7.3.
• Streaming file length. The streaming result is said to come from "the 10 minute streaming test audio clip", while the released streaming file is described as one hour long §7.2. Do not expect to reproduce 49.0/46.0/3.0/0.0 exactly on the released file. The four numbers depend on the time tolerance, suppression and threshold settings (code defaults listed above), so report those settings with any streaming result.
• Unlisted word and prompt count. "Visual" (1,592 clips) appears in the counts fig. 1 but in neither word list §5.2. The stated 135 prompts per session §5.3 cannot be rebuilt from the listed words (24 core × 5 + 10 auxiliary × 1 = 130). Build your label set from the folder names, not from the paper's lists.
• Imbalanced classes. Core command words have about 3,700 to 4,050 clips; auxiliary words and the four V2 additions have about 1,550 to 2,130 fig. 1. Training on all 35 words as separate classes needs reweighting or sampling.
• Label noise is possible. Each clip was checked by a single crowd worker §5.6, and alignment comes from a loudness heuristic that assumes the word is the loudest part §5.5. Expect some mislabeled, off-centre or partly cut-off clips.
• Generated noise can wrap around. The noise command scales noise / 3 by 32767 and casts to int16 with no clipping §5.8. Any sample with |noise| > 3 wraps around instead of saturating. If you regenerate noise, clip before casting.
• Recording conditions are skewed. No iOS devices (WebAudioAPI was not available), English only, speakers recruited through social media, one session per browser §5.1 §5.3. No demographic labels exist §5.1, so accent or gender bias cannot be measured from the metadata.
• Baseline hyperparameters are not in the paper. The paper gives only train.py commands §7.3; the table above holds today's code defaults, which may have changed since 2018.

Lineage

• Builds on: TIDIGITS (the author's earlier keyword-spotting experiments used it) §3; the baseline model follows Sainath & Parada, "Convolutional Neural Networks for Small-footprint Keyword Spotting" (Interspeech 2015) §7.3.
• Compared against (as datasets, qualitatively): Mozilla Common Voice, LibriSpeech, TIDIGITS, CHiME-5 §3.
• Used by (V1, as listed in the paper) §7.4: CMSIS-NN, "Listening to the World", "Did you Hear That" (adversarial attacks), "Deep Residual Learning for Small Footprint Keyword Spotting", "Raw Waveform-based Audio Classification", "Keyword Spotting Through Image Recognition".