--- kind: concept slug: spectrogram title: Spectrogram summary: >- A time-frequency grid of signal strength, computed from short-time Fourier transform frames. Log-mel is the common audio variant, used as classifier input, generation target, VAE input or robot-policy input. Papers often leave the FFT, hop and bin settings unstated, and the released settings are not interchangeable. tags: [audio, signal-processing, time-frequency] papers: ["1609.04243", "2105.06337", "2111.02813", "2503.23377", "2502.12048", "2005.13249", "2010.05113", "2503.04404", "2406.19464"] related: [mel-frequency-cepstral-coefficients, audio-deepfake-detection, latent-diffusion, data-augmentation, convolutional-recurrent-network, intrusion-detection, imitation-learning] status: draft updated_at: 2026-10-25 --- ## Overview A spectrogram is a grid with time on one axis and frequency on the other. Each cell holds how strong that frequency is at that moment. Cards in this wiki build it from short-time Fourier transform (STFT) frames: a fixed window and a fixed hop turn a waveform into a sequence of spectra [arxiv:2111.02813 §2.1]. Many audio papers then map the linear frequency bins onto a mel scale and compress magnitudes with a logarithm [arxiv:1609.04243 §3] [arxiv:2105.06337 §4]. The invariant all variants share is the framing: window length, hop and frequency resolution set how much time and frequency detail the grid keeps. The cards use spectrograms in four roles: model input, generation target, latent input to a generative model, and augmentation surface. The one non-audio use is an exploratory picture of network traffic [arxiv:2503.04404 §5.5]. ## Variants ### Log-mel spectrogram as classifier input A convolutional recurrent network for music tagging takes a 29-second clip as a single-channel log-amplitude mel-spectrogram with 96 bands, 12 kHz audio and a hop of 256 samples, giving 1366 frames [arxiv:1609.04243 §3]. The paper says it chose this over STFT, MFCC and linear-amplitude mel input [arxiv:1609.04243 §3]. The card does not give the numbers behind that choice. Card: [Convolutional Recurrent Neural Networks for Music Classification](/papers/1609.04243). ### Mel spectrogram as the target of a text-to-speech model Grad-TTS generates an 80-bin mel-spectrogram from text, and a HiFi-GAN vocoder turns that into audio for every compared system [arxiv:2105.06337 §3.1] [arxiv:2105.06337 §4]. The paper gives the 22.05 kHz sample rate and 80 bins. The released code also uses an FFT and window of 1024, a hop of 256, and an 8 kHz top frequency, none of which the paper states [arxiv:2105.06337 §4]. Card: [Grad-TTS](/papers/2105.06337). A survey of EEG-to-audio work describes the same pattern: EEG is mapped to a mel-spectrogram, which a neural vocoder such as HiFi-GAN renders [arxiv:2502.12048 §6.3]. Card: [A Survey on Bridging EEG Signals and Generative AI](/papers/2502.12048). ### Mel spectrogram encoded by a frozen VAE JavisDiT encodes the audio mel-spectrogram with a frozen AudioLDM2 VAE at 16 kHz, and diffusion runs on the latents [arxiv:2503.23377 app. C.1]. The paper gives no mel-bin count. The card records 64 bins from the released config. Card: [JavisDiT](/papers/2503.23377). ### Log-mel as input to a robot policy's audio encoder ManiWAV resamples contact-microphone audio from 48 kHz to 16 kHz, takes 64 log-mel bins, scales the values to [-1, 1] and passes the result to an audio spectrogram transformer (AST) trained from scratch [arxiv:2406.19464 §3.2] [arxiv:2406.19464 app. A.2]. The appendix gives a 400-sample window and a 160-sample hop. The released code instead calls the transformers library's AST feature extractor, so the framing comes from that library, and its scaling uses fixed bounds from the config unless statistics are computed first. The paper's audio window is 2 to 3 s depending on task, while the released config uses 2 s for every task. Card: [ManiWAV](/papers/2406.19464). ### Linear STFT power, then filterbanks and a cosine transform For detection features, WaveFake takes the STFT power, applies a linear filterbank for LFCC or a mel filterbank for MFCC, then log-compresses and keeps 20 cepstral coefficients with delta and double-delta [arxiv:2111.02813 §2.1] [arxiv:2111.02813 app. C]. The card gives FFT size 512 and a 20-band linear filterbank from the code, and says that the window and hop are not set anywhere in the repository [arxiv:2111.02813 app. C]. Card: [WaveFake](/papers/2111.02813). The cepstral step is its own concept: [Mel-frequency cepstral coefficients](/concepts/mel-frequency-cepstral-coefficients). ### Time and frequency masks as augmentation SpecAugment-style masks on a time axis and a frequency axis appear in CLOCS, a contrastive method for ECG, with mask width 0.2 and R = 1 [arxiv:2005.13249 app. B.2]. The released code does not contain the STFT masking, so the card cannot reproduce those results [arxiv:2005.13249 app. B.2]. The review lists frequency and temporal masking of the mel-spectrogram among audio data transformations [arxiv:2010.05113 §III.B.2]. Cards: [CLOCS](/papers/2005.13249), [Contrastive Representation Learning: A Framework and Review](/papers/2010.05113). ### Time-frequency picture of non-audio series For network traffic, the NetFlow analysis draws a spectrogram of the most repeated pattern for each attack class in NF3-UNSW-NB15 [arxiv:2503.04404 §5.5]. The window, hop, FFT size, normalisation and the signal chosen are not stated, and the claims rest on the plots alone [arxiv:2503.04404 §5.5]. Card: [Temporal Analysis of NetFlow Datasets](/papers/2503.04404). ## When to use which | Situation | Use | Why | |---|---|---| | Classifier on music or general audio, no pretrained model to match | Log-mel, modest band count (96 in the tagging paper) | The tagging paper chose it over STFT, MFCC and linear mel [arxiv:1609.04243 §3]; the card gives no comparison numbers | | Feeding a pretrained vocoder or a released checkpoint | The vocoder's exact mel definition (sample rate, bins, FFT, hop, floor) | A different mel definition does not match the released HiFi-GAN pipeline, per the Grad-TTS card's code review [arxiv:2105.06337 §4] | | Robot policy that takes a contact-microphone stream | Log-mel with 64 bins at 16 kHz, into an audio transformer trained from scratch | ManiWAV uses this and reports that a from-scratch transformer beats both a pretrained and a from-scratch CNN encoder [arxiv:2406.19464 §3.2]; the card gives no separate numbers for the spectrogram settings | | Generative model over audio with a frozen autoencoder available | Mel-spectrogram into a frozen VAE, diffusion on latents | JavisDiT uses this setup [arxiv:2503.23377 app. C.1]; the card does not isolate the spectrogram choice | | Detector with a classical model such as a GMM | Log filterbank plus DCT: MFCC, or LFCC with a linear bank | WaveFake's GMM detectors use both [arxiv:2111.02813 §2.1]; the two are reported in separate tables, so no ranking follows from the card | | Augmentation for a non-audio signal | Time and frequency masks, only if you implement them yourself | The CLOCS masks are in the paper but not in its code [arxiv:2005.13249 app. B.2] | | Exploring a non-audio time series | Spectrogram as a picture, with the parameters written down | The traffic paper reports visual findings only, with no detector [arxiv:2503.04404 §5.5] [arxiv:2503.04404 §6] | No card reports a controlled, numeric comparison of spectrogram settings. Choose the setting that the model you reuse was trained with, and record the FFT, window, hop and bin count in your own code. ## Implementation sketch Common core of the audio variants. Parameter values are placeholders, not recommendations: each card gives its own. ```pseudocode # Log-mel spectrogram, the core shared by the audio variants # Input: wav (samples,), sample_rate, n_fft, hop, n_mels # Output: log_mel (n_mels, frames) def log_mel_spectrogram(wav, sample_rate, n_fft, hop, n_mels): # Short-time Fourier transform: one spectrum per hop [arxiv:2111.02813 §2.1] # Grad-TTS's released code uses sqrt(|X|^2 + 1e-9) as its magnitude, # a choice the paper does not state [arxiv:2105.06337 §4] spectrum = stft_magnitude(wav, n_fft=n_fft, hop=hop) # (n_fft/2+1, frames) # Mel scale: f_mel = 2595 * log10(1 + f / 700) [arxiv:2111.02813 eq. 1] # Triangular bands spaced evenly on that scale, n_mels of them filters = mel_filterbank(sample_rate, n_fft, n_mels) # (n_mels, n_fft/2+1) mel = filters @ spectrum # (n_mels, frames) # Log compression with a floor. Floors in the cards differ: 1e-5 in # Grad-TTS, 1e-6 in WaveFake, both from released code, not the papers. # ManiWAV's code leaves log and framing to the AST feature extractor return log(mel + 1e-5) ``` Cepstral features, if a detector needs them, add a discrete cosine transform over the bands and keep the first coefficients [arxiv:2111.02813 eq. 3]. Masking, if used, applies to the output grid after this function. ## Contradictions None known between papers: the cards report no incompatible numbers for the same setting, and they use spectrograms for different purposes. Inconsistencies inside individual cards: - The tagging card states 1366 frames for a 29-second clip at hop 256 [arxiv:1609.04243 §3]. The card derives about 1359 frames from the same settings, and notes that the framing rule is not stated. - The WaveFake paper states that silences longer than 2 s are removed, while the released code trims silences longer than 0.2 s at a 1% volume threshold [arxiv:2111.02813 app. C]. The card flags this as a difference between the paper and the code. - The ManiWAV card's appendix gives torchaudio-style 400/160 STFT settings, but its released code uses the transformers AST feature extractor, so matching the appendix gives a different input [arxiv:2406.19464 app. A.2]. - The NetFlow card's text and table disagree on which attack occurs on which day, so use its spectrogram figures only as visual examples [arxiv:2503.04404 §5.3] [arxiv:2503.04404 tab. 4]. ## Papers - [Convolutional Recurrent Neural Networks for Music Classification](/papers/1609.04243): log-mel input for music tagging, with the stated choice of representation. - [Grad-TTS](/papers/2105.06337): mel-spectrogram as the output of a diffusion text-to-speech model. - [ManiWAV](/papers/2406.19464): log-mel input to an audio transformer inside a diffusion policy, for contact sound. - [WaveFake](/papers/2111.02813): LFCC and MFCC features from the spectrum, and the mel scale formula. - [JavisDiT](/papers/2503.23377): mel-spectrogram encoded by a frozen audio VAE for joint audio-video generation. - [A Survey on Bridging EEG Signals and Generative AI](/papers/2502.12048): EEG-to-mel-spectrogram pipelines and spectrogram metrics, as summarised by the survey. - [CLOCS](/papers/2005.13249): time and frequency masking of spectrograms, not present in the released code. - [Contrastive Representation Learning: A Framework and Review](/papers/2010.05113): masking and warping of mel-spectrograms as data transformations. - [Temporal Analysis of NetFlow Datasets for Network Intrusion Detection Systems](/papers/2503.04404): spectrograms of traffic series for exploration only.