Imaging Time-Series to Improve Classification and Imputation

In plain words

Image models such as convolutional networks work well on pictures, but a time series (a sequence of numbers over time, like a sensor trace) is not a picture. This paper turns each series into a square image in two different ways, so that image-style models can read it.

The first way, the Gramian angular field, maps every value to an angle and then looks at how pairs of angles add up or subtract. Each cell of the image therefore compares two moments in time. The second way, the Markov transition field, counts how often the series jumps from one value range to another, and places those jump statistics at the matching pairs of time steps. Both images can be stacked as colour channels.

The authors feed these images to a convolutional network and report strong classification results on 20 public benchmark datasets. They also show that the angle-based image can be turned back into the original numbers, which lets an autoencoder fill in missing values by repairing the image and reading the values back out. That fill-in step beats filling from the raw series in their tests.

TL;DR

• Encode a series as GASF, GADF (angle sums and differences) and MTF (quantile transition probabilities) images §2, §2.1, §2.2.
• Stack the three as one image and classify with Tiled CNNs plus a linear SVM, tuned per dataset §3.2, §3.3.
• Author-reported: best error on 9 of 20 UCR datasets tab. 1.
• A denoising autoencoder on GASF images lowers interpolation MSE by 12.18% to 48.02% against the raw series §4.2, tab. 2.

Problem

Deep convolutional networks rely on local spatial structure, such as edges and neighbouring pixels. A time series has no such structure, and earlier time-series-to-graph or recurrence-map encodings do not have an exact inverse §1. Two problems follow. Classifiers on raw series may miss patterns that image models pick up, and imputation of missing points from the raw series tends to fit the observed points much better than the missing ones §4.2. The paper asks whether an invertible image encoding helps both tasks §1, §4.

Method

Overview. The paper builds two encodings and one learning pipeline.

Encodings. The Gramian angular field (GAF) rescales the series, reads each value as the cosine of an angle, and forms pairwise angle sums (GASF) or differences (GADF). The Markov transition field (MTF) discretises the series into quantile bins, estimates a first-order transition matrix over bins, and places the transition probabilities at the matching pairs of time steps §2, §2.2.

Rescaling. The series is scaled to [0,1] or [-1,1] using its own minimum and maximum §2.1:

\tilde{x}_{[0,1]} = \frac{x - \min X}{\max X - \min X} §2.1

\tilde{x}_{[-1,1]} = \frac{2x - \max X - \min X}{\max X - \min X} §2.1

Polar encoding. Each scaled value gives an angle, and the time index gives the radius: \phi = \arccos(\tilde{x}) and r = t/N, where N is a constant that regularises the span §2. The [0,1] scaling maps into angles in [0, \pi/2], and the [-1,1] scaling maps into [0, \pi] §2.1.

GASF and GADF. With \phi from above §2.1:

\mathrm{GASF}_{ij} = \cos(\phi_i + \phi_j) = \tilde{x}_i \tilde{x}_j - \sqrt{1-\tilde{x}_i^2}\,\sqrt{1-\tilde{x}_j^2} §2.1

\mathrm{GADF}_{ij} = \sin(\phi_i - \phi_j) = \sqrt{1-\tilde{x}_i^2}\,\tilde{x}_j - \tilde{x}_i\,\sqrt{1-\tilde{x}_j^2} §2.1

The paper gives both as matrix products §2.1. The main diagonal of GASF is \cos(2\phi_i) §4.

MTF. Assign each value to one of Q quantile bins. Count transitions between consecutive points, normalise, and fill the n \times n field so that cell (i, j) holds the transition probability between the bin of time i and the bin of time j §2.2. The field is then shrunk by averaging non-overlapping m \times m blocks §2.2, §3.2.

Classification pipeline.

1. Encode each series as GASF, GADF and MTF images. The GAF size is set by the PAA bin count S_{GAF}, and the MTF is shrunk to S_{MTF} §3.2.
2. Stack GASF, GADF and MTF of equal size as three channels §3.3.
3. For each image size and quantile count, pretrain a Tiled CNN with TICA on the full unlabeled set, training and test together §3.2, fig. 3.
4. Train a linear soft-margin SVM on the top features and pick C by 5-fold cross-validation on the training set §3.2, fig. 3.
5. Pick \{S, Q, C\} with the lowest training-set error; on ties prefer the larger S, then the larger Q §3.2.

The network is two stacked Tiled CNN layers with 6 maps each, tile sizes k \in \{1, 2, 3\}, 8 \times 8 receptive fields in the first layer and 3 \times 3 in the second, and 3 \times 3 TICA pooling blocks fig. 3.

Imputation pipeline (denoising autoencoder on GASF).

# Gramian angular summation field of one series  [§2.1]
# Input:  series (n,), mode "[0, 1]" or "[-1, 1]"
# Output: gasf (n, n)

def gasf_image(series, mode):
    # Rescale the series with its own min and max  [§2.1]
    scaled = rescale(series, mode)                  # (n,)
    # Read each scaled value as the cosine of an angle  [§2]
    angles = arccos(scaled)                         # (n,)
    gasf = zeros(n, n)
    for i in range(n):
        for j in range(n):
            # Cell (i, j) is the cosine of the angle sum  [§2.1]
            gasf[i, j] = cos(angles[i] + angles[j])
    return gasf

# Impute missing points through a denoising autoencoder on GASF  [§4, §4.1]
# Input:  series (n,), boolean missing_mask (n,), trained autoencoder
# Output: series (n,) with missing points filled

def impute_with_gasf(series, missing_mask, autoencoder):
    # Set missing points to zero, as the training noise did  [§4.1]
    corrupted = where(missing_mask, 0, series)      # (n,)
    # Rescale to [0, 1], the range where the diagonal inverse holds  [§4]
    scaled = rescale(corrupted, "[0, 1]")           # (n,)
    # Build the broken GASF and let the trained model repair it  [§4]
    broken = gasf_image(scaled, "[0, 1]")           # (n, n)
    recovered = autoencoder.reconstruct(broken)     # (n, n)
    # Invert the diagonal: cos(phi) = sqrt((cos(2 phi) + 1) / 2)  [§4]
    imputed = sqrt((diagonal(recovered) + 1) / 2)   # (n,)
    # Undo the rescaling; which min and max are used is not stated  [§4]
    return undo_rescale(imputed, scaled_params)

Denoising autoencoder. A single hidden layer of 500 sigmoid units, trained with mean squared error on the GASF images. During training, 20% of the raw points are set to zero before encoding, and the input is not noised again §4, §4.1.

Hyperparameters

Results

Author-reported. Setup: 20 datasets from the UCR time-series repository, each pre-split into train and test §3. Classifier: Tiled CNN on compound GASF-GADF-MTF images with a linear SVM §3.3. Baselines are error rates taken from earlier publications, not rerun by the authors §3. Imputation: DA on GASF versus DA on raw series, on five datasets, MSE averaged over 10 runs §4.1. Hardware is not stated.

Classification error on 20 UCR datasets tab. 1. Bold marks the best value in each row as in the paper. The last column holds the GASF-GADF-MTF result with the selected (PAA size, quantile size).

Headline: the GASF-GADF-MTF column has the lowest error on 9 of 20 datasets, the most of any column in tab. 1. The paper's abstract describes the result as "highly competitive" with nine other methods §1, §3.3. The 9 comparison methods are three classic baselines and six recent methods tab. 1 caption.

Imputation MSE tab. 2. "Full" is the MSE over the whole recovered series. "Interpolation" is the MSE over only the missing points. The table header uses the word "Interpolation"; the text calls the same quantity "imputation MSE" §4.2.

The paper reports interpolation MSE reductions of 12.18% to 48.02% §4.2. The per-dataset reductions in tab. 2 are 48.02% (ECG), 24.2% (CBF), 21.3% (Gun Point), 12.18% (SwedishLeaf) and 35.6% (7 Misc). These percentages were computed here from tab. 2 and are not stated in the paper. GASF has a higher full MSE in all five rows tab. 2.

Paper vs code

No official code.

The paper names no repository, and the arXiv metadata carries no code link. The search covered four GitHub queries: "imaging time series gramian", "zhiguang wang time series gramian", "markov transition field" and "GASF GADF MTF tiled cnn". None returned a repository owned by the authors or their lab. A user search for "zhiguang wang" returned one account whose public repositories are all forks of infrastructure tools, with no link to this paper. It was not used.

Pitfalls

• Parameter grid versus Table 1. The text gives the grid S \in \{16, 24, 32, 40, 48\} and Q \in \{8, 16, 32, 64\} §3.2. Table 1 reports selected values outside it: S = 8 and S = 64 (for example ECG, FaceAll and Trace), and Q = 24, 40 and 48 (for example CBF, Beef and Trace) tab. 1. Either the grid or the table is wrong. Use the table's values for reproduction and keep the grid as a reported range until the authors clarify.
• Search cost. The grid has 5 sizes and 4 quantile counts, so there are 20 pretraining runs per dataset, plus a 5-fold search over nine values of C for each one §3.2. The number of runs per dataset is not stated beyond this.
• Single S for compound images. The three channels must share one size §3.3, so the table gives one (S, Q) pair per dataset, not one per channel.
• Index convention in MTF. The paper defines w_{ij} as the frequency with which a point in quantile q_j is followed by one in q_i, then normalises with \sum_j w_{ij} = 1 §2.2. That sums over the origin index, not the destination, so the rows do not give a standard transition matrix from the stated index. The pseudocode above uses the standard row-normalised convention (from-bin, to-bin). Check the normalisation against the authors' code if it is released.
• Quantile binning is not specified. Tie handling, bin boundaries and whether bins hold equal counts are not stated §2.2.
• Uneven MTF blocks. The block size m = \lceil n / S_{MTF} \rceil §3.2 can produce a field that does not divide evenly into S_{MTF} blocks. Padding or truncation is not stated.
• Test data in pretraining. TICA pretraining uses all series, test included, without labels §3.2. This is transductive. Reported test errors are not from a model that never saw the test inputs.
• Selection on training error. Hyperparameters are selected on training-set error §3.2, while C uses 5-fold CV on the same training set §3.2. Whether the training-set error is a held-out value is not stated.
• Baselines not rerun. The competitor numbers are copied from earlier publications §3, so protocol differences cannot be ruled out. Results are single-run values; no variance is reported for classification tab. 1.
• Tiled CNN details missing. Learning rate, epochs, pretraining iterations, activation, initialisation and stride are not stated §3.1, fig. 3. The paper defers the Tiled CNN method to its original reference §3.1, which is not in this wiki.
• Invertibility holds only for [0, 1]. The diagonal inverse \cos\phi = \sqrt{(\cos 2\phi + 1)/2} holds only for \phi \in [0, \pi/2], that is for [0, 1]-rescaled data §4. The paper says the [-1, 1] map is not invertible in this sense §2.1, §4. The rescaling used in the imputation runs is not stated explicitly. An implementer must store the min and max of each series to invert the scaling.
• Corruption uses zeros. Missing points are simulated by setting values to 0 §4.1. A real zero in the data cannot be told apart from a removed point unless a mask is kept. The test-time corruption rate is not stated §4.1.
• Full MSE gets worse. GASF imputation raises full-series MSE in every row of tab. 2. The gain is confined to the missing points. Do not describe it as a general improvement in reconstruction.
• Equation numbers not verified. Equations are cited by section because the compiled PDF was not available to count numbers. Labels in the LaTeX source include eqn:polar, eqn:GASF, eqn:GADF, eqn:MTF and eqn:bijection.
• Naming. The text says "imputation MSE", the table says "Interpolation MSE" §4.2, tab. 2. Both refer to MSE over missing points only.
• Figures. Figures 1 and 2 illustrate the GAF and MTF pipelines, figure 3 the Tiled CNN, figure 4 the imputation pipeline, figure 5 the reconstructions from feature maps, and figure 6 the filters learned by the autoencoder [fig. 1–6]. None is reproduced here, per the no-images rule.

Lineage

• Builds on: Tiled convolutional networks, trained with topographic ICA pretraining §3.1. Also the duality between time series and complex networks and the Markov-matrix encoding of Campanharo et al. §2, §2.2, which the paper extends to keep time order.
• Compared against: 1NN with Euclidean distance (RAW), 1NN-DTW with best warping window (BWW) and no warping window (nWW), Fast-Shapelets, SAX-BoP, SAX-VSM, RPCD, SMTS and TSBF §3, tab. 1.
• Superseded by: not stated.