MultiRocket: Multiple pooling operators and transformations for fast and effective time series classification

In plain words

Time series classification means giving a label to a sequence of measurements, such as a sensor trace, by looking at its shape over time. The most accurate methods are large ensembles of many models and take hours or days to train. MiniRocket is a much faster method. It slides many small fixed filters over the signal, summarises each filter's output with a simple statistic, and trains a plain linear classifier on those numbers.

MultiRocket keeps that recipe and changes two things. It also looks at how the signal changes from one step to the next, which brings out slopes and sudden jumps. It also summarises each filter's output in four ways instead of one: how often the output is positive, how large the positive values are on average, where in the sequence they tend to fall, and the longest unbroken run of positive values. More views and more summaries give the same kind of linear classifier a richer picture.

The result is accuracy close to the best slow ensembles on a standard benchmark, at a small fraction of their cost. The filters are still a fixed recipe rather than a learned network, and the authors test the method only on univariate series.

TL;DR

• Keeps MiniRocket's pipeline: convolve with fixed kernels, pool the outputs, train a linear classifier §3.
• Adds the first-order difference of the series as a second input, and adds three pooling operators to PPV, for four in total per kernel and dilation §3.1 §3.4.
• Default: 49,728 features (6,216 × 2 × 4), a ridge classifier §1 §3.5.
• Author-reported: significantly more accurate than MiniRocket and not significantly less accurate than HIVE-COTE 2.0 on 109 UCR datasets over 30 resamples §4.1.
• Roughly 10 times slower than default MiniRocket (10,000 features), but about 5 minutes for all 109 datasets on 32 threads §1 §4.1 tab. 2.

Problem

The most accurate methods for time series classification, such as HIVE-COTE 2.0, are computationally expensive. The authors report that one CPU thread needs more than two weeks to train and test HIVE-COTE 2.0 on the 109 UCR datasets §2.1. MiniRocket is far faster, but it is somewhat less accurate than the HIVE-COTE family §1. The authors argue that accuracy comes from diversity of representations. HIVE-COTE 2.0 gets that diversity from ensembling, while MiniRocket uses a single representation §3.1. The question is whether MiniRocket can be extended with more diverse features at a small added cost §1 §3.

Method

Overview. MultiRocket uses MiniRocket's architecture. It transforms each series with convolution kernels, computes features from the outputs, and trains a linear classifier §3. It differs in two ways: it adds the first-order difference as a second representation, and it pools each convolution output with four operators instead of one §3.

Steps.

1. Representations. The base series X (length L) is used as is, and also transformed to its first-order difference X', where X'_t = x_t − x_{t−1} for t = 2..L. So X' has length L − 1 §3.1.
2. Kernels. There are 84 fixed kernels, each of length 9 with weight 2 at three taps and weight −1 at the other six. The 84 kernels are all ways to choose the three taps §3.2 multirocket.py.
3. Dilations. Each kernel is applied at a set of dilations. The exponents are spread evenly from 0 to a maximum that depends on the input length §3.2. The exact maximum formula is in Pitfalls and Paper vs code.
4. Biases. For each kernel and dilation, the convolution is run on one randomly chosen training series. Quantiles of that output become the bias values, one per feature §3.2.
5. Padding. Within each representation, the kernel and dilation combinations alternate between padded (full-length output) and unpadded (edge values trimmed) §3.2. The code's parity rule is in Paper vs code.
6. Convolution. Each representation is convolved with its own set of dilations and biases, so the base and difference series have different parameters §3.3.
7. Pooling. For each convolution output z and bias b, the four operators are PPV, MPV, MIPV and LSPV. PPV is the proportion of positive values §3.4.1. MPV is the mean of the positive values §3.4.2. MIPV is the mean index of the positive values §3.4.3. LSPV is the longest stretch of positive values §3.4.4. Algorithm 1 gives step-by-step code for all four alg. 1 app. A.
8. Features. Concatenate all pooled values from both representations. The default has 6,216 kernel/dilation/bias combinations per representation, giving 49,728 features in total §3.5 §1.
9. Classifier. A ridge regression classifier is the default. Logistic regression is an option for larger datasets §3.5.

# Fit dilations, feature counts and biases for one representation  [§3.2]
# Input:  train (N, L) float; per_kernel = features per kernel (74 default)
# Output: dilations, feature count per dilation, biases

def fit_representation(train, per_kernel):
    length = train.shape[1]
    # Exponents spread evenly from 0 to the maximum  [§3.2]
    # [code: multirocket/multirocket.py#L105-L107]
    max_exponent = log2((length - 1) / (9 - 1))
    exponents = linspace(0, max_exponent, 32)
    dilations, counts = unique(floor(2 ** exponents), return_counts=True)
    # Share the features of each kernel across its dilations  [code#L103-L109]
    per_dilation = floor(counts * per_kernel / 32)
    # Hand leftover features to the dilations one at a time  [code#L111-L117]
    per_dilation = spread_remainder(per_dilation, per_kernel)
    biases = []
    for dilation_index, dilation in enumerate(dilations):
        for kernel_index in range(84):
            # Bias source: one random training series  [§3.2] [code#L60]
            series = train[random_index(len(train))]
            # Full-length response, even when the combination is trimmed later
            # [code: multirocket/multirocket.py#L86-L91]
            response = convolve_full(series, kernel_index, dilation)
            # Quantile levels for this kernel and dilation  [code#L121-L123]
            levels = next_quantile_levels(per_dilation[dilation_index])
            biases += quantile(response, levels)
    return dilations, per_dilation, biases

# Feature vector for one univariate series  [§3.1] [§3.3] [§3.4]
# Input:  series (L,) float; fitted parameters for both representations
# Output: features, four per kernel/dilation/bias combination

def multirocket_features(series, base_params, diff_params):
    # First-order difference, one value shorter  [§3.1]
    diffed = series[1:] - series[:-1]        # (L-1,)
    base = pool_all(series, base_params)
    diff = pool_all(diffed, diff_params)
    return concatenate([base, diff])

# Pool one representation over all kernels, dilations and biases  [§3.2]
# Input:  x (n,) series; params = (dilations, per-dilation counts, biases)
# Output: list of pooled values

def pool_all(x, params):
    dilations, counts, biases = params
    features = []
    for dilation_index, dilation in enumerate(dilations):
        for kernel_index in range(84):
            # Alternate padding across combinations, so half are padded  [§3.2]
            # [code: multirocket/multirocket.py#L191-L192] [#L223]
            same_length = (dilation_index + kernel_index) % 2 == 0
            response = convolve_full(x, kernel_index, dilation)   # (n,)
            if not same_length:
                # Keep only values that do not touch the zero edges
                # [code: multirocket/multirocket.py#L266]
                response = response[4 * dilation : -4 * dilation]
            for bias in biases_for(dilation_index, kernel_index):
                # Four pooled values for this response and bias  [§3.4]
                features += pool_four(response, bias)
    return features

# Taps of one kernel at one dilation, zero outside the series  [§3.2] [§3.3]
# Weights: 2 at the three chosen taps, -1 at the other six  [§3.2]
# Tap offsets run from -4 * dilation to +4 * dilation  [code#L207-L230]

def convolve_full(x, kernel_index, dilation):
    # Tap positions chosen for this kernel, out of nine  [code#L23-L24]
    chosen = kernel_taps[kernel_index]
    response = zeros(len(x))
    for tap in range(9):
        offset = (tap - 4) * dilation
        weight = 2 if tap in chosen else -1
        # Add weight times the shifted series, zero where it runs off the end
        response += weight * shift(x, offset)
    return response

# Four pooling operators for one response  [§3.4] [alg. 1]
# Input:  response (n,) convolution output; bias (scalar)
# Output: [ppv, mpv, mipv, lspv]

def pool_four(response, bias):
    # Threshold convention: a value counts as positive when it is above
    # the bias, following the code  [code: multirocket/multirocket.py#L244]
    shifted = response - bias                 # (n,)
    positive = shifted > 0
    count = sum(positive)
    # Proportion of positive values  [§3.4.1]
    ppv = count / len(shifted)
    # Mean of the positive values, or 0 when there are none  [§3.4.2]
    mpv = mean(shifted[positive]) if count > 0 else 0
    # Mean index of the positive values, or -1 when there are none  [§3.4.3]
    positions = arange(len(shifted))
    mipv = mean(positions[positive]) if count > 0 else -1
    # Longest stretch of positive values  [§3.4.4] [alg. 1]
    lspv = longest_stretch(shifted)
    return [ppv, mpv, mipv, lspv]

# Longest stretch as Algorithm 1 defines it  [alg. 1]
# Input:  shifted (n,) values; positive means shifted > 0
# Output: integer stretch length

def longest_stretch(shifted):
    # Index of the last non-positive value, starting at 0  [alg. 1]
    last_break = 0
    longest = 0
    for position in range(len(shifted)):
        if shifted[position] > 0:
            continue
        # Gap back to the previous non-positive value  [alg. 1]
        longest = max(longest, position - last_break)
        last_break = position
    # Gap from the last non-positive value to the end  [alg. 1]
    longest = max(longest, len(shifted) - 1 - last_break)
    return longest

Hyperparameters

Results

Author-reported. Setup: 109 univariate UCR datasets, 30 resamples each, average accuracy; baselines are HIVE-COTE 2.0, TS-CHIEF, InceptionTime, MiniRocket, Arsenal, DrCIF, TDE, STC and Proximity Forest, with results taken from the HIVE-COTE 2.0 paper and the MiniRocket repository §4.1.

Headline: MultiRocket is significantly more accurate than MiniRocket. It is not significantly less accurate than HIVE-COTE 2.0, TS-CHIEF or InceptionTime, and it is the only method whose difference from HIVE-COTE 2.0 is not significant after Holm correction §4.1 fig. 4 fig. 5. The average ranks are shown only in figures, so the paper gives no numbers for them in the text fig. 1 fig. 4.

Runtime, total training time as reported in the paper tab. 2. MultiRocket and MiniRocket rows are single-thread on the stated CPU. Other rows are from the HIVE-COTE 2.0 paper, and the paper says the InceptionTime row was trained on a GPU and is not directly comparable §4.2:

Wall-clock times on 32 threads: MultiRocket with 50,000 features takes about 5 minutes for all 109 datasets, against about 2 minutes for MiniRocket with 10,000 features §1 §4.1 fig. 2.

Paper vs code

Checked against the official repository at code_commit (GitHub, owned by the first author). The code is GPL-3.0, so this card cites it by path and line and contains no copied code LICENSE.

1. Dilation maximum. Paper: maximum exponent typeset as log2(l_input − 1) divided by (l_kernel − 1) §3.2. Code: log2((L − 1) / (9 − 1)) multirocket.py. The card uses the code form.
2. Dilations per kernel. Code caps distinct dilations at 32 and shares each kernel's feature count across them multirocket.py. The paper does not state a cap.
3. Bias quantile levels. Code assigns levels with a golden-ratio low-discrepancy sequence multirocket.py and computes all biases on the full-length response [code#L86-L91]. The paper only says quantiles of one random series' output are used §3.2.
4. Threshold sign. Code treats a value as positive when it is above the bias [code#L244] [#L279]. The paper's text says a positive bias lets PPV capture weak matches §3.4.1, which corresponds to the opposite sign. The card follows the code.
5. MPV offset. Code adds the bias to each positive value [code#L247] [#L282] [#L368] [#L403]. The paper defines MPV as the mean of the positive values, and Algorithm 1 averages those values §3.4.2 alg. 1. With the code's threshold, that gives the mean minus the bias, so the code's value differs by twice the bias. The card uses the paper's definition.
6. LSPV. The equation gives the maximum of j − i over runs of positive values §3.4.4. Algorithm 1 and the code measure the gap between consecutive non-positive positions alg. 1 [code#L248-L253]. For a run between two non-positive values, that gap is the run length plus one. Table 1's value for example D is 4 tab. 1; the Algorithm 1 rule gives 5 for D (hand-traced from the algorithm and code, not executed). The card follows Algorithm 1.
7. Ties. Code treats a value equal to the bias as neither positive nor a break [code#L244] [#L248]. Algorithm 1 treats it as non-positive alg. 1. The card follows Algorithm 1.
8. Empty positives. MPV is undefined when there are no positive values, and MIPV is −1 §3.4.2 §3.4.3. Code sets MPV to 0 and MIPV to −1 [code#L262-L264]. The card does the same.
9. Ridge classifier. Code standardises features and searches α over logspace(−3, 3, 10) with RidgeClassifierCV [code#L502-L509]. The paper gives no scaling or α grid §3.5.
10. Logistic classifier. Code uses Adam with learning rate 1e-4 and a maximum of 200 epochs passed by MultiRocket logistic_regression.py [#L58-L63] multirocket.py. The paper gives no details §3.5.
11. Experiments. The paper says all its experiments used the ridge classifier §3.5. The 109-dataset script uses the default ridge classifier main_ucr_109.py. The 128-dataset script uses logistic regression main_ucr.py. That script is not described in the paper.
12. Non-finite values. Code replaces NaN features with 0 [code#L492] [#L547]. The paper does not say so.
13. Multivariate. The repository includes a multivariate module multirocket_multivariate.py. The paper covers univariate series only §5.

No other discrepancies found. The code implements the kernels, dilations, padding, first-order difference, four pooling operators and classifier described in the paper.

Pitfalls

• Kernel count wording. The paper calls the 6,216 combinations "kernels" in its feature count §3.5. They are kernel/dilation/bias combinations; there are only 84 fixed kernels.
• Dilation formula. The typeset formula in §3.2 would give a much smaller maximum dilation than the code's log2((L − 1) / 8) [code#L105]. Use the code's form, and check the maximum dilation for your series length.
• Short series. For L < 9 the code's maximum exponent is negative, so the dilations round down to 0 [code#L105-L107]. The code does not guard against this. This is a reading of the code, not a tested result.
• Trimmed lengths. Unpadded combinations pool a response that is 8 × dilation values shorter than the series [code#L266] [#L278-L293]. PPV divides by that shorter length, so the scale of PPV varies across combinations. Biases are computed on the full-length response [code#L86-L91], which the paper does not describe.
• Sign and offset. Threshold sign and the MPV offset (Paper vs code, items 4 and 5) change feature values. Decide which convention to implement, and check the result against the code's output on a few series before comparing with published numbers.
• LSPV off-by-one. The equation and the algorithm disagree, and Table 1 disagrees with the algorithm on example D (Paper vs code, item 6).
• Randomness. The only random choice is the training series used for each bias, and the paper says so §3.2. The code calls NumPy's random generator inside a Numba-compiled function [code#L60]. Numba keeps its own generator state, so seeding NumPy outside the compiled function may not make results repeatable. This is a known Numba behaviour, not something tested here.
• Memory. The feature matrix is float32 with 49,728 columns by default, so about 199 kB per series (derived from the stated feature count). About 2 GB for 10,000 series.
• Compute claims disagree. Section 4.2 says the 50k-feature version is about 4 times slower than MiniRocket with 50k features on one thread §4.2. The Table 2 totals give about 3 times (15.77 / 5.25) tab. 2. Section 1 says it is about 10 times slower than MiniRocket at 10k features §1. The timings measure different things and use different dataset counts, so do not mix them.
• Dataset counts disagree. Most of the paper uses 109 datasets §1 §4. Table 2's caption and the HIVE-COTE 2.0 timings in Section 4.2 say 112 tab. 2 §4.2.
• Repeated count. The count "76 datasets" appears for two different baselines: MiniRocket with 50k features §4.3.3 and MiniRocket with 10k features app. D. Check which one applies before quoting it.
• Per-dataset accuracies. The accuracies quoted for SemgHandMovementCh2 and PigAirwayPressure do not say which method each value belongs to §4.1. The source contains a comment flagging this [raw/src/2102.00457/main-ecmlpkdd.tex#L632].
• Tuning overlap. Design choices were tuned on 40 development datasets, "to avoid overfitting" §3. The paper does not say whether these overlap with the 109 evaluation datasets.
• Cost on a single thread. Single-thread training on 109 datasets takes about 40 minutes for MultiRocket with 50k features, against about 4 minutes for MiniRocket §2.1. Multithreading is the recommended setting §4.2.
• Tap alignment. The paper describes convolution only as a sliding dot product §3.3. A commented-out equation in the source uses forward offsets, while the code centres the nine taps on each output position [code#L207-L230]. The card follows the code (convolve_full).

Lineage

• Builds on: MiniRocket (Dempster et al., KDD 2021), which uses the same 84 kernels, PPV and bias sampling §2.2 §3. ROCKET (Dempster et al., 2020) is the base method §2.2. Neither has a card in this wiki, and the source bibliography gives no arXiv ids.
• Draws on: DrCIF's use of diverse representations, including first-order differences §3.1.
• Compared against: HIVE-COTE 2.0, TS-CHIEF, InceptionTime, MiniRocket, Arsenal, DrCIF, TDE, STC and Proximity Forest §4.1.
• Superseded by: not known from the wiki content.