Random convolutional kernels

Overview

The idea is to turn a time series into a fixed-length feature vector without training any convolution filters. The series is convolved with a bank of short kernels at several dilations. Each response is summarised by pooling operators, such as the proportion of positive values. A linear classifier is then fitted on the pooled numbers 2102.00457 §3.

The name fits two forms of this idea. MultiRocket uses a fixed set of kernels: 84 of them, with dilations set by a formula 2102.00457 §3.2. Its only random choice is which training series supplies the bias values 2102.00457 §3.2. Arsenal, a module of HIVE-COTE 2.0, draws its kernels at random: the weights, biases, dilations and padding are all sampled 2104.07551 §3.3, alg. 4.

The invariant the variants share: the feature extractor is built without using the labels, from the series length and either a fixed recipe or random draws. Only the final classifier is fitted to the labels 2102.00457 §3.5 2104.07551 §3.3. The MultiRocket authors argue that accuracy comes from diverse representations of the same series. HIVE-COTE 2.0 gets that diversity by combining four different modules 2102.00457 §3.1 2104.07551 §3. Arsenal gets it inside one family, from 25 members with 2,000 random kernels each 2104.07551 tab. 2.

The page draws on two cards. ROCKET, the origin of the Arsenal design, and MiniRocket are named in these cards, but neither has a card here. This page uses ROCKET only through the HIVE-COTE 2.0 result tables.

Variants

The building blocks below are the ones the MultiRocket card describes. Each is a design choice that other variants in this family can change. The Arsenal section that follows changes the kernels and the classifier.

• Kernels. 84 fixed kernels of length 9. Each has weight 2 at three taps and −1 at the other six. The 84 kernels are all ways to choose the three taps 2102.00457 §3.2.
• Dilations. Each kernel runs at a set of dilations. Their exponents are spread evenly from 0 to a maximum that depends on the input length 2102.00457 §3.2. The paper's typeset maximum differs from the code's formula (see Contradictions).
• Biases. For each kernel and dilation, the convolution runs on one random training series. Quantiles of that output become one bias each 2102.00457 §3.2.
• Padding. About half of the kernel and dilation combinations are padded, so the output keeps the full length. The rest are trimmed at the edges 2102.00457 §3.2.
• Pooling operators. Four are used per response and bias: PPV, the proportion of positive values; MPV, the mean of the positive values; MIPV, the mean index of the positive values; and LSPV, the longest stretch of positive values 2102.00457 §3.4, alg. 1.
• Input representations. The base series and its first-order difference, each with its own dilations and biases 2102.00457 §3.1, §3.3.

MultiRocket (default, 50k features)

Keeps the pipeline above and uses both representations with all four operators. That gives 6,216 kernel/dilation/bias combinations per representation, times 2 representations and 4 operators, for 49,728 features 2102.00457 §3.5, §1. The default classifier is a ridge regression. The paper says logistic regression is an option for larger datasets but gives no details 2102.00457 §3.5. Card: MultiRocket.

Author-reported results on 109 univariate UCR datasets, 30 resamples each 2102.00457 §4.1:

• Significantly more accurate than MiniRocket. Against MiniRocket with 10k features, MultiRocket (50k) wins on 76 datasets, loses on 29 and ties on 4 2102.00457 app. D.
• Not significantly less accurate than HIVE-COTE 2.0, TS-CHIEF or InceptionTime. HIVE-COTE 2.0 is still ahead on 59 datasets 2102.00457 §4.1, fig. 6.

Training time, total single-thread figures from the paper's table 2102.00457 tab. 2:

On 32 threads, the 50k version takes about 5 minutes for all 109 datasets, against about 2 minutes for MiniRocket with 10k features 2102.00457 §1, §4.1.

MultiRocket, 10k features

A smaller variant with the same operators. The paper uses it as the setting matched to MiniRocket's default feature count 2102.00457 §4, app. D. Its author-reported wins over MiniRocket (10k) are 70 datasets 2102.00457 app. D. Its total single-thread time is 4.38 minutes 2102.00457 tab. 2. Card: MultiRocket.

Arsenal: random kernels in a ridge ensemble

Arsenal is one of the four modules of HIVE-COTE 2.0. It replaces the single ROCKET ridge classifier with 25 ROCKET-style members, each built on 2,000 random kernels 2104.07551 §3.3, alg. 4, tab. 2. Each kernel has:

• a length of 7, 9 or 11;
• weights drawn from N(0, 1) and then mean-centred;
• a bias drawn from U(−1, 1);
• an exponentially spaced dilation, and a random padding flag 2104.07551 §3.3.

Each kernel gives two features per series: the maximum response, and the proportion of positive responses 2104.07551 alg. 4. Each member is a ridge classifier with its regularisation chosen by built-in cross-validation 2104.07551 §3.3, alg. 4. For multivariate data, a kernel draws a random number of dimensions 2104.07551 §3.3. Card: HIVE-COTE 2.0.

Arsenal is reported inside the ensemble, not alone. The card gives no standalone accuracy for it. What it does give:

• Time for one resample of the 112 UCR problems, single core: 27.91 h for Arsenal, against 2.85 h for the ROCKET baseline and 340.21 h for HIVE-COTE 2.0 2104.07551 tab. 4.
• Against the ROCKET baseline, HIVE-COTE 2.0 has a mean accuracy difference of −2.49% on UCR, and wins on 97 of 112 datasets 2104.07551 tab. 3. The baseline is a single ROCKET ridge classifier with 10,000 kernels 2104.07551 tab. 2.
• On the 26 UEA multivariate problems, HIVE-COTE 2.0 beats ROCKET with a mean difference of −2.52%, and wins on 19 datasets 2104.07551 tab. 5. These rows measure the full ensemble, not Arsenal alone.

When to use which

Implementation sketch

The common core, in Python-flavoured pseudocode. The steps follow MultiRocket; variants differ in the parts marked in the notes below.

# Fit the dilations and biases for one representation  [arxiv:2102.00457 §3.2]
# Inputs: train (N, L); features per kernel (74 by default)

def fit_representation(train, per_kernel):
    length = train.shape[1]
    # Dilation exponents spread evenly up to a length-based maximum
    # [arxiv:2102.00457 §3.2] [code: multirocket/multirocket.py#L105]
    max_exp = log2((length - 1) / 8)
    dilations = unique(floor(2 ** linspace(0, max_exp, 32)))
    biases = []
    for dilation in dilations:
        for kernel in range(84):
            # Bias source: one random training series [arxiv:2102.00457 §3.2]
            series = train[random_index(len(train))]
            response = convolve_full(series, kernel, dilation)
            # Quantiles of the response become the biases
            biases += quantile(response, quantile_levels(per_kernel))
    return dilations, biases


# Feature vector for one series  [arxiv:2102.00457 §3.1, §3.3, §3.4]

def features(series, params_base, params_diff):
    # First-order difference, one value shorter  [arxiv:2102.00457 §3.1]
    diffed = series[1:] - series[:-1]
    # Pool each representation over its kernels, dilations and biases
    return concat([pool_all(series, params_base),
                   pool_all(diffed, params_diff)])


# Four pooling operators for one response and one bias  [arxiv:2102.00457 §3.4]

def pool_four(response, bias):
    # A value is positive when it is above the bias
    shifted = response - bias
    positive = shifted > 0
    ppv = mean(positive)
    # Mean of the positive values; 0 when there are none
    mpv = mean(shifted[positive]) if any(positive) else 0
    # Mean index of the positive values; -1 when there are none
    mipv = mean(where(positive)) if any(positive) else -1
    # Longest run of positive values  [arxiv:2102.00457 alg. 1]
    lspv = longest_run(positive)
    return [ppv, mpv, mipv, lspv]

# Classifier: ridge regression on the feature matrix, with the default
# settings in the paper's text  [arxiv:2102.00457 §3.5]

Notes on where implementations differ, all from the card's comparison of paper and code 2102.00457 §3.2, §3.4 multirocket.py:

• Threshold sign. The code counts a value as positive when it is above the bias. The paper's text reads as the opposite sign 2102.00457 §3.4.1.
• MPV offset. The code adds the bias back into MPV. The paper defines MPV as the mean of the positive values, so with the code's threshold the two differ by twice the bias 2102.00457 §3.4.2, alg. 1.
• Longest run. Algorithm 1 and the code measure gaps between non-positive positions. That gives one more than the run length for a run between two non-positive values, which differs from the equation in §3.4.4 2102.00457 §3.4.4, alg. 1.
• Trimmed responses. Unpadded combinations are 8 × dilation values shorter than the series. PPV divides by that shorter length, so its scale varies across combinations 2102.00457 §3.2.
• Randomness. The paper's only random choice is the bias series 2102.00457 §3.2. The code draws it inside a Numba-compiled function, so seeding NumPy outside may not make runs repeatable. This is a reading of the code, not a tested result multirocket.py.

To get Arsenal from the sketch, replace the 84 fixed kernels with random draws as listed in its section, and keep two features per kernel: the maximum and the proportion of positive responses 2104.07551 alg. 4. The paper's text describes a majority vote over members, but the code weights each member by its cross-validated ridge score raised to the power 4 (see Contradictions).

Contradictions

Across cards, the two papers report different things rather than contradicting each other on the same quantity:

• Timings across papers. MultiRocket reports 15.77 min for 109 UCR datasets 2102.00457 tab. 2. HIVE-COTE 2.0 reports 2.85 h for ROCKET, and 27.91 h for Arsenal, on 112 UCR datasets 2104.07551 tab. 4. The dataset sets, implementations and hardware differ, so these numbers should not be divided into a speed-up.
• Dataset sets across papers. MultiRocket uses 109 univariate UCR datasets. HIVE-COTE 2.0 uses 112 UCR problems 2102.00457 §4.1 2104.07551 §4. Accuracy claims from the two cards are not measured on the same set.
• Arsenal vote. The HIVE-COTE 2.0 paper says new cases are classified by majority vote 2104.07551 §3.3. The code weights each member by its ridge cross-validation score raised to the power 4 Arsenal.java Arsenal.java. The paper gives no formula for the probabilities 2104.07551 §3.3.
• Arsenal normalisation. The paper does not mention normalising the input. The code defaults normalise to true Arsenal.java.
• Arsenal padding. The algorithm writes the padded length as m + 2p, while the text describes padding as a yes/no flag. The padding amount is not stated 2104.07551 alg. 4, §3.3.

Within the MultiRocket paper, these sources disagree:

• Dilation maximum. The paper typesets the maximum exponent with a different denominator than the code's log2((L − 1) / 8). The paper's form gives a much smaller maximum dilation 2102.00457 §3.2 multirocket.py.
• Speed. Section 4.2 gives about 4× for the 50k version over MiniRocket (50k) on one thread 2102.00457 §4.2. The totals in the table give about 3× (15.77 / 5.25) 2102.00457 tab. 2. Section 1 gives about 10× over MiniRocket (10k) 2102.00457 §1, while the table gives about 6.5× (15.77 / 2.44). The timings measure different things, so do not mix them.
• Dataset count. Most of the paper uses 109 datasets. The table caption and the HIVE-COTE 2.0 timings say 112 2102.00457 §1, §4 2102.00457 tab. 2, §4.2.
• Per-dataset values. The accuracies quoted for two single datasets do not say which method each value belongs to 2102.00457 §4.1.

Papers

• MultiRocket, arXiv:2102.00457: adds the first-difference input and the PPV, MPV, MIPV and LSPV operators to the MiniRocket pipeline, with a 50k-feature default.
• HIVE-COTE 2.0, arXiv:2104.07551: a meta ensemble whose Arsenal module is 25 ROCKET-style ridge members built on random kernels; reports the ensemble's results, not Arsenal's alone.