Evaluation of deep learning models for multi-step ahead time series prediction

In plain words

The task is forecasting: given the last five values of a signal, predict the next ten. The paper compares seven neural networks on this same task, using the same windows of past values, on seven series. Four series come from simulated systems and three from real measurements (solar activity, a Santa Fe competition series and stock prices).

The authors report that the networks which read a window in both directions, and the design that squeezes the window into one summary and unrolls it into ten outputs, do best across the series. Plain feedforward nets and simple recurrent nets do worse. Each network is run 30 times, and the paper reports the average error with a confidence interval.

TL;DR

Seven neural forecasters are compared on seven univariate series, with 5 lagged inputs and 10 outputs trained as one direct multi-output problem §4.1 §5. The authors report that BD-LSTM or ED-LSTM has the lowest test RMSE on five of the seven series, and LSTM on the other two tab. 5 tab. 7. The released code is one script; it departs from the paper in several places (see Paper vs code).

Problem

Multi-step prediction can be done recursively, feeding each prediction back as input, or directly, predicting all horizons at once §1. Recursive error accumulates over horizons §1. The paper asks how well LSTM-type and other deep models perform on multi-step univariate prediction, which it says had not been evaluated for this task §1. It focuses on univariate series and a 10-step horizon §4.1.

Method

Data and preprocessing

• Series. Simulated: Mackey-Glass, Lorenz, Henon and Rossler. Real-world: Sunspot (solar activity; 2000 points, November 1834 to June 2001), ACI-finance (800 points, closing prices December 2006 to February 2010) and Lazer (Santa Fe competition, 500 points) §4.1.
• Scaling and split. Series are scaled to [0, 1]. The first 1000 points are used; the first 60% of them are for training and the rest for testing §4.1.
• Embedding. The series is embedded with dimension D = 5 and lag T = 1 (Takens embedding), with a 10-step-ahead target §3.1 §4.1. The embedded vector is Y(t) = [x(t), x(t-T), ..., x(t-(D-1)T)] §3.1.
• Windows. Each sample is 5 consecutive values as input and the next 10 as target, stride 1 Data processing.py [code: data/Data processing.py#L84].

# Sliding windows for direct 10-step prediction  [§3.1] [§4.1]
def make_windows(series, n_in=5, n_out=10):
    # Collect one (input, target) pair per start position  [§3.1]
    inputs, targets = [], []
    for i in range(len(series)):
        end_in = i + n_in
        end_out = end_in + n_out
        # Stop at the end  [code: data/Data processing.py#L22-L26]
        if end_out > len(series):
            break
        inputs.append(series[i:end_in])
        targets.append(series[end_in:end_out])
    return inputs, targets

Models

All models use ReLU activations §4.1. Hidden sizes and layer counts are in the configuration table below tab. 1. Adam is used for FNN-Adam, LSTM, BD-LSTM, ED-LSTM, the simple RNN and the CNN; SGD for FNN-SGD §4.1.

• FNN-SGD and FNN-Adam §3.2. A feedforward net (multilayer perceptron) trained by stochastic gradient descent, or by Adam. Single learning rate for SGD §3.2.
• Simple RNN §3.3. An Elman network with a context layer that stores the previous hidden state. Its update is Eq. (1) §3.3 eq. 1. Trained with backpropagation through time §3.3.
• LSTM §3.4. Input, forget and output gates with a cell state. The update is Eq. (2) §3.4 eq. 2. The paper reports h_0 = 0 and c_0 = 0 §3.4.
• BD-LSTM §3.5. Two LSTMs run in opposite time directions; the forward and backward states are combined at each step §3.5.
• ED-LSTM §3.6. An encoder LSTM maps the input window to a fixed-length vector; a decoder LSTM produces the output sequence from that vector, followed by a time-distributed output layer §3.6 tab. 1.
• CNN §3.7. A one-dimensional convolution with 64 filters, kernel size 3 and max-pooling of size 2, followed by dense layers tab. 1 [code: FNN/Code.py#L382-L386].

Adam update, as written in the paper §3.2 (unnumbered display):

# Adam update for parameters theta at step t  [§3.2]
# The paper gives no values for lr, beta1, beta2 or eps
def adam_step(theta, grad, m, v, t, lr, beta1, beta2, eps):
    # Running averages of the gradient and of its square  [§3.2]
    m = beta1 * m + (1 - beta1) * grad
    v = beta2 * v + (1 - beta2) * grad * grad
    # Bias correction for the zero initialisation  [§3.2]
    m_hat = m / (1 - beta1 ** t)
    v_hat = v / (1 - beta2 ** t)
    # Parameter step  [§3.2]
    theta = theta - lr * m_hat / (sqrt(v_hat) + eps)
    return theta, m, v

ED-LSTM as the released code builds it:

# Encoder-decoder LSTM for 10-step prediction  [§3.6]
# Code: FNN/Code.py lines 269-274
def ed_lstm(window):
    # Input is 5 past values with one feature  [§4.1]  # (B, 5, 1)
    # Encoder: 10 units, ReLU; keeps only the final state  [§3.6]
    code_vector = lstm_encoder(window)              # (B, 10)
    # Repeat the fixed-length vector once per horizon  [§3.6]
    repeated = repeat(code_vector, 10)              # (B, 10, 10)
    # Decoder: 10 units, one state per horizon  [§3.6]
    decoded = lstm_decoder(repeated)                # (B, 10, 10)
    # Per-horizon linear map to one value, ReLU output  [§3.6]
    return relu(dense_per_step(decoded))            # (B, 10, 1)

Evaluation protocol

• Metric. RMSE, defined in Eq. (3) as the square root of the mean squared error over N observations §4.1 eq. 3. Per-horizon RMSE is also reported §4.2 tab. 5.
• Repetitions. 30 runs per model and series, with different initial network weights §4.2. Mean and 95% confidence interval are reported for train and test RMSE and for each horizon §4.2.
• Confidence interval. The paper does not give a formula. The released code uses the mean plus or minus 1.96 times the standard deviation divided by the square root of 30 (see Pitfalls) Code.py.

# Summary of one model on one series  [§4.2]
# Code uses the same rule  [code: FNN/Code.py#L526-L534]
def summarise(test_rmse_per_run):
    # Mean of the 30 per-run test RMSE values  [§4.2]
    mean = average(test_rmse_per_run)
    # Population std, numpy default ddof=0  [code: FNN/Code.py#L531-L532]
    std = std_population(test_rmse_per_run)
    # Normal approximation, 95% half width  [code: FNN/Code.py#L533-L534]
    half_width = 1.96 * std / sqrt(30)
    return mean, half_width

Configuration tab. 1

Hyperparameters

Results

Author-reported. Setup: seven univariate series, 10-step-ahead direct prediction, 30 runs per model, RMSE on scaled data. Values are mean plus or minus the 95% confidence-interval half width, copied from the appendix tables of the paper. The released FNN/Results/OverallAnalysis.csv matches the ACI-finance FNN-Adam row we checked (train 0.1628, test 0.0885, step 1 0.0165).

Headline. Lowest test RMSE on the overall test row: BD-LSTM on Sunspot (0.1147) and Rossler (0.0349); ED-LSTM on Henon (0.3294), Lorenz (0.0234) and Mackey-Glass (0.0602); LSTM on ACI-finance (0.0860) and Lazer (0.1910). Mean rank across the seven series tab. 2: BD-LSTM 2.00, ED-LSTM 2.14, LSTM 2.42, CNN 3.85, RNN 4.85, FNN-Adam 5.42, FNN-SGD 7.00. The paper says FNN-SGD is worst on all series §5. It also says CNN beats LSTM on Mackey-Glass and Rossler §4.2.

Performance rank tab. 2

The paper gives no basis for the rank; see Pitfalls for two rows that do not follow from the appendix test values.

Per-series RMSE, train, test and horizons 1 to 10 tab. 5-tab. 11

Figures 6 to 12 plot RMSE with 95% interval error bars, for train and test across horizons and for each model at 10 steps: ACI-finance fig. 6, Sunspot fig. 7, Lazer fig. 8, Henon fig. 9, Lorenz fig. 10, Mackey-Glass fig. 11, Rossler fig. 12. Figures 13 to 17 show actual against predicted values for the best run of ED-LSTM on ACI-finance, Sunspot, Henon, Lazer and Mackey-Glass fig. 13-fig. 17.

ACI-finance tab. 5

Sunspot tab. 6

Lazer tab. 7

Henon tab. 8

Lorenz tab. 9

Mackey-Glass tab. 10

Rossler tab. 11

Literature comparison tab. 3 tab. 4

The paper also lists published numbers from other methods (RTRL, ESN, EKF, UKF, GPF, ANFIS, KELM, CMTL, SVR and its variants) for some series and horizons tab. 3 tab. 4. The authors say the comparison is not fair: the other methods may use different models, data processing and error measures, and some report the best run rather than mean and spread §4.3. Use these values only with their original sources.

Paper vs code

The released repository is sydney-machine-learning/deeplearning_timeseries, pinned at commit 4efda2d. It contains one script with all seven models (FNN/Code.py) and a data-preparation script (data/Data processing.py); the Keras and torch models are inside FNN/Code.py, not in separate files. It has no LICENSE file.

• RNN optimiser. Paper: Adam for the simple RNN §4.1. Code: torch SGD with learning rate 0.01, one sample per update Code.py Code.py.
• RNN architecture. Paper: Elman network with a context layer §3.3. Code: one RNNCell of 10 units on the 5-value window, with the hidden state set to zeros on every call, followed by a linear output layer Code.py. The 5 lags therefore enter as one step, not as a sequence. Table 1 says 2 hidden layers tab. 1.
• Independent runs. Paper: 30 runs with different initial weights §4.2. Code: the Keras models (LSTM, BD-LSTM, ED-LSTM, CNN) are built once Code.py Code.py Code.py Code.py, and fit is called 30 times on the same model Code.py. Keras keeps weights between fit calls, so each run continues the previous one. The RNN builds a new model per run Code.py. The MLPs reuse one estimator; scikit-learn's fit starts from new weights unless warm_start is set (library behaviour, not run here) Code.py Code.py.
• RMSE normalisation. Paper: Eq. (3), a root mean square over N observations §4.1 eq. 3. Code: the helper sums squared errors over all columns and divides by the number of rows Code.py. With 10 outputs the overall RMSE is therefore the square root of the sum of the 10 per-horizon mean squared errors, i.e. sqrt(10) times the per-element RMSE. Our arithmetic on the ACI-finance FNN-Adam step values in tab. 5 gives about 0.088 for this form, matching the reported test RMSE of 0.0885; the paper's form gives about 0.028.
• LSTM activation. Paper: Eq. (2) uses tanh for the candidate state and output §3.4 eq. 2; text says ReLU in all models §4.1. Code: activation='relu' for LSTM layers Code.py Code.py Code.py Code.py.
• Output activation. Paper: ReLU in all models §4.1. Code: ReLU only on the ED-LSTM output Code.py; linear outputs for LSTM, BD-LSTM Code.py Code.py and CNN Code.py.
• Batch size and shuffling. Not in paper. Code: batch size 64, shuffle=False for the Keras models Code.py Code.py Code.py Code.py.
• MLP settings. Not in paper. Code: alpha=0.1, tol=0, max_iter=1000 Code.py Code.py. With tol=0 and no early stopping, scikit-learn still stops after about 10 epochs without improvement (library default; not run here), so MLPs may train for fewer than 1000 epochs.
• Only CNN runs by default. main() loops over range(7, 8), so only method 7 (CNN) is run as committed Code.py. The other six methods require widening the loop. All results are written to the same OverallAnalysis.csv Code.py.
• Confidence interval. Code: np.std (ddof 0), mean plus or minus 1.96 times std over the square root of 30 Code.py. Paper: no formula.
• Best-run figures. The plotted predictions come from the run with the lowest test RMSE Code.py Code.py. They do not show a typical run.
• Data. The series are rebuilt from the values at zero-based indices 3 and 4 of the pre-split train.txt and test.txt rows Data processing.py, and windows are made with split_sequence Data processing.py. The file format, the scaler and the split ratio are not described in the code read here. scaled_dataset.txt for Sunspot has 1000 lines, matching the paper's first 1000 points.
• Simulated series. Parameters, integration step and length of Mackey-Glass, Lorenz, Henon and Rossler are not stated in the paper. The repository ships the raw .dat files lorenz.dat and other data files only.

Pitfalls

• Overall RMSE scale. The paper's Eq. (3) and the code's helper do not agree for multi-output models Code.py. To reproduce the tables, use the code's form; to match the paper's text, use the average. Check which one your metric uses before comparing numbers.
• Repeated runs are not independent for Keras. The 30 "runs" of LSTM, BD-LSTM, ED-LSTM and CNN continue training one network for 1000 epochs each. The spread across runs therefore does not measure initial-weight variance as the paper describes §4.2. Re-create the model inside the loop to get the paper's setup. To reproduce the published intervals, use the code's std and 1.96 rule (see Evaluation protocol).
• Seeds. No random seed is set in FNN/Code.py. Repeat runs will not match the published numbers exactly.
• Adam settings. The paper gives no learning rate, beta1, beta2 or epsilon §3.2 §4.1. The code leaves them to the Keras and scikit-learn defaults Code.py Code.py. Note the RNN uses a torch SGD learning rate of 0.01 Code.py.
• Equation (2) is wrong as printed. The cell-state update is wrapped in a sigmoid §3.4 eq. 2; the standard update is a sum without that outer sigmoid, and the paper's text gives no bias terms in the equations although it names b. The weight letters are also swapped against Eq. (1): input weights are W in Eq. (1) and U in Eq. (2) §3.3 §3.4. Implement the standard LSTM and check against the code.
• Table 2 ranks. Mackey-Glass CNN is ranked 1, tied with ED-LSTM. On the appendix test totals the order is ED-LSTM 0.0602, BD-LSTM 0.0765, CNN 0.0868, LSTM 0.0897, so CNN is third tab. 10 tab. 2. On Lazer, FNN-Adam (0.2537) is better than RNN (0.2580) on the test total, but the table ranks RNN 5th and FNN-Adam 6th tab. 7 tab. 2. The paper does not state the ranking basis, so these rows cannot be checked. Do not cite the rank table alone.
• Abstract wording. "BD-LSTM and ED-LSTM" hold on mean rank and on five of seven series, but LSTM is best on ACI-finance and Lazer tab. 5 tab. 7.
• Figure reference. The text cites Figure 7(b) for the ACI-finance comparison §5. By float order the ACI-finance figure is Figure 6 fig. 6.
• Literature numbers are not comparable. Several entries use a different embedding dimension, and the starred entries are not fair by the authors' own account §4.3.
• Unstated details to decide. Learning-rate schedule, batch size for the MLPs and the RNN, initial weights, the scaler's fit range, and the split mechanism are not in the paper. Each one must be chosen before a rerun.
• Cost not reported. Training time per run is not given in the paper.

Lineage

• Builds on the Elman network, the LSTM, bidirectional RNNs, sequence-to-sequence encoder-decoder LSTMs and Adam, as the paper cites them §2.2 §3.2-§3.6.
• Compared against published multi-step methods in the paper's literature tables: reinforced RTRL and echo state networks §2.1 §4.3, extended, unscented and particle filters §4.3, multi-output KELM §4.3, coevolutionary multi-task learning §2.1 §4.3, and SVR variants §4.3.
• Superseded by: not established from this wiki's content.