CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

In plain words

Forecasting a time series, such as electricity load or a weather reading, is usually learned end to end: one network is trained to predict the next values directly. This paper argues for a different order. First learn a compact description of the series, then fit a simple regressor on top of it.

The description has two parts. One captures the slow trend and the other the repeating seasonal pattern. Each part is trained with a contrastive objective, which teaches the network to treat two altered copies of the same series as alike and other series as different. The seasonal part does this in the frequency domain, so the period of the pattern does not have to be known in advance.

Once the features are learned, a ridge regression predicts the future values. The authors report that this beats the best earlier representation method and the end-to-end forecasters they compare with, with the largest gains on the multivariate benchmarks.

TL;DR

• CoST learns a per-timestep representation with a trend branch and a seasonal branch, each trained with a contrastive loss, then fits a ridge regressor on the frozen features §3 §4.1.
• Author-reported: average multivariate MSE 0.590 against 0.750 for TS2Vec (21.3% lower) and 0.972 for the best end-to-end baseline, TCN (39.3% lower) §4.2 tab. 1.
• The code is official (salesforce/CoST, BSD-3-Clause). Several details differ from the paper: projection head, number of trend experts, batch size, augmentations and the inputs of the frequency loss. See Paper vs code.

Problem

End-to-end forecasters learn features and prediction jointly from noisy observations. The paper argues this can pick up spurious correlations, and that the learned features transfer poorly when one component of the data-generating process shifts §1. It models a series as a sum of trend, seasonal and error terms. It argues that trend and seasonal representations are invariant to interventions on the error term, so they should support forecasting better §1 §2.

Two practical costs are named. Choosing an autoregressive lookback window by grid search is too expensive, so the trend branch mixes several windows §3.1. Determining the seasonal period in advance is avoided by a frequency-domain loss §1.

Method

The backbone maps the input series X (length h, m channels) to features V of shape h x d, with d = 320 §3 §4.1. The backbone is a temporal convolution network as in TS2Vec: a linear input layer to 64 channels, ten dilated residual blocks (kernel 3, dilation 2^i, GELU), and a final block to 320 channels app. E. Two feature disentanglers split the output into a trend part V^(T) and a seasonal part V^(S), each with d/2 = 160 channels. The two parts are concatenated at the end §3. The paper's equations are unnumbered in the source, so they are cited by section.

\mathcal{L} = \mathcal{L}_{\mathrm{time}} + \frac{\alpha}{2} \left( \mathcal{L}_{\mathrm{amp}} + \mathcal{L}_{\mathrm{phase}} \right) \quad [§3]

Trend branch (Trend Feature Disentangler). §3.1 fig. 3b

1. A mixture of L+1 causal 1-D convolutions. Expert i has kernel size 2^i and maps d channels to d_T. The paper sets L = floor(log2(h/2)) §3.1.
2. The trend representation is the average of the expert outputs §3.1: V^{(T)} = \frac{1}{L+1} \sum_{i=0}^{L} \mathrm{CausalConv}(\tilde V, 2^i) \quad [§3.1]
3. Time-domain loss: MoCo-style InfoNCE. A random time step t is chosen, a one-layer MLP projection gives the query, and the key comes from a momentum encoder. Negatives come from a queue, with temperature tau §3.1 §4.1. \mathcal{L}_{\mathrm{time}} = \sum_{i=1}^{N} -\log \frac{\exp(q_i \cdot k_i / \tau)} {\exp(q_i \cdot k_i / \tau) + \sum_{j=1}^{K} \exp(q_i \cdot k_j / \tau)} \quad [§3.1]

Seasonal branch (Seasonal Feature Disentangler). §3.2 fig. 3c

1. A discrete Fourier transform over time maps the features to F = floor(h/2) + 1 complex frequency bins §3.2 app. B.
2. A learnable Fourier layer applies a separate complex affine map to each frequency, with weights A in C^{F x d x d_S} and bias B in C^{F x d_S}. It is not translation invariant by design §3.2: V^{(S)}_{i,k} = \mathcal{F}^{-1}\!\left( \sum_{j=1}^{d} A_{i,j,k} \, \mathcal{F}(\tilde V)_{i,j} + B_{i,k} \right) \quad [§3.2]
3. The inverse DFT returns the seasonal representation to the time domain §3.2.
4. Frequency-domain loss. Each complex bin is split into amplitude and phase. For each bin, the augmented view of a sample is the positive, and the other N-1 samples in the batch are negatives. There is no temperature. Amplitude and phase losses are averaged over bins and samples §3.2: \mathcal{L}_{\mathrm{amp}} = \frac{1}{FN} \sum_{i=1}^{F} \sum_{j=1}^{N} -\log \frac{\exp(|F^{(j)}_{i}| \cdot |F'^{(j)}_{i}|)} {\exp(|F^{(j)}_{i}| \cdot |F'^{(j)}_{i}|) + \sum_{k \neq j}^{N} \exp(|F^{(j)}_{i}| \cdot |F^{(k)}_{i}|)} \quad [§3.2] \mathcal{L}_{\mathrm{phase}} has the same form with the phase phi(F) in place of |F| §3.2.

Augmentations. Scale, shift and jitter, each applied with probability 0.5 and with a Gaussian draw of scale 0.5. The paper treats them as interventions on the error term app. C §2.

Training. Self-supervised training uses the training split only. SGD, initial learning rate 1e-3, momentum 0.9, weight decay 1e-4, cosine annealing, batch size 256. The MoCo queue has 256 entries, momentum 0.999 and temperature 0.07. Training runs 200 iterations for datasets under 100,000 samples and 600 otherwise §4.1.

Evaluation. The encoder is frozen. A ridge regression maps the representation of the last time step to the whole forecast horizon. The ridge penalty is chosen on the validation split from 13 values. Inputs are zero-mean normalised, the split is 60/20/20, and MSE and MAE are reported on the test split §4.1.

The training step, following the paper and the code. Where the code departs from the paper, the code is used and the difference is listed under Paper vs code.

# One CoST training iteration  [§3] [app. C] [code: cost.py#L141-L172]
# Input:  series (B, T, m), crops of T = 201 in the scripts
# Output: scalar loss

def training_step(series, queue, encoder_q, encoder_k, alpha):
    # Two views, each with random scale, shift and jitter  [app. C]
    view_q = augment(series)
    view_k = augment(series)

    # Trend and seasonal features, each (B, T, 160)  [§3]
    trend_q, season_q = encoder_q(view_q)
    # Key trend features from the momentum encoder  [§3.1] [code: cost.py#L150]
    trend_k, _ = encoder_k(view_k)

    # Time-domain loss at one random step t  [§3.1] [code: cost.py#L143]
    t = random_step(T)
    query = normalize(head_q(trend_q[:, t]))
    key = normalize(head_k(trend_k[:, t]))
    loss_time = info_nce(query, key, queue, temperature=0.07)
    queue = enqueue(queue, key)

    # Seasonal key: second query-encoder pass  [code: cost.py#L162]
    _, season_k = encoder_q(view_k)
    # Frequency view of each seasonal output, (B, F, 160)  [§3.2]
    freq_q = rfft(normalize(season_q), dim=time)
    freq_k = rfft(normalize(season_k), dim=time)

    # Contrast amplitudes and phases; other batch items are negatives  [§3.2]
    loss_amp = contrast(abs(freq_q), abs(freq_k))
    loss_phase = contrast(phase(freq_q), phase(freq_k))

    # Weighted sum; alpha = 5e-4 in the experiments  [§3] [§4.3]
    return loss_time + (alpha / 2) * (loss_amp + loss_phase)

Inference. The encoder is run on every time step, with a causal window of 201 steps ending at that step, and the last step's trend and seasonal features are concatenated. The ridge regression then predicts the next pred_len values.

# Frozen-feature forecasting  [§4.1] [code: tasks/forecasting.py#L26-L33]
# Input:  standardised series (N, T, m), causal window 201
# Output: forecasts (N, T, pred_len * m)

def forecast(series, encoder, ridge, pred_len):
    # Left-pad with NaN to 201 steps  [code: models/encoder.py#L115-L116]
    for t in range(T):
        window = pad_nan_left(series[:, max(0, t - 200) : t + 1], 201)
        trend, season = encoder(window)
        # Last-step features, 320 dims in total  [§3] [code: cost.py#L334]
        features[:, t] = concat(trend[:, -1], season[:, -1])
    # Step t's feature predicts t+1 .. t+pred_len  [code: tasks/forecasting.py#L9]
    return ridge.predict(features)

Hyperparameters

Results

Author-reported. Setup: five public benchmarks in the multivariate and univariate settings. Representation learners are scored with a ridge regression on top, and end-to-end forecasters are trained with a supervised loss §4.1 §4.2. The tables give MSE / MAE for each cell. Bold marks the best value in the source and is not reproduced. Table numbers follow the source order of the LaTeX.

Headline numbers §4.2 tab. 1

• Multivariate average MSE / MAE: CoST 0.590 / 0.524, TS2Vec 0.750 / 0.607, TCN 0.972 / 0.666.
• Univariate average MSE / MAE: CoST 0.193 / 0.283, TS2Vec 0.203 / 0.298, TCN 0.236 / 0.367 tab. 7.

Table 1: multivariate forecasting, MSE / MAE tab. 1

The Informer and LogTrans values are copied from other papers, as noted in app. E. See Pitfalls.

Table 7: univariate forecasting, MSE / MAE tab. 7, appendix G §4.2

Other multivariate baselines, average MSE / MAE tab. 8 app. H: Triplet 0.969 / 0.732, CPC 0.880 / 0.663, TST 0.893 / 0.671, TCC 1.021 / 0.701. TSFresh (hand-crafted features) has per-dataset numbers in tab. 8 but no average. The per-dataset numbers for all of them are in tab. 8.

Table 2: sensitivity to alpha, ETT tab. 2 §4.3

Bold 5E-04 is the value used for all other experiments §4.3.

Table 3: component ablation, ETT averages tab. 3 §4.4

Abbreviations used in the table: TFD is the trend branch, MARE the mixture of autoregressive experts inside it, SFD the seasonal branch, LFL the learnable Fourier layer, and FDCL the frequency-domain loss.

Table 4: backbone encoders, ETT averages tab. 4 §4.4

Table 5: downstream regressors, ETT averages tab. 5 §4.4

Table 12: M5, multivariate, MSE / MAE tab. 12 app. J

Table 6: runtime in seconds, training and inference tab. 6 app. F. For representation methods, A + B is encoder time plus ridge time. The source repeats the CoST and TS2Vec training times for every horizon.

The paper attributes CoST's extra time over TS2Vec to running the trend experts one after another, and says parallel methods could speed this up app. F.

Paper vs code

The official repository is salesforce/CoST at commit 7bb5271c2b66cd5a1d634c27f5df308f190b50ec. The paper's Method and Implementation sections and the code differ in the following places. Each difference is listed with both citations.

• Projection head. The paper says the head is a one-layer MLP §3.1. The code builds two linear layers with a ReLU between them cost.py.
• Number of trend experts. The paper sets L = floor(log2(h/2)) §3.1. For h = 201 that gives kernels up to 64. The scripts use kernels 1 to 128, which is 8 experts ETT_CoST.sh train.py. The paper does not give the h used in its experiments.
• Batch size. The paper says 256 §4.1. The scripts pass 128 ETT_CoST.sh. The class default is 16 cost.py.
• Iteration rule. The paper says "fewer than 100,000 samples" §4.1. The code tests train_data.size, the total number of elements cost.py.
• Scale and shift augmentation. The paper defines scale as x * epsilon and shift as x + epsilon, each with a single random scalar app. C. The code draws one value per channel for both, and scales by 1 + epsilon cost.py.
• Frequency-loss input. The paper computes amplitude and phase on the pre-inverse-FFT complex output F §3.2. The code takes the FFT of the time-domain seasonal output, after L2 normalisation over channels and training-time dropout cost.py encoder.py.
• Frequency-loss negatives. The paper uses the other N-1 original samples as negatives §3.2. The code uses all other 2B-1 items, including the other augmented view cost.py.
• Seasonal key. The paper does not say which encoder produces the seasonal key. The code runs the query encoder a second time on the second view, with gradients cost.py.
• Random time step. The paper picks a random time step §3.1. The code draws one index per iteration, shared by the whole batch cost.py.
• Phase and amplitude numerics. The code adds 1e-6 to the real part before atan2, and to both parts in the amplitude cost.py. The paper's formulas have no epsilon.
• Seasonal dropout. The code applies dropout with probability 0.1 to the seasonal output during training encoder.py encoder.py. The paper does not mention it.
• Encoder masking. The encoder can apply a binomial input mask in training encoder.py. CoST's calls use the default mask='all_true', so the mask is not applied cost.py.
• Ridge model selection. The paper says the validation set picks the penalty §4.1. The code chooses by RMSE plus MAE on validation _eval_protocols.py. It also caps the ridge training set at 100,000 rows _eval_protocols.py.
• Evaluation windows. The paper uses the representation of the final step §3. The code evaluates every step on a causal 201-step window with NaN left-padding forecasting.py cost.py.
• Calendar covariates. The code appends seven calendar features as input channels for CSV datasets, and drops them from the targets at evaluation datautils.py forecasting.py. The paper does not mention them.
• Normalisation. The paper says "zero-mean normalized" §4.1. The code uses a StandardScaler fit on the training split datautils.py.
• Reported metric scale. The paper does not say whether its tables use the normalised or the raw scale. The code computes both forecasting.py.
• Seeds. The paper does not say whether it averages seeds §4.1. The scripts run seeds 0 to 4 ETT_CoST.sh.
• Data preparation. The Electricity, Weather and M5 data are not in the repository. They must be downloaded and prepared with the scripts described in the README.

Pitfalls

• Effective lookback is 201 steps, not h. At inference each step sees the 201 steps ending at it, with NaN padding on the left forecasting.py. The seasonal layer is built for 201 steps: its weights cover 101 frequency bins encoder.py. Longer inputs are only written into the first 101 bins, so the rest of the spectrum is zero encoder.py. Keep the window at 201 or re-derive the bin count.
• Frequency loss scale. The paper says the frequency loss is about three orders of magnitude larger than the time-domain loss. That is why alpha is small, and the paper says a small alpha does not mean seasonality matters less §4.3.
• Larger alpha helps some ETTh2 horizons. At ETTh2 horizons 168 and 336, the largest value tested (0.1) gives the lowest multivariate MSE. The paper uses 5E-04 for all other experiments tab. 2 §4.3.
• MoCo batch size must divide the queue. The code asserts K % batch_size == 0 with K = 256 cost.py. Batch sizes such as 100 fail.
• Numbers are author-reported and partly copied. TS2Vec, TNC, MoCo, Triplet, CPC, TST and TCC are reproduced by the authors. Informer and LogTrans are copied from TS2Vec and Informer app. E. TS2Vec was run with its default settings app. E. The settings used for the other reproduced baselines are listed in app. E.
• Table averages do not fully recompute. The paper gives a 4.71% univariate gain over the best feature-based baseline §4.2. The rounded averages in tab. 7 give about 4.9% (0.193 against 0.203). Use the paper's text values when comparing with it.
• Source and metadata differ. The LaTeX source is the ICLR 2022 camera-ready. The arXiv metadata's abstract omits the phrase "long sequence". The numbers in this card come from the source.
• Old NumPy calls. datautils.py uses np.float datautils.py, which NumPy 1.24 and later reject. Replace it with float before running.
• Runtime is A100-specific. Runtime numbers come from one A100 app. F. The paper gives no other hardware.

Lineage

• Builds on: MoCo for the time-domain loss §3.1; the TS2Vec backbone and evaluation protocol §4.1 app. E; structural time-series models for the trend and seasonal decomposition §2.
• Compared against: TS2Vec, TNC, MoCo, Triplet, CPC, TST, TCC, TSFresh for representation learning; Informer, LogTrans, LSTNet and TCN for end-to-end forecasting §4.2 app. E.
• Superseded by: not stated in the content on this wiki.