UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting

In plain words

Most forecasting models are built for one kind of data, such as electricity use or weather, and are trained on that data alone. UniTime trains one model on many kinds of time series at once. Their lengths, the number of measured channels and the sampling rates may all differ.

Two things help it tell the sources apart. Each input starts with a short sentence written by a person that says what the data is. The series itself is cut into small chunks, and parts of it are hidden during training so that the model cannot simply memorise one dataset. The chunks and the sentence go into a pretrained language model, which is used as the core of the network.

The authors report that one such model beats models trained on a single dataset in many cases. They also report reasonable forecasts on three datasets that the model never trained on, when it reuses the sentence of a training domain.

TL;DR

Use this card if you need one forecaster for several time-series sources, or want to see how a pretrained language model can be reused for numeric sequences. The model is a GPT-2 backbone with a domain instruction prepended to patch tokens, trained with masked inputs and an extra reconstruction term. The code is Apache-2.0 and available. Numbers are author-reported and the training code differs from the paper in several places (see Paper vs code).

Problem

Dedicated forecasters need one model per dataset §1. The authors name three obstacles to a single cross-domain model §1:

• Varying data characteristics. Domains differ in channel count, history length and horizon length, and many architectures fix the channel count §1.
• Domain confusion. Mixed training on domains with different temporal patterns degrades accuracy §1, §4.2.
• Domain convergence-speed imbalance. Simple domains converge early and overfit, while others still need more steps §1, §4.1.

The paper says that channel-mixing designs cannot share one encoder across domains with different semantics, which motivates channel independence §3.

Method

Notation: a multivariate series from domain τ has C channels, lookback L and forecast length T §3. The method treats each channel as a separate univariate series (channel independence) and processes it with the shared network §3, §4.

1. Instance normalisation. For each channel, compute the mean and standard deviation over observed (unmasked) values, subtract and divide. These statistics are not back-propagated through. The paper calls this series stationarisation and does not give the formula §4.1. The code is in unitime.py.
2. Padding and patching. Pad the end of each series by repeating its last value, so that the windows fit exactly. Cut windows of length P = 16 with stride S §4.1. The count is N = ceil((L − P) / S) + 1 §4.1. The padding and unfolding follow unitime.py.
3. Gated fusion with the mask. Hidden values are set to zero in the input. The mask is also cut into patches, and it is embedded by its own linear layer. The gate is a sigmoid of two linear maps of the value and mask embeddings. The output mixes the two embeddings with the gate weights eq. 1 eq. 2. The paper does not add a projection after the mix. The code does §4.1 unitime.py.
4. Instruction prefix. The domain instruction (a plain sentence per domain, listed in instruct.json) is embedded with the language model's token embedding. It is placed before the patch tokens §4.2.
5. Language-TS Transformer. Run the first six GPT-2 blocks over the instruction tokens followed by the patch tokens, with causal attention and all weights trained §4.2 §5.1. The block equations are eq. 3 and eq. 4; the causal attention is eq. 5, with its mask in eq. 6. The code keeps the first six blocks of the pretrained model [code: models/unitime.py#L33-L35].
6. Padding to a fixed length. Append copies of one learned pad vector until each sequence has R = 17 tokens §4.3. The total number of instruction and patch tokens must not exceed R unitime.py.
7. Light decoder. Run a two-layer transformer over the R tokens, with L_light = 2 §4.3 §5.1. The paper gives the padding and the light transformer in eq. 7, and the linear head in eq. 8.
8. Linear head. Flatten each channel's R tokens into one vector and map it to a fixed number of output steps with one linear layer. The paper states the output length as the maximum forecast length O, set to 720 §4.3 §5.1. The code outputs 96 backcast steps plus O steps [code: models/unitime.py#L78-L80, #L148-L158]. The output is then undone from step 1.
9. Training objective. The paper uses the mean squared error for the forecast, plus a reconstruction of the past window, each averaged over its own length eq. 9 §4.4. The mask hides about half of each series during training. The stated reason is that masking stops the model from fitting trivial, dataset-specific patterns on domains that overfit §4.1. The reconstruction target is the original unmasked window [code: engines/engine_forecasting.py#L25-L29].
10. Batches from one domain. Each batch holds samples from a single domain, because channel counts and lengths differ. Domains are oversampled when they have few samples §4.4. Illness is the oversampled domain in the paper app. B.

# UniTime: one training step on a batch from one domain  [§4.4] [eq. 9]
# Inputs:  x (B, L, C) lookback, y (B, T, C) future, text (instruction)
# Output:  scalar loss

def training_step(x, y, text):
    # Hide each value with probability r_m = 0.5, independently per step
    # and per channel; mask value 0 means hidden  [§4.1]
    # [code: engines/engine_forecasting.py#L25-L29]
    mask = random_uniform(B, L, C) >= 0.5
    masked_x = x * mask
    # Predict the backcast and the forecast from the masked input  [§4.3]
    # [code: engines/engine_forecasting.py#L31-L43]
    out = model(masked_x, mask, text)            # (B, 96 + O, C)
    pred = out[:, 96 - L : 96 + T]               # (B, L + T, C)
    target = concatenate(x, y, axis=time)        # (B, L + T, C)
    # Plain mean over all L + T steps; not the 1/T and 1/L split of [eq. 9]
    # [code: engines/engine_forecasting.py#L42-L45]
    loss = mean((pred - target) ** 2)
    return loss

# UniTime forward pass  [§4.1] [§4.2] [§4.3]
# Inputs:  x (B, L, C) series, mask (B, L, C) with 1 = observed, text
# Output:  (B, 96 + O, C) predictions in the original scale

def forward(x, mask, text):
    # Per-channel statistics from observed values only; detached  [§4.1]
    # [code: models/unitime.py#L117-L124]
    mean = sum(x * mask, axis=time) / sum(mask, axis=time)
    x = (x - mean) * mask
    std = sqrt(sum(x ** 2, axis=time) / sum(mask, axis=time) + 1e-5)
    x = x / std
    # Repeat the last value to pad, then cut patches of length 16  [§4.1]
    # [code: models/unitime.py#L84-L97]
    x = replicate_pad_end(x, stride=S)
    mask = replicate_pad_end(mask, stride=S)
    values = unfold(x, size=16, step=S)          # (B*C, N, 16)
    indicators = unfold(mask, size=16, step=S)   # (B*C, N, 16)
    # Gated fusion of value and indicator embeddings  [eq. 1] [eq. 2]
    # [code: models/unitime.py#L103-L109]
    value_emb = linear_value(values)             # (B*C, N, D)
    indicator_emb = linear_indicator(indicators) # (B*C, N, D)
    gate = sigmoid(W_g1(value_emb) + W_g2(indicator_emb) + b_g)
    tokens = gate * value_emb + (1 - gate) * indicator_emb
    tokens = linear_after_gate(tokens)           # in code only
    # Instruction tokens first, then patches; first 6 GPT-2 blocks  [§4.2]
    # [code: models/unitime.py#L130-L137]
    prefix = gpt2_token_embedding(tokenize(text))  # (I, D)
    hidden = gpt2_blocks_causal(concat(prefix, tokens))
    # Pad to R = 17 tokens with one learned vector  [§4.3]
    # [code: models/unitime.py#L139-L143]
    hidden = append_copies(hidden, pad_vector, up_to=17)
    # Two-layer light transformer over the R tokens  [§4.3]
    hidden = light_transformer(hidden)           # (B*C, R, D)
    # Map each channel's flattened tokens to the output steps  [eq. 8]
    # [code: models/unitime.py#L148-L154]
    flat = reshape(hidden, (B, C, 17 * D))
    out = transpose(linear_head(flat))           # (B, 96 + O, C)
    # Undo the normalisation  [code: models/unitime.py#L156-L157]
    return out * std + mean

Key equations, as written in the paper (numbering as compiled):

\boldsymbol{Z} = Gate \odot \boldsymbol{Z} + (1-Gate) \odot \boldsymbol{M} eq. 1

Gate = \sigma(\boldsymbol{Z} \boldsymbol{W}_{g1} + \boldsymbol{M} \boldsymbol{W}_{g2} + \boldsymbol{b}_g) eq. 2

\mathcal{L}_{\tau} = \frac{1}{c_\tau} \sum_{i=1}^{c_\tau} \left( \frac{1}{T_\tau}\|\hat{\boldsymbol{x}}^{T_\tau}_{\tau,i} - \boldsymbol{x}^{T_\tau}_{\tau,i}\|_2^2 + \frac{1}{L_\tau}\|\hat{\boldsymbol{x}}^{L_\tau}_{\tau,i} - \boldsymbol{x}^{L_\tau}_{\tau,i}\|_2^2 \right) eq. 9

Equations 3 to 8 (layer equations, causal attention, light transformer padding and linear head) are in §4.2 and §4.3; the code for them is cited in the steps above.

Hyperparameters

Results

Author-reported. Setup: one UniTime model trained across the datasets (the training list includes Illness) and evaluated per dataset and horizon. Baselines are GPT4TS and PatchTST modified to train across datasets (†), GPT4TS and PatchTST with official code, input length and epochs reset, trained per dataset (*), and TimesNet, DLinear, NSformer, FEDformer, Autoformer and Informer trained per dataset, with numbers taken from the TimesNet paper §5.1 tab. 2. Mean squared error (MSE) and mean absolute error (MAE) are reported, each cell as MSE / MAE. Average over the horizons.

UniTime per horizon tab. 2:

Illness uses horizons 24, 36, 48 and 60: 2.460 / 0.954, 1.998 / 0.912, 1.979 / 0.912 and 2.109 / 0.938; average 2.137 / 0.929 tab. 2.

Average over horizons, UniTime against the baselines tab. 2:

The table omits DLinear, NSformer, FEDformer, Autoformer and Informer, which are in tab. 2. Author claims: UniTime is best among models trained across datasets in 79 of 80 entries, and best overall in 37 of 80 entries §5.2. The table shows that a per-dataset TimesNet has the lowest Electricity average.

Zero-shot, average over horizons, for models trained on ETTh1, ETTm1 and ETTm2 only tab. 5 §5.4:

"Repeat" repeats the last observed value §5.4. On the Weather average, Repeat ties UniTime on MAE (0.320) and has a higher MSE.

Ablation, average over horizons, each cell MSE / MAE tab. 3:

Language-model design choices, same format tab. 4:

The paper also reports that T5 is not better than GPT-2 app. D, tab. 9, and that rephrased instructions change results only slightly [app. C, tab. 7 and tab. 8]. Figures: fig. 3 shows validation loss curves with and without masking, and fig. 4 a t-SNE of hidden states, with and without instructions. Both are described in the text §5.3.

Paper vs code

Code: official repository https://github.com/liuxu77/UniTime, commit 09acfbe9c63fc67db7539590094ac5f328230654, Apache-2.0 [LICENSE]. The paper links it in a footnote on the corresponding author. The repository has no dataset files; the README points to an external archive that we did not download or check.

• Loss weighting. Paper: the forecast and the reconstruction each have their own length-normalised term, weighted equally eq. 9 §4.4. Code: one mean squared error over the concatenated L + T steps [code: engines/engine_forecasting.py#L39-L45]. For L = 96 and T = 720, reconstruction therefore gets 96 / 816, about 12% of the weight, and the forecast gets about 88%. Under eq. 9 the two terms would have equal weight.
• Projection after the gate. Paper: the gate output is the token eq. 1. Code applies one more linear layer to the gated output [code: models/unitime.py#L107-L109].
• Output length. Paper: the linear layer outputs the maximum forecast length O §4.3. Code: it outputs 96 backcast steps plus O [code: models/unitime.py#L78-L80] run.py. The backcast region is aligned to the end, so training slices [96 − L : 96 + T] [code: engines/engine_forecasting.py#L39-L41].
• Decoder layers. Paper: two light transformer layers §4.3 §5.1. Code: PyTorch encoder layers with feed-forward width 4D, pre-norm, the PyTorch default ReLU activation, no positional encoding and no final normalisation [code: models/unitime.py#L69-L76]. None of this is in the paper.
• Masking pattern. Paper: a binary mask with ratio r_m §4.1; it does not say whether hidden steps are contiguous. Code: each step and channel is hidden independently with probability 0.5 engine_forecasting.py.
• Domain sampling. Paper: batches come from one domain, chosen at random, and small domains are oversampled §4.4. Code: a training epoch visits every batch of every domain once in a random order. A domain's share of steps is therefore its batch count, not a uniform share engine.py. Illness is listed 13 times in the default training list [code: execute_list/train_all.csv#L9-L21], against 12 in the paper's table app. B.
• Validation. Paper: validation loss is computed for all datasets involved §5.1. Code: the default list validates five datasets at horizon 720 only (ETTh1, ETTm1, ETTm2, Electricity, Weather). ETTh2, Exchange and Illness have no validation loader train_all.csv.
• Model selection. Paper: keep the model with the lowest average validation loss §5.1. Code: keep a checkpoint when the sum over validation sets of the relative improvement over the stored best is at least zero. The stored best starts at 5.0 per set engine.py, [code: engines/engine.py#L152-L159]. A checkpoint can therefore be saved even if the average loss rose.
• Learning-rate schedule. Not in the paper. Code: cosine annealing with T_max = 20 and eta_min = 1e-6, stepped once per epoch engine.py, engine.py.
• Zero-shot protocol. Paper: choose the instruction on 0.5% of test samples, using the loss on the second part of the lookback §5.4. Code: no such selection. The instruction is passed as a string flag run.py [code: engines/engine.py#L83-L84]. No zero-shot training list is in the repository.
• Dataset normalisation. Not in the paper. Code: StandardScaler fitted on each dataset's training part data_loader.py.

No further discrepancies found in the patch, token and attention code.

Pitfalls

• Loss weights. The paper's loss gives the reconstruction term equal weight; the code gives it L / (L + T) of the weight. Decide which one you implement and label it. We did not rerun the code, so we cannot say which one the reported numbers match.
• Slicing. The output always has 96 backcast steps, right-aligned to the forecast. For Illness (L = 36) training uses output steps 60 to 156 at T = 60. An implementation that uses only O will misalign the reconstruction target.
• Normalisation chain. Mask, then compute mean and standard deviation on observed values, then zero the hidden values, then divide. The statistics are detached and the variance has 1e-5 inside the square root [code: models/unitime.py#L117-L124]. The scaler is fitted on training data before this, and the metrics are computed on scaled values [code: engines/engine_forecasting.py#L95-L104]. MSE values are therefore not in original units.
• Token budget. R = 17 must hold the instruction tokens plus every patch. The code stops the process with exit(0) if they do not fit [code: models/unitime.py#L144-L146]. Long instructions for datasets with many patches are the usual cause.
• Two meanings of stride. Patch stride S (16 or 4) is not the window step. The window step is 1 for training, validation and testing [code: data_provider/data_loader.py#L54-L63].
• Compute is constant. The model always produces 96 + O steps, so a short horizon costs as much as the longest one. Truncate after the forward pass.
• Instruction order. The instruction must come first, so that patch tokens can attend to it under causal attention §4.2 §5.5. The paper reports a small gap when the order is reversed, which it attributes to the decoder reading the instruction tokens §5.5.
• Zero-shot is not reproducible from the repository. See Paper vs code.
• Memory. Full fine-tuning of six GPT-2 blocks with a 720-step output on one 80 GB GPU. The paper lowered the Electricity batch to 24 for memory app. B.
• Versions. Python 3.9, PyTorch 1.13.1 with CUDA 11.6 and transformers 4.31.0 are the versions the authors used README.md.
• Author-reported only. Single-dataset baselines are numbers from the TimesNet paper, except where starred tab. 2.

Lineage

• Builds on: GPT-2 (pretrained backbone, §4.2); patching and channel independence from PatchTST §3; masking and reconstruction for training §4.4; the TimesNet experimental protocol §5.1. None of these has a paper card.
• Compared against: GPT4TS, PatchTST, TimesNet, DLinear, NSformer, FEDformer, Autoformer and Informer §5.1.
• Superseded by: not known from content in this wiki.