Time-Series Representation Learning via Temporal and Contextual Contrasting

In plain words

Sensor and brain recordings are cheap to collect but expensive to label, and most deep models need many labels. This paper asks whether a model can first learn useful features from unlabeled recordings, so that only a small classifier needs labels afterwards.

The core idea is to show the model two slightly different versions of the same recording and reward it for matching them. The two versions come from different kinds of distortion, one gentle and one stronger, so the model cannot just memorise a single version. A small attention-based model then reads the beginning of one version and has to guess the later part of the other. That forces it to learn how the signal develops over time. A second objective pulls together the summaries of the same recording and pushes apart the summaries of different recordings.

After this pretraining, a simple linear classifier on the frozen features is trained on the labels. The authors report that the result comes close to training the whole network with labels, and that it holds up when labels are scarce.

TL;DR

TS-TCC is an unsupervised pretraining method for time series. It builds two augmented views of each sample, encodes them with a three-block 1-D convolutional encoder, and trains a Transformer to predict future latent steps of one view from the past of the other (temporal contrasting). A SimCLR-style loss on the Transformer contexts adds a contextual contrasting term §3. The authors report linear-probe accuracy close to supervised training on HAR, Sleep-EDF and Epilepsy §5.1. Read the Pitfalls and Paper vs code sections before reimplementing: the public code differs from the paper in several augmentation and loss-scale details.

Problem

Time series rarely have labels that a person can read at a glance, so supervised deep learning is hard to apply §1. Contrastive methods from images do not transfer directly, for two reasons: they do not model temporal dependencies, and their image augmentations such as colour distortion do not suit signals §1. Earlier contrastive methods for time series were built for single applications such as EEG or ECG and did not generalise §1, §2.2.

The setting is an unlabeled pool of series with temporal dynamics, followed by a labeled downstream task that may have few labels or come from another domain §1, §5.2, §5.3.

Method

Overview §3: each raw sample is turned into a weakly and a strongly augmented view. An encoder maps each view to a sequence of latent steps. A temporal contrasting module uses an autoregressive Transformer to summarise one view up to a cut point and predicts the next K latent steps of the other view. A contextual contrasting module applies a contrastive loss to projected contexts from both views. The total loss is a weighted sum of the two terms eq. 6.

Step by step:

1. Augmentation §3.1. The weak view uses jitter and scaling: random variation is added and the magnitude is scaled up. The strong view splits the signal into a random number of segments (at most M), shuffles them, and then adds jitter. The paper notes that M should grow with sequence length and that the jitter should be smaller for normalised data §3.1.
2. Encoder §3.1 model.py. A three-block 1-D convolutional encoder, following the baseline in §3.1. Each block is a convolution, batch normalisation, ReLU and max-pooling. The first block has a dataset-specific kernel and stride; the second and third use kernel 8 and stride 1. The output is a latent sequence of T' steps with d channels. The channel widths (32, 64, then d) are given only in the code.
3. Temporal contrasting §3.2. The latent steps of a view are fed through a linear projection to width h, a learned token c is prepended, and L pre-norm Transformer layers run over the sequence. The output at token c is the context c_t eq. 3, eq. 4. For each k in 1..K, a linear map W_k sends c_t to the latent space, and the log-bilinear score against the true future latent is contrasted with the other samples in the batch eq. 1, eq. 2. Each context predicts the other view, so the two terms are the strong-to-weak and weak-to-strong losses.
4. Contextual contrasting §3.3. The contexts are passed through a non-linear projection head, as in SimCLR. With N samples there are 2N contexts; each one's positive is the context of the same sample in the other view, and the other 2N-2 contexts are negatives. Similarity is cosine with temperature tau eq. 5.
5. Total loss §3.3 eq. 6. The temporal terms are weighted by lambda1, the contextual term by lambda2.

# TS-TCC pretraining loss for one batch  [§3]
# Input:  x (B, C, T) raw unlabeled series
# Output: scalar loss

def pretrain_loss(x):
    # Two correlated views: weak and strong augmentation  [§3.1]
    x_weak = augment_weak(x)
    x_strong = augment_strong(x)
    # Latent sequences from the three-block encoder  [§3.1]
    z_weak = encoder(x_weak)                  # (B, d, T')
    z_strong = encoder(x_strong)              # (B, d, T')
    # Temporal contrasting, one term per direction  [§3.2] [eq. 1, eq. 2]
    loss_tc_s, context_s = temporal_loss(z_strong, z_weak)
    loss_tc_w, context_w = temporal_loss(z_weak, z_strong)
    # Projected contexts for the contextual term  [§3.3]
    proj_s = projection_head(context_s)       # (B, d')
    proj_w = projection_head(context_w)       # (B, d')
    # NT-Xent over the 2N contexts of the batch  [§3.3] [eq. 5]
    loss_cc = contextual_loss(proj_s, proj_w, tau)
    # Weighted total; the paper and code both use lambda1 = 1, lambda2 = 0.7
    # [§4.2] [eq. 6]
    return lambda1 * (loss_tc_s + loss_tc_w) + lambda2 * loss_cc

# Temporal contrasting: the context of one view predicts the future of
# the other  [§3.2]
# Input:  z_context (B, d, T'), z_target (B, d, T')
# Output: scalar loss, context c_t (B, h)

def temporal_loss(z_context, z_target):
    # Random cut point t, drawn once per call  [code: models/TC.py#L35]
    t = sample_cut_point(T' - K)
    # Transformer summary of the context up to t  [§3.2] [eq. 3, eq. 4]
    context = autoregressive_transformer(z_context[:, : t + 1])   # (B, h)
    loss = 0
    for k in 1 .. K:
        # Predict the latent k steps ahead from the context  [§3.2]
        prediction = W[k](context)                               # (B, d)
        # Dot product with every sample's true future in the batch  [eq. 1]
        scores = prediction @ z_target[:, t + k].T               # (B, B)
        log_probs = log_softmax(scores, axis=1)
        # The diagonal is the positive pair; average over K  [eq. 1]
        loss = loss - mean(diagonal(log_probs)) / K
    return loss, context

# Contextual contrasting over 2N projected contexts  [§3.3]
# Input:  proj_a (N, d'), proj_b (N, d'), temperature tau
# Output: scalar loss

def contextual_loss(proj_a, proj_b, tau):
    # All 2N contexts; the positive of i is its twin in the other view  [§3.3]
    contexts = concat(proj_a, proj_b)                            # (2N, d')
    # Cosine similarity scaled by temperature  [§3.3] [eq. 5]
    sims = cosine_matrix(contexts) / tau                         # (2N, 2N)
    # Exclude each context's similarity with itself  [eq. 5]
    sims = mask_diagonal(sims)
    # Negative log-likelihood of the positive among the other 2N-1  [eq. 5]
    per_anchor = -log_softmax(sims, axis=1)[i, twin(i)]          # (2N,)
    # Eq. 5 sums over N anchors; the code averages over 2N  [see Paper vs code]
    return mean(per_anchor)

Key equations, as printed in the paper (numbering follows the LaTeX order):

Temporal contrasting, strong context predicting the weak view:

\mathcal{L}_{TC}^s=-\frac{1}{K} \sum_{k=1}^{K} \log \frac{\exp ((\mathcal{W}_k (c_t^s))^T z_{t+k}^{w})}{\sum_{n \in \mathcal{N}_{t, k}} \exp ((\mathcal{W}_k (c_t^s))^T z_{n}^w)} eq. 1

The weak-context, strong-target term is the same with s and w swapped eq. 2:

\mathcal{L}_{TC}^w=-\frac{1}{K} \sum_{k=1}^{K} \log \frac{\exp ((\mathcal{W}_k (c_t^w))^T z_{t+k}^{s})}{\sum_{n \in \mathcal{N}_{t, k}} \exp ((\mathcal{W}_k (c_t^w))^T z_{n}^s)} eq. 2

Transformer layers, with pre-norm residual blocks eq. 3, eq. 4:

\tilde{\psi_\ell} = \operatorname{MHA}(\operatorname{Norm}(\psi_{\ell-1})) + \psi_{\ell-1}, \quad 1 \leq \ell \leq L eq. 3

\psi_\ell = \operatorname{MLP}(\operatorname{Norm}(\tilde{\psi_\ell})) + \tilde{\psi_\ell}, \quad 1 \leq \ell \leq L eq. 4

Contextual contrasting eq. 5:

\mathcal{L}_{CC}= -\sum_{i=1}^{N} \log \frac{\exp \left(\operatorname{sim}\left(\boldsymbol{c}_t^{i}, \boldsymbol{c}_t^{i^+}\right) / \tau\right)}{\sum_{m=1}^{2 N} \mathbb{1}_{[m \neq i]} \exp \left(\operatorname{sim}\left(\boldsymbol{c}_t^{i}, \boldsymbol{c}_t^{m}\right) / \tau\right)} eq. 5

Total loss eq. 6:

\mathcal{L} = \lambda_1 \cdot (\mathcal{L}_{TC}^s + \mathcal{L}_{TC}^w ) + \lambda_2 \cdot \mathcal{L}_{CC} eq. 6

Downstream use. The linear and fine-tuning evaluations use the encoder output, flattened over channels and time, followed by one linear layer. The Transformer and projection head are not used downstream model.py model.py. The paper describes the downstream classifier only as a single MLP layer on the frozen encoder §5.1.

Hyperparameters

Values are the paper's unless the Where column says code. Dataset-specific values are given per dataset.

Results

Author-reported. Setup: four datasets tab. 1 with a linear classifier, a semi-supervised fine-tuning run and a transfer experiment §5. The baselines are random initialisation, supervised training, SSL-ECG, CPC and SimCLR, with time-series augmentations used to adapt SimCLR §5.1. Metrics are accuracy (ACC) and macro F1 (MF1) over 5 seeds §4.2.

Datasets tab. 1 §4.1:

Linear evaluation, frozen encoder tab. 2 §5.1:

Headline claims in the text: TS-TCC is best on two of three datasets, close to supervised training on the third, and better than CPC, SimCLR and SSL-ECG overall §5.1. Contrastive methods beat the pretext-based SSL-ECG, and CPC beats SimCLR, which the authors read as temporal features mattering more than generic ones §5.1.

Semi-supervised training §5.2 fig. 3. Fine-tuning uses 1%, 5%, 10%, 50% and 75% of the training labels. The text reports about 70% MF1 on HAR and about 90% on Epilepsy with 1% of labels, and says that 10% of labels gives performance comparable to supervised training on 100% of labels on all three datasets. These numbers appear only in the figure and text; no table is given.

Transfer learning, fault diagnosis, accuracy tab. 3 §5.3:

Ablation, linear evaluation tab. 4 §5.4:

The text reads the ablation as: the cross-view task adds more than 5 points on HAR and about 1 point on Sleep-EDF and Epilepsy; the contextual term adds further gains; using one augmentation type alone hurts HAR and Sleep-EDF but not Epilepsy §5.4.

Sensitivity, HAR only §5.5 fig. 4. Predicting 40% of the feature length performs best (panel a). lambda2 near 1 works, with 0.7 best (panel c). With lambda2 = 0.7, lambda1 = 1 is best and the model is insensitive to lambda1 below 10 (panel b).

Paper vs code

Checked against the official repository at code_commit (github.com/emadeldeen24/TS-TCC, MIT, owned by the first author). The code was read, not executed.

• Weak augmentation. The paper describes it as jitter and scaling §3.1. The code applies scaling only augmentations.py.
• Scaling factor. The paper sets the scaling ratio to 2 §4.2. The code draws the factor from N(2, sigma), with the mean fixed at 2 and sigma taken from the config augmentations.py. The configured values are HAR 1.1, Sleep-EDF 1.5, Epilepsy 0.001 and FD 2 [code: config_files/HAR_Configs.py#L32] sleepEDF_Configs.py Epilepsy_Configs.py pFD_Configs.py.
• Max segments M. The paper gives M = 10 for HAR, 12 for Epilepsy and 20 for Sleep-EDF §4.2. The code uses 8 for HAR [code: config_files/HAR_Configs.py#L34], 5 for Epilepsy [code: config_files/Epilepsy_Configs.py#L35], 12 for Sleep-EDF [code: config_files/sleepEDF_Configs.py#L36] and 5 for FD [code: config_files/pFD_Configs.py#L39].
• Future steps K. The paper sets K to 40% of d §5.5. The code uses fixed values: HAR 6 HAR_Configs.py (40% of 18 is 7.2), Epilepsy 10 Epilepsy_Configs.py (40% of 24 is 9.6), Sleep-EDF 50 sleepEDF_Configs.py (40% of 127 is 50.8), FD 50 pFD_Configs.py (40% of 162 is 64.8).
• Batch size. The paper uses 128 §4.2. The FD config uses 64 [code: config_files/pFD_Configs.py#L20]. The other configs use 128.
• Hidden width. Matches the paper for all three datasets it names: HAR 100, Epilepsy 100, Sleep-EDF 64 HAR_Configs.py Epilepsy_Configs.py sleepEDF_Configs.py.
• Temporal loss normalisation. Equations 1 and 2 have a 1/K factor and no batch average eq. 1, eq. 2. The code sums over the batch and over K, then divides by batch size times K TC.py.
• Contextual loss normalisation. Equation 5 sums over i = 1..N eq. 5. The code uses cross-entropy with sum reduction over 2N anchors and divides by 2N loss.py loss.py. Taken literally, the paper's term is about N times the code's, so lambda2 = 0.7 is tuned to the code's scale.
• Latent normalisation. The code L2-normalises the encoder output over the channel axis before the temporal module trainer.py. The paper does not mention this step.
• Direction naming. The code's first temporal call uses the weak view as the context and the strong view as the target trainer.py. The paper's L_TC^s is the reverse. Both terms are summed, so the total is the same.
• Augmentation sampling. The code generates both augmented views once per dataset, before training dataloader.py. The paper does not say when augmentations are drawn.
• Projection head. Matches "non-linear, as in SimCLR" §3.3 in shape only; the code's widths are Linear(h, d/2), BatchNorm, ReLU, Linear(d/2, d/4) [code: models/TC.py#L17-L22].
• Encoder dropout. 0.35 in every config HAR_Configs.py. The paper gives 0.1 for the Transformer only, which matches the code's Transformer default attention.py.
• Learning-rate schedule. The code applies ReduceLROnPlateau on the validation loss in every downstream mode trainer.py trainer.py. The paper does not mention a schedule.
• Downstream optimiser. The same Adam settings and 40 epochs are used for downstream modes as for pretraining main.py.
• Not in the repository. The few-label subsets (1% to 75%), the 12 cross-domain fault-diagnosis runs and the 5-seed aggregation are not scripted in the repository. The file list has no script for them.

Pitfalls

• Scale factor shape. The factor is drawn with shape (batch, time) and the same factor is multiplied into every channel [code: dataloader/augmentations.py#L20-L24]. The scaling is therefore a random gain per time step, not a per-series constant, and the paper does not say which. Read from the code: a Normal(2, sigma) draw is negative with probability Phi(-2/sigma). That is about 9% for Sleep-EDF (sigma 1.5), 3% for HAR (sigma 1.1) and 16% for FD (sigma 2), computed analytically from the configured values, not by running the code. A negative factor flips the sign of that time step.
• Strong permutation may copy channel 0. The permutation assigns pat[0, warp], a one-channel ordering, into a (C, T) array [code: dataloader/augmentations.py#L43]. Read from the code and not executed here: for the 9-channel HAR data, any sample with more than one segment would then have identical channels before jitter. Check this before reusing the strong view for multichannel data.
• Segment count. np.random.randint(1, max_segments) never returns max_segments, so the number of segments is 1 to M-1 [code: dataloader/augmentations.py#L31]. The paper's "at most M" is therefore one segment fewer than its maximum.
• Fixed views. The views are sampled once for the whole run [code: dataloader/dataloader.py#L32-L33]. With 40 epochs, each sample is seen with the same two views every epoch. Re-sampling per epoch is a different method.
• Attention scale. The attention scale is dim^-0.5 where dim is the full model width h, not the per-head width attention.py. With h = 100 and 4 heads, the code's scale is 0.1, and the per-head scale would be 0.2. Copying a standard "1/sqrt(head_dim)" formula changes the model.
• No position information. The Transformer has no positional embedding and no attention mask attention.py. The context token attends to all latent steps up to t, in both directions.
• BatchNorm and dropout during linear training. Parameters are frozen with requires_grad main.py, but the trainer puts the model in train mode for every downstream epoch trainer.py. Batch-norm running statistics and dropout stay active. The paper describes a frozen encoder §5.1. A faithful reimplementation must decide which one to follow.
• Downstream features. The linear and fine-tuning models use the flattened encoder output, not the Transformer context model.py model.py. A reader following the method section might use c_t instead.
• Claim check, transfer. The text says TS-TCC improves by at least 7 points in 7 of its 8 winning scenarios, except D→B §5.3. Table 3 does not match: of the seven non-D→B wins, A→C (+6.56) and B→C (+6.71) fall below 7 points, so only five of seven exceed it tab. 3. The average gain of about 4 points (67.82 against 63.83) does match.
• Notation for d. §3.1 uses d for the channel width of z_i. §5.5 uses d for the number of latent steps (the feature length). The code uses 128 channels and T' steps HAR_Configs.py [code: config_files/HAR_Configs.py#L11].
• Equation reference. §5.5 cites "Eq. eqn:lcc" for lambda1 and lambda2, but the weights appear in the total loss, eq. 6 §5.5 eq. 6.
• Equation numbers. Numbers are taken from the LaTeX order (1 and 2 the temporal terms, 3 and 4 the Transformer layers, 5 the contextual term, 6 the total). Check against the compiled PDF before citing externally.
• Costs the abstract hides. Each step runs the temporal module twice, once per direction trainer.py, and the contextual loss builds a 2N x 2N similarity matrix loss.py. Both grow with batch size.
• Gaps in the method. Not stated: the strong jitter sigma, the Transformer MLP width, the encoder dropout, the projection head widths, and the few-label batch size of 32 (code has no config for it).

Lineage

• Builds on: SimCLR's projection head and NT-Xent-style loss for the contextual term §3.3; CPC's prediction of future latents with a log-bilinear model §2.2, §3.2, arXiv:1807.03748 (no card in this wiki); BERT's learned classification token, adapted as the context token §3.2.
• Compared against: random initialisation, supervised training, SSL-ECG, CPC and SimCLR §5.1.
• Extended by: the repository README points to CA-TCC, a semi-supervised extension, arXiv:2208.06616 README.md.