Multi-task Self-Supervised Learning for Human Activity Detection

In plain words

Phones and watches record motion all day, but almost none of that data has activity labels, and labelling it by hand is slow and raises privacy concerns. This paper asks whether a model can learn useful features from the raw motion signal without anyone labelling it.

The trick is to create its own quiz. Take an unlabelled motion recording, make a changed copy (add noise, flip the signs, reverse it in time, shuffle the axes, and so on), and train a network to say which changes were applied. To answer, the network has to learn what normal movement looks like. Afterwards, its early layers are reused, and a small classifier is trained on a few labelled examples to name the activity.

The authors report that these features classify activities almost as well as a network trained with full labels, and clearly better than features from an autoencoder. They tested this on six public smartphone datasets. The numbers come from their own experiments, so the same setup may not transfer to other sensors or tasks.

TL;DR

• Self-supervised pre-training for accelerometer-based human activity recognition (HAR): a multi-task 1-D CNN predicts which of eight signal transforms were applied to a window §3.1 §3.2.
• Reused conv features give kappa 0.76 to 0.88 with a frozen trunk and 0.76 to 0.89 with the last conv layer fine-tuned, across six datasets tab. 2.
• Frozen features beat autoencoder features on all six datasets, but trail supervised training on three of them (UniMiB, WISDM, MotionSense) tab. 2.
• No official code was found. The protocol has several gaps: transform parameters, task weights, batch size and seeds are not stated §3.2 alg. 1.

Problem

Activity classifiers need large labelled datasets. In smartphone sensing, collecting and labelling such data is costly and privacy-sensitive, so a model that can start from unlabelled data would help §1. Autoencoders are the standard unsupervised baseline here, and the authors argue that a reconstruction goal gives a weak signal and can learn features that do not help activity recognition §3.1. The paper assumes the window length, the sensor axes and the signal transforms are chosen by the practitioner; it does not study how sensitive results are to these choices §4.2.

Method

Step 1. Build the pretext data §3.1 alg. 1. Each unlabelled window x has shape (N, C) with N = 400 samples and C channels (see Pitfalls for C) §4.2. For each transform J_t in the set T (|T| = 8) §3.2, the window x is added to task t's dataset with label False, and its transformed copy J_t(x) with label True. Every window therefore contributes two items to each of the 8 task datasets alg. 1. The transforms and their parameters are listed in the table below.

Step 2. The network (TPN, Transformation Prediction Network) §3.1 §3.3 fig. 3. One trunk of three 1-D convolutional blocks, each followed by ReLU and dropout, then global max pooling to a 96-dimensional vector §3.3. Eight task heads sit on the trunk, one per transform. Each head is a dense layer with 256 units, then a single sigmoid output §3.3. The trunk is shared (hard parameter sharing) §3.3.

# Shared trunk: three Conv1D blocks  [§3.3] [fig. 4]
def trunk(window):                          # window: (B, N, C)
    # ConvA: 32 maps, kernel 24, stride 1  [§3.3]
    h = dropout(relu(conv1d(window, 32, 24)), rate=0.1)
    # ConvB: 64 maps, kernel 16, stride 1  [§3.3]
    h = dropout(relu(conv1d(h, 64, 16)), rate=0.1)
    # ConvC: 96 maps, kernel 8, stride 1  [§3.3]
    h = dropout(relu(conv1d(h, 96, 8)), rate=0.1)
    # Global max pooling over time  [§3.3]
    return global_max_pool(h)               # (B, 96)

# One task head per transform  [§3.3]
def pretext_head(features):                 # features: (B, 96)
    # Dense layer with 256 units and ReLU  [§3.3]
    hidden = relu(dense(features, 256))
    # Sigmoid: probability that the window is transformed  [§3.1]
    return sigmoid(dense(hidden, 1))        # (B,)

Step 3. The loss eq. 2 §3.1 alg. 1. For each task, binary cross-entropy averaged over that task's m_t examples eq. 2. The task losses are summed with weights psi_t eq. 2 eq. 1. The algorithm also adds an L2 penalty beta * ||theta||^2 alg. 1.

Step 4. Pre-training §3.3 alg. 1. Mini-batches are drawn from all tasks at once, and the total loss is updated together, so the trunk gets a gradient from every head at each step §3.1 alg. 1. Adam with learning rate 0.0003 is used, for a maximum of 30 epochs unless stated otherwise §3.3. Pre-training uses only the unlabelled training split §4.2.

# Pre-train trunk and 8 heads on the pretext data  [alg. 1] [§3.3]
def pretrain(trunk, heads, pretext_sets, psi, max_epochs):
    for epoch in range(max_epochs):         # max 30 [§3.3]
        # Batches mix all tasks  [alg. 1]
        for batch in sample_batches_all_tasks(pretext_sets):
            loss = 0.0
            for t in heads:
                inputs, labels = batch[t]
                probs = heads[t](trunk(inputs))  # (B,)
                # Task loss [eq. 2], weighted by psi_t (values not stated)
                loss += psi[t] * bce(probs, labels)
            # L2 penalty; beta = 0.0001 is read from [§3.3] (see Pitfalls)
            loss += beta * l2_norm_squared(trunk, heads)
            adam_update(trunk, heads, loss, lr=0.0003)  # [§3.3]

Step 5. Transfer to activity recognition alg. 1 §3.1 §3.3. The trunk's conv weights are copied into a new activity model. The pretext heads are discarded. The activity model adds a dense layer with 1024 units and a softmax over the activity classes §3.3. Either all copied layers stay frozen, or only the last conv block (ConvC) is fine-tuned, depending on the experiment §3.1 §4.3.2. Early stopping is used "if the network fully converges on the training set" §3.3; no rule is given for that.

Step 6. Semi-supervised and transfer variants §4.3.4 §4.3.5. Semi-supervised: pre-train on the full unlabelled set of the target dataset, then train the classifier on 2, 5, 10, 20, 50 or 100 labelled windows per class §4.3.4. Transfer: pre-train on unlabelled MobiAct, then train the activity model on the full labelled set of another dataset §4.3.5 tab. 3. The small-data transfer curves fig. 7 use the same MobiAct pre-training.

Step 7. Analysis tools §4.4. SVCCA similarity uses the average of the top 20 canonical correlations §4.4 fig. 8. Saliency is the absolute gradient of the negative log-likelihood with respect to each input step §4.4, unnumbered equation. t-SNE projects the 96-dimensional global-max-pool features to 2-D §4.4 fig. 10.

Hyperparameters

Results

Author-reported. Setup: six public smartphone HAR datasets (HHAR, UniMiB, UCI HAR, MobiAct, WISDM, MotionSense), a 1-D CNN with the architecture above, 10 runs, and an autoencoder with the same encoder architecture as the baseline §4.1 §4.3.2. Cohen's kappa, mean over 10 runs tab. 2:

Kappa with 5-fold user-split cross-validation tab. 4:

Transfer from unlabelled MobiAct, fully supervised on the target, kappa tab. 3: HHAR 0.6816 to 0.7130; UniMiB 0.7815 to 0.8214; UCI HAR 0.8789 to 0.8879; WISDM 0.8211 to 0.8517; MotionSense 0.8763 to 0.8843 (scratch to transfer).

Per-layer features, HHAR, kappa §4.3.1 fig. 4: ConvB 0.774, ConvC 0.679. The authors then used ConvC throughout for consistency.

Semi-supervised with 2 to 10 labels per class: the abstract reports kappa of 0.7 to 0.8 at 10 labels per class. These curves appear only in fig. 6 and their values are not stated numerically in the text.

Analysis: SVCCA shows strong similarity between the self-supervised and supervised networks, with a grid-like pattern at the last layers, where the supervised networks have been trained on labels §4.4 fig. 8. Saliency maps on MotionSense highlight similar regions for both networks fig. 9. t-SNE clusters of self-supervised features follow the activity classes fig. 10.

Paper vs code

No official code. Searched: the TeX source has no URL or repository link (raw/src/1907.11879/main.tex); the arXiv metadata has no comment or project page; a GitHub search by the method and title words returned no match; the first author's GitHub account (aqibsaeed) has no repository for this method (its nearest repo, Sensor-Transformer, is a different transformer model). A repository by a third party (Alessandro-Gobbetti/MTSSL_for_Label-Efficient_Learning) is a master's thesis that is not by the authors and was not used.

Pitfalls

• Rotated and Negated overlap. "Rotated" is described as inverting the sample signs, and "Negated" as negating samples within a time window, while the paper also calls Negated an instance of Rotated §3.2. Whether the two differ only by the window extent is not stated. Decide whether to keep both, and say so in your configuration.
• Transform parameters are missing. Noise magnitude, scalar range, segment count for permutation, warp strength and the channel-permutation set are not stated §3.2. Expect the transform set to be the most tunable part of the method.
• Task weights are missing. Psi_t appears in the loss eq. 2 alg. 1 with no values given. The text notes that some tasks (channel shuffle, permutation, rotation) work better than others §4.3.3, which suggests weighting matters, but no weighting is specified.
• L2 term placement. The algorithm writes beta * ||theta||^2 inside the per-task bracket, which would add it once per task, or once in total alg. 1. The value 0.0001 is given as the L2 "rate" §3.3, and beta is not defined in the algorithm. Implementing it once per task applies eight times the penalty.
• The pretext set is fixed. Algorithm 1 builds the transformed dataset once, before the training loop alg. 1. The paper does not say whether transforms are resampled each epoch.
• Pretext set size. Each window adds two items to each of eight task datasets, so 16 items per window alg. 1. The HHAR subsample (4000 per user) and the UniMiB 5x factor §4.2 are the only size controls described.
• Input channels are unclear. The method text says accelerometer §3.2, while the dataset sections list gyroscope, orientation and altitude for several datasets §4.1. The channel count C, which channels each dataset uses, and the axis count for channel shuffle (which assumes three axes §3.2) are not stated.
• Window duration varies. Windows are 400 samples §4.2, but the sampling rates differ across datasets: 50 to 200 Hz for HHAR, 50 Hz for UniMiB, UCI HAR and MobiAct, and 20 Hz for WISDM §4.1. The paper does not say whether signals were resampled. Under the same sample count, the windows cover different durations. Check each dataset's rate before reusing the window length.
• Layer choice conflicts with the per-layer result. The frozen ConvC HHAR kappa is 0.679 in the per-layer analysis §4.3.1, but 0.731 in the main table tab. 2. Both are described as the last conv layer. The paper gives no explanation. ConvB (0.774) did better in the per-layer analysis, yet the authors kept ConvC for consistency §4.3.1.
• Frozen features do not beat supervised everywhere. In tab. 2, frozen features are 1.7 to 2.3 points of kappa below supervised training on UniMiB, WISDM and MotionSense. Under 5-fold user-split CV tab. 4, frozen features are below supervised on five of six datasets (HHAR is the exception, +0.9 points). The text says the gap is "reduced" §4.3.2 rather than closed.
• Dataset counts disagree. MobiAct: 67 users in tab. 1, "66 participants" in §4.1, and 61 used in §4.1. The transfer section says MobiAct has "twelve activities" §4.3.5, while the dataset section lists 11 activities of daily living §4.1 (and four falls, which are not used as classes). WISDM: 36 users in tab. 1 and user IDs 1 to 36 §4.1, but the text says "29 volunteers".
• Table typos. The UniMiB P value for the transfer table is written "00.8557" tab. 3; read it as 0.8557.
• Transfer protocol wording. tab. 3 says the target model is trained "in an end-to-end fashion", while §4.3.5 says "fine-tuning". Whether the transferred conv layers are frozen in these runs is not stated.
• Small-data numbers are in figures only. The 10-label kappa in the abstract, and the 2 to 100 label curves, are only in figures fig. 6 fig. 7. Read them from the figures only with care and mark them as approximate.
• Reproducibility gaps. Batch size, weight initialisation, padding, the activity classifier's learning rate and epochs, the random seeds, the exact user splits and the meaning of "±" are not stated (see Hyperparameters).
• Cost. Eight heads on each window, each with an original and a transformed copy alg. 1, multiply the pretext work. The paper says the method has "an equivalent computational cost to supervised learning" §3.1, but gives no runtime or memory figures to back that up.
• Saliency equation is unnumbered. The formula for S_k sits in a starred environment in the source §4.4, so cite it as "unnumbered equation in §4.4".

Lineage

• Builds on: multi-task learning §2.2 (Caruana, 1997); self-supervised pretext tasks from vision, such as rotation prediction §1 (Gidaris et al., 2018) and the multi-task self-supervision of Doersch and Zisserman (2017) §1; the time-series transformations used as pretext tasks §3.2 (Batista et al., 2011; Um et al., 2017), and augmentation of wearable data §5 (Um et al., 2017).
• Compared against: random init. (frozen), a supervised network trained end to end on labels, and an autoencoder with the same encoder §4.3.2; single-transform pretext models §4.3.3.
• Superseded by: not stated in this card's sources.