Explaining the Unique Nature of Individual Gait Patterns with Deep Learning

In plain words

The paper asks whether a person's walk is distinctive enough to pick that person out of a group, and whether a neural network can show why it picked someone. Fifty-seven healthy adults each walked 20 times over a force plate while their joint angles and foot forces were recorded. Each walk was turned into a fixed-size table of numbers over one stride.

Simple linear classifiers, small and large fully connected networks, and convolutional networks were trained to name the walker. Most of them were almost always right. The authors then used a method called Layer-wise Relevance Propagation to trace each prediction backwards and mark which moments in the stride and which measurements pushed the answer toward the right person. The authors also tested how well each model held up when the input was corrupted with noise, and how stable the marks were from trial to trial. Deeper convolutional networks did best on both counts.

TL;DR

• Subject identification (57 classes) from one stride per trial. Inputs are ground reaction forces (GRF) and joint angles, each as a channels-by-101 time-normalised array.
• Models: a linear SGD classifier, a linear SVM, fully connected networks (MLP) and several CNN variants. Ten-fold cross-validation.
• Explanation: LRP attributes each prediction to input values over time, for the true class only.
• Author-reported: the linear SVM is at 99.7 to 100.0% on all five feature sets. Most networks are above 95% on GRF and on the full-body and lower-body joint-angle sets. Small MLPs are weakest on the sagittal-only sets (LBJAX: 55.6% for MLP (2, 64)). Linear SGD is the least robust to noise and the least stable in relevance.
• Only one session of 57 subjects, with the same 20 trials per subject spread across training and test splits. Generalisation to other sessions or to patients is not tested.

Problem

Gait analysis usually reduces time-continuous signals to a few pre-chosen variables such as range of motion or step length, which discards information and relies on the analyst's choices § Introduction. Machine learning can use full time courses, but the models are black boxes, which hampers clinical acceptance § Introduction.

The paper's aim is threefold: show that gait patterns identify individuals with neural networks, check that the networks use meaningful gait features, and describe the features that identify a specific person § Introduction.

Variability within and between individuals is treated as an inherent feature of human motor control, so robustness and reliability are both required § Prediction Accuracy and Model Robustness.

Method

1. Data acquisition § Experimental protocol and data acquisition § Subjects and ethics statement. The sample is 57 active adults, 29 women and 28 men, with no gait disorder and no lower-limb injury. Each walked barefoot at a self-chosen pace on a 10 m path, 20 trials each, in one session with no intervention. Ten infrared cameras recorded 62 retro-reflective markers at 250 Hz fig. 5. Two force plates recorded three-dimensional ground reaction forces at 1000 Hz.

2. Stride extraction and normalisation § Data processing. One stride per trial is kept, from right-foot heel strike to left-foot toe-off, detected with a 10 N vertical force threshold. Marker trajectories are low-pass filtered at 12 Hz and forces at 50 Hz (both second-order, bidirectional Butterworth). Forces are normalised to body weight. Joint angles in the elbow, shoulder, spine, hip, knee and ankle, in the sagittal, transverse and coronal planes, are computed with Visual3D. Every channel is time-normalised to 101 points, z-transformed, and scaled to [-1, 1]. Kinematic channels are z-transformed per trial; kinetic channels across all trials.

3. Input vectors § Neural Network Architectures. Each sample is a channels x time array with C channels and T = 101 points. Five feature sets are used:

Linear and MLP models receive the array flattened by row concatenation. CNNs use the array directly § Data Analysis.

4. Classifiers § Data Analysis tab. ST4–tab. ST9.

• Linear (SGD): one dense layer from D to L classes, trained with SGD. No SoftMax is listed for this model tab. ST4.
• Linear (SVM): a linear support vector classifier, C = 0.1. The paper says it used scikit-learn's multi-class linear SVC, but the exact estimator class and multi-class strategy are not named. It is not trained with SGD § Data Analysis.
• MLP (2, H): dense D to H, ReLU, dense H to L, SoftMax. One hidden layer tab. ST4.
• MLP (3, H): dense D to H, ReLU, dense H to H, ReLU, dense H to L, SoftMax. Two hidden layers tab. ST4.
• H is 64, 128, 256, 512 or 1024 tab. ST1.
• CNN-A, CNN-A3, CNN-A6, CNN-C3, CNN-C3-3 and CNN-C6: convolutional stacks with ReLU and a final dense layer to L classes with SoftMax. The layer specifications differ by feature set. See the Architectures table below.

5. Training § Data Analysis. All models except the SVM are trained as L-way classifiers with SGD on mini-batches of 5 randomly chosen training samples, for up to 30,000 iterations. The learning rate starts at 5e-3 and is lowered to 1e-3 and then 5e-4 every 10,000 iterations. Initial weights are drawn from a zero-mean normal distribution whose standard deviation is the inverse square root of each layer's fan-in (the number of inputs feeding one output unit) § Data Analysis.

# Preprocess one stride  [§ Data processing]
# Input:  marker trajectories and forces for one trial
# Output: input array x, shape (C, 101), values in [-1, 1]

def preprocess(markers, forces):
    # Low-pass filter: 12 Hz for markers, 50 Hz for forces  [§ Data processing]
    markers = butterworth_bidirectional(markers, cutoff_hz=12.0)
    forces = butterworth_bidirectional(forces, cutoff_hz=50.0)
    # Forces normalised to body weight  [§ Data processing]
    forces = forces / body_weight
    # Joint angles from the filtered markers (Visual3D)  [§ Data processing]
    angles = joint_angles(markers)
    # Resample to 101 points; method not stated  [§ Data processing]
    channels = resample_to_points(concat(forces, angles), 101)
    # z-score: kinematics per trial, kinetics over all trials
    # [§ Data processing]
    # Scaling to [-1, 1]; the exact rule is not stated  [§ Data processing]
    return scale_to_unit_range(zscore(channels))

# Train one classifier  [§ Data Analysis]
# Input:  train_x (N, C, 101), train_y (N,), validation split, model
# Output: trained model

def train(model, train_x, train_y, val_x, val_y):
    # Normal weights, mean 0, std = fan_in ** -0.5  [§ Data Analysis]
    initialise_weights(model)
    for step in range(30000):   # up to 3e4 iterations  [§ Data Analysis]
        # 5e-3, then 1e-3, then 5e-4, switching every 1e4 steps
        # [§ Data Analysis]
        rate = learning_rate(step)
        # Mini-batch of 5 random training samples  [§ Data Analysis]
        batch_x, batch_y = sample_batch(train_x, train_y, size=5)
        # Loss function not stated in the paper
        loss = training_loss(model(batch_x), batch_y)
        model.sgd_step(loss, rate)
        # A learning-rate stage may end early once validation converges
        # (criterion not stated)  [§ Data Analysis]
        if validation_converged(model, val_x, val_y):
            step = next_stage_start(step)
    return model

6. Cross-validation § Data Analysis. Ten-fold cross-validation with 8 parts for training, 1 for validation and 1 for testing. On average 912 samples per split are used for training (derived: 0.8 x 57 x 20 = 912). The 20 samples of each subject are spread at random over all partitions.

7. Explanation with Layer-wise Relevance Propagation § Layer-wise Relevance Propagation. For a linear model, the score of class c is f_c(x) = sum_i w_ic x_i + b_c eq. 1. Each input's term w_ic x_i is its contribution. LRP generalises this to any network. The output relevance is set to the class score, and each neuron passes its relevance to its inputs in proportion to the contribution z_ij / z_j eq. 2. Relevance arriving at an input is summed over its outputs eq. 3. The total relevance is conserved, so the input relevances add up to the score. For one linear layer, the rule reduces to R_i = x_i w_i. The paper explains predictions for the true class only fig. 1. Positive relevance supports the class, negative relevance contradicts it, and near-zero means no influence.

# Explain the output for one class  [§ Layer-wise Relevance Propagation]
# Input:  layers in forward order, activations per layer, class index c
# Output: relevance of the input, shape (C, 101)

def lrp_explain(layers, activations, c):
    # Start with the class score as the output's relevance  [eq. 1]
    relevance = zeros_like(activations[-1])
    relevance[c] = activations[-1][c]
    for index in reversed(range(len(layers))):
        # Share each output's relevance among its inputs by z_ij / z_j  [eq. 2]
        # The propagation rule used in the experiments is not stated
        shares = layers[index].redistribute(relevance, activations[index])
        # Sum the shares that reach each input  [eq. 3]
        relevance = shares.sum(axis=0)
    return relevance

8. Robustness to noise § Prediction Accuracy and Model Robustness tab. 2. For each test set, a random order over the input components is drawn per input. The n-th perturbation is applied to all test samples at once, and accuracy is measured after each step. Fifty steps are run and the experiment is repeated 10 times. The area over the perturbation curve (AOPC, from § Results: Prediction Accuracy and Model Robustness) is summarised per model.

# Noise robustness curve  [§ Prediction Accuracy and Model Robustness]
# Input:  model, test_x (N, C, 101), test_y (N,), noise_fn
# Output: accuracy after each of 50 perturbation steps

def robustness_curve(model, test_x, test_y, noise_fn):
    # Random order of the components, drawn for each input  [§ Prediction...]
    orders = random_orders(test_x)
    accuracies = []
    for step in range(50):   # 50 steps, repeated 10 times  [§ Prediction...]
        # Perturb the step-th component of every sample; effects accumulate
        test_x = apply_noise(test_x, orders, step, noise_fn)
        accuracies.append(accuracy(model(test_x), test_y))
    # AOPC is then computed from this curve; the exact formula is not stated
    return accuracies

9. Stability of relevance § Results: Coefficient of variation of relevance. For each subject and each input component, the 20 relevance values (one per trial) are summarised with a coefficient of variation: the root-mean-square normalised standard deviation around the relevance signal. Lower means more reliable across trials and cross-validation splits tab. 3.

Architectures

Convolutional models are specified as Conv(Fc, Ft | Sc, St | H): channel span, time span, channel stride, time stride, number of filters. Dense sizes are the flattened inputs to the final dense layer. Each network ends with SoftMax. Sizes were checked by the output-length arithmetic (no padding, except where noted), and they match the paper's tables.

CNN-C3-3 is not reported for GRF, FBJAX or LBJAX tab. ST1. Source: GRF tab. ST5, FBJA tab. ST6, FBJAX tab. ST7, LBJA tab. ST8, LBJAX tab. ST9.

# CNN-A on the GRF input  [tab. ST5]
# Input:  x, shape (6, 101): channels x time
# Output: class probabilities, shape (L,)

def cnn_a_grf(x):
    # One conv: 6x6 filters spanning all channels, 32 filters  [tab. ST5]
    hidden = relu(conv2d(x, kernel=(6, 6), filters=32))  # (1, 96, 32)
    # Flatten to 3072 values, then a dense layer to L classes  [tab. ST5]
    return softmax(dense(flatten(hidden), out_features=L))

Hyperparameters

Results

Author-reported. Setup: 57 subjects, one stride per trial, 10-fold cross-validation, subject identification with L = 57 classes. Values are mean (SD), in percent, as reported in tab. 1. The main Table 1 rows are a subset of ST1 and agree with it. Full ST1 values are below.

Prediction accuracy tab. 1 tab. ST1

Headline statements: the linear SVM is at or near 100% on every set. Most networks are above 95% on GRF and on the full-body and lower-body joint-angle sets. Small MLPs (64 units) are weakest, especially on the sagittal-only sets tab. ST1 § Prediction Accuracy and Model Robustness.

Robustness: AOPC for GRF, lower is more robust tab. 2 tab. ST2

Table 2 in the main text reports the same rows, but with a different Linear (SVM) row. See Pitfalls.

Noise definitions tab. ST2 caption: Gaussian adds white noise; Salt- sets inputs to -1; Salt+ sets inputs to 1; Pepper sets inputs to 0; Shot sets inputs to a random value from {-1, 0, 1}.

Robustness was measured for GRF only tab. ST2. Text summary: the linear SGD network is the least robust. The paper reports that wider fully connected models and deeper architectures are more robust than the smaller ones § Prediction Accuracy and Model Robustness.

Reliability of relevance: coefficient of variation, lower is more stable tab. 3 tab. ST3

Text summary: lowest reliability for Linear (SGD). Reliability rises with MLP width and depth. The linear SVM and the CNNs are most reliable § Results: Coefficient of variation of relevance.

Qualitative findings fig. 1 fig. 2 fig. 4 fig. SF1–fig. SF4

• Relevance is spread over many variables and time windows. Single individual characteristics are rare, and most models use complex combinations § Results § Discussion. Subject-specific examples are given in fig. 2 and in the supplementary figures fig. SF1–fig. SF4.
• Left-right symmetric relevance patterns appear for most models except CNN-C3 fig. 4 § Discussion.
• The models were not given any physiological information, so these patterns are learned, not imposed § Results.

Paper vs code

No official code is used on this card. The paper's Data Availability section links an authors' repository, sebastian-lapuschkin/interpretable-deep-gait § Data Availability. Its python/LICENSE file limits use to academic purposes, which excludes commercial use LICENSE. Because of that, the code is not cited, and no code, code_commit or code_license field is set. Discrepancies between the paper and that code were therefore not checked. The paper's own text is the only source for this card.

Pitfalls

• CNN-A3 and CNN-A6 time spans are inverted in the text. The text says CNN-A3 reads Ft = 6 and CNN-A6 reads Ft = 3 in the input layer § Data Analysis. The tables say the reverse: CNN-A3 uses Conv(6,3) and CNN-A6 uses Conv(6,6) tab. ST5. The dense sizes (3104 for A3, 2912 for A6) match the tables. Implement the tables.
• Linear (SVM) robustness row differs between Table 2 and ST2. Table 2 gives 4.0 (3.9), 18.6 (7.0), 35.4 (4.7), 31.3 (7.7), 2.5 (2.4), 32.8 (8.7), 32.4 (8.6). ST2 gives 7.4 (4.6), 26.2 (5.6), 39.9 (3.7), 34.0 (6.9), 5.0 (3.2), 35.9 (7.2), 35.9 (7.5). The other rows agree. The paper does not say which is correct. This card reports both.
• Loss function not stated. The text gives SoftMax outputs for the networks, but never names the loss. The linear SGD model has no SoftMax in its listed architecture tab. ST4, so its loss is also unstated.
• Optimiser details missing. Momentum, weight decay, and whether mini-batches are drawn with or without replacement are not stated § Data Analysis.
• Learning-rate schedule is ambiguous. The text says the rate is "gradually lowered" to 1e-3, then 5e-4, "after every 1e4" iterations § Data Analysis. The reading used above is switches at 1e4 and 2e4. Early termination of each stage can shorten it, and its criterion is not stated.
• Possible normalisation leakage. Kinetic channels are z-scored "for all trials" § Data processing. The paper does not say whether these statistics come from training folds only. Compute the statistics from training data to avoid leakage.
• Scaling and resampling are underspecified. The rule for scaling to [-1, 1] and the interpolation method for the 101-point resampling are not stated § Data processing.
• MLP depth notation. "L" is described as "the number of layers" § Data Analysis. The tables show L counts dense layers including the output. MLP (2, H) has one hidden layer; MLP (3, H) has two tab. ST4.
• CNN depth varies by feature set. The text says the number of convolution layers is C in {1, 2, 3} § Data Analysis. In practice, CNN-C6 has one conv layer for GRF and LBJAX, and three for FBJA, FBJAX and LBJA tab. ST5–tab. ST9. Use the tables.
• Time-padding for CNN-C3-3. The time axis is padded to 102 with one extra column of zeros § Neural Network Architectures. Which side is padded is not stated. The stated dense sizes (6720 for FBJA and 1920 for LBJA) match a 102-point input.
• Channel lists are not enumerated. The paper gives only the channel counts (6, 33, 10, 18, 6). It does not list which variables they are.
• Cross-validation details. How the ten test folds are formed, and whether the reported mean (SD) is over the folds, is not explicitly stated § Data Analysis. Validation and test partitions are random per subject, and the paper does not say whether trials of one subject are kept together.
• Within-session generalisation only. The 20 trials per subject are spread at random over the partitions § Data Analysis, so a test trial's subject also has training trials from the same session. The results do not show that identification holds across sessions, days or people with pathology. The paper's claims of uniqueness rest on this design.
• SVM specifics. The paper names scikit-learn's multi-class linear SVC § Data Analysis, but not the exact estimator class, multi-class strategy or loss.
• AOPC formula. The paper refers to Samek et al. for AOPC § Prediction Accuracy and Model Robustness. The aggregation over steps is not restated in this paper.
• Coefficient of variation. The paper gives a description in words only: "root mean square normalised band of standard deviation around the relevance signal" § Results: Coefficient of variation. The normalisation constant and whether it is per time point or per component are not stated.
• LRP propagation rule not stated. The general relevance rule eq. 2 and pooling eq. 3 are given, and the linear case is worked out. The rule for dense and convolution layers used in the experiments (for example any stabiliser term) is not stated. The paper cites Bach et al. for details.
• Relevance is for the true class. The paper analyses the true-class prediction only fig. 1. Relevance maps for a wrong predicted class will differ.
• Abstract and body disagree on inputs. The abstract names ground reaction forces and full-body joint angles as the inputs [Abstract]. The results and subject figures use lower-body joint angles (LBJA and LBJAX) § Results fig. 2.
• Pre-print. This card follows the arXiv pre-print (v2, dated 2019-02-27) and says it is a pre-print of a Scientific Reports article § Acknowledgements. The journal version may differ.
• Dataset not used or checked. The data are on Mendeley Data (DOI 10.17632/svx74xcrjr.1) § Data Availability. This card does not check the dataset's license. Check it before reuse.
• All numbers are the authors'. Accuracy, AOPC and CV values are author-reported, on one dataset.

Lineage

• Builds on: Layer-wise Relevance Propagation, Bach et al., cited as [BachPLOS15] in the paper. No card yet.
• Uses AOPC from Samek et al., cited as [SamTNNLS17] in the paper. No card yet.
• Baselines inside the paper: linear SGD and linear SVM, against MLPs and CNNs § Results.
• Superseded by: not known from the content of this paper.