Online Adaptation for Myographic Control of Natural Dexterous Hand and Finger Movements

In plain words

The goal is to let amputees control a robotic hand with the muscle signals left in the forearm. This paper reads those signals with a band of sensors worn around the forearm and turns them into the position of the wrist and each finger, all at once, rather than picking one gesture from a short list.

The model is a neural network that looks at the most recent stretch of signal, so it reacts almost as the movement happens. A glove worn by the test person records the real hand position during training, which gives the network its answers. Instead of a fixed training routine, the person simply moves their hand however they like, and the network keeps learning from that movement while it runs.

The paper tests this on people with intact hands, not on amputees, using a virtual robotic hand shown in real time on a display. The authors say the work is still in progress and the numbers may change.

TL;DR

• Predicts 7 wrist and finger degrees of freedom continuously and together from 16 surface-EMG channels §2.1.2 §2.1.3.
• Uses causal temporal convolutional (TCN) and LSTM models on time-domain features; a support vector regressor (SVR) is the frame-wise baseline §2.1.6 tab. 1.
• Author-reported, 9 non-amputee subjects: TCN and LSTM reach 8.0 to 10.4 degrees RMSE across the standard and free-form sessions, against 11.3 to 15.4 for SVR §3.1 §3.2.
• Training uses a free-form session in which the model is updated trial by trial as the subject moves; the paper reports a further improvement of about 4 degrees over that session §2.1.5 §3.3.
• Relevant if you work on EMG or myoelectric regression and want a causal, low-latency baseline. Treat every number as preliminary, since the authors state the experiments are ongoing §3.1.

Problem

Most EMG prosthesis control picks among a set of pre-chosen movements, or maps muscle force to speed. The authors argue that training on cued movements teaches the model only the set it was shown, and that rest-to-movement-to-rest transitions teach nothing about moving between two active states §2.1.4 §1. Cued movements also tend to need strong contractions, which fatigue the user §2.1.4 §1.

Regression, which estimates the joint position or velocity directly, has been shown for 2 to 3 degrees of freedom §1. The stated aim is natural upper-limb function for amputees §5. Multi-finger regression and frame-wise models with fixed feature windows have been reported, but the authors note that frame-wise models add delay that is set by the window size §1 §4. The paper assumes a human is in the control loop, so a long delay can cause overshoot and compensating movements §3.2 fig. 6.

Method

Signals and targets. A 16-channel EMG armband records at 2000 Hz. Five time-domain features are computed from the raw signal in real time §2.1.2. A data glove records 7 degrees of freedom: flexion and extension of the five fingers, flexion and extension of the wrist, and abduction and adduction of the wrist §2.1.3. Each subject first explores the full range of each degree of freedom for 15 s. The minimum and maximum glove values from this calibration map each degree of freedom to a display angle and to a normalised value in [0, 1] §2.1.3. The paper does not write the mapping formula.

Timing. Features, glove angles and normalised targets are recorded every 25 ms §2.1.3. The prediction step is Δt = 25 ms tab. 1.

Features. Five time-domain features per channel, called TD5: average absolute amplitude, curve length, variance, and two counts of sign behaviour (slope reversals and zero crossings) §2.1.2 tab. 1. Thresholds for the two counts are not stated.

Models. Three models are compared §2.1.6:

1. TCN: temporal convolutional network with M = 128 and d = 25 as listed in the paper; the meaning of M and d is not defined in the text §2.1.6.
2. LSTM: long short-term memory network with 64 nodes §2.1.6.
3. SVR: support vector regression with a Gaussian radial basis kernel, gamma = 0.01 and C = 1, on a single 200 ms feature window §2.1.6 tab. 1.

The TCN and LSTM end in sigmoid outputs, use mean squared error as the loss, and are strictly causal: each prediction uses only earlier EMG §2.1.6. The paper's window and sequence settings are in tab. 1. Sequence length is given as a count; the text does not say the unit, which is probably 25 ms steps (see Pitfalls).

Standard experiment. Each subject does 3 trials of cued movements, with 12 active movement types fig. 1. Each trial cues each movement once in random order, with 20 s active cues alternating with 5 s rest cues §2.1.4. The first 2 trials train the models and the third tests them §2.1.4.

Free-form experiment. One 1-minute trial trains the first model. Then 15 consecutive 30-second trials follow. The subject moves freely §2.1.5. The model is tested on each trial, then updated with a "reinforcement update" using that trial's data, and that data is discarded afterwards, although saved to file §2.1.5. TCN predictions are shown to the subject in real time, while LSTM predictions run in the background §2.1.5. The total session is about 9 minutes §2.1.5. Offline analysis uses the first 60 percent of each subject's data to train and the last 40 percent to test §2.1.5.

Reinforcement update. The paper gives no optimizer, learning rate, epoch count or warm-start rule for the update. It says the models are "further trained" on the new data §2.1.5.

Metrics. Angular RMSE per degree of freedom in degrees, the coefficient of determination r², and response delay: the time shift that maximises the correlation between prediction and ground truth. Group p-values are from a Kruskal-Wallis test. Shaded regions are standard error of the mean §2.1.6.

Pseudocode. Written from the text. Comments give the source; anything not stated is marked.

# Glove targets: calibrated range mapped to [0, 1]  [§2.1.3]
# Input:  glove_raw (T, 7) per 25 ms step; calib_min, calib_max (7,)
# Output: targets (T, 7) in [0, 1]

def normalize_targets(glove_raw, calib_min, calib_max):
    # The paper gives the range-to-range mapping, not the formula  [§2.1.3]
    span = calib_max - calib_min
    return (glove_raw - calib_min) / span

# TD5 features per window  [§2.1.2] [tab. 1]
# Input:  raw_emg (16, S); S = 50 ms x 2000 Hz samples  (derived)
# Output: features (16, 5)

def td5_features(raw_emg):
    # Five time-domain features per channel  [§2.1.2]
    mean_abs = mean(abs(raw_emg), axis=-1)
    wave_len = sum(abs(diff(raw_emg, axis=-1)), axis=-1)
    variance = var(raw_emg, axis=-1)
    # Thresholds for these two counts are not stated  [§2.1.2]
    slope_sign = count_slope_sign_changes(raw_emg)
    zero_cross = count_zero_crossings(raw_emg)
    return stack([mean_abs, wave_len, variance, slope_sign, zero_cross],
                 axis=-1)

# Causal sequence regressor  [§2.1.6] [tab. 1]
# Input:  feature_seq (B, S, F); S = sequence length; F = feature size
# Output: dof_pred (B, 7), each value in (0, 1)

def predict_dofs(feature_seq):
    # Each output uses only the current and earlier windows  [§2.1.6]
    hidden = causal_temporal_layers(feature_seq)  # TCN or LSTM
    # Assumed: the last time step is read out; not stated  [§2.1.6]
    # Sigmoid output, one value per degree of freedom  [§2.1.6]
    dof_pred = sigmoid(dense(hidden[:, -1]))
    return dof_pred

# Supervised fit with mean squared error  [§2.1.6] [tab. 1]
# Epochs: TCN 15, LSTM 40; batch size 24; optimizer not stated

def fit(model, windows, targets, epochs):
    for epoch in range(epochs):
        for batch_x, batch_y in minibatches(windows, targets, size=24):
            # Error between predictions and normalised glove targets
            loss = mean((model(batch_x) - batch_y) ** 2)
            update_weights(model, loss)  # optimizer, rate not stated
    return model

# Free-form session with trial-by-trial updates  [§2.1.5]
# Input:  emg_stream (16, 2000 Hz); glove_stream (7, every 25 ms)
# Output: updated model

def freeform_session(emg_stream, glove_stream, model, epochs):
    # One 1-minute trial trains the first model  [§2.1.5]
    # Epochs for the session are not stated; [tab. 1] gives the sweep values
    first = record(emg_stream, glove_stream, seconds=60)
    model = fit(model, first.windows, first.targets, epochs)
    discard(first)  # kept on disk for later analysis only  [§2.1.5]
    for trial_index in range(15):
        # 30 s test, then update  [§2.1.5]
        # Reading of the text: the update uses the trial just tested;
        # the paper's wording is ambiguous  [§2.1.5]
        trial = record(emg_stream, glove_stream, seconds=30)
        predictions = model(trial.windows)  # causal, live output
        log(predictions, trial.targets)
        # Update epochs and optimizer not stated  [§2.1.5]
        model = fit(model, trial.windows, trial.targets, epochs)
        discard(trial)
    return model

# Per-degree-of-freedom error and response delay  [§2.1.6]
# Input:  predictions, targets (T, 7) in degrees
# Output: rmse (7,), delay_ms (7,)

def evaluate(predictions, targets):
    # Angular RMSE per degree of freedom, in degrees  [§2.1.6]
    rmse = sqrt(mean((predictions - targets) ** 2, axis=0))
    # Shift that maximises correlation, in 25 ms steps  [§2.1.6]
    # Sign convention for the shift is not stated
    shift = argmax_shift(correlate(predictions, targets))
    return rmse, shift * 25  # milliseconds

Hyperparameters

Results

Author-reported. The paper says the experiments are ongoing and the final numbers may differ §3.1. Setup: 9 non-amputee subjects, a virtual Modular Prosthetic Limb display §2.1.2, and SVR as the only non-sequential baseline §2.1.6. The paper does not compare against prior published systems, by the authors' choice §4.

• Both sequential models beat SVR on RMSE and r² in both experiments; the paper reports p < 0.001 for the SVR comparison §3.1 §3.2.
• The paper states that the online RMSE and r² improved significantly (p < 0.001) for TCN and LSTM over the session §3.3. It does not give starting values in the text fig. 7.
• In both experiments, wrist and thumb degrees of freedom had lower RMSE than individual fingers fig. 5 §3.1 §3.2.
• Preliminary sonomyography (ultrasound) results with TCN on 5 finger degrees of freedom are shown in fig. 8. They are qualitative only, with no numbers given. The input was down-sampled and masked, cutting its size by about 12 times for one subject fig. 8 §4.1.

Paper vs code

No official code. The paper has no code link, and no project page. A GitHub search found no repository that clearly belongs to the authors. Searches: user search for "Betthauser" (8 accounts); the Betthauser-Neuro-Lab organisation, which has two unrelated repositories; the account jlbettha, which has no public repositories; repository search for "Betthauser" (two results, both unrelated); and repository search for "myographic prosthesis freeform", which returned an error. Other author accounts were not checked individually. The Keras and TCN libraries cited in §2.1.6 are third-party code and are not used here.

Pitfalls

• Small, ongoing study. Nine non-amputees (eight in the free-form analysis) with the authors stating the experiments are ongoing §2.1.1 §2.1.5 §3.1. No amputees were tested, and the robotic arm was not used §4.1.
• No comparison to prior work. The authors decline to compare against other models or prior research §4. The only baseline is SVR, so the "state of the art" framing in the abstract rests on the authors' own comparisons.
• Reinforcement update is undefined. The text gives no optimizer, rate, epoch count or whether training restarts from scratch. "Further trained" suggests warm start, but this is not stated §2.1.5.
• Test-then-update order is ambiguous. The text does not say which trial's data each update uses. The pseudocode above reads it as "test, then update on that trial" §2.1.5.
• Feedback confound. The subject sees TCN output while moving, and LSTM output in the background, so the subject's movements may adapt to TCN. The TCN and LSTM free-form comparison may therefore not be clean §2.1.5.
• Sequence unit not stated. TCN sequence length 40 and LSTM 24 tab. 1. The unit is probably 25 ms steps, which would give histories of 1.0 s and 0.6 s. This is an inference from the Δt footnote, not a stated fact tab. 1.
• Undefined TCN parameters. M = 128 and d = 25 are listed, but the text does not define them. Kernel size, dilation schedule and layer count are not stated §2.1.6.
• Targets are glove angles, not intent. Ground truth is the glove reading. Its calibration sets the [0, 1] scale for training and the display angles, both fixed by a 15 s calibration §2.1.3. Calibration error would shift every RMSE.
• Response delay is quantised. A delay under 25 ms is reported as "within a single step". The shift sign convention is not stated §2.1.6 §3.1.
• Inference cost. The time per prediction on the GPU is not stated, although the paper argues for near-zero delay §3.2.
• Hyperparameters from preliminary data. Window, sequence and epoch settings come from parameter sweeps on earlier, unpublished data. The sweep values are not in the text, and the paper does not say whether the preliminary subjects overlap the test subjects §2.1.6 fig. 4.
• Statistics. p-values come from a Kruskal-Wallis test. The unit of analysis (subject, degree of freedom or trial) is not stated, so the p-values may not reflect the number of independent observations §2.1.6.
• Licence. The paper is CC BY-SA 4.0, so this page is a derivative of a share-alike work. No figure is reproduced. A human reviewer should confirm how the share-alike terms apply to the page.

Lineage

• Builds on: the group's earlier work on temporal convolutional networks for EMG classification and reinforcement adaptation, cited as Betthauser et al. 2019 (IEEE TBME and IEEE/EMBS NER) §1. Neither is on arXiv under a known id from the paper, so neither has a card.
• Uses: temporal convolutional networks §2.1.6 §1, which are the subject of [[temporal-convolutional-network]], and LSTM networks §2.1.6, which are covered by [[long-short-term-memory]].
• Compared against: SVR for EMG regression §2.1.6, covered by [[support-vector-machine]].
• Related sensors: the virtual Modular Prosthetic Limb §2.1.2 and the sonomyography sensor work by the George Mason University group §4.1.
• Source note: the paper states that the text is modified from Chapter 5 of a 2020 dissertation by the first author. The dissertation was not used.
• Superseded by: not known.