Long short-term memory (LSTM)

Overview

An LSTM keeps a cell state that is updated additively and scaled by gates. That additive path is the "constant error carousel" the paper builds on 2405.04517 §2.1. The paper names three limits of the classic LSTM: it cannot revise a stored value once a better match appears later in the sequence, it compresses everything into scalar cell states, and its hidden-to-hidden connections force sequential processing 2405.04517 §1. The variants below address these limits. The invariant they all keep is the gated, additive cell-state update.

LSTMs are also used as a recurrent controller rather than a language model. The bipedal-locomotion card uses a two-layer, 128-unit LSTM as the policy that maps proprioceptive input and a clock to joint set points 2204.04340 §III-A. That use is covered in its own variant below.

Outside language modelling and locomotion, the LSTM is a classifier, a forecaster, a residual corrector and a regressor over sensor streams. It is also the sequence predictor inside an unsupervised anomaly detector. It is the encoder and decoder of a task-oriented dialogue model and of a speech-to-text translator, and the per-step reader of skeleton joints. Each of these has its own variant below, and each card reports results on its own task, so the numbers are not comparable across variants.

Variants

Exponential gating with normalizer and stabilizer

The input gate, and optionally the forget gate, use the exponential function. A normalizer state tracks the overall gate strength, and a stabilizer state is subtracted inside the exponentials so they do not overflow. The stabilized and unstabilized forms give the same outputs and parameter gradients 2405.04517 §2.2; app. A.2. In the 15B ablation, adding exponential gating to the vanilla-style backbone lowers perplexity from 26.01 to 17.70, and adding matrix memory lowers it further to 13.48 2405.04517 tab. 2.

sLSTM: scalar memory with memory mixing

The sLSTM keeps the scalar cell and adds recurrent matrices from the previous hidden state to every gate. These matrices are block-diagonal, one block per head, so heads do not mix 2405.04517 §2.2; app. A.2. Because of this recurrence it is not parallelisable over the sequence 2405.04517 §5 (i). Card: xLSTM: Extended Long Short-Term Memory.

mLSTM: matrix memory with covariance update

The mLSTM replaces the scalar cell with a d x d matrix. Keys and values are written with an outer-product (covariance) rule and read back with a query. Its gates depend only on the input, not on the previous output, so training can use a parallel form over the whole sequence [arxiv:2405.04517 §2.3; app. A.3]. Card: xLSTM: Extended Long Short-Term Memory.

xLSTM blocks and the a:b ratio

sLSTM and mLSTM sit in residual blocks with pre-LayerNorm. The notation xLSTM[a:b] gives the ratio of mLSTM to sLSTM blocks 2405.04517 §2.4. The paper reports two families: xLSTM[1:0], which is mLSTM only, and xLSTM[7:1], which has sLSTM blocks at a few fixed positions 2405.04517 §4.2; app. B.

Recurrent policy trained with PPO

The locomotion paper's policy is an LSTM with two hidden layers of 128 units. Its inputs are pelvis orientation and rotational velocity, joint positions and velocities, a sine and cosine clock whose period is the cycle time, and the desired forward velocity. Its outputs are joint set points at 40 Hz 2204.04340 §III-A. It is trained with PPO using a clipped objective and gradient clipping 2204.04340 §III-A. Card: Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads.

The policy input does not change with the load. Only the simulated robot model changes 2204.04340 §III-B. Training on the specific load raises the Carry Pole pass rate from 0.0014 for the base policy to 0.9578 2204.04340 tab. 2. The card records no non-recurrent policy as a comparison, so these results cannot be credited to the LSTM alone.

Recurrent classifier on raw squeeze signals

The tactile paper feeds each squeeze's raw sensor series to an LSTM. The series is zero-padded to a common length. The output at the last non-zero step goes through two linear layers and a softmax, trained with cross-entropy and Adam 2204.06343 §4.2.3. The grid covers 2 or 4 layers, 32 to 256 units and a log range of learning rates. The paper trained 336 such models, and the best were typically 4 layers with 256 units 2204.06343 §4.2.3. A support vector machine on hand-made features is the other main classifier in the same study 2204.06343 §4.2.2. Card: Single-grasp Deformable Object Discrimination.

CNN features feeding an LSTM forecaster

The teleoperation paper first classifies six glovebox tasks with a CNN. It then feeds CNN features into an LSTM that forecasts the next 500 frames, which is 5 s at 100 Hz 2402.10220 §IV-A; §IV-E; tab. III. The baseline LSTM instead takes the mean over the 24 input channels at each step 2402.10220 §IV-E.1. The LSTM stack is 100, 100, 50 and 50 units, read from a Netron export, with a 200-unit dense layer after it 2402.10220 fig. 8d. Card: Towards Semi-Autonomous Robotic Arm Manipulation Operator Intention Detection.

LSTM residual correction to a physics model

The off-road driving paper keeps a bicycle-type physics model for the tyre and drive forces. An LSTM predicts a correction to those forces from a 0.2 s history of states, controls and measurements 2412.00581 §4; eq. 5. In the visual variant, a small per-wheel encoder maps terrain features into extra LSTM inputs, and the whole model is trained end to end 2412.00581 §5; eq. 6; fig. 1. Card: Dynamics Modeling using Visual Terrain Features.

Causal LSTM regressor for EMG

The myoelectric paper maps 16 surface-EMG channels, reduced to time-domain features, to seven wrist and finger angles. Its LSTM has 64 nodes, a sigmoid output and an MSE loss. The window is 50 ms, and the model trains for 40 epochs with batch size 24 2412.17991 §2.1.6; tab. 1. The model is strictly causal, so each prediction uses only earlier EMG 2412.17991 §2.1.6. After each trial the model is updated on that trial's data. Both sequential models improve by about 4 degrees RMSE over the session, which the paper reports as significant 2412.17991 §3.3. Card: Online Adaptation for Myographic Control.

Attention BiLSTM predictor inside an autoencoder anomaly detector

The CAE-M model trains a convolutional autoencoder jointly with a bidirectional LSTM. The LSTM reads the earlier time steps of each sample's feature sequence and predicts the next one. Attention over the step outputs gives a softmax-weighted sum that serves as the prediction [arxiv:2107.12626 §3.4.1; eq. 13–17]. The LSTM has 512 units, and dropout of 0.2 sits after the attention. The text merges the two directions by sum, while eq. 13 writes concatenation. The paper does not say whether 512 is per direction 2107.12626 §3.4.1; eq. 13; §4.3. The LSTM term is weighted 0.5 in the total loss, next to a linear predictor and the reconstruction term 2107.12626 §3.5; eq. 20.

On three datasets the paper reports the same ablation for attention. Removing it lowers mF1 from 0.9639 to 0.9213 on PAMAP2, from 0.9961 to 0.9271 on CAP, and from 0.9960 to 0.9578 on fatigue. These are single values with no spread 2107.12626 tab. 6. Removing both predictors (λ2 = λ3 = 0) gives 0.8063 on PAMAP2, but that also removes the linear predictor, so it does not isolate the LSTM tab. 6. Against the ten baselines, CAE-M reports the highest mF1 on all three datasets. The LSTM autoencoder baseline reports 0.8296 on PAMAP2, 0.6671 on CAP and 0.6870 on fatigue 2107.12626 tab. 2. The paper releases no code and does not state the learning rate, so an implementation must fill in those values 2107.12626 §4.3. Card: Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals.

Stacked and hierarchical LSTM over skeleton joints

The skeleton paper's temporal stream runs LSTMs over per-frame joint coordinates in one of two ways. The stacked version has two layers of 512 units on NTU RGB+D. The hierarchical version runs one LSTM per body part (two arms, two legs and the trunk), concatenates their outputs at each step, and feeds a second LSTM for the whole body, with 128 units per part and 512 for the body 1704.02581 §4.1; §5.2; tab. 2. The LSTM has peephole connections from the cell to the gates 1704.02581 eq. 1. In the paper's ablation on NTU, the stacked stream reaches 66.1% cross-subject and 68.9% cross-view, and the hierarchical stream 67.8% and 70.5% [arxiv:1704.02581 tab. 1]. A second stream reads the joints as an ordered sequence, one time window per joint. Fusing the two streams gives 68.6% and 71.7%. Adding the 3D rotation, scaling and shear augmentation gives 71.3% and 79.5% 1704.02581 tab. 1. The augmentation ranges are not stated 1704.02581 §4.3. Card: Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks.

Key-value attention over a knowledge base in an LSTM decoder

The in-car dialogue paper uses an LSTM encoder and an LSTM decoder with attention over the encoder states 1705.05414 §2.1; §2.2. It adds one score per knowledge-base key to the decoder's output logits, so the model can emit a canonical token such as a meeting time. A lookup then swaps that token for its stored value 1705.05414 §2.3; eq. 7–9. The embedding, hidden and cell sizes are all 200. Training is token-level cross-entropy with Adam, and no belief tracker is used 1705.05414 §5.2. On the test split, the model reports 13.2 BLEU and 48.0 entity F1. Without encoder attention it reports 10.8 and 40.9, Copy Net reports 11.0 and 37.0, and the rule-based system 6.6 and 43.8 1705.05414 tab. 3. Card: Key-Value Retrieval Networks for Task-Oriented Dialogue.

Bidirectional encoder with attention decoder for speech translation

The speech-translation paper maps French speech to English text with one attention-based encoder-decoder and no source transcript [arxiv:1612.01744 §1]. The encoder is a hierarchical bidirectional LSTM. It takes 40 MFCCs plus frame energy per frame, behind two fully connected layers of size 256. Layers 2 and 3 each read every other output of the layer below, so the encoded sequence is a quarter of the frame count 1612.01744 §2; §3. The decoder is an LSTM that takes the previous target word. Its attention adds a convolutional term over the previous step's attention weights, which the authors say stops the model translating the same part of the signal twice 1612.01744 §2; eq. 6. Training uses Adam with an initial learning rate of 0.001, mini-batch size 64 and dropout 0.5 between LSTM layers 1612.01744 §2. Card: Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation.

On the held-out speaker's test1 set with seven references, the single greedy model reaches 37.6 BLEU. Beam search, a trigram language model and a 5-model ensemble raise it to 40.1, 41.8 and 48.8. The Google ASR plus SMT pipeline reaches 52.7 1612.01744 tab. 3. On the training speaker the ensemble reaches 50.4 against 51.9 for the pipeline tab. 3. All speech data is synthetic, and the paper reports no natural-speech result [arxiv:1612.01744 §5]. The card records ensemble weights as not stated, so the ensemble gain cannot be reproduced exactly from the paper.

When to use which

All numbers come from the authors. The paper's language-model results use SlimPajama at 15B tokens, where xLSTM[1:0] reaches 13.43 against 13.70 for Mamba and 14.25 for Llama 2405.04517 tab. 1. At 300B tokens with a 1.3B model, the figures are 8.89, 9.14 and 9.44 2405.04517 tab. 3. The paper did not tune the large architectures 2405.04517 §5 (vi).

Implementation sketch

The shared core is the exponential-gated update with a stabilizer, shown for the scalar cell. The mLSTM follows the same pattern with a matrix state and a normalizer division. The sigmoid forget gate is shown; the paper also allows an exponential one 2405.04517 §2.2.

# Exponential input gate, forget gate in log space, stabilizer  [§2.2]
# State per unit: c (cell), n (normalizer), m (stabilizer)
def exp_gated_step(z, i_pre, f_pre, c_prev, n_prev, m_prev):
    # Forget gate in log space; the input gate is exponential  [§2.2]
    log_f = log_sigmoid(f_pre)
    # Stabilizer keeps every exp() argument at or below zero  [§2.2; app. A.2]
    m = max(log_f + m_prev, i_pre)
    i_gate = exp(i_pre - m)
    f_gate = exp(log_f + m_prev - m)
    # Additive gated memory and normalizer update  [§2.2]
    c = f_gate * c_prev + i_gate * z
    n = f_gate * n_prev + i_gate
    return c, n, m

# Hidden output: output gate times the normalized cell  [§2.2]
# o_gate and z are computed from the input and the previous hidden state
h = o_gate * (c / n)

Where the variants differ: the mLSTM replaces c with a matrix, uses a covariance update and divides by a normalizer threshold [arxiv:2405.04517 §2.3]. Released code and paper also differ in several details, listed on the card under "Paper vs code" [/papers/2405.04517].

The recurrent controller uses a different loop. It takes the robot's sensor inputs and a clock at each control step and returns joint set points. The locomotion card has no official code, so the sketch follows its text only 2204.04340 §III-A.

# One control step of the recurrent policy  [arxiv:2204.04340 §III-A]
# lstm_state is carried from one control step to the next
def policy_step(obs, clock, cmd_vel, lstm_state):
    # Sine and cosine of the gait clock, from its period  [§III-A]
    x = concat(obs, sin(clock), cos(clock), cmd_vel)
    # Two stacked recurrent layers of 128 units  [§III-A]
    h, lstm_state = lstm_2x128(x, lstm_state)
    # Joint set points at 40 Hz; the output head is not described  [§III-A]
    action = output_head(h)
    return action, lstm_state

Contradictions

The xLSTM card and the locomotion card study different uses of the LSTM, a language model and a locomotion controller. Their numbers are not comparable and do not conflict. Within the locomotion paper, the text and the table differ on the Cart force gain. The text gives 60 N more force than the unloaded robot, while the table gives 40 N 2204.04340 §IV-A tab. 2. Within the xLSTM paper, the equations and the released code differ in several places: the gate inputs, the output gate, the normalizer threshold and the sLSTM bias initialisation. Follow one source and state which one you chose. See the card's Paper vs code section [/papers/2405.04517]. The paper also marks some table values as re-evaluations, which the card notes [/papers/2405.04517].

The anomaly paper's text and equations disagree on two points. It merges the BiLSTM directions by sum in the text but writes concatenation in eq. 13, and it sums attention over all h steps in eq. 17 but over h − 1 steps in its implementation details 2107.12626 §3.4.1; eq. 13; eq. 17; §4.3. Its card lists both under Pitfalls, and the page does not choose between them.

The skeleton and dialogue cards add two more tasks, and they do not conflict with the others either. The skeleton paper's §5.5 swaps the meanings of λ and τ between its text and the Figure 7 caption, so the window sweep cannot be read with confidence 1704.02581 §5.5; fig. 7.

The robotics cards do not conflict with each other, because they study different tasks. Their own internal inconsistencies matter for implementation. In the teleoperation paper, the forecast table labels its metric RMSE, while the text and figure captions report MSE 2402.10220 tab. III 2402.10220 fig. 13. Its forecast loss is also called RMSE in one section and MSE in another 2402.10220 §III-G 2402.10220 §IV-E. In the off-road paper, the LSTM initialisation is described as "20 hidden layers", while the predictor's hidden size is 4 2412.00581 §6. These may refer to the same unit, and the paper does not say. In the EMG paper, the unit of the sequence length is not stated 2412.17991 tab. 1.

Papers

• xLSTM: Extended Long Short-Term Memory: introduces exponential gating, sLSTM, mLSTM and the xLSTM block stack, with author-reported language-model results.
• Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks: stacked and hierarchical LSTM streams over skeleton joints, fused with a spatial stream that reads the joints as a sequence.
• Key-Value Retrieval Networks for Task-Oriented Dialogue: an LSTM encoder-decoder that adds knowledge-base key scores to its output logits, with author-reported dialogue results.
• Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads: uses a two-layer LSTM policy trained with PPO to walk a simulated and a real bipedal robot under dynamic loads.
• Single-grasp Deformable Object Discrimination: the Effect of Gripper Morphology, Sensing Modalities, and Action Parameters: compares SVM, k-NN and an LSTM on raw squeeze signals from four robot hands and grippers.
• Towards Semi-Autonomous Robotic Arm Manipulation Operator Intention Detection from Force Data: a CNN classifies glovebox tasks and a CNN+LSTM forecasts the next 5 s of arm data.
• Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving: an LSTM corrects a bicycle-type vehicle model's forces, optionally using DINOv2 terrain features.
• Online Adaptation for Myographic Control of Natural Dexterous Hand and Finger Movements: a causal LSTM and TCN decode seven wrist and finger angles from forearm EMG, with online updates.
• Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals: a convolutional autoencoder trained with an attention BiLSTM and an autoregressive predictor, scored by summed loss, with author-reported F1.
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation: a hierarchical BiLSTM encoder and LSTM decoder with convolutional attention translate French speech directly to English text, with author-reported BLEU on synthetic speech.