Continuous online sequence learning with an unsupervised neural network model

In plain words

Many real signals arrive as a never-ending stream: sensor readings, network traffic, city transport counts. A useful model has to learn from each new item as it arrives, notice when the patterns change, and cope with noisy data. It also helps to hold several possible futures at once when the recent past does not settle which one will happen.

This paper studies a model inspired by how the brain's outer layer is thought to store sequences. Each input is turned into a sparse pattern in which only a small set of units is on. Those units are grouped so that a few of them can signal a prediction for the next input, and the model updates its connections after each item without storing a training set. When two sequences share a stretch, the model keeps both possible endings open until later input picks one.

The authors compare it with a statistical forecaster and with several neural networks on two kinds of data: artificial sequences designed to need memory of earlier items, and real taxi-passenger counts. The model does well on both, and it recovers quickly when the pattern changes.

TL;DR

• Sequence memory with 2048 columns and 32 cells per column, updated with a Hebbian-style rule after each element §3.1 tab. 1.
• Each element is a sparse binary code; the cells that predict the next element are the ones on dendritic segments with enough active inputs §3.2 eq. 1.
• It learns in one pass with no gradient descent and no buffered training set §3.3 §6.1.
• Artificial sequences: perfect prediction on high-order sequences, up to 100th order, and on two and four simultaneous endings fig. 4 fig. 6 fig. 7.
• NYC taxi demand, 2.5 hours ahead: error comparable to the best LSTM and lower than ARIMA, ELM and ESN (author-reported) §5 fig. 10.
• Under temporal noise present throughout training, LSTM is slightly better, and on the Reber grammar task HTM is below LSTM §4.5 §6.4.

Problem

Stream learning needs models that keep adapting when the data statistics change, that do not need a fixed buffer of recent data, that tolerate noise and lost inputs, and that work without per-task hyperparameter search §1 §2. Batch-trained models must be retrained on a buffer, and ELM needs a fixed time lag set in advance §1 §4 §7.2. The paper asks whether HTM sequence memory meets these criteria and how its prediction accuracy compares with ARIMA, ELM, ESN and LSTM §1 §2.

The authors also ask whether a model built on cortical principles can match standard sequence learners on accuracy §1 §6.3.

Method

Overview. Each stream element becomes a sparse binary vector of 2048 columns in which about 2% are active §3.2 §7.1. Cells inside each active column carry the context. A cell is predictive when enough of its distal synapses point to cells that were active in the same layer eq. 1. Columns that receive input and contain a predictive cell fire only those cells eq. 2. Otherwise the whole column fires, which lets the model create a new context eq. 2 §3.3. Synapses are then updated by a Hebbian-style rule that strengthens synapses to recently active cells and weakens the rest eq. 3. A segment that predicted a cell which did not fire is weakened slightly eq. 4 §3.3.

Steps.

1. Encode each symbol as a random binary vector with 40 of 2048 bits on §7.1. For taxi data, three encoders (passenger count, time of day, day of week) produce bit vectors that are combined by spatial pooling into 40 active bits of 2048 §7.1.
2. Choose winning columns. Each column is connected to a random 50% of the inputs, and the top 2% of columns by number of active proximal synapses win §3.3. Proximal synapses are fixed during learning §3.3.
3. Compute predictive cells at time t from the active cells of time t. A distal segment is active when the number of its connected synapses to active cells exceeds θ = 15 eq. 1 tab. 1. A cell is predictive when one of its segments is active eq. 1 §3.3.
4. Compute active cells at time t from the winning columns and the predictive cells of time t-1 eq. 2. A winning column with predictive cells activates only those cells. A winning column with no predictive cell activates all of its cells eq. 2 §3.3.
5. Reinforce segments. For each cell that was predicted and became active, reinforce the segment that made the prediction §3.3 eq. 3.
6. For a winning column with no predicted cell, pick the cell whose matching segment has the most active potential synapses and reinforce that segment §3.3. If no segment matches, create a new segment on the least-used cell of the column app. 7.1.
7. New synapses start at permanence 0.21, below the connection threshold of 0.5 app. 7.1 tab. 1. At most 32 are added per step tab. 1.
8. Decay false predictions. A segment that was active at t-1 but whose cell stayed silent at t loses permanence on its connected synapses by p-- eq. 4 §3.3.

The rule is Hebbian-like. The authors state that they did not derive it from gradient descent on a cost function §3.3.

# Predictive cells from distal segment activity  [eq. 1] [§3.3]
# Input:  active_cells (A^t), segments with their connected synapses
# Output: predictive_cells (pi^t)

def predictive_cells(active_cells, segments, theta):
    predictive = set()
    for segment in segments:
        # Count connected synapses whose presynaptic cell is active  [eq. 1]
        overlap = count_connected_synapses_to(segment, active_cells)
        # The segment is active when the overlap is above theta  [eq. 1]
        if overlap > theta:
            predictive.add(segment.cell)
    return predictive

# Cell activation in the winning columns  [eq. 2] [§3.3]
# Input:  winning_columns (W^t), predictive_prev (pi^{t-1})
# Output: active_cells (A^t)

def activate_cells(winning_columns, predictive_prev):
    active_cells = set()
    for column in winning_columns:
        predicted = cells_of(column) & predictive_prev
        # Predicted cells in a winning column become active  [eq. 2]
        if predicted:
            active_cells |= predicted
        # No predicted cell: the whole column becomes active  [eq. 2]
        else:
            active_cells |= cells_of(column)
    return active_cells

# Hebbian permanence update for one reinforced segment  [eq. 3] [§3.3]
# Input:  segment (its connected synapses), active_prev (A^{t-1})
# Output: segment permanences, updated in place

def reinforce(segment, active_prev):
    for synapse in connected_synapses(segment):
        # Synapses to cells active at t-1 are strengthened by p+  [eq. 3]
        if synapse.cell in active_prev:
            synapse.permanence += p_plus
        # The others are weakened by p-  [eq. 3]
        else:
            synapse.permanence -= p_minus
        # Permanences are bounded to [0, 1]  [§3.3]
        synapse.permanence = clip(synapse.permanence, 0.0, 1.0)

# Decay after a false prediction  [eq. 4] [§3.3]
# Input:  segment, cell_active_now (a^t), segment_active_prev (bool, t-1)
# Output: segment permanences, updated in place

def decay_false_prediction(segment, cell_active_now, segment_active_prev):
    # Only when the cell stayed silent and the segment was active  [eq. 4]
    if not cell_active_now and segment_active_prev:
        for synapse in connected_synapses(segment):
            # Every connected synapse loses the small amount p--  [eq. 4]
            synapse.permanence -= p_decay

Equations (as printed in the paper; notation: A^t is the M×N binary matrix of active cells, \Pi^t the predictive cells, \mathbf{D}^d_{ij} the permanence matrix of segment d of cell i in column j, and \tilde{\mathbf{D}}^d_{ij} the binary matrix of connected synapses):

\pi^t_{ij} = \begin{cases} 1 & \text{if } \exists_d \left\| \tilde{\mathbf{D}}^d_{ij} \circ A^t \right\|_1 > \theta \\ 0 & \text{otherwise} \end{cases} eq. 1

a^t_{ij} = \begin{cases} 1 & \text{if } j \in W^t \text{ and } \pi^{t-1}_{ij} = 1 \\ 1 & \text{if } j \in W^t \text{ and } \sum_i \pi^{t-1}_{ij} = 0 \\ 0 & \text{otherwise} \end{cases} eq. 2

\Delta \mathbf{D}^d_{ij} = p^+ \tilde{\mathbf{D}}^d_{ij} \circ A^{t-1} - p^- \tilde{\mathbf{D}}^d_{ij} \circ (1 - A^{t-1}) eq. 3

\Delta \mathbf{D}^d_{ij} = p^{--} \tilde{\mathbf{D}}^d_{ij} \quad \text{where } a^t_{ij} = 0 \text{ and } \left\| \tilde{\mathbf{D}}^d_{ij} \circ A^{t-1} \right\|_1 > \theta eq. 4

Here \circ is element-wise multiplication, W^t is the set of winning columns, and p^{--} \ll p^- eq. 4.

The classifier used to decode predictions from the SDR output (Section 3.4) is:

a_j = \sum_{i=1}^{N} w_{ij} x_i eq. 5

y_k = P(C_k \mid \mathbf{x}) = \frac{e^{a_k}}{\sum_{K=1}^{K} e^{a_K}} eq. 6

\Delta w_{ij} = \lambda \left( y_j - z_j \right) x_i eq. 7

where \mathbf{x} is the cell activation pattern, z_j is the target distribution, and \lambda is the learning rate §3.4.

Hyperparameters

Results

Author-reported. Setup: artificial high-order sequences with 50,000 noise symbols §4.1 and taxi counts from April 2015 at 30-minute resolution §5 fig. 10. The baselines are ARIMA, ELM, ESN and LSTM, with the batch and online variants described above §5 app. 7.2. The paper states that all models use the stated configuration; it gives no per-run variance.

Paper vs code

Code: official repository https://github.com/numenta/nupic (linked in the paper's appendix §7.1), pinned at 7281482def2a96fbda663e6c39e8351a1886dec7, which is the shallow clone's HEAD. The clone is 2024-era code (copyright 2013-2024, version 1.0.6.dev0). The commit from 2016 used for the paper was not checked; that would need the full history. Code is MIT-licensed, and no code is quoted.

The repository has no script that reproduces the paper's experiments in Sections 4 and 5. The closest files are examples/tm/tm_high_order.py and examples/opf/clients/nyctaxi/model_params.py, and they use different parameters from Table 1. Comparisons below use the library defaults in the Python reference TemporalMemory class.

• Activation threshold. Table 1 says 15 tab. 1. The class default is 13 temporal_memory.py. The taxi example uses 13 model_params.py. The high-order example uses 15 tm_high_order.py.
• Comparison operator. Equation 1 uses "greater than" eq. 1. The code uses "greater than or equal to" for active segments temporal_memory.py and for matching segments temporal_memory.py.
• Max new synapses per step. Table 1 says 32 tab. 1. The class default is 20 temporal_memory.py. The taxi example uses 20 model_params.py and the high-order example uses 40 tm_high_order.py.
• Max synapses per segment. Table 1 says 128 tab. 1. The class default is 255 temporal_memory.py. The taxi example uses 32 model_params.py.
• Max segments per cell. Table 1 says 128 tab. 1. The class default is 255 temporal_memory.py. The taxi example uses 128 model_params.py.
• Predicted-segment decrement p--. Table 1 says 0.01 tab. 1. The class default is 0.0 temporal_memory.py. The high-order example uses 0.01 tm_high_order.py. The taxi example does not set it.
• Scope of the p-- decay. Equation 4 decays all connected synapses of the segment eq. 4. The code decays only synapses to cells active at t-1, and the condition uses matching segments in an inactive column rather than segments active under θ temporal_memory.py temporal_memory.py.
• Which synapses the Hebbian update touches. Equation 3 updates only connected synapses, through the mask D̃ eq. 3. The code updates every synapse on the segment, including those below the connection threshold temporal_memory.py.
• Growth source. Appendix 7.1 says new synapses sample from previously active cells app. 7.1. The code samples from the previous winner cells, both when a predicted column grows synapses and when a bursting column creates a new segment temporal_memory.py temporal_memory.py temporal_memory.py.
• New segment creation. The code creates a segment on a bursting column only when there is at least one previous winner cell temporal_memory.py. The paper does not state this condition.
• Growth amount on predicted columns. The code asks for maxNewSynapseCount minus the number of active potential synapses on the segment temporal_memory.py. The paper does not state this.
• Matching segments. Bursting picks the matching segment with the most active potential synapses temporal_memory.py, which matches Section 3.3. The code requires a matching segment to have at least minThreshold potential synapses; the default is 10 temporal_memory.py temporal_memory.py. The paper does not state this threshold.
• Cleanup not in the paper. The code drops synapses below permanence EPSILON, removes segments left with no synapses, evicts the least recently used segment at the cap, and drops the weakest synapses when a segment is full temporal_memory.py temporal_memory.py temporal_memory.py. The paper says nothing about these.
• Spatial pooling. The top-2% step is not in the temporal memory class. It is in the spatial pooler, which was not reviewed here.

Pitfalls

• Strict versus non-strict activation. At exactly θ the paper's segment is inactive, and the code's is active. This matters only for segments with exactly 15 (or 13 in the taxi example) connected active synapses, but it changes the prediction set.
• Table 1 is not one configuration. The code default, the taxi example and the high-order example disagree on θ, the segment size cap, new synapses per step and the p-- value. Implement Table 1 as the paper states it, and treat the code defaults as unverified for the paper's results.
• Decay scope and trigger. Equation 4 weakens connected synapses of a segment that was active at t-1 and whose cell is silent now. The code's rule is narrower and uses different segments. Pick one and say which.
• Only connected synapses change under equation 3. A synapse that drops below the threshold stops changing under the equation as written. In the code those synapses keep changing. The paper does not discuss this.
• Growth source is ambiguous. "Previously active cells" in the appendix and "previous winner cells" in the code differ when a column bursts.
• Winning-column count. The paper says 2% of columns win, which is about 41 of 2048, but the encoder output has 40 active bits. It does not say whether the top-2% step applies to the artificial inputs at all; Section 7.1 mentions it only for taxi data.
• Equation numbers clash. The main text and the appendix both number equations (1) to (4). Cite the appendix ones as app. 7.3 eq. 1 and so on, not as eq. 1.
• Negative log-likelihood sign. The printed NLL formula has no minus sign app. 7.3 eq. 4, yet Figure 10C shows positive values. Treat the plotted quantity as the negative of the average log-probability. This is inferred from the plot, not stated.
• Figure-only numbers. Values for Figures 9, 10 and 11 are not in the text. Cite them qualitatively. The Reber grammar figure (98.4% for HTM and 100% for LSTM) comes from a rendered page of the PDF, so check it against the source before quoting.
• Single runs. No repeats or seeds are stated. Figure error bars come from different sequence sets fig. 6.
• Learning rate for the classifier. λ is not given eq. 7.
• Draft status. The title page says this is a draft and that the final paper will differ [title page]. The arXiv record here is version 2, dated 2016-04-28 [raw metadata].
• Sequence boundaries. The artificial streams have no start or end markers, and noise symbols separate sequences §4.1. The paper does not describe a reset signal. The code has a reset() method temporal_memory.py, which the paper does not mention.
• Noise robustness is limited. Temporal noise (replacing a symbol inside a sequence) hurts HTM badly, while spatial noise and cell death do not §4.5 §6.4. The paper says a hierarchy might help, but does not test one.

Lineage

• Builds on: the HTM neuron model, the sparse distributed representation and the sequence memory described in Hawkins and Ahmad (2016), which is cited as the reference for the cortical model §1 §3 §6.2. That paper is not in this wiki, and no arXiv id is given here.
• Spatial pooling used for the taxi task is described in Hawkins et al. (2011), which the paper cites §3.3 §7.1.
• Compared against: ARIMA (Durbin and Koopman, 2012) §1 §5; online sequential ELM (Liang et al., 2006) §4 §7.2; LSTM (Hochreiter and Schmidhuber, 1997) §1 §4; echo state networks (Jaeger and Haas, 2004) §1 §7.2.
• Related work the paper names: Lavin and Ahmad (2015) for anomaly detection §6.2. Not in this wiki.
• Superseded by: not stated in this wiki.