Sparse distributed representations (SDRs)

Overview

A sparse distributed representation (SDR) stores an input as a binary vector in which only a few units are on. The paper's sequence model uses 2048 columns and keeps about 2% of them active for each stream element 1512.05463 §3.2 1512.05463 §7.1. The shared invariant is the fixed sparsity: every input has the same small number of active bits, so the model can compare two inputs by counting the bits they share 1512.05463 §3.4.

The paper reports that spatial noise in the input and random cell death do not hurt the sequence predictions much, with no drop up to about 30% cell death 1512.05463 §4.5 1512.05463 §4.6. Temporal noise, where a symbol is replaced inside a sequence, hurts far more 1512.05463 §4.5.

All variants below come from one source, the paper Continuous online sequence learning with an unsupervised neural network model. It reports experiments on artificial sequences and on NYC taxi counts, and gives no SDR-only ablation.

Variants

Random codes per symbol

Each discrete symbol gets its own random set of 40 active bits out of 2048 1512.05463 §7.1. The artificial sequence experiments use this scheme 1512.05463 §4. Because the codes are random, the paper does not build similarity between symbols into the encoding. Similarity is left to the sequence model.

Pooled multi-field codes

For real-valued streams the paper uses three encoders, for passenger count, time of day and day of week. Their outputs are combined by spatial pooling into one code of 40 active bits out of 2048 1512.05463 §7.1. The paper points to an earlier spatial pooling method for this step, which it does not restate in full 1512.05463 §3.3 1512.05463 §7.1. The encoder widths are not stated 1512.05463 app. 7.1.

Reading the code back: overlap or a trained softmax

For artificial sequences, the decoder picks the symbol whose stored code shares the most active bits with the observed code 1512.05463 §3.4. For taxi counts, the cell activation pattern feeds a single-layer softmax over 22 disjoint count buckets, trained with a gradient-style update 1512.05463 §3.4 1512.05463 eq. 5 1512.05463 eq. 6 1512.05463 eq. 7. The learning rate for that readout is not stated 1512.05463 eq. 7.

When to use which

The paper compares no SDR variants against each other, so the choices below follow from what each variant needs and what the paper shows for it.

Implementation sketch

The sketch shows the two steps that the paper's SDR variants share: encode a symbol into a fixed-size set of active bits, then decode by overlap. The pooled encoders and the softmax readout are described in prose above, because the paper gives their sizes only in part.

# Random SDR for one symbol: 40 active bits out of 2048  [§7.1]
# Output: a set of bit positions

def random_sdr(n_bits=2048, n_active=40):
    # Choose the active bit positions uniformly at random  [§7.1]
    return set(random.sample(range(n_bits), n_active))

# Decode an observed SDR by overlap with the stored codes  [§3.4]
# Input: observed (set of active bits), codebook (symbol -> stored SDR)
# Output: best-matching symbol

def decode_by_overlap(observed, codebook):
    # Count shared active bits with each stored code  [§3.4]
    # and return the symbol with the largest overlap
    return max(codebook, key=lambda s: len(codebook[s] & observed))

Contradictions

None known between papers, since only one paper is on wikillm for this concept. Within the paper, the text says the top 2% of columns win 1512.05463 §3.3. Two percent of 2048 is about 41 columns, while the encoder output is stated as 40 active bits 1512.05463 §7.1. The paper does not say whether the 2% rule applies to the artificial inputs, which it describes with 40-bit codes 1512.05463 §4 1512.05463 §7.1.

Papers

• Continuous online sequence learning with an unsupervised neural network model: the HTM sequence memory, which takes SDR inputs and predicts the next element online, with results on artificial sequences and NYC taxi counts.