---
kind: concept
slug: sparse-distributed-representations
title: Sparse distributed representations (SDRs)
summary: >-
  Encode each input as a small set of active bits out of many, here 40 of 2048 (about 2%), so a sequence memory can match inputs by shared active bits. Variants use random codes per symbol or pool several encoded fields into one code, and read the code back by overlap or a trained softmax. Trade-off: only this one sparsity level is tested, and the multi-field encoder sizes are not stated.
tags: [sparse-coding, representation-learning, sequence-learning]
papers: ["1512.05463"]
related: [recurrent-neural-network, time-series-forecasting]
status: draft
updated_at: 2026-10-18
---

## Overview

A sparse distributed representation (SDR) stores an input as a binary vector in which only a few units are on. The paper's sequence model uses 2048 columns and keeps about 2% of them active for each stream element [arxiv:1512.05463 §3.2] [arxiv:1512.05463 §7.1]. The shared invariant is the fixed sparsity: every input has the same small number of active bits, so the model can compare two inputs by counting the bits they share [arxiv:1512.05463 §3.4].

The paper reports that spatial noise in the input and random cell death do not hurt the sequence predictions much, with no drop up to about 30% cell death [arxiv:1512.05463 §4.5] [arxiv:1512.05463 §4.6]. Temporal noise, where a symbol is replaced inside a sequence, hurts far more [arxiv:1512.05463 §4.5].

All variants below come from one source, the paper [Continuous online sequence learning with an unsupervised neural network model](/papers/1512.05463). It reports experiments on artificial sequences and on NYC taxi counts, and gives no SDR-only ablation.

## Variants

### Random codes per symbol

Each discrete symbol gets its own random set of 40 active bits out of 2048 [arxiv:1512.05463 §7.1]. The artificial sequence experiments use this scheme [arxiv:1512.05463 §4]. Because the codes are random, the paper does not build similarity between symbols into the encoding. Similarity is left to the sequence model.

### Pooled multi-field codes

For real-valued streams the paper uses three encoders, for passenger count, time of day and day of week. Their outputs are combined by spatial pooling into one code of 40 active bits out of 2048 [arxiv:1512.05463 §7.1]. The paper points to an earlier spatial pooling method for this step, which it does not restate in full [arxiv:1512.05463 §3.3] [arxiv:1512.05463 §7.1]. The encoder widths are not stated [arxiv:1512.05463 app. 7.1].

### Reading the code back: overlap or a trained softmax

For artificial sequences, the decoder picks the symbol whose stored code shares the most active bits with the observed code [arxiv:1512.05463 §3.4]. For taxi counts, the cell activation pattern feeds a single-layer softmax over 22 disjoint count buckets, trained with a gradient-style update [arxiv:1512.05463 §3.4] [arxiv:1512.05463 eq. 5] [arxiv:1512.05463 eq. 6] [arxiv:1512.05463 eq. 7]. The learning rate for that readout is not stated [arxiv:1512.05463 eq. 7].

## When to use which

The paper compares no SDR variants against each other, so the choices below follow from what each variant needs and what the paper shows for it.

| Situation | Use | Why |
|---|---|---|
| Discrete symbols with no known similarity | Random codes per symbol, overlap readout | This is the setup of the artificial experiments, where the paper reports perfect prediction on high-order sequences [arxiv:1512.05463 §4.4] [arxiv:1512.05463 §7.1] |
| Several numeric fields at each step | Encode each field, then pool into one 40-of-2048 code | This is what the taxi experiment does [arxiv:1512.05463 §7.1]. Encoder sizes are missing, so budget a tuning pass |
| Output must be a probability over ranges | Softmax readout on the active cells | The paper trains one this way for the taxi forecast [arxiv:1512.05463 §3.4] |
| Output must be one symbol from a known set | Overlap readout against stored codes | This is the decoder the paper uses for symbol sequences [arxiv:1512.05463 §3.4] |
| Input stream may lose or corrupt units | Keep the sparse code | The paper reports tolerance to spatial noise and cell death [arxiv:1512.05463 §4.5] [arxiv:1512.05463 §4.6]. It reports poor tolerance to temporal noise, so do not rely on the code alone for that |
| Need a sparsity level other than about 2% | No support in the paper | The paper reports no experiment that varies the number of active bits |

## Implementation sketch

The sketch shows the two steps that the paper's SDR variants share: encode a symbol into a fixed-size set of active bits, then decode by overlap. The pooled encoders and the softmax readout are described in prose above, because the paper gives their sizes only in part.

```pseudocode
# Random SDR for one symbol: 40 active bits out of 2048  [§7.1]
# Output: a set of bit positions

def random_sdr(n_bits=2048, n_active=40):
    # Choose the active bit positions uniformly at random  [§7.1]
    return set(random.sample(range(n_bits), n_active))

# Decode an observed SDR by overlap with the stored codes  [§3.4]
# Input: observed (set of active bits), codebook (symbol -> stored SDR)
# Output: best-matching symbol

def decode_by_overlap(observed, codebook):
    # Count shared active bits with each stored code  [§3.4]
    # and return the symbol with the largest overlap
    return max(codebook, key=lambda s: len(codebook[s] & observed))
```

## Contradictions

None known between papers, since only one paper is on wikillm for this concept. Within the paper, the text says the top 2% of columns win [arxiv:1512.05463 §3.3]. Two percent of 2048 is about 41 columns, while the encoder output is stated as 40 active bits [arxiv:1512.05463 §7.1]. The paper does not say whether the 2% rule applies to the artificial inputs, which it describes with 40-bit codes [arxiv:1512.05463 §4] [arxiv:1512.05463 §7.1].

## Papers

- [Continuous online sequence learning with an unsupervised neural network model](/papers/1512.05463): the HTM sequence memory, which takes SDR inputs and predicts the next element online, with results on artificial sequences and NYC taxi counts.