---
kind: concept
slug: memory-augmented-neural-network
title: Memory-augmented neural network
summary: >-
  A network that keeps state in a store separate from its weights and reads it with attention-style addressing. Four cards show four designs: rewritten key-value slots (knowledge tracing), a frame bank with an object summary (video segmentation), learned prototype banks (anomaly detection) and persistent state tokens updated per image (3D reconstruction). Results are author-reported.
tags: [memory, memory-networks, attention, knowledge-tracing, video-object-segmentation, anomaly-detection, 3d-reconstruction]
papers: ["1910.13197", "2302.05160", "2310.12982", "2501.12387"]
related: [long-short-term-memory, recurrent-neural-network, attention-mechanism, cross-attention, object-queries, prototype-learning, anomaly-detection, weakly-supervised-learning, agent-memory, retrieval-augmented-generation, feed-forward-reconstruction, multi-view-reconstruction, monocular-depth-estimation]
status: draft
updated_at: 2026-11-06
---

## Overview

A memory-augmented network stores its working state in a separate store that
the model can address, instead of only in one hidden vector or in its weights.
The model computes an address from the current input, reads the slots it
points to, and, in most designs, writes back to them. The store can be small
and rewritten (a few key-value slots), a bank of recent past frames with
first-in-first-out eviction, a fixed set of learned prototypes that is trained
but not rewritten per input, or a fixed-size set of state tokens that every
frame of a stream updates.

The invariant shared by the first three cards is the read: a weighted sum of
stored values, where the weights come from inner products between a query and
stored keys [arxiv:1910.13197 §3.2.2] [arxiv:2310.12982 §3.4.1]
[arxiv:2302.05160 §3.3]. Two of those cards normalise the weights with a
softmax across slots. The third applies a sigmoid to each slot on its own, so
its weights do not sum to one [arxiv:2302.05160 eq. 3]. The fourth card reads
its state with cross-attention inside each decoder block [arxiv:2501.12387 §3.1].
That card does not restate how the attention is normalised, so this page does
not say. The designs differ in what is stored, how it is written (if at all),
how often it changes, and how far back it reaches.

Four cards cover this concept, in four domains (student modelling, video
object segmentation, weakly supervised video anomaly detection and 3D
reconstruction), with different metrics. The page therefore describes design
choices rather than a settled best design.

## Variants

### Fixed key-value slot memory with erase and add writes

A key matrix of N slots is fixed after training. A value matrix of N slots holds
the state, and each answer rewrites every slot with an erase vector and an add
vector, each weighted by that slot's attention [arxiv:1910.13197 §3.2.3] [eq. 5]
[eq. 7]. The model is SKVMN, which builds on the DKVMN key-value memory [arxiv:1910.13197 §3.1].
Card: [Knowledge Tracing with Sequential Key-Value Memory Networks](/papers/1910.13197).
The authors report AUC of 84.0 on Synthetic-5 against 82.7 for DKVMN, and
between 2.0 and 2.4 points higher on the four real datasets [arxiv:1910.13197 tab. 3].

### Recurrent state linked along identity chains

SKVMN also runs an LSTM over the exercises, but it links each exercise only to
the most recent earlier exercise with the same identity vector, so unrelated
steps are skipped [arxiv:1910.13197 §3.3.1] [arxiv:1910.13197 §3.3.2].
The identity rule is not fully specified in the paper (see the card's Pitfalls).
The authors say this makes inference faster, but they report no runtime or
FLOP numbers [arxiv:1910.13197 §1] [arxiv:1910.13197 §5]. Card: [Knowledge Tracing with Sequential Key-Value Memory Networks](/papers/1910.13197).

### Frame-level memory bank with a bounded readout

Each memory frame contributes keys and values to a pixel memory, which is read
by matching the query frame's keys and keeping the top entries. Cutie uses 30
entries and at most five memory frames, the first frame always kept and the
rest in first-in-first-out order [arxiv:2310.12982 §3.4.4]. Card: [Putting the Object Back into Video Object Segmentation](/papers/2310.12982).
The base readout is inherited from XMem, which the authors do not claim as new
[arxiv:2310.12982 §3.4.1].

### Compact object summary memory

Cutie also keeps N = 16 object summaries. Each summary is a weighted average of
object features over the pixels of the memory frames, pooled separately for the
object and the background [arxiv:2310.12982 §3.3] [eq. 6] [eq. 8]. It is
updated as a streaming average, so its cost does not grow with video length,
which the authors report [arxiv:2310.12982 §3.3] [app. G.2]. The object memory
is read by object queries inside the object transformer [arxiv:2310.12982 §3.2.1].
Card: [Putting the Object Back into Video Object Segmentation](/papers/2310.12982).

### Dual learned prototype banks with sigmoid matching

Two banks of M = 60 learned prototypes, one for normal and one for abnormal
snippets, are read by the same rule. Each snippet is matched to every slot by
a sigmoid of a scaled inner product, not a softmax, and the augmented feature
is the match-weighted sum of the slots [arxiv:2302.05160 eq. 3]. A snippet's
bank match is the mean of its top-K slot matches [arxiv:2302.05160 eq. 4]. A
dual memory loss trains each bank to fire on its own videos and stay quiet on
the other kind [arxiv:2302.05160 eq. 5]. The banks are trained parameters: the
card's method has no write step at test time, so this variant sits at the edge
of the definition above. Card: [Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection](/papers/2302.05160).
The authors report 86.97% frame AUC on UCF-Crime and 81.66% AP on XD-Violence
with I3D RGB features [arxiv:2302.05160 tab. 1] [tab. 2]. The repository's
shipped records that match these numbers were selected on the test split, so
they are not held-out results (see the card's Paper vs code).

### Persistent state tokens updated once per image

The state is a fixed set of 768 learned tokens of width 768, shared by all
scenes at the start of a stream [arxiv:2501.12387 §3.1] [§3.4]. Each new image
passes through two decoders that run on the image tokens and the state tokens
together. In each block the two sides cross-attend, so the state takes in the
new image and the image reads the stored context [arxiv:2501.12387 §3.1 eq. 2].
The state is therefore rewritten on every frame. The card describes this update
as attention, not as an erase-and-add rule [arxiv:2501.12387 §3.1]. A virtual
camera given as a raymap can read the state without writing to it, to predict
an unseen view [arxiv:2501.12387 §3.2]. The authors report 16.58 FPS on KITTI
at 512x144 on one A100 [arxiv:2501.12387 tab. 2]. Memory use and training time
are not reported, and the authors say recurrent training is time-consuming
[arxiv:2501.12387 §5]. Card: [Continuous 3D Perception Model with Persistent State](/papers/2501.12387).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Few concepts, per-step explicit update of state | Fixed slots with erase and add | Slot count is the capacity; each write is one formula [arxiv:1910.13197 §3.2.3] |
| Need to inspect what each slot holds | Slots, with caution | The authors report clusters that overlap less than DKVMN's, checked by hand on a few questions with no clustering metric [arxiv:1910.13197 §5.3] [tab. 4] |
| Dense inputs, long streams, fixed memory budget | Bounded frame bank with FIFO | Cutie-small reaches 61.1 test J&F on BURST at 1.35G peak memory, against 57.9 for DeAOT with unbounded memory at 34.9G [arxiv:2310.12982 tab. 2] |
| Object-level consistency on top of pixel matching | Object summary memory plus object queries | Removing the object transformer costs 2.2 J&F on the small model [arxiv:2310.12982 tab. 3] |
| Expecting the object summary to carry most of the gain | Do not assume it | Removing the object memory costs 0.4 J&F, about the size of the default's standard deviation [arxiv:2310.12982 tab. 5] |
| Only video-level labels, memory of normal and abnormal patterns wanted | Dual prototype banks with a dual memory loss | Dual banks reach 86.97 UCF AUC against 85.19 for the normal bank alone and 83.72 for the abnormal bank alone [arxiv:2302.05160 tab. 3] |
| Expecting the dual banks to cut false alarms | Do not assume it | On UCF-Crime the dual model's false-alarm rate (1.05) is slightly above the normal bank alone (0.98) [arxiv:2302.05160 tab. 3] |
| Choosing the number of prototype slots | Check the sweep before picking | 20 slots give 86.71 UCF AUC, 60 give 86.97, 80 give 85.16, with no variance reported [arxiv:2302.05160 tab. 4] |
| Frames arrive as a stream, camera poses unknown, no per-scene optimisation wanted | Persistent state tokens updated per frame | Online: 16.58 FPS on KITTI, against 0.31 to 0.76 for the global-alignment baselines, which the paper classes as offline [arxiv:2501.12387 tab. 2] |
| Expecting drift-free output on very long sequences | Do not assume it | The authors say long sequences can drift, since the method has no global alignment [arxiv:2501.12387 §5] |
| Building a commercial product from the official code | Implement from the paper and the card | The official code is CC BY-NC-SA 4.0, which is non-commercial (see [the card](/papers/2501.12387)) |

For agents that need to recall text across sessions, the memory in
[agent memory](/concepts/agent-memory) is an external text store, not a learned
slot store, and these cards do not cover it.

## Implementation sketch

Language-agnostic sketch of a slot read and write. Dimensions follow the
slot-memory card. Choices the paper leaves open are marked.

```pseudocode
# Read: attention over N fixed key slots  [arxiv:1910.13197 §3.2.1] [eq. 1]
# keys: (N, d_k) fixed after training; values: (N, d_v) per student
def read(keys, values, query_vec):
    # Softmax over inner products; no scaling factor stated  [eq. 1]
    # UR-DMU uses a per-slot sigmoid here instead  [arxiv:2302.05160 eq. 3]
    # CUT3R reads its state by cross-attention in each decoder block  [§3.1]
    w = softmax(keys @ query_vec)            # (N,)
    # Weighted sum of value slots  [arxiv:1910.13197 §3.2.2] [eq. 2]
    r = w @ values                           # (d_v,)
    return w, r

# Write: erase then add, each slot scaled by its weight  [eq. 5] [eq. 7]
def write(values, w, erase, add):            # erase, add: (d_v,) in (0, 1), (-1, 1)
    for i in range(N):
        values[i] = values[i] * (1 - w[i] * erase) + w[i] * add
    return values

# Object summary update: streaming weighted mean  [§3.3] [eq. 6]
# Keep running sums and area; read sums / area. Code adds 1e-4 to the area
# [code: cutie/model/transformer/object_transformer.py#L132]
def update_summary(sums, area, feats, pool_weight):
    sums = sums + pool_weight @ feats        # (N, C)
    area = area + pool_weight.sum(axis=1)    # (N,)
    return sums, area

# Streaming state update, one frame at a time  [arxiv:2501.12387 §3.1]
# state: S = 768 learned tokens of width 768  [§3.4]
# The paper's pose token is left out; see the card's Paper vs code
def stream_step(state, image_tokens):
    # Decoder blocks: state and image tokens cross-attend  [eq. 2]
    state, image_out = decoder(state, image_tokens)
    # Heads read image_out; the updated state goes to the next frame
    return state, image_out
```

## Contradictions

None across cards: the four papers study different tasks and metrics. Within
papers:

- SKVMN says it has fewer parameters than DKVMN on ASSISTments2009 and
  JunyiAcademy [arxiv:1910.13197 §5.1]. At matching (d, N) it has more
  parameters in [arxiv:1910.13197 tab. 2] (for example 7.8k against 7k).
- Cutie's abstract describes running time as similar to XMem, while its main
  table gives 36.4 FPS for Cutie-base against 22.6 for XMem on the same
  hardware [arxiv:2310.12982 tab. 1]. The card's Pitfalls use the table.
- UR-DMU's paper gives K = floor(M/16) + 1 = 4 for every top-K operation
  [arxiv:2302.05160 §4.2]. The repository uses that value for the memory read,
  but 13 for snippet selection and video pooling with 200 training snippets
  [code: model.py#L72]. Which one an implementation follows changes the bank
  matches and the video score.
- For the normal bank on an abnormal video, UR-DMU's paper sets the target to 1
  [arxiv:2302.05160 eq. 5]. The repository's target is 1 on the complement of
  the anomaly mask, which pushes that score towards 0 on abnormal clips
  [code: train.py#L32-L33]. The card records this as a paper-code difference.
- CUT3R's paper describes one learnable pose token placed in front of each
  image [arxiv:2501.12387 §3.1 eq. 2]. The official code reads the pose token
  for frames after the first from a learned memory, which the paper does not
  describe [code: src/dust3r/model.py#L140-L223]. The card records this as a
  paper-code difference.

## Papers

- [Knowledge Tracing with Sequential Key-Value Memory Networks](/papers/1910.13197): key-value slot memory with erase and add writes, plus a chain-linked LSTM, for student mastery.
- [Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection](/papers/2302.05160): two learned normal and abnormal prototype banks, read by per-slot sigmoid match and trained with a dual memory loss.
- [Putting the Object Back into Video Object Segmentation](/papers/2310.12982): frame-level pixel memory with an object-level summary memory and object queries for video segmentation.
- [Continuous 3D Perception Model with Persistent State](/papers/2501.12387): a fixed set of learned state tokens, updated by decoder cross-attention for each streamed image, giving per-frame pointmaps and camera poses.