wikillm

Paper · 2020-04-15 · cs.CL, cs.LG

Entities as Experts: Sparse Memory Access with Entity Supervision

Entities as Experts (EaE) inserts an entity-memory layer after four Transformer layers. Mention spans retrieve learned entity embeddings (top-k at inference), supervised by Wikipedia hyperlinks, and the result is merged into the token stream. Pre-trained from scratch on Wikipedia. Author-reported: 43.2 TriviaQA dev exact match at 367M parameters, above T5-11B (42.3).

Memory-augmented neural networkKnowledge injectionMasked language modelingMixture of experts (MoE)entity-memoryknowledge-probinglamatriviaqaopen-domain-qapretrainingmixture-of-experts

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A normal language model stores facts about the world spread across all of its weights, so the notes on "Charles Darwin" get mixed up with those on "Charles River". This model keeps a separate memory slot for each named entity, such as a person, place or work. While it reads a sentence, each name it spots looks up its slot and mixes that memory into the text representation. The slots are learned from Wikipedia text alone, and the model is trained to fill in hidden words and to pick the right entity for each name. Only the slots for names in the text are read, so most of the stored knowledge stays untouched for a given sentence. The authors use it for fact-recall tests and quiz questions, where it beats some larger text-only models on these tests.

TL;DR

Entities as Experts (EaE) is a 367M-parameter Transformer with one entity-memory layer in the middle. It is pre-trained from scratch on English Wikipedia with masked spans, mention detection and entity linking supervised by hyperlinks. The authors report stronger results than a similar-size BERT on LAMA probes and than T5-11B on TriviaQA dev, but not better than T5-11B+SSM on TriviaQA Wiki test. No official code was found; implementers must work from this card.

Problem

Language models store facts only implicitly in shared weights, so a name with several meanings draws on blended representations §1. Earlier ways to inject entity knowledge either use fixed, pre-computed entity vectors or a fixed candidate detector §3, and the paper argues a fixed detector and frozen vectors limit what the model can learn §3. Scaling by adding parameters is expensive. The paper's aim is a model that stores knowledge per entity and only reads the memories for entities mentioned in a given text §1, §2.

Method

Notation and data

Let E be a set of N entities and V a token vocabulary with a [MASK] token. A context is a token sequence. Each context carries its mentions: a linked entity (or a null entity when unlinked), a start index and an end index §2.

Architecture §2.1 fig. 2

The model is a Transformer, split in two. The token embedding feeds 4 Transformer layers (l0 = 4). Their output goes into the entity-memory layer. The result is added to the pre-memory state, layer-normalised, and fed into 8 more Transformer layers (l1 = 8). Task heads sit on top: TokenPred (masked-token softmax, as in BERT) and EntityPred (entity linking) §2.1.

The layer-wise equations as printed in the paper §2.1:

  • token embedding, then Transformer with l0 layers;
  • entity memory layer, then the sum with its input and LayerNorm;
  • Transformer with l1 layers, then task heads.

Entity memory layer §2.1 eq. 1 eq. 2

For each mention m with start s and end t:

  1. Build a pseudo entity vector h from the start and end (the paper says "tail") states of the preceding Transformer output: a linear map W_f applied to their concatenation eq. 2. W_f has shape (d_ent, 2 · d_emb).
  2. Score h against all rows of the entity table E (N × d_ent) by dot product and keep the top k rows. Training uses k = N; inference uses k = 100 §2.1.
  3. Softmax over those k dot-product scores gives weights α_j. The entity representation E_m is the weighted sum of the selected embeddings §2.1.
  4. The output at position s is W_b · E_m, where W_b maps d_ent to d_emb eq. 1. Other positions stay empty, so the layer is sparse §2.1.
Pseudocode
# Entity memory layer  [§2.1] [eq. 1] [eq. 2]
# hidden: (L, d_emb) output of the preceding Transformer layers
# entity_table: E, (N, d_ent)

def entity_memory_layer(hidden, mentions, entity_table, k):
    # Positions without a mention start stay zero  [§2.1] [eq. 1]
    output = zeros_like(hidden)
    for start, end in mentions:
        # Pseudo entity from the start and end states  [eq. 2]
        span = concat(hidden[start], hidden[end])      # (2 * d_emb)
        pseudo = W_f @ span                            # (d_ent)
        # Keep the k entities with the highest dot product  [§2.1]
        top = top_k_rows(entity_table, pseudo, k)
        # Softmax over the kept scores  [§2.1]
        weights = softmax(entity_table[top] @ pseudo)
        # Weighted sum of kept entity embeddings  [§2.1]
        entity = weights @ entity_table[top]           # (d_ent)
        # Project to model width, place at the start position  [eq. 1]
        output[start] = W_b @ entity                   # (d_emb)
    return output

Block-level combination §2.1:

Pseudocode
# One EaE forward pass through the trunk  [§2.1]
# Trunk widths: BERT-base sized Transformer layers  [§4.1]

def eae_trunk(tokens, mentions, entity_table, k):
    # Token embedding, then l0 = 4 Transformer layers  [§2.1]
    hidden = transformer(token_embed(tokens), num_layers=4)
    # Entity memory output, then residual sum and LayerNorm  [§2.1]
    memory = entity_memory_layer(hidden, mentions, entity_table, k)
    merged = layer_norm(memory + hidden)
    # l1 = 8 further Transformer layers, then task heads  [§2.1]
    return transformer(merged, num_layers=8)

Mention detection at inference §2.1

A BIO classifier sits on the output of the first Transformer block. The whole BIO sequence is decoded with constraints that forbid inconsistent tags. The inferred spans are the ones used for all experiments §2.1. This replaces an external mention detector at test time.

Training data §2.2.1

  • Corpus: English Wikipedia, 2019-04-14 dump, split into 500-byte contexts (32M contexts, 17M linked mentions) app. A.
  • Entity vocabulary: the 1M most frequent entities (86% of linked mentions) app. A.
  • Mention boundaries: hyperlinks plus the Google Cloud Natural Language API, giving 140M mentions. Unlinked mentions are also annotated §2.2.1 app. A.
  • Masking: 20% of randomly chosen mentions have all their boundary tokens replaced by [MASK] §2.2.1.

Learning objective §2.2.2

The pre-training loss is the sum of three terms: a mention-boundary loss, an entity-linking loss and a masked-language-modelling loss §2.2.2. The paper gives no loss weights, so an unweighted sum is the reading that matches the text.

Pseudocode
# Pre-training loss: sum of three terms  [§2.2.2]
# Weights are not stated in the paper; unweighted sum is assumed.

def pretraining_loss(batch):
    # Mention boundaries: BIO cross-entropy on every token  [§2.2.2]
    boundary = bio_cross_entropy(bio_logits, bio_labels)
    # Entity linking on hyperlinked mentions only  [§2.2.2]
    linking = entity_linking_term(hyperlinked_mentions)
    # Masked tokens: cross-entropy over the vocabulary  [§2.2.2] [§2.2.1]
    masked = token_cross_entropy(token_logits, masked_labels)
    return boundary + linking + masked

Entity linking term §2.2.2. The paper writes the softmax probability of the gold entity over all N entities, not its negative log. Whether a minus sign or a log is applied is not stated. Standard practice is a cross-entropy, i.e. the negative log, but that choice is an assumption of this card:

Pseudocode
# Entity linking term for one hyperlinked mention  [§2.2.2]
# Paper writes the softmax probability; sign and log not stated.
# Cross-entropy (negative log) is assumed here.

def entity_linking_term(pseudo, gold_entity, entity_table):
    # Dot product against every entity in the vocabulary  [§2.2.2]
    scores = entity_table @ pseudo                 # (N)
    return -log(softmax(scores)[gold_entity])      # scalar

Only hyperlinked mentions receive this loss; the paper says it is not applied to the 88% of mentions without hyperlinks, whose memory access is unsupervised §2.2.2.

Optimisation: Adam, learning rate 1e-4, linear-style warmup over the first 5% of steps then decay (decay shape not stated), gradient norm clipped at 1.0, 1M steps at batch 4096, trained from scratch on 64 TPUs app. A.

Downstream fine-tuning §5.3 app. B app. C

  • Open-domain QA: each question gets a special answer-position token. The model predicts the answer entity at that position with the entity-linking loss. Mention spans in the question get the mention-detection loss §5.3.
  • Entity typing (9 classes, Ultra-Fine): entity embeddings and mention detection are frozen; the span representation is projected to 9 classes app. C.

Hyperparameters

NameValueWhereNotes
Transformer layers before memory (l0)4§2.1
Transformer layers after memory (l1)8§2.1
Transformer layer sizeidentical to BERT-base§4.1
Entity embedding size (d_ent)256§4.1Variant with 512 in tab. 3
Entity vocabulary N1Mapp. A86% of linked mentions
Top-k during trainingk = N (all entities)§2.1
Top-k at inferencek = 100§2.1 tab. 5k = 10 gives nearly the same TriviaQA score
Mention masking rate20% of mentions§2.2.1
Context500-byte chunks, BERT lowercase word-pieces, max 128 tokensapp. A
Dev / test split of pre-training data0.2% / 0.2%app. ATest size discrepancy: see Pitfalls
OptimizerAdamapp. Abetas, epsilon not stated
Learning rate1e-4app. A
Warmupfirst 5% of stepsapp. ADecay schedule shape not stated
Gradient clippingnorm 1.0app. A
Pre-training steps1,000,000app. A
Pre-training batch size4096app. A
Pre-training hardware64 Cloud TPUsapp. A2 days to a week per run
Weight decay, dropout, loss weightsnot stated
Entity embedding init (main EaE)random§7 tab. 6Random init is the EaE baseline in tab. 6
TriviaQA fine-tuninglr 5e-6, batch 64, 50,000 stepsapp. B
WebQuestions fine-tuninglr 3e-5, batch 32, 700 stepsapp. B
QA search gridlr {5e-5, 3e-5, 1e-5, 5e-6, 3e-6}; batch {32, 64, 128}app. BSelected on validation
QA fine-tuning hardware4 TPUsapp. B
Entity typing fine-tuninglr 3e-5, batch 32, 700 steps, label smoothing 0.1app. C5 seeds, best on validation
Entity typing F1 thresholdchosen on validationapp. C

Results

Author-reported. All numbers use the paper's own pre-training on Wikipedia. Dataset and setup per row are given in the Source column. Parameter counts come from the tables.

SettingMetricBaselineThis paperSource
Masked hyperlink prediction (32k test examples, see Pitfalls)entity accuracyNo-EaE 58.661.8 (367M)tab. 1
Masked hyperlink predictiontoken accuracy / perplexityMM-large 53.4 / 10.356.9 / 11.0tab. 1
LAMA probes, averageaccuracyBERT-large 19.920.0tab. 2
LAMA T-RExaccuracyBERT-large 32.337.4tab. 2
LAMA ConceptNetaccuracyBERT-large 19.210.7tab. 2
TriviaQA dev (unfiltered), closed-bookexact matchT5-11B 42.343.2 (367M)tab. 3
TriviaQA dev, emb 512exact match43.2 (EaE)45.7 (623M)tab. 3
TriviaQA Wiki testexact matchT5-11B+SSM 61.653.4tab. 3
WebQuestions-open devexact matchT5-11B+SSM 43.539.0tab. 3
TACRED revised / weighted F1F1KnowBERT 79.3 / 58.480.6 / 61.3tab. 7
TACRED original F1F1KnowBERT 71.570.2tab. 7

Ablations and analysis. Figures and tables here are from the paper, not re-derived:

  • No-EaE (same parameter count, no memory layer) gets 37.7 TriviaQA dev tab. 3.
  • Unsupervised memory (EaE-unsup, no entity-linking loss on the memory) reaches 46.9 token accuracy and 16.9 perplexity, against 56.9 and 11.0 for EaE tab. 1. On LAMA it averages 18.0 against 20.0 tab. 2. The paper calls the gap significant §5.1; no TriviaQA number is given for it.
  • Entity typing with learned Wikidata-initialised embeddings gives the best typing accuracy in tab. 6 (75.1).

Paper vs code

No official code. Searches: the paper's TeX source links only SLING, a phrase table used for answer linking, and the Google Cloud Natural Language API, which is a third-party dependency app. B §2.2.1. A GitHub repository search for "entities as experts" returned unrelated repositories; a search for "entities experts" restricted to google-research returned none. No project page was found.

Pitfalls

  • Entity-linking loss as printed. The paper writes the softmax probability of the gold entity with no minus sign or log §2.2.2. Minimising that probability would push it the wrong way. The intended form is almost certainly a cross-entropy, but this is not stated.
  • Test-set size mismatch. Section 5.1 reports 32k test examples §5.1. The pre-training appendix sets aside 0.2% for test; of 32M contexts that is about 64k app. A. Which is correct is not resolved here.
  • Training uses all N entities; inference uses top-100. Training computes dot products and a softmax over the full 1M-entry table for every mention §2.1. The paper says its implementation does not yet use efficient top-k routing and that a naive version would bring no large speedup §4.1 §6.3. The "fraction of parameters accessed" claim depends on fast nearest-neighbour search, which the paper leaves to future work §4.1.
  • Ablation at inference differs. EaE-unsup uses full attention over the memory at inference, not top-k §4.2.
  • Parameter-count claim in the abstract. The abstract says EaE outperforms a model with 10x its parameters. Table 3 lists T5-3B at 3B against 367M (about 8x) tab. 3. The comparison is on different metrics and splits (T5-3B appears only in the Wiki-test column), so check the table before quoting.
  • Mention detector. Training boundaries come from hyperlinks and an external API (Google Cloud Natural Language, 140M mentions) §2.2.1. An implementer needs a substitute detector; the paper does not say how close it must be.
  • Span end definition. The pseudo-entity uses the "tail" state at the end index t §2.1 eq. 2. Whether the end index is inclusive is not stated.
  • Unlinked entities. Only hyperlinked mentions get the linking loss, so 88% of mentions are unsupervised for memory access §2.2.2. The authors say this unsupervised access is still useful but do not quantify it separately.
  • Fixed vocabulary. The model cannot represent entities outside the 1M vocabulary §8. QA training drops questions whose answers are outside the vocabulary app. B. Oracle TriviaQA dev is 85% app. B.
  • Unequal comparison with T5. EaE does not retrain on train plus dev after tuning; T5 does app. D. Table 3's dataset notes differ by row.
  • Entity embedding dimension. The main EaE uses 256; the 512 variant is reported as adding parameters (623M) with little inference-time cost §5.3. The text does not state the exact per-mention cost.
  • Overconfidence. The authors suggest EaE is overconfident when wrong on token prediction; this is a hypothesis, not tested §5.1.
  • Hyperparameter search. Pre-training was not tuned because of cost app. A, so the chosen values carry no tuning evidence.

Lineage

  • Builds on: BERT, for the Transformer and masked-LM objective; EaE uses BERT-base layer sizes §4.1.
  • Related entity-memory and knowledge-injection work: KnowBERT (similar memory layer, but starts from the BERT checkpoint and uses a fixed candidate detector) §3; ERNIE (fixed, pre-computed entity vectors) §3; RELIC (dual encoder for entity retrieval) §4.3.
  • Compared against: T5 and T5+SSM §5.3; BM25+BERT, ORQA and GraphRetriever §4.3 app. E; LAMA probes §5.2.
  • Memory-network background: the paper relates the layer to key-value memory networks and product-key memories §3.

Source and license

Adapted from “Entities as Experts: Sparse Memory Access with Entity Supervision” by Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, Tom Kwiatkowski (arXiv:2004.07202), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.