---
kind: concept
slug: coreference-resolution
title: Coreference resolution
summary: >-
  Coreference resolution groups all mentions that refer to one entity into clusters. Options run from hand-written rules to end-to-end neural span models, which the survey reports at 68.8 CoNLL F1 on English. Quoref tests coreference inside reading comprehension. Scores are author-reported and the metrics differ, so compare only within one metric and setup.
tags: [nlp, coreference, evaluation-metrics, reading-comprehension, benchmarks]
papers: ["1805.11824", "1908.05803"]
related: [anaphora-resolution, extractive-question-answering, benchmark-validity, sentiment-analysis]
status: draft
updated_at: 2026-11-09
---

## Overview

Coreference resolution finds the mentions in a text that refer to the same
entity and puts them into one cluster. A mention can be a name, a noun phrase
or a pronoun. The survey treats it as the clustering side of a wider pair of
tasks: anaphora resolution links a reference to one earlier item, while
coreference resolution groups all mentions of an entity [arxiv:1805.11824 §1].
The two are often confused, so check which one a paper or benchmark measures
[arxiv:1805.11824 §1]. The link view is on the [anaphora resolution](/concepts/anaphora-resolution) page.

The invariant all variants share: each mention either joins an existing
cluster or starts a new one. Mention-pair and mention-ranking models make this
choice by scoring candidate earlier mentions [arxiv:1805.11824 §8.2]
[arxiv:1805.11824 §8.3]. Systems differ in how they find mentions, which
features or learned vectors they use, and how they train [arxiv:1805.11824 §8].

The survey covers English NLP only [arxiv:1805.11824 §1]. This page rests on
two cards: the survey, which gives the taxonomy, metrics and reported scores,
and Quoref, a benchmark whose questions need coreference to answer
[arxiv:1908.05803 §1]. The systems themselves are described in their original
papers, which have no cards here. Check those before implementing any of them.

## Variants

### Rule-based systems

Early systems use discourse properties and syntactic and semantic constraints.
The survey names the BFP centering algorithm as an early discourse-based method
for pronouns [arxiv:1805.11824 §8.1]. It cites a multi-sieve deterministic system
as the best rule-based system on CoNLL 2012 [arxiv:1805.11824 §8.1]
[arxiv:1805.11824 tab. 4]. The survey's table gives the rule-based row (Lee et al.
2013) a CoNLL F1 of 54.82 [arxiv:1805.11824 tab. 7]. Card: [Anaphora and Coreference Resolution: A Review](/papers/1805.11824).

### Statistical and machine-learning systems

These use mention-pair or mention-ranking models trained on hand-built
features. The group also includes joint inference with integer linear
programming and graph partitioning with conditional random fields
[arxiv:1805.11824 §8.2] [arxiv:1805.11824 tab. 5]. Reported CoNLL F1 runs from
61.40 (Durrett and Klein 2013) to 63.02 (Clark and Manning 2015, entity-centric)
[arxiv:1805.11824 tab. 7]. Card: [Anaphora and Coreference Resolution: A Review](/papers/1805.11824).

### Neural mention-ranking systems

A feed-forward network scores each candidate antecedent for a mention. Wiseman
et al. 2015 uses tanh layers over mention and pair features and a
latent-variable loss [arxiv:1805.11824 §8.3] [arxiv:1805.11824 eq. 23]
[arxiv:1805.11824 eq. 26]. Wiseman et al. 2016 adds an RNN that reads each
candidate cluster [arxiv:1805.11824 §8.3]. Clark and Manning 2016 pools
mention-pair vectors into cluster-pair vectors and then chooses merge or pass
actions. Its deep form replaces the heuristic loss with a reinforcement-learning
objective [arxiv:1805.11824 §8.3] [arxiv:1805.11824 fig. 4]. CoNLL F1 across
these rows runs from 63.39 to 65.73 [arxiv:1805.11824 tab. 7]. See the
Contradictions section for two numbers in this group that the text and the
table disagree on. Card: [Anaphora and Coreference Resolution: A Review](/papers/1805.11824).

### End-to-end span models

Lee et al. 2017 detects mentions and links them in one model. A bi-LSTM reads
the word vectors, every span up to width 10 is scored, the top-scoring spans are
kept, and each is paired with the preceding 250 mentions [arxiv:1805.11824 §8.3]
[arxiv:1805.11824 tab. 6]. The survey reports 68.80 CoNLL F1 for it, the highest
figure in its table [arxiv:1805.11824 tab. 7], and calls it the best system it
knew of at the time of its posting [arxiv:1805.11824 §8.3]. The fraction of
spans kept is not stated [arxiv:1805.11824 §8.3]. Card: [Anaphora and Coreference Resolution: A Review](/papers/1805.11824).

## Metrics

The survey defines seven scores. Each is a precision and recall over clusters
or links [arxiv:1805.11824 §5]:

- B-cubed, per mention, with weights [arxiv:1805.11824 §5.1] [arxiv:1805.11824 eq. 1].
- MUC, link-based: it counts the link changes needed to turn the output into the gold clusters [arxiv:1805.11824 §5.2] [arxiv:1805.11824 eq. 3].
- CEAF, which first finds the best one-to-one mapping between output and gold clusters, then scores the mapped pairs [arxiv:1805.11824 §5.3] [arxiv:1805.11824 eq. 6].
- ACE value, similar to CEAF but without normalisation [arxiv:1805.11824 §5.4].
- CoNLL score, the unweighted mean of MUC, B-cubed and CEAF F1, as used in the CoNLL 2012 shared task [arxiv:1805.11824 §5.5] [arxiv:1805.11824 eq. 12].
- BLANC, a Rand-index style score over coreferent and non-coreferent links [arxiv:1805.11824 §5.6].
- LEA, link-based and weighted by entity size, which the survey introduces to avoid the mention-identification effect [arxiv:1805.11824 §5.7].

The survey notes that MUC ignores singletons [arxiv:1805.11824 §5.8]. It also
argues that published results are hard to compare because the metrics, splits
and mention-detection setups differ [arxiv:1805.11824 §5.8] [arxiv:1805.11824 §9].
Several formulas in the survey need care before you implement them: the B-cubed
weights are described differently in the text and the equation, the CoNLL
formula does not name the CEAF variant, and one BLANC term carries the wrong
label [arxiv:1805.11824 eq. 1] [arxiv:1805.11824 eq. 12] [arxiv:1805.11824 eq. 16].
Check each against the original paper.

```pseudocode
# CoNLL 2012 headline score  [arxiv:1805.11824 §5.5] [arxiv:1805.11824 eq. 12]
# Input:  F1 values for MUC, B-cubed and CEAF, in the same units
# Output: CoNLL F1, the unweighted mean of the three

def conll_score(f1_muc, f1_b3, f1_ceaf):
    # Equal weight for each metric  [arxiv:1805.11824 §5.5]
    return (f1_muc + f1_b3 + f1_ceaf) / 3
```

## When to use which

| Situation | Use | Why |
|---|---|---|
| You must compare with published English numbers | Report the CoNLL average on OntoNotes 5.0 with the CoNLL 2012 splits | The survey's table uses this setup [arxiv:1805.11824 tab. 2] [arxiv:1805.11824 tab. 7]. Compare only rows that share the setup [arxiv:1805.11824 §5.8]. |
| You want a highest-reported English score and can train a neural span model | End-to-end span model ([Lee et al. 2017](/papers/1805.11824)) | 68.80 CoNLL F1 [arxiv:1805.11824 tab. 7], author-reported. The survey gives no learning rate, optimiser or epoch settings, so the original paper is needed before you build it [arxiv:1805.11824 §8.3]. |
| You already have a trusted mention detector | Mention-ranking neural model | The early neural systems take mentions from external tools [arxiv:1805.11824 tab. 6]. The survey treats mention detection as an input, not as a contribution [arxiv:1805.11824 tab. 6]. |
| You want a transparent pipeline with no learned model | Rule-based | The rule-based row has the lowest CoNLL F1 in the survey's table, 54.82 [arxiv:1805.11824 tab. 7]. Expect a lower score. The lack of a learned model is an inference from the method class, not a claim the survey makes. |
| You need to test whether a reader handles coreference | A span-QA benchmark such as Quoref | Quoref asks questions whose answers need the coreference link [arxiv:1908.05803 §3]. Its scores are QA F1, so they do not replace the clustering metrics [arxiv:1908.05803 §4]. See Benchmarks below. |
| Pronoun resolution feeds aspect-level sentiment | Any system that resolves pronouns well | One cited study reports about a 10% average gain over baseline coreference modules on its sentiment task. Another reaches about half the gain that perfect anaphora resolution would give [arxiv:1805.11824 §11]. The survey does not name the systems in these studies. |

## Benchmarks

### Quoref: coreference inside span-QA

Quoref has about 24K questions over 4.7K Wikipedia paragraphs
[arxiv:1908.05803 tab. 1]. Workers must find two or more co-referring spans and
write a question whose answer depends on knowing that they refer to the same
thing [arxiv:1908.05803 §2]. Each question is checked by an adversary, an
uncased BERT base QA model trained on SQuAD 1.1. A question is accepted only
if the adversary's answer does not match the worker's answer
[arxiv:1908.05803 §2]. The paper does not define "match".

The authors judged 78% of 100 sampled questions to need coreference
[arxiv:1908.05803 §3]. On test, the best baseline (XLNet QA, base) reaches 70.51
F1, BERT QA reaches 66.39, and a passage-only baseline reaches 28 to 29
[arxiv:1908.05803 tab. 3]. The authors estimate human performance at 93.41 F1
on 400 test questions [arxiv:1908.05803 §4.3]. In an error analysis of 100 dev questions, about half
of BERT QA's correct answers point at the first or most frequent entity in the
paragraph, against 12% of its wrong answers [arxiv:1908.05803 §4.4]. Those shortcuts make the scores harder to read
without a passage-only control; see [benchmark validity](/concepts/benchmark-validity).
Card: [Quoref](/papers/1908.05803).

## Implementation sketch

Most neural systems in the survey share a mention-ranking core: score each
earlier candidate for a mention, add a "no antecedent" choice, and link to the
best one. The survey gives the scoring form for Lee et al. 2017 but not every
term [arxiv:1805.11824 eq. 28]. It also does not say how scores become
probabilities in the training objective [arxiv:1805.11824 eq. 29].

```pseudocode
# Greedy mention ranking  [arxiv:1805.11824 §8.2] [arxiv:1805.11824 §8.3]
# Input:  mentions in document order, pair scorer s(i, j), null score
# Output: cluster id per mention

def resolve(mentions, score, null_score):
    cluster = {}
    for i, m in enumerate(mentions):
        # Candidates: earlier mentions, plus a "start new entity" option
        best_j, best = None, null_score(m)
        for j in range(i):
            s = score(mentions[j], m)  # pair score  [arxiv:1805.11824 eq. 28]
            if s > best:
                best_j, best = j, s
        # Join the best antecedent's cluster, or open a new one
        cluster[i] = cluster[best_j] if best_j is not None else i
    return cluster
```

## Contradictions

- **Clark and Manning 2016 (improving), CoNLL F1.** The text gives 65.39 [arxiv:1805.11824 §8.3]. The table gives 65.29 for the same system [arxiv:1805.11824 tab. 7]. Use the table value and check the original paper.
- **Clark and Manning 2016 (deep RL), component scores.** The text assigns 65.73 and 63.4 to CEAF and B-cubed [arxiv:1805.11824 §8.3]. The table lists 65.73 as the CoNLL score, with CEAFe 59.23 and B-cubed 63.40 [arxiv:1805.11824 tab. 7]. The table is internally consistent with the CoNLL average, so it is the more likely reading.
- **Quoref learning rate.** The paper typesets the learning rate as "3^{-5}". The intended value is probably 3e-5, but the exponent base is not written out [arxiv:1908.05803 app. B]. Check the stated settings against any released training configuration before you reuse them [arxiv:1908.05803 app. B].

## Papers

- [Anaphora and Coreference Resolution: A Review](/papers/1805.11824): survey of reference types, the seven metrics, rule, statistical and neural systems, and datasets, with a CoNLL 2012 comparison table (author-reported).
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](/papers/1908.05803): span-QA benchmark whose questions need coreference, filtered by a live adversary.